compmoc blog

SCALA TIPS : remove unnecessary tests when working with collections...

noreply@blogger.com (david crosson) — Tue, 08 May 2012 19:54:00 +0000

Thanks to scala Option type and scala returning the right type behavior, many various kinds of checks can be removed. Example :



val m = Map("A" -> 1, "B" ->3)

var l1 = List.empty[Int]
var l2 = List.empty[Int]

// Don't write such thing :
if (m.contains("A")) l1 ::= m.get("A").get
if (m.contains("Z")) l1 ::= m.get("Z").get
println(l1)

// Instead just write something like :
l2 ++:= m.get("A")
l2 ++:= m.get("Z")
println(l2)

Because concatenating List[Int] with Option[Int] will get you a List[Int], and if the Option[Int] type value is None, nothing will be added to the list :)

Variance and bounds example

noreply@blogger.com (david crosson) — Tue, 01 May 2012 05:59:00 +0000


class A
class B extends A
class C extends B
class D extends C

class E

class X[+T] {                        // T covariant for input types
  def add[U >: T](x: U): X[U] = this // U contravariant for return type
}

class Y[+T <: A] {                   // This time with an upper bound A
  def add[U >: T <: A](y: U): Y[U] = this
}

val a = new A
val b = new B
val c = new C
val d = new D

val x = new X[C]
val x2 = x.add(b)    // x2 type becomes X[B]
val x3 = x.add(d)    // x3 remains of type X[C]
val x4 = x.add(10)   // x4 type becomes X[Any]
val x6 = x.add("Z")  // x5 type becomes X[java.lang.Object]

val y = new Y[C]
val y2 = y.add(b)    // y2 type becomes X[B]
val y3 = y.add(d)    // y3 remains of type X[C]
//val y4 = y.add(2)  // Won't compile as Y has an upper bound, it must inherits from A


class Z {
  def add[T<%E](e:T) = this  // View Bound, it exists an implicit conversion to E
}
val z  = new Z
//val z1 = z.add(a)  // Won't compile as there no implicit conversion from A to E
implicit def a2E(a:A) = new E
val z2 = z.add(a)

sbt scala drools expert system skeleton project

noreply@blogger.com (david crosson) — Thu, 26 Apr 2012 21:51:00 +0000

Just released a simple scala drools project skeleton. The Knowledge base is used to evaluate drools-mvel / scala interaction.
The Knowledge Base :


package dummy;

dialect "mvel"

import java.util.LinkedList;

// --------------------------------------------------------------

rule "who is young"
  when
    Someone($name:name, age < 30)
  then
    System.out.println($name+" is very young")
end

// --------------------------------------------------------------

rule "people cars"
  when
    $someone:Someone($name:name)
    $car : Car(someone == $someone)
  then
    System.out.println("One of "+$name+"'s car is "+$car.color.name+" "+$car.model)
end

// --------------------------------------------------------------

rule "who own at least one red car"
  when
    $someone:Someone($name:name)
    exists Car(someone == $someone, color == Color.red)
  then
    System.out.println($name+" has at least one red car")
end

// --------------------------------------------------------------

rule "who doesn't have a home"
  when
    $someone:Someone($name:name)
    not Home(someone == $someone)
  then
    insert(new InformationRequest($someone, $name+"'s home is unknown"))
end

// --------------------------------------------------------------

rule "who doesn't have an address"
  when
    $someone:Someone($name:name)
    Home(someone == $someone, !address.isDefined)
  then
    insert(new InformationRequest($someone, $name+" address is unknown"))
end

// --------------------------------------------------------------

rule "process information requests"
  when
    InformationRequest($who:someone, $what:message)
  then
    System.out.println("INFORMATION REQUESTED FOR "+$who+" : "+$what)
end

// --------------------------------------------------------------

rule "who owns more than 1 car"
  when
    $someone : Someone($name:name)
    $cars    : LinkedList( size>1 ) from collect(Car(someone == $someone))
  then
    System.out.println($name+" has "+$cars.size+" cars")  
end

// --------------------------------------------------------------

And the startup code :


package dummy

// ----------------------------------------------------------------
// DOMAIN MODEL

case class Someone(name:String, age:Int)

case class Car(someone:Someone, model:String, year:Int, color:Color)

case class Color(name:String)

object Color {
  val red = Color("red")
  val blue = Color("blue")
  val green = Color("green")
  val black = Color("black")
}

case class Address(street:String, town:String, country:String)

case class Home(someone:Someone, address:Option[Address]) 

case class InformationRequest(someone:Someone, message:String)




// ----------------------------------------------------------------


import java.io.FileInputStream
import java.io.InputStreamReader
import org.drools.RuleBaseFactory
import org.drools.audit.WorkingMemoryFileLogger
import org.drools.compiler.PackageBuilder


object Dummy {

  def main(args: Array[String]) {
    System.setProperty("drools.dialect.mvel.strict", "false")
    
    val rulesfilename = "src/main/resources/KBExpertise.drl"
    val source = new InputStreamReader(new FileInputStream(rulesfilename))
    
    val builder = new PackageBuilder()
    builder.addPackageFromDrl(source)
    if (builder.hasErrors()) {
      System.out.println(builder.getErrors().toString())
      throw new RuntimeException("Unable to compile " + rulesfilename + ".")
    }

    val pkg = builder.getPackage()
    val ruleBase = RuleBaseFactory.newRuleBase()
    ruleBase.addPackage(pkg)

    val session = ruleBase.newStatefulSession()
    //session.addEventListener(new org.drools.event.DebugAgendaEventListener())
    //session.addEventListener(new org.drools.event.DebugWorkingMemoryEventListener())
    
    // setup the audit logging
    val logger:WorkingMemoryFileLogger = new WorkingMemoryFileLogger(session)
    logger.setFileName("drools")
    logger.writeToDisk()

    val martine = Someone(name="Martine", age=30)
    val martin  = Someone(name="Martin", age=40)
    val jack    = Someone(name="Jack", age=12)
    val martineCar = Car(martine, "Ford", 2010, Color.blue)
    val martinCar  = Car(martin, "GM", 2010, Color.black)
    val martinCar2 = Car(martin, "Ferrari", 2012, Color.red)
    val martinCar3 = Car(martin, "Porshe", 2011, Color.red)
    
    val martinHome = Home(martin, None)
    val jackHome   = Home(jack, Some(Address("221B Baker Street", "London", "England")))
    
    List(martine, martin, jack, 
        martineCar, martinCar, martinCar2, martinCar3,
        martinHome, jackHome
    ).foreach(session.insert(_)) 

    session.fireAllRules()

    session.dispose()
  }
}

To finish, the build.sbt file, to build and run the project :


import AssemblyKeys._

seq(assemblySettings: _*)

name := "ScalaDroolsDummyProject"

version := "0.0.1"

scalaVersion := "2.9.2"

mainClass in assembly := Some("dummy.Dummy")

jarName in assembly := "dummy.jar"



libraryDependencies += "org.drools" % "drools-compiler" % "5.3.1.Final"

libraryDependencies += "org.drools" % "drools-core" % "5.3.1.Final"

libraryDependencies += "org.drools" % "drools-jsr94"  % "5.3.1.Final"

libraryDependencies += "org.drools" % "drools-decisiontables"  % "5.3.1.Final"

libraryDependencies += "org.drools" % "knowledge-api"  % "5.3.1.Final"



libraryDependencies += "com.thoughtworks.xstream" % "xstream" % "1.4.2"
            


libraryDependencies += "org.scalatest" %% "scalatest" % "1.7.2" % "test"

libraryDependencies += "junit" % "junit" % "4.10" % "test"

resolvers += "JBoss third party releases repository" at "https://repository.jboss.org/nexus/content/repositories/thirdparty-releases"

Test it by your self :


$ sbt run
INFORMATION REQUESTED FOR Someone(Martin,40) : Martin address is unknown
One of Martin's car is red Porshe
Martin has 3 cars
Martin has at least one red car
One of Martin's car is red Ferrari
One of Martin's car is black GM
One of Martine's car is blue Ford
Jack is very young
INFORMATION REQUESTED FOR Someone(Martine,30) : Martine's home is unknown
[success] Total time: 6 s, completed 27 avr. 2012 22:35:37

And for those who don't know about SBT or GIT :

Download SBT jar file
- You'll get a java jar executable named sbt-launch.jar
Download the scala drools skeleton zip archive
- You'll get a file which will look like : "dacr-scala-drools-dummy-project-6b43143.zip"
- unzip the content and rename the unzipped directory to scala-drools-dummy-project


test@localhost /tmp/test $ unzip dacr-scala-drools-dummy-project-6b43143.zip 
Archive:  dacr-scala-drools-dummy-project-6b43143.zip
6b43143f369bda03361ca2c5f2376d51ba7d5524
   creating: dacr-scala-drools-dummy-project-6b43143/
  inflating: dacr-scala-drools-dummy-project-6b43143/.gitignore  
  inflating: dacr-scala-drools-dummy-project-6b43143/README  
 extracting: dacr-scala-drools-dummy-project-6b43143/RELEASE-NOTES  
  inflating: dacr-scala-drools-dummy-project-6b43143/build.sbt  
  inflating: dacr-scala-drools-dummy-project-6b43143/cleanup.sh  
   creating: dacr-scala-drools-dummy-project-6b43143/project/
  inflating: dacr-scala-drools-dummy-project-6b43143/project/plugins.sbt  
   creating: dacr-scala-drools-dummy-project-6b43143/src/
   creating: dacr-scala-drools-dummy-project-6b43143/src/main/
   creating: dacr-scala-drools-dummy-project-6b43143/src/main/resources/
  inflating: dacr-scala-drools-dummy-project-6b43143/src/main/resources/KBExpertise.drl  
   creating: dacr-scala-drools-dummy-project-6b43143/src/main/scala/
   creating: dacr-scala-drools-dummy-project-6b43143/src/main/scala/dummy/
  inflating: dacr-scala-drools-dummy-project-6b43143/src/main/scala/dummy/Dummy.scala  
   creating: dacr-scala-drools-dummy-project-6b43143/src/test/
   creating: dacr-scala-drools-dummy-project-6b43143/src/test/scala/
   creating: dacr-scala-drools-dummy-project-6b43143/src/test/scala/dummy/
  inflating: dacr-scala-drools-dummy-project-6b43143/src/test/scala/dummy/DummyTest.scala  
test@localhost /tmp/test $ mv dacr-scala-drools-dummy-project-6b43143 scala-drools-dummy-project
test@localhost /tmp/test $ cd scala-drools-dummy-project
test@localhost /tmp/test/scala-drools-dummy-project $ java -jar ~/Downloads/sbt-launch.jar run
[info] Loading project definition from /tmp/test/scala-drools-dummy-project/project
[info] Set current project to ScalaDroolsDummyProject (in build file:/tmp/test/scala-drools-dummy-project/)
[info] Updating {file:/tmp/test/scala-drools-dummy-project/}default-d556eb...
[info] Resolving org.scala-lang#scala-library;2.9.2 ...
[info] Resolving org.drools#drools-compiler;5.3.1.Final ...
...
[info] Compiling 1 Scala source to /tmp/test/scala-drools-dummy-project/target/scala-2.9.2/classes...
[info] Running dummy.Dummy 
INFORMATION REQUESTED FOR Someone(Martin,40) : Martin address is unknown
One of Martin's car is red Porshe
Martin has 3 cars
Martin has at least one red car
One of Martin's car is red Ferrari
One of Martin's car is black GM
One of Martine's car is blue Ford
Jack is very young
INFORMATION REQUESTED FOR Someone(Martine,30) : Martine's home is unknown
[success] Total time: 8 s, completed 30 avr. 2012 12:59:17

JAJMX scala script to list remote JVM threads using JMX...

noreply@blogger.com (david crosson) — Fri, 20 Apr 2012 21:36:00 +0000

A new example script using JAJMX API. This script, named lsthreads, connects to a remote JVM using just the host adress and the used JMX port, and then list all active threads and theirs current states. Some java options (JAVA_OPTS) have been added to allow using this script against itself.


#!/bin/sh
JAVA_OPTS=""
JAVA_OPTS=$JAVA_OPTS" -Dcom.sun.management.jmxremote"
JAVA_OPTS=$JAVA_OPTS" -Dcom.sun.management.jmxremote.port=9999"
JAVA_OPTS=$JAVA_OPTS" -Dcom.sun.management.jmxremote.authenticate=false"
JAVA_OPTS=$JAVA_OPTS" -Dcom.sun.management.jmxremote.ssl=false"
exec java $JAVA_OPTS -jar jajmx.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#
import jajmx._
import javax.management.openmbean.CompositeData

val host = if (args.size>=1) args(0) else "localhost" 
val port = if (args.size>=2) args(1).toInt else 9999

case class ThreadLock(name:String, ownerId:Option[Long], ownerName:Option[String])

case class ThreadInfo(name:String, id:Long, state:String,lock:Option[ThreadLock],
                      blockedCount:Long, waitedCount:Long)

object ThreadInfo {
  def apply(ti:CompositeData) :ThreadInfo = {
   val id           = ti.get("threadId").toString.toLong
   val name         = ti.get("threadName").toString
   val state        = ti.get("threadState").toString
   val blockedCount = ti.get("blockedCount").toString.toLong
   val waitedCount  = ti.get("waitedCount").toString.toLong
   val lock = Option(ti.get("lockName")) map {_.toString} filter {_.size>0} collect {
                case lockName =>
                    val lockOwnerId = Option(ti.get("lockOwnerId")) map {_.asInstanceOf[Long]} filterNot {_ == -1}
                    val lockOwnerName = Option(ti.get("lockOwnerName")) map {_.toString.trim}
                    ThreadLock(lockName, lockOwnerId, lockOwnerName)
              }
   ThreadInfo(name, id, state, lock, blockedCount, waitedCount)
  }
}

JMX.once(host,port) { jmx =>
  for (threading <- jmx.threading) {
    val ids = threading.get[Array[Long]]("AllThreadIds")
    val rawThreadsInfos = threading.call[Array[CompositeData]]("getThreadInfo", ids).get
    val threadsInfos = rawThreadsInfos map {ThreadInfo(_)} toList
    val countByState = threadsInfos groupBy {_.state} map { case (s,l) => s -> l.size}
    val countByStateStr = countByState map {case (s,l) => s+":"+l} mkString " "
    println("Total %d  %s".format(threadsInfos.size, countByStateStr))
    for ( ti <- threadsInfos sortBy {_.id } ) {
      println("%d - %s - %s".format(ti.id, ti.state, ti.name) )
    }
  }
}

Usage examples, with itself and then with a remote apache tomcat:


$ lsthreads
Total 13  TIMED_WAITING:4 WAITING:2 RUNNABLE:7
1 - RUNNABLE - main
2 - WAITING - Reference Handler
3 - WAITING - Finalizer
4 - RUNNABLE - Signal Dispatcher
9 - RUNNABLE - RMI TCP Accept-0
10 - RUNNABLE - RMI TCP Accept-9999
11 - RUNNABLE - RMI TCP Accept-0
12 - RUNNABLE - RMI TCP Connection(1)-127.0.0.1
13 - TIMED_WAITING - RMI Scheduler(0)
14 - TIMED_WAITING - RMI RenewClean-[127.0.0.1:45376]
15 - TIMED_WAITING - GC Daemon
16 - RUNNABLE - RMI TCP Connection(2)-127.0.0.1
17 - TIMED_WAITING - JMX server connection timeout 17

$ lsthreads 192.168.1.10 1099
Total 17  TIMED_WAITING:6 WAITING:2 RUNNABLE:9
1 - RUNNABLE - main
2 - WAITING - Reference Handler
3 - WAITING - Finalizer
4 - RUNNABLE - Signal Dispatcher
10 - RUNNABLE - RMI TCP Accept-0
11 - RUNNABLE - RMI TCP Accept-1099
12 - RUNNABLE - RMI TCP Accept-0
13 - TIMED_WAITING - GC Daemon
16 - TIMED_WAITING - ContainerBackgroundProcessor[StandardEngine[Catalina]]
17 - RUNNABLE - http-bio-8080-Acceptor-0
18 - TIMED_WAITING - http-bio-8080-AsyncTimeout
19 - RUNNABLE - ajp-bio-8009-Acceptor-0
20 - TIMED_WAITING - ajp-bio-8009-AsyncTimeout
22 - RUNNABLE - RMI TCP Connection(5)-127.0.0.1
23 - TIMED_WAITING - RMI Scheduler(0)
24 - RUNNABLE - RMI TCP Connection(6)-127.0.0.1
27 - TIMED_WAITING - JMX server connection timeout 27

First nasdaq stock query using scala and JASeries

noreply@blogger.com (david crosson) — Tue, 17 Apr 2012 22:10:00 +0000

The following script is a first attempt to make a stock query over nasdaq using JASeries (scala time series API).


#!/bin/sh
exec java -jar jaseries.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#

import io.Source
import fr.janalyse.series._

// Get nasdaq codes
val codes = 
  Source.fromURL("http://www.nasdaqtrader.com/dynamic/SymDir/nasdaqlisted.txt")
    .getLines
    .map(_.split("[|]",2).head)
    .toList.tail.init //.take(10)

// Get nasdaq stocks history
val stocks =
  codes.flatMap(code =>
    try {
     Some(code -> CSV2Series.fromURL("http://ichart.finance.yahoo.com/table.csv?s="+code))
    } catch {
      case x:java.io.FileNotFoundException => 
        println("Could'nt find %s stock data".format(code))
        None
    }
  )

// get close series
val closes = stocks flatMap { case (code, allSeries) => 
  allSeries.get("Close").map(_.rename(code+" Close"))
}

val top10incr =
  closes
    .sortBy( _.stat.linearApproximation.slope)
    .take(20)
    
// print highest slope top20, but remember that all history data is taken into
// account, so the last value may not be quiter higher than the first one.
for(stock <- top10incr) {
  println("%s : %f -> %f".format(stock.name, stock.head.value, stock.last.value))
}

This script is the starting point to next JASeries improvements and new features.

JAJMX scala script to list all available JVM numerical JMX values

noreply@blogger.com (david crosson) — Fri, 13 Apr 2012 22:02:00 +0000

Using JAJMX it becomes very easy to list all available numerical values of a given JVM JMX plateform :


#!/bin/sh
JAVA_OPTS=""
JAVA_OPTS=$JAVA_OPTS" -Dcom.sun.management.jmxremote"
JAVA_OPTS=$JAVA_OPTS" -Dcom.sun.management.jmxremote.port=9999"
JAVA_OPTS=$JAVA_OPTS" -Dcom.sun.management.jmxremote.authenticate=false"
JAVA_OPTS=$JAVA_OPTS" -Dcom.sun.management.jmxremote.ssl=false"
exec java $JAVA_OPTS -jar jajmx.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#

import fr.janalyse.jmx._

val host  = args.headOption.getOrElse("localhost")
val port  = if (args.size>1) args.tail.head.toInt else 9999

JMX.once(host, port) { jmx =>
  for(mbean <- jmx.mbeans ;
      attr  <- mbean.attributes collect {case n:RichNumberAttribute => n};
      value <- mbean.getLong(attr)) {
    println("%s - %s = %d".format(mbean.name, attr.name, value))
  }
}

Some jvm parameters (JAVA_OPTS) have been added in order to make possible to test this script against itself. So without any arguments it will give you such results :


$ ./lsnum
java.lang:type=Memory - ObjectPendingFinalizationCount = 0
java.lang:type=MemoryPool,name=PS Eden Space - CollectionUsageThreshold = 0
java.lang:type=MemoryPool,name=PS Eden Space - CollectionUsageThresholdCount = 0
java.lang:type=MemoryPool,name=PS Survivor Space - CollectionUsageThreshold = 0
java.lang:type=MemoryPool,name=PS Survivor Space - CollectionUsageThresholdCount = 0
java.lang:type=MemoryPool,name=Code Cache - UsageThreshold = 0
java.lang:type=MemoryPool,name=Code Cache - UsageThresholdCount = 0
java.lang:type=GarbageCollector,name=PS MarkSweep - CollectionCount = 1
java.lang:type=GarbageCollector,name=PS MarkSweep - CollectionTime = 90
java.lang:type=Runtime - StartTime = 1334354060679
java.lang:type=Runtime - Uptime = 5497
java.lang:type=ClassLoading - LoadedClassCount = 4806
java.lang:type=ClassLoading - UnloadedClassCount = 0
java.lang:type=ClassLoading - TotalLoadedClassCount = 4806
java.lang:type=Threading - DaemonThreadCount = 12
java.lang:type=Threading - PeakThreadCount = 13
java.lang:type=Threading - CurrentThreadCpuTime = 40000000
java.lang:type=Threading - CurrentThreadUserTime = 40000000
java.lang:type=Threading - ThreadCount = 13
java.lang:type=Threading - TotalStartedThreadCount = 13
java.lang:type=Compilation - TotalCompilationTime = 7144
java.lang:type=MemoryPool,name=PS Perm Gen - CollectionUsageThreshold = 0
java.lang:type=MemoryPool,name=PS Perm Gen - CollectionUsageThresholdCount = 0
java.lang:type=MemoryPool,name=PS Perm Gen - UsageThreshold = 0
java.lang:type=MemoryPool,name=PS Perm Gen - UsageThresholdCount = 0
java.lang:type=GarbageCollector,name=PS Scavenge - CollectionCount = 3
java.lang:type=GarbageCollector,name=PS Scavenge - CollectionTime = 57
java.lang:type=OperatingSystem - MaxFileDescriptorCount = 4096
java.lang:type=OperatingSystem - OpenFileDescriptorCount = 19
java.lang:type=OperatingSystem - CommittedVirtualMemorySize = 4598407168
java.lang:type=OperatingSystem - FreePhysicalMemorySize = 10819850240
java.lang:type=OperatingSystem - FreeSwapSpaceSize = 10742177792
java.lang:type=OperatingSystem - ProcessCpuTime = 13030000000
java.lang:type=OperatingSystem - TotalPhysicalMemorySize = 16816541696
java.lang:type=OperatingSystem - TotalSwapSpaceSize = 10742177792
java.lang:type=OperatingSystem - AvailableProcessors = 6
java.lang:type=OperatingSystem - SystemLoadAverage = 0
java.lang:type=MemoryPool,name=PS Old Gen - CollectionUsageThreshold = 0
java.lang:type=MemoryPool,name=PS Old Gen - CollectionUsageThresholdCount = 0
java.lang:type=MemoryPool,name=PS Old Gen - UsageThreshold = 0
java.lang:type=MemoryPool,name=PS Old Gen - UsageThresholdCount = 0

JASeries (JAnalyseSeries) 1.1.0 released

noreply@blogger.com (david crosson) — Fri, 13 Apr 2012 20:11:00 +0000

JASeries is a time numeric series operations API. The goal is to make simple, time series summaries generation, using sampling and various kind of cells merging. ( ** project page **)

Latest changes :

Statistics : alive added = OpenCell.time - CloseCell.time = series alive period of time
realias() without any argument means take the name as the alias.
added braces to delta, compact, zeroBased
Chart seriesScales method renamed to scales
Chart new method : colors which returns a map of [Series[Cell], Color]
Added a package object jaseries to define shortcuts to fr.janalyse.series.CSV2Series, fr.janalyse.series.view.Chart classes and objects

Google stock summary catchup scala script


#!/bin/sh
exec java -jar jaseries.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#

val allSeries = jaseries.CSV2Series.fromURL("http://ichart.finance.yahoo.com/table.csv?s=GOOG")
val closeSeries = allSeries("Close")

println("GOOGLE stock summary")
println("Higher : "+closeSeries.max)
println("Lowest : "+closeSeries.min)
println("Week Trend : "+closeSeries.stat.linearApproximation.daySlope*7)
println("Latest : "+closeSeries.last)

Apple trend chart generation scala script


#!/bin/sh
exec java -jar jaseries.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#

val allSeries = jaseries.CSV2Series.fromURL("http://ichart.finance.yahoo.com/table.csv?s=AAPL")
val closeSeries = allSeries("Close").realias("Apple stock value")    
jaseries.Chart(closeSeries).toFile("AppleStockTrend.jpg")

JAJMX (JAnalyseJMX) 0.3.0 released

noreply@blogger.com (david crosson) — Fri, 13 Apr 2012 19:15:00 +0000

JAJMX is high level scala JMX API. The goal is to simplify to the maximum JMX operations. ( ** project page **)

Latest changes :

JMXOptions now contains an extra field "name" which allow user to friendly identify a remote jmx system
added a package object jajmx to define shortcuts to fr.janalyse.jmx.JMX class and object
JMX.connect renamed to JMX.once
scaladoc generation now works fine, follow hack described here : https://github.com/harrah/xsbt/issues/85

jvm force gc script


#!/bin/sh
exec java -jar jajmx.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#

jajmx.JMX.once("127.0.0.1", 1099) { _.memory map {_ call "gc"} }

jmx grep script


#!/bin/sh
exec java -jar jajmx.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#

import jajmx._

if (args.size < 2) {
  println("Usage   : jmxgrep host port searchMask1 ... searchMaskN")
  System.exit(1)
}
val host  = args(0)
val port  = args(1).toInt
val masks = args.toList.drop(2) map {s=>("(?i)"+s).r}

def truncate(str:String, n:Int=60) = if (str.size>n) str.take(n)+"..." else str

JMX.once(host, port) { jmx =>
  for(mbean <- jmx.mbeans ; attr <- mbean.attributes; value <- mbean.getString(attr)) {
    val found = List(mbean.name, attr.name, value) exists { 
         item => masks exists {_.findFirstIn(item).isDefined }
    }
    if (found) println("%s - %s = %s".format(mbean.name, attr.name, truncate(value)))
  }
}

JASSH (JAnalyseSSH) 0.7.3 released

noreply@blogger.com (david crosson) — Thu, 12 Apr 2012 06:53:00 +0000

JASSH is a high level scala SSH API for easy and fast operations on remote servers. ( ** project page **)

Latest changes :

JSCH updated to release 0.1.47
SSHOptions now contains an extra field "name" which allow user to friendly identify a remote ssh system
SSHOptions password type is now of SSHPassword type instead of String. Implicit conversions are provided from String, Option[String]
SSHShell batch method renamed to execute

hello scala script


#!/bin/sh
exec java -jar jassh.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#

print(jassh.SSH.shell("localhost", "test", "testtest") {_ execute "echo Hello `hostname`" } )

remote vmstat scala script


#!/bin/sh
exec java -jar jassh.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#

jassh.SSH.once("localhost", "test", "testtest")  {
  _.run("vmstat 1 10", println(_.getOrElse("")).waitForEnd
}

JASSH (JAnalyseSSH) 0.7.2 released

noreply@blogger.com (david crosson) — Fri, 06 Apr 2012 19:57:00 +0000

JASSH is a high level scala SSH API for easy and fast operations on remote servers. ( ** project page **)

Latest changes :

added a package object jassh to define shortcuts to fr.janalyse.ssh.SSH class and object
SSHOptions, host parameter is now in first position !

These small changes enable even simpler scripts :

hello scala script


#!/bin/sh
exec java -jar jassh.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#

println(jassh.SSH.shell("localhost", "test", Some("testtest")) { _ execute "echo -n Hello `hostname`"})

remote vmstat scala script


#!/bin/sh
exec java -jar jassh.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#

jassh.SSH.once("localhost", "test", password=Some("testtest"))  {
  _.run("vmstat 1 10", println(_.getOrElse("")).waitForEnd
}

Leave it crash, akka will restart it for you !!!

noreply@blogger.com (david crosson) — Sun, 01 Apr 2012 20:28:00 +0000

Akka is an awesome tool, even using default configuration, you do not have anymore to worry about failure management and recovery details. I've been bluffed to discover that my new monitoring system prototype (akka/scala based) was able to recovery by itself from errors, while I didn't yet implement such features !!

So In order to illustrate this feature, I've written a simple example that illustrates this behavior. The RobustActor actor keeps a ssh connection to a remote system (using janalyse-ssh scala ssh api), and execute/print every 5s the "date" command :


package dummy

import akka.actor.ActorSystem
import akka.actor.Actor
import akka.util.duration._
import com.typesafe.config.ConfigFactory
import akka.actor.Props

import fr.janalyse.ssh._

object DummySSH {
 def main(args:Array[String]) {
      val system=ActorSystem("DummySSHSystem",ConfigFactory.load.getConfig("dummySSH"))
      system.actorOf(
        Props(new RobustActor(system)),
        name="RobustActor")
 }
}


class RobustActor(system:ActorSystem) extends Actor {
  
  val sh = SSH(host="localhost", username="test", password=Some("testtest")).newShell
  
  override def preStart() {
    system.scheduler.schedule(1 seconds, 5 seconds, self, "doit")
  }
  
  override def postStop() {
    sh.close()
  }
    
  def receive = {
    case "doit" => print(sh execute "date")
  }
}

If we start this example code, we wait a little, we identify the sshd process that manages the established ssh connection and then we kill the sshd process, akka will intercept RobustActor internal exception and restart this actor for us !


$ sbt run
[info] Loading project definition from /home/dcr/dev-new/akka-sandbox/project
[info] Set current project to AkkaSandbox (in build file:/home/dcr/dev-new/akka-sandbox/)
[info] Compiling 1 Scala source to /home/dcr/dev-new/akka-sandbox/target/scala-2.9.1/classes...

Multiple main classes detected, select one to run:

 [1] dummy.Dummy
 [2] dummy.DummySSH

Enter number: 2

[info] Running dummy.DummySSH 
dim. avril  1 22:15:32 CEST 2012
dim. avril  1 22:15:37 CEST 2012
dim. avril  1 22:15:42 CEST 2012
dim. avril  1 22:15:47 CEST 2012
[ERROR] [04/01/2012 22:15:53.92] [DummySSHSystem-akka.actor.default-dispatcher-3] [akka://DummySSHSystem/user/RobustActor] Pipe closed
java.io.IOException: Pipe closed
 at java.io.PipedInputStream.checkStateForReceive(PipedInputStream.java:244)
 at java.io.PipedInputStream.receive(PipedInputStream.java:210)
 at java.io.PipedOutputStream.write(PipedOutputStream.java:132)
 at java.io.OutputStream.write(OutputStream.java:58)
 at fr.janalyse.ssh.SSHShell$Producer.sendCommand(SSHAPI.scala:480)
 at fr.janalyse.ssh.SSHShell.sendCommand(SSHAPI.scala:475)
 at fr.janalyse.ssh.SSHShell.execute(SSHAPI.scala:448)
 at dummy.RobustActor$$anonfun$receive$1.apply(DummySSH.scala:34)
 at dummy.RobustActor$$anonfun$receive$1.apply(DummySSH.scala:33)
 at akka.actor.Actor$class.apply(Actor.scala:290)
 at dummy.RobustActor.apply(DummySSH.scala:21)
 at akka.actor.ActorCell.invoke(ActorCell.scala:617)
 at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:179)
 at akka.dispatch.Mailbox.run(Mailbox.scala:161)
 at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:505)
 at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
 at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:997)
 at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1495)
 at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

dim. avril  1 22:15:54 CEST 2012
dim. avril  1 22:15:58 CEST 2012

To identify the process which manages our example ssh connection, just use the following commands, list all sshd processes which belongs to test user, then take the process that do not use your current test shell session.


test@lanfeust ~ $ ps -ef | grep "^test"
test      8824  8815  0 11:20 ?        00:00:00 sshd: test@pts/7 
test      8825  8824  0 11:20 pts/7    00:00:00 -bash
test     32468 32458  0 22:15 ?        00:00:00 sshd: test@pts/8 
test     32469 32468  0 22:15 pts/8    00:00:00 -bash
test     32474  8825  0 22:15 pts/7    00:00:00 ps -ef
test     32475  8825  0 22:15 pts/7    00:00:00 grep --colour=auto ^test
test@lanfeust ~ $ tty
/dev/pts/7
test@lanfeust ~ $ kill -9 32468
test@lanfeust ~ $

This example source code is available here : akka-sandbox

JASSH (JAnalyseSSH) 0.7.0 released

noreply@blogger.com (david crosson) — Sun, 01 Apr 2012 13:50:00 +0000

JAnalyseSSH is a jsch based scala SSH API which aims to simplify SSH operations.

The hello world script...

(It requires a local user named "test" with password "testtest", remember that you can remove the password, if your public key has been added in authorized_keys file of the test user)


#!/bin/sh
exec java -jar jassh.jar -usejavacp "$0" "$@"
!#
import fr.janalyse.ssh._
SSH.shell(host="localhost", username="test", password=Some("testtest")) { sh =>
  println(sh.execute("""echo -n "Hello World from `hostname`" """))
}

... and the remote vmstat script


#!/bin/sh
exec java -jar jassh.jar -usejavacp "$0" "$@"
!#
import fr.janalyse.ssh._
SSH.once(host="localhost", username="test", password=Some("testtest"))  { ssh =>
  val executor = ssh.run("vmstat 1 10", (l)=> l foreach {println(_)})
  executor.waitForEnd
}

Releases notes for 0.7.0

Added new method to SSH : newShell & newSftp for user to manage themselves shell and sftp session
Some internal changes to SSHExec class, in order to try to remove actor dependency. Mixing actors systems looks problematic
SSHShell new implementation, no more actors used, better performances and behavior, ... (throughput : 504 cmd/s using persistency)
SSHExec last result line is no longer lost
SSHOptions : new parameter : "prompt" to enable custom shell or console command to be use.
prompt provide to SSHShell the way to separate command results
SSHOptions : connectionTimeout renamed into timeout
Various cleanup and enhancements
Tests : compare performances persistent SSHShell versus SSHExec commands throughputs
SSH : Add an execute immediate method which rely on SSHExec, not SSHShell ! (throughput : 93cmd/s)
execOnce & execOnceAndTrim
SSHExec : Do not rely on DaemonActor/Actor anymore
SSHShell : Removed init Thread.sleep => Better performances (throughput : 37 cmd/s instead 1cmd/s)
SSH.connect becomes SSH.once
Removing apply in SSH class as it may encourage bad usage, and close not called

Dependency injection using scala traits

noreply@blogger.com (david crosson) — Sun, 11 Mar 2012 13:57:00 +0000

Consider the following code example which does not depend on any kind of configuration system. We've just defined a trait, XConfiguration, which describes how we can get simple configuration data.


trait XConfigurable {
  def getSetting(key:String):Option[String] = None
}

trait XPlugin extends XConfigurable {
  def domain:String
}

trait XSniffPlugin extends XPlugin {
  val period=getSetting("period") map {_.toLong} getOrElse 5000L
  val periodInS=period/1000
}

abstract class XSSHPlugin extends XSniffPlugin {
  val host = getSetting("host") getOrElse "127.0.0.1"
  val username = getSetting("username") getOrElse util.Properties.userName
  val password = getSetting("password") 
  val port = getSetting("port") map {_.toInt} getOrElse 22
  def cmd:String
}

case class XTopPlugin(domain:String) extends XSSHPlugin {
  val ignoreKeywords = getSetting("ignoreKeywords")
  def cmd="top -c -b -d %d".format(periodInS)
  override def toString = 
    "top each %ds through ssh to %s%s@%s".format(
        period/1000, username, password map {":"+_} getOrElse "", host
        )
}

Now I would like to modify XTopPlugin behavior in order to let it takes its configuration through typesafe config library, BUT without modifying anything in the code.

The idea here is to create a new trait which inheritates from XConfigurable :


object PlayWithTrait {
  import com.typesafe.config.Config
  import com.typesafe.config.ConfigFactory
  import com.typesafe.config.ConfigException.{Missing,WrongType}
    
  private lazy val context = ConfigFactory.parseString("""
        |Defaults {
        |  period=5000
        |}
        |SSHPlugin = ${Defaults}
        |SSHPlugin = {
        |  host:"127.0.0.1"
        |  port:22
        |}
        |TopPlugin = ${SSHPlugin}
        |TopPlugin = {
        |  ignoreKeywords:"tty$"
        |} """.stripMargin).resolve()

  trait XConfigurableUsingConfig extends XConfigurable {
    def config:Config
    override def getSetting(key:String):Option[String] = {
      try {
        Some(config.getString(key))
      } catch {
        case x:Missing => None
        case x:WrongType => None
      }
    }
  }

...

And then we'll take advantage of scala trait feature which allow us to stack a new behavior to an object at instanciation time :


  def main(args: Array[String]) {
    import collection.JavaConversions._
    
    val localCfg = Map(
        "host" -> "192.168.2.1",
        "username" -> "test",
        "password" -> "testtest"
        )
    
    val topConfig = ConfigFactory.parseMap(localCfg).withFallback(context.getConfig("TopPlugin"))
    
    val top = new XTopPlugin("myserver") with XConfigurableUsingConfig {def config=topConfig}
    
    println(top)
  }

So now our XTopPlugin instance is taking its configuration through typesafe "Config" library, using a kind of closure to give the configuration context (topConfig).

One can object that it is less flexible than using for example spring and its xml configuration files, but remember that scala compiler can be easily embedded in any application, so you just have to use a scala configuration file which will get compiled at runtime... it will bring you very awesome dependencies injection features using pure scala.

Akka actors : 10 millions messages processed (1s / message) in 75 seconds !

noreply@blogger.com (david crosson) — Thu, 01 Mar 2012 21:17:00 +0000

In the previous POST, Akka actors versus Scala actors : control the message throughput of actors, we've seen how to control message throughput and avoid OutOfMemory problem in particular when actors doesn't have the same messages processing rate.

Now we have to find a way to find a better way to simulate our 1 second delay, because using Thread.sleep breaks actors management as it will monopolize all threads of the used executor. In fact we would like to simulate a not-blocking 1s processing time.

So we should remove the following Thread.sleep call :


class MyMessageProcessor extends Actor {
  def receive = {
    case _:DoItMessage => Thread.sleep(1000)
  }
}

The solution is straightforward as Akka comes with a scheduler system which allow you to send a message after a given delay.


class MySimulator(system:ActorSystem) extends Actor {
  def receive = {
    case _:DoItMessage =>
      system.scheduler.scheduleOnce(1000 milliseconds, sender, "Done") 
  }
}

Using scheduleOnce, we didn't monopolize system resources, no thread is used, in fact we achieve a fake processing which simulates an asynchronous response which come 1s later. In the following code, the response is even received asynchronously by a future :


package dummy

import akka.actor._
import akka.util.duration._
import akka.util.Timeout
import akka.pattern.ask
import akka.dispatch.Future

sealed trait MyMessage
case class DoItMessage(cmd:String) extends MyMessage
case class DoneMessage extends MyMessage


object Dummy {
  def main(args:Array[String]) {
    
    val howmanyjob=10*1000000
    
 import com.typesafe.config.ConfigFactory
    implicit val system=ActorSystem("DummySystem",ConfigFactory.load.getConfig("dummy"))
    
    val simu = system.actorOf(
        Props(new MySimulator(system))
        .withDispatcher("simu-dispatcher"),
        name="simulator")

    val appManager = system.actorOf(
        Props(new ApplicationManager(system, howmanyjob))
        .withDispatcher("simu-dispatcher"),
        name="application-manager")

        
    import akka.routing.RoundRobinRouter
    val processor = system.actorOf(
        Props(new MyMessageProcessor(appManager, simu))
        .withDispatcher("workers-dispatcher")
        .withRouter(RoundRobinRouter(10)),
        name="default")
        
    for(i <- 1 to howmanyjob) {
      processor ! DoItMessage("Do the job with ID#%d now".format(i))
    }
    print("All jobs sent")
  }
}

class MyMessageProcessor(appManager:ActorRef, simu:ActorRef) extends Actor {
  def receive = {
    case msg:DoItMessage =>
      implicit val timeout = Timeout(5 minutes)
      val receivedTime = System.currentTimeMillis()
      val future = simu ? msg
      future.onComplete { 
        case result:Either[Throwable, String] =>
          assert(System.currentTimeMillis()-receivedTime >= 1000)
          appManager ! DoneMessage
      }
  }
}

class MySimulator(system:ActorSystem) extends Actor {
  def receive = {
    case _:DoItMessage =>
      // Fake processing, somewhere, the job is executed and we get 
      // the results 1s later asynchronously
      system.scheduler.scheduleOnce(1000 milliseconds, sender, "Done") 
  }
}

class ApplicationManager(system:ActorSystem, howmanyjob:Int) extends Actor {
  val startedTime = System.currentTimeMillis()
  var count=0
  def receive = {
    case DoneMessage => 
      count+=1
      if (count%(howmanyjob/20)==0) println("%d/%d processed".format(count, howmanyjob))
      if (count == howmanyjob) {
        val now=System.currentTimeMillis()
        println("Everything processed in %d seconds".format((now-startedTime)/1000))
        system.shutdown() 
      }
  }
}

The application configuration is the following (application.conf file) :


dummy {
  akka {
    loglevel = WARNING
    actor {
      default-dispatcher {
      }
    }
    scheduler {
      tick-duration = 50ms
      ticks-per-wheel = 1000
    }
  }
  
  simu-dispatcher {
    type = Dispatcher
    mailbox-capacity = 100000
  }
  
  workers-dispatcher {
    mailbox-capacity = 10000
    executor = "fork-join-executor"
    fork-join-executor {
      parallelism-min = 0
      parallelism-max = 6000
      parallelism-factor = 3.0      
    }
  }
}

The build.sbt SBT file (To get dependencies, build, test & run) is the following :


name := "AkkaSandbox"

version := "0.1"

scalaVersion := "2.9.1"

libraryDependencies += "com.typesafe.akka" % "akka-actor" % "2.0-RC3"

resolvers += "Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/"

Everything is processed in 75 seconds using :

linux 3.2.1-gentoo-r2 64bits
AMD Phenom(tm) II X6 1090T (6 CPU cores)
Java HotSpot(TM) 64-Bit Server 1.6.0.31

CPU usage ~80%, java memory footprint is below 300Mo; each message is requiring 50-60 bytes, so the 10 million messages are requiring at least 480Mb (but not simultaneously).

Akka actors versus Scala actors : control the message throughput of actors

noreply@blogger.com (david crosson) — Thu, 23 Feb 2012 17:50:00 +0000

In the post following this one, I show how to enhance the example code in order to be able to process a very high number of messages : Akka actors : 10 millions messages processed (1s / message) in 75 seconds !

Scala default actor system doesn't come with any mechanism to control messages throughput between actors, so it is up to you to implement such feature in order to avoid OutOfMemory problem, in particular when actors are not processing messages at the same rate.
Akka framework is very interesting because it brings you all the features to control and manage throughputs. Let's see how it work through a simple example :


import akka.actor._

sealed trait MyMessage
case class DoItMessage(cmd:String) extends MyMessage

object Dummy {
  def main(args:Array[String]) {
    val system=ActorSystem("DummySystem")
    val processor = system.actorOf(Props[MyMessageProcessor],name="default")
    for(i <- 1 to 10000000) {
      processor ! DoItMessage("Do the job with ID#%d now".format(i))
    }
    println("All jobs sent")
  }
}

class MyMessageProcessor extends Actor {
  def receive = {
    case _:DoItMessage => Thread.sleep(1000)
  }
}

Each message sent is more than 50 bytes, so a JVM with a 512Mo of maximum heap size will quickly run out of memory.

The good news is that akka support bounded actor mailboxes, just add a configuration file with such content :


dummy {
  akka {
    loglevel = WARNING
    actor {
      default-dispatcher {
         mailbox-capacity = 10000
      }
    }
  }
}

And make this configuration file taken into account with the following code update :


    import com.typesafe.config.ConfigFactory
    val system=ActorSystem("DummySystem",ConfigFactory.load.getConfig("dummy"))

This time if you run again the previous code, you'll get a stable memory footprint :

Of course it will take a long time to process as each message requires 1s (and actors are blocked during 1s because of Thread.sleep call, which monopolize a thread during the same duration => waste of resources)... If we want to reduce the processing time although each message requires 1s to be taken into account, we'll have to load-balance message processing accross a high number of actors.

CONTEXT : Scala 2.9.1 / SBT 0.11.2 / AKKA 2.0-rc2 / sbteclipse 2.0.0

Revisiting Git/SVN Update script... and let's try to make it faster

noreply@blogger.com (david crosson) — Thu, 23 Feb 2012 10:24:00 +0000

The following script is an enhancement of the one shown in the previous post. It now detects all git or svn projects in the current directory, and then update and compile them.

It could have been straightforward to make it faster by just turning "project" collection into a parallel one (projects.par) but unfortunately, it won't work because of the use of a global mutable variable used to store the current worker directory (Classical trap) ! So more work is mandatory to make this faster.


#!/bin/sh
exec scala -deprecation -savecompiled "$0" "$@"
!#
import java.io.File
import sys.process.Process
import sys.process.ProcessBuilder._

// ======================================================================
case class CurDir(cwd:File)
def file(f:String)=new File(f)
def file(d:File, f:String)=new File(d, f)
implicit def stringToCurDir(d:String) = CurDir(file(d))
implicit def fileToCurDir(d:File) = CurDir(d)
implicit def stringToFile(f:String) = file(f)
implicit def stringToProcess(cmd: String)(implicit curDir:CurDir) = Process(cmd, curDir.cwd)
implicit var cwd:CurDir=file(".").getPath
def cd(dir:File=file(util.Properties.userDir)) = cwd=CurDir(dir)
// ======================================================================

trait Project {
  val dir:File
}
case class GitProject(dir:File) extends Project
case class SvnProject(dir:File) extends Project

// Find GIT or SVN project directories
val projects = file(".").listFiles filter { _.isDirectory} flatMap { d =>
  if (file(d, ".git").exists) Some(GitProject(d))
  else if (file(d, ".svn").exists) Some(SvnProject(d))
  else None
}

// Update compile project (update also eclipse conf files generated by sbteclipse plugin)
for(project<- projects) {
  println(project)
  cd(project.dir)
  project match {
    case GitProject(_) => "git pull" !
    case SvnProject(_) => "svn update" !
  }
  "sbt eclipse compile" !
}

Let's now revisit the current directory hack to make it compatible with parallel processing and avoid border side effect.
Therefore, we have to modify the "cd" command hack in order to remove the mutable cwd variable :


#!/bin/sh
exec scala -deprecation -savecompiled "$0" "$@"
!#
import java.io.File
import sys.process.Process
import sys.process.ProcessBuilder._

// ======================================================================
case class CurDir(cwd:File)
def file(f:String)=new File(f)
def file(d:File, f:String)=new File(d, f)
implicit def stringToCurDir(d:String) = CurDir(file(d))
implicit def fileToCurDir(d:File) = CurDir(d)
implicit def stringToFile(f:String) = file(f)
implicit def stringToProcess(cmd: String)(implicit curDir:CurDir) = Process(cmd, curDir.cwd)
def cd[A](dir:File=file(util.Properties.userDir)) (indir:(CurDir)=>A) = indir(CurDir(dir))
// ======================================================================

trait Project {
  val dir:File
}
case class GitProject(dir:File) extends Project
case class SvnProject(dir:File) extends Project

// Find GIT or SVN project directories
val projects = file(".").listFiles filter { _.isDirectory} flatMap { d =>
  if (file(d, ".git").exists) Some(GitProject(d))
  else if (file(d, ".svn").exists) Some(SvnProject(d))
  else None
}

// Update compile project (update also eclipse conf files generated by sbteclipse plugin)
for(project<- projects.par) {
  println(project)
  cd(project.dir) { implicit cwd =>
    project match {
      case GitProject(_) => "git pull" !
      case SvnProject(_) => "svn update" !
    }
    "sbt eclipse compile" !
  }
}

println("%d project updated".format(projects.size))

So we couln't keep the classical shell cd behavior, because it implies a mutable state, so in this script we turn the cd "classical" definition into a new one, with a new argument, a function which will take as argument, the given working directory to use.

In my case (7 projects), using "projects.par", the script executes in 10s instead of 33s !

CONTEXT : Scala 2.9.1

Small script to update & compile a list of svn project

noreply@blogger.com (david crosson) — Wed, 01 Feb 2012 20:53:00 +0000

Scala script skeleton to update a set of subversion projects.


#!/bin/sh
exec scala -deprecation -savecompiled "$0" "$@"
!#

// ======================================================================
import sys.process.Process
import sys.process.ProcessBuilder._

case class CurDir(cwd:java.io.File)
implicit def stringToCurDir(d:String) = CurDir(new java.io.File(d))
implicit def stringToProcess(cmd: String)(implicit curDir:CurDir) = Process(cmd, curDir.cwd)
implicit def stringSeqToProcess(cmd:Seq[String])(implicit curDir:CurDir) = Process(cmd, curDir.cwd)

implicit var cwd:CurDir=scala.util.Properties.userDir
def cd(dir:String=util.Properties.userDir) = cwd=dir
// ======================================================================

val updateList=List("project1", "project2", "project3")

for(dir<- updateList) {
  println("----------------------------------------")
  println("Processing %s".format(dir))

  cd(dir)

  "svn update" !

  "sbt eclipse compile" !
}

Simplifying scala scripts : Adding #include support to your scripts

noreply@blogger.com (david crosson) — Mon, 30 Jan 2012 22:30:00 +0000

Test a #include support example by executing the following 5 lines :


$ wget http://dnld.crosson.org/bootstrap.tar.gz
$ tar xvfz bootstrap.tar.gz
$ cd bootstrap
$ sbt assembly
$ ./scripts/test.scala

The test.scala example script is the following :


#!/bin/sh
DIRNAME=`dirname "$0"`
exec java -jar "$DIRNAME"/bootstrap.jar "$0" "$@"
!#

#include "shell.scala"

cd("/etc/")

"ls" #| "grep net" !

go to /etc directory, and prints files which contain net keyword in their names.

test.scala script includes the following file : scripts/include/shell.scala


import sys.process.Process
import sys.process.ProcessBuilder._
 
case class CurDir(cwd:java.io.File)
implicit def stringToCurDir(d:String) = CurDir(new java.io.File(d))
implicit def stringToProcess(cmd: String)(implicit curDir:CurDir) = Process(cmd, curDir.cwd)
implicit def stringSeqToProcess(cmd:Seq[String])(implicit curDir:CurDir) = Process(cmd, curDir.cwd)

implicit var cwd:CurDir=scala.util.Properties.userDir
def cd(dir:String=util.Properties.userDir) = cwd=dir

This file contains some definitions to make possible for the user to change current directory.

All the include mechanism logic in defined as follow :


package fr.janalyse.script

import scala.tools.nsc.ScriptRunner
import scala.tools.nsc.GenericRunnerCommand
import scala.io.Source
import java.io.File

object Bootstrap {
  val defaultOptions = List("-nocompdaemon","-usejavacp","-savecompiled", "-deprecation")
  val defaultExpandedScriptExt = ".pscala"
  
  val includeRE = """\s*#include\s+"(.+)"\s*"""r
  
  def expand(file:File, availableIncludes:List[File]) : List[String] = {
    val content=Source.fromFile(file).getLines().toList
    // First we remove "shell" startup lines, everything between #! and !#
    val cleanedContent = content.indexWhere { _.trim.startsWith("!#") } match {
      case -1 => content
      case i  => content.drop(i+1)
    }
   // Then we expand #include directives
   cleanedContent flatMap {
     case includeRE(filename) =>
       val fileOpt = availableIncludes find {_.getName() == filename}
       fileOpt orElse {
          throw new RuntimeException("%s : Couln't find include file '%s' ".format(file.getName, filename))
       }
       fileOpt map { file => expand(file, availableIncludes)} getOrElse List.empty[String]  
     case line => line::Nil
   }
  }
    
  def main(cmdargs:Array[String]) {
    val command = new GenericRunnerCommand(defaultOptions ++ cmdargs.toList)
    val scriptDir = new File(cmdargs(0)).getParentFile()
    val includePath = List(new File(scriptDir, "include"), scriptDir)
    val availableIncludes = includePath filter {_.exists()} flatMap {_.listFiles()}
    val scriptname = command.thingToRun 
    val script = new File(scriptname)
    val richerScript = new File(scriptname.replaceFirst(".scala", defaultExpandedScriptExt))
    
    if (script.exists()) {
      val jars = util.Properties.javaClassPath.split(File.pathSeparator) map {new File(_)} collect {
        case f if (f.exists() && f.isFile()) => f 
      }
      val jarsLastModified = (jars map {_.lastModified()} max)
      
      if (!richerScript.exists ||  // -- nothing already available
          (jarsLastModified > richerScript.lastModified) ||   // -- Bootstrap jar is newer
          (script.lastModified > richerScript.lastModified)) { // -- Script has been modified
        val newcontent =  expand(script, availableIncludes).mkString("\n")
        new java.io.FileOutputStream(richerScript) {
          write(newcontent.getBytes())
        }.close()
      }
    }
    ScriptRunner.runScript(command.settings, richerScript.getPath, command.arguments)
  }
}

How does it work :
The principle is to override scala standard script startup mechanism by introducing an additionnal step which consist to expand the script with all includes it contains, and then gives to scala the new script resulting of expansion process.
test.scala becomes test.pscala which will generate the savedcompile file test.pscala.jar. No recompilation will be required as soon as no change occured on test.scala or bootstrap.jar file.

You should also notice that the script is started using 'exec java -jar "$DIRNAME"/bootstrap.jar "$0" "$@"' and not 'exec scala ...' because bootstrap is an assembly jar which contains everything to run and compile scala scripts, and even more if you want, as it can include any third parties you may need, just add library dependencies ! So you only need one file, bootstrap.jar, to run any scala scripts, nothing to install, just one file to upload.

SBT build configuration : bootstrap/build.sbt


import AssemblyKeys._

seq(assemblySettings: _*)

name := "bootstrap"

version := "0.1"

scalaVersion := "2.9.1"

libraryDependencies <++=  scalaVersion { sv =>
   ("org.scala-lang" % "scala-swing" % sv) ::
   ("org.scala-lang" % "jline"           % sv  % "compile") ::
   ("org.scala-lang" % "scala-compiler"  % sv  % "compile") ::
   ("org.scala-lang" % "scala-dbc"       % sv  % "compile") ::
   ("org.scala-lang" % "scalap"          % sv  % "compile") ::
   ("org.scala-lang" % "scala-swing"     % sv  % "compile") ::Nil   
}

mainClass in assembly := Some("fr.janalyse.script.Bootstrap")

jarName in assembly := "bootstrap.jar"

SBT Plugins configuration : bootstrap/project/plugins.sbt file


resolvers += Classpaths.typesafeResolver

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.0.0-M3")

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.7.2")

Simplifying scala scripts

noreply@blogger.com (david crosson) — Thu, 26 Jan 2012 22:42:00 +0000

I've already written several dozens of scala scripts (series computations, ssh automation, various jmx operations, remote administration, garbage collector log analysis, ...) and found this language quite interesting for use as a script language. There's many reasons for that :

Script automatic compilation reduce runtime error. I am often amazed at the first attempt to get a script that works and without any runtime error !
Take benefits of scala powerfull collections that make possible to write "sql" like operations
It becomes straightforward to parallelize tasks using Actors; one of my favorite use case is a short script that trigger an explicit garbage collection on several dozens of remote jvm in a very short time

But I miss some features that will help to make scala scripts even simpler and concise :

A #include like feature within script
A way to modify default imports, to avoid adding always the same imports in all scripts
the #! !# shell scala bootstrap can become long (and not DRY) once you want to add many external java dependencies

In fact those missing features are no so difficult to implement, the following source code is a proof of concept that shows it no so difficult to implement those features. It defines a class, Bootstrap, which can be use to start a scala script and that will bring new imports and definitions to your script.


package fr.janalyse.script

import scala.tools.nsc.ScriptRunner
import scala.tools.nsc.GenericRunnerCommand
import scala.io.Source

object Bootstrap {

  val header = 
"""// WARNING
// Automatically generated file - do not edit !
import sys.process.Process
import sys.process.ProcessBuilder._
 
case class CurDir(cwd:java.io.File)
implicit def stringToCurDir(d:String) = CurDir(new java.io.File(d))
implicit def stringToProcess(cmd: String)(implicit curDir:CurDir) = Process(cmd, curDir.cwd)
implicit def stringSeqToProcess(cmd:Seq[String])(implicit curDir:CurDir) = Process(cmd, curDir.cwd)

implicit var cwd:CurDir=scala.util.Properties.userDir
def cd(dir:String=util.Properties.userDir) = cwd=dir

"""

  val footer = 
"""
"""

  def main(cmdargs:Array[String]) {

    def f(name:String) = new java.io.File(name)
    
    val na = List("-nocompdaemon","-usejavacp","-savecompiled", "-deprecation") ++ cmdargs.toList
    
    val command = new GenericRunnerCommand(na)
    
    import command.settings
    
    val scriptname = command.thingToRun 
    val script = f(scriptname)
    val richerScript = f(scriptname.replaceFirst(".scala", ".scala-plus"))
    
    if (script.exists()) {
      if (!richerScript.exists || (script.lastModified > richerScript.lastModified)) {
        val content=Source.fromFile(script).getLines().toList
        val cleanedContent = content.dropWhile(x => !x.startsWith("!#")).tail.mkString("\n")
        val newcontent =  List(header, cleanedContent, footer).mkString("\n")
        new java.io.FileOutputStream(richerScript) {
          write(newcontent.getBytes())
        }.close()
      }
    }
    
    val args = command.arguments
    
    ScriptRunner.runScript(settings, richerScript.getName, args)
  }  
}

Then generate a standalone executable jar with this class and all needed dependencies, thanks to such SBT build specification :


import AssemblyKeys._

seq(assemblySettings: _*)

name := "bootstrap"

version := "0.1"

scalaVersion := "2.9.1"

libraryDependencies <++=  scalaVersion { sv =>
   ("org.scala-lang" % "scala-swing" % sv) ::
   ("org.scala-lang" % "jline"           % sv  % "compile") ::
   ("org.scala-lang" % "scala-compiler"  % sv  % "compile") ::
   ("org.scala-lang" % "scala-dbc"       % sv  % "compile") ::
   ("org.scala-lang" % "scalap"          % sv  % "compile") ::
   ("org.scala-lang" % "scala-swing"     % sv  % "compile") ::Nil   
}

mainClass in assembly := Some("fr.janalyse.script.Bootstrap")

jarName in assembly := "bootstrap.jar"

you'll be able to directly run any scala script like that :


#!/bin/sh
exec java -jar bootstrap.jar "$0" "$@"
!#

cd("/etc/")

"ls" #| "grep net" !

Thanks to the assembly SBT plugin, you've generated a standalone executable jar, which contains the scala compiler, and our custom scala script startup mechanism.
In a next POST, I'll describe more in detail a new bootstrap implementation that will bring #include feature to scala script.

A simple approach to generate reports using scala

noreply@blogger.com (david crosson) — Tue, 24 Jan 2012 21:19:00 +0000

I was wondering if it was possible to create one executable script containing both the report template and its configuration. So I tried to write a scala script to test this idea :


#!/bin/sh
exec scala -nocompdaemon -usejavacp -savecompiled -deprecation "$0" "$@"
!#

import sys.process._
import java.io.{File, FileOutputStream}

println("Let's generate and display a report")

val message = "hello"
val guys = List("Marge", "Bart", "Omer")

val smallreport = 
< html>
  < head>
    < title>Hello guys report< /title>
  < /head>
  < body>
    < h1>{message}< /h1>
    {guys map {guy=> 
      < h2>{guy}< /h2>
    } }
  < /body>
< /html>

val reportfile = File.createTempFile("report", ".html")
reportfile.deleteOnExit

new FileOutputStream(reportfile) {
  write(smallreport.toString.getBytes)
}.close()

List("firefox", "-new-window", reportfile.toURI.toURL.toString) !

Note: I've added one space inside each HTML TAG, looks like SyntaxHighlighter doesn't like HTML or XML embedded within scala

Make this script executable (chmod u+x reportingScript.scala) and execute it (./reportingScript.scala), if firefox is available on your operating system, the following window will be displayed :

Now we can say :
- it is possible to combine a report template, and data access in a single executable script !
- By using a custom scala startup command, it should be possible to simplify many operations, extending defaults imports, ...

In a next post I'll try to make such script simpler.

How to generate google stock trend chart in 3 lines...

noreply@blogger.com (david crosson) — Sun, 22 Jan 2012 17:25:00 +0000

Using janalyse-series scala API, it becomes very simple to generate a trend chart, 3 lines are enough in order to download, parse CSV data and then generate the chart.


#!/bin/sh
exec java -jar jaseries.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#

import fr.janalyse.series.CSV2Series
import fr.janalyse.series.view.Chart

val allSeries = CSV2Series.fromURL("http://ichart.finance.yahoo.com/table.csv?s=GOOG")
val closeSeries = allSeries("Close").rename("Google stock value")    
Chart(closeSeries).toJpgFile(new java.io.File("googleStockTrend.jpg"), 800, 400)

The following chart is created.

CSV files merge, based on series names...

noreply@blogger.com (david crosson) — Sun, 22 Jan 2012 08:45:00 +0000

This example illustrates how janalyse-series API can be use to merge CSV files together while respecting series names. It takes a set of directory containing CSV files, merge all series and write results in a new directory. The merge is only based on series names, not file names, so if for one series, different file names were used, then only one of them will be chosen in the final destination directory, the shortest one.


#!/bin/sh
exec jaseries -deprecation -savecompiled "$0" "$@"
!#
/*
 * Copyright 2011 David Crosson
 * 
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 * 
 *   http://www.apache.org/licenses/LICENSE-2.0
 * 
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
import fr.janalyse.series._
import java.io.File

if (args.size < 2) {
  println("usage : csv-merge srcDir1 ... srcDirN destNewDir")
  println("  Merge and reduce numeric series stored in CSV files")
  System.exit(0)
}

case class InputSeries(name:String, filebasename:String, series:Series[Cell])
def file(filename:String) = new File(filename)
def file(dirname:String, filename:String) = new File(dirname, filename)
def file(dir:File, filename:String) = new File(dir, filename)

def csvFileFilter(f:String):Boolean = {f.endsWith(".csv") ||f.endsWith(".jxtcsv")}
val basename1RE="""(.*)(?:[-_]\d+)[.].+"""r
val basename2RE="""(.*)[.].+"""r
def basename(filename:String) = filename match {
  case basename1RE(basename) => basename
  case basename2RE(basename) => basename
  case other => other
}

val toMerge = args.init
val destNewDir = file(args.last)

destNewDir.mkdirs()

// --- Read everything
var inputSeriesList=List.empty[InputSeries]
for( dirname<-toMerge ; 
     filename<-file(dirname).list filter csvFileFilter;
     (seriesname, series)<-CSV2Series.fromFile(file(dirname, filename))) {
  val filebasename = basename(filename)
  inputSeriesList = InputSeries(seriesname, filebasename, series)::inputSeriesList
}

// --- Merge and Reduce
val seriesGroupByName=inputSeriesList groupBy {is => is.name}
val mergedSeriesList = for( (seriesname, inputSeriesList4Name) <- seriesGroupByName) yield {
  val mergedseries   = inputSeriesList4Name.map(_.series) reduceLeft {_ <<< _}
  val mergedbasename = inputSeriesList4Name.map(_.filebasename).min
  (mergedseries, mergedbasename)
}

// --- Write everything
val mergedSeriesGroupByBasename = mergedSeriesList groupBy { case (_, basename) => basename}
for( (mergedbasename, mergedTuples) <- mergedSeriesGroupByBasename) {
  val seriesList = mergedTuples map {case (series,_) => series}
  CSV2Series.toFile(seriesList, file(destNewDir, mergedbasename+".csv"))
}

Build Play20 with latest sources, and let's play.

noreply@blogger.com (david crosson) — Thu, 05 Jan 2012 13:31:00 +0000

All the steps to run your first play 2 web application using latest play improvements; until the first stable release will be ready. The idea here is to build yourself Play. Eclipsify is now taken into account, so once all the steps described here are done, you can import it into eclipse IDE.


$ cd ~/workdir

$ git clone https://github.com/playframework/Play20.git

$ cd Play20/framework

$ ./build
> build-repository
> exit

$ export PATH=$PATH~/workdir/Play20

$ cd ~/workdir

$ play new firstapp

Getting play console_2.9.1 2.0-RC1-SNAPSHOT ...
:: retrieving :: org.scala-tools.sbt#boot-app
 confs: [default]
 5 artifacts copied, 0 already retrieved (5161kB/19ms)
       _            _ 
 _ __ | | __ _ _  _| |
| '_ \| |/ _' | || |_|
|  __/|_|\____|\__ (_)
|_|            |__/ 
             
play! 2.0-RC1-SNAPSHOT, http://www.playframework.org

The new application will be created in /home/dcr/experiments/firstapp

What is the application name? 
> firstapp

Which template do you want to use for this new application? 

  1 - Create a simple Scala application
  2 - Create a simple Java application
  3 - Create an empty project

> 1

OK, application firstapp is created.

Have fun!

$ cd firstapp

$ play 
> eclipsify
> run
--- (Running the application from SBT, auto-reloading is enabled) ---

[info] play - Listening for HTTP on port 9000...

(Server started, use Ctrl+D to stop and go back to the console...)

Now everything is ready, to test and modify the simple scala project application using eclipse. More information can be found on : play20 and scala ide 20 tutorial.

Easy SSH scala scripting

noreply@blogger.com (david crosson) — Wed, 14 Dec 2011 22:46:00 +0000

Visit janalyse-ssh, and download "jassh.jar" standalone executable jar.
In the same directory where you've downloaded "jassh.jar"file, create a file named "helloworld", and copy&paste the following content in it :


#!/bin/sh
exec java -jar jassh.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#
import fr.janalyse.ssh._
SSH.connect(host="localhost", username="test", password="testtest") { ssh =>
  println(ssh.execute("""echo -n "Hello World from `hostname`" """))
}

Change the specified user to an existing user, or create a "test" user. Give the right password or remove it if you've exported your public SSH key in remote user ~test/.ssh/authorized_keys file
Then make your "helloworld" file executable (chmod u+x helloworld), and execute it :


toto@myhost $ ./helloworld 
Hello World from myhost

Easy ! And you'll take all benefits of the scala language using just vi, in order to make any kind of remote operations ! No graphical user interface required, no compiler (automatic & transparent compilation on the fly when needed), high performances (I've reached a command throughput up to 545 cmd/s on my host), parallelism using actors, easy parsing using combinators, ...

JMX grep scala script

noreply@blogger.com (david crosson) — Sun, 11 Dec 2011 20:36:00 +0000

A simple script to search for mbeans/attributes-names/attribute-values matching regular expressions given as script parameters. This script is using janalyse-jmx scala API; jajmx.jar java executable is used to startup the script.
This jmx grep script can be used against itself, with its own embedded jmx platform. This feature is enabled through java options (JAVA_OPTS) specified within script startup header.


$ ./jmxgrep localhost 9999  operating
java.lang:type=OperatingSystem - MaxFileDescriptorCount = 4096
java.lang:type=OperatingSystem - OpenFileDescriptorCount = 18
java.lang:type=OperatingSystem - CommittedVirtualMemorySize = 4520407040
java.lang:type=OperatingSystem - FreePhysicalMemorySize = 7900864512
java.lang:type=OperatingSystem - FreeSwapSpaceSize = 10742177792
java.lang:type=OperatingSystem - ProcessCpuTime = 1490000000
java.lang:type=OperatingSystem - TotalPhysicalMemorySize = 16848113664
java.lang:type=OperatingSystem - TotalSwapSpaceSize = 10742177792
java.lang:type=OperatingSystem - Name = Linux
java.lang:type=OperatingSystem - AvailableProcessors = 6
java.lang:type=OperatingSystem - Arch = amd64
java.lang:type=OperatingSystem - SystemLoadAverage = 0.0
java.lang:type=OperatingSystem - Version = 3.0.6-gentoo

$ ./jmxgrep localhost 9999 threadcount
java.lang:type=GarbageCollector,name=PS MarkSweep - LastGcInfo = javax.management.openmbean.CompositeDataSupport(compositeTyp...
java.lang:type=Runtime - SystemProperties = javax.management.openmbean.TabularDataSupport(tabularType=ja...
java.lang:type=Threading - DaemonThreadCount = 12
java.lang:type=Threading - PeakThreadCount = 13
java.lang:type=Threading - ThreadCount = 13
java.lang:type=Threading - TotalStartedThreadCount = 13
java.lang:type=GarbageCollector,name=PS Scavenge - LastGcInfo = javax.management.openmbean.CompositeDataSupport(compositeTyp...

jmxgrep code (just copy paste this code into a file named jmxgrep, and make it executable using chmod a+x jmxgrep) :


#!/bin/sh
JAVA_OPTS=""
JAVA_OPTS=$JAVA_OPTS" -Dcom.sun.management.jmxremote"
JAVA_OPTS=$JAVA_OPTS" -Dcom.sun.management.jmxremote.port=9999"
JAVA_OPTS=$JAVA_OPTS" -Dcom.sun.management.jmxremote.authenticate=false"
JAVA_OPTS=$JAVA_OPTS" -Dcom.sun.management.jmxremote.ssl=false"
SCA_OPTS="-nocompdaemon -usejavacp -savecompiled"
exec java $JAVA_OPTS -jar jajmx.jar $SCA_OPTS "$0" "$@"
!#

import fr.janalyse.jmx._
import JMXImplicits._

if (args.size < 2) {
  println("Usage   : jmxgrep host port searchMask1 ... searchMaskN")
  println("Example : jmxgrep localhost 1099  vendor")
  println("   will self connect to jmx server, and looks for vendor keyword")
  System.exit(1)
}
val host  = args(0)
val port  = args(1).toInt
val masks = args.toList.drop(2) map {s=>("(?i)"+s).r}

def truncate(str:String, n:Int=60) = if (str.size>n) str.take(n)+"..." else str

JMX.connect(host, port) { implicit jmx =>
  for(on <- jmx.browse ; attr <- on.browse) {
    val value = try { on.get[Any](attr).toString} catch { case _ => "**error**"}
    val found = List(on.toString, attr, value) exists { item =>
      masks exists {re => (re findFirstIn item).isDefined} 
    }
    if (found || masks.isEmpty) println("%s - %s = %s".format(on, attr, truncate(value)))
  }
}

Notice the "-savecompiled" scala option which enable the script compilation result to be stored in a file named "jmxgrep.jar". When started, already compiled code will be reuse if the script hasn't been modified since last compilation time. This allow very fast script startup.