Thursday, June 28, 2012

SSH intricated tunnels

Let's simulate intricated tunnels using JASSH (high level scala SSH API).

In the following example, we simulate bouncing between 9 SSH hosts, using SSH tunnel intrication :
  • A:22 -> B:22 -> C:22 -> D:22 -> E:22 -> F:22 -> G:22 -> H:22 -> I:22 ; (typical case : a given host is only accessible through the previous one)
  • All "foreign" hosts become directly accessible using new ssh local ports
  • A->10022, B-> 10023, ... I->10030, so now I (and all others) are direcly accessible from local ssh client host !
    // From host/port, bring back locally remote fhost/fport to local host using tport. 
    case class Sub(host:String, port:Int, fhost:String, fport:Int, tport:Int)
    
    val intricatedPath = Iterable(
        Sub("localhost", 22,    "127.0.0.1", 22, 10022), // A 
        Sub("localhost", 10022, "127.0.0.1", 22, 10023), // B
        Sub("localhost", 10023, "127.0.0.1", 22, 10024), // C
        Sub("localhost", 10024, "127.0.0.1", 22, 10025), // D
        Sub("localhost", 10025, "127.0.0.1", 22, 10026), // E
        Sub("localhost", 10026, "127.0.0.1", 22, 10027), // F
        Sub("localhost", 10027, "127.0.0.1", 22, 10028), // G
        Sub("localhost", 10028, "127.0.0.1", 22, 10029), // H
        Sub("localhost", 10029, "127.0.0.1", 22, 10030)  // I
        )
        
    def intricate[T](path:Iterable[Sub], curSSHPort:Int=22)(proc:(SSH)=>T):T = {
      path.headOption match {
        case Some(curSub) =>
          SSH.once(curSub.host, "test", port=curSub.port) { ssh =>
            ssh.remote2Local(curSub.tport, curSub.fhost, curSub.fport)
            intricate(path.tail, curSub.tport)(proc)
          }
        case None => 
          SSH.once("localhost", "test", port=curSSHPort) { ssh =>
            proc(ssh)
          }
      }
    }
    
    // Build the intricated tunnels and execute a ssh command on the farthest host (I)
    val result = intricate(intricatedPath) {ssh =>
      ssh.executeAndTrim("echo 'Hello intricated world'")
    }

    println(result)
So now let's automate complex SSH tunnels construction, with automatic rebuild on failure and direct integration into tools, when some direct access are not possible...

Wednesday, June 27, 2012

JASSH (JAnalyseSSH) 0.9.3 released

JASSH is a high level scala SSH API for easy and fast operations on remote servers. It is JSCH based.

Latest changes for 0.9.3

  • now using sbt-assembly 0.8.3
  • fixes relatives to implicit conversions with SSHPassword
  • fixes relatives to implicit conversion to SSHCommand and SSHBatch
  • For SSHBatch : execute, executeAndTrim, executeAndTrimSplit
    renamed to : executeAll, executeAllAndTrim, executeAllAndTrimSplit
  • Using Iterable instead of List
  • external (package) usage tests completed (ExternalSSHAPITest.scala)
  • small fix about how private key passphrase is taken into account (when pub-key auth is used)

Latest changes for 0.9.2

  • date '+%Y-%m-%d %H:%M:%S %z' %z and %Z gives the same result on AIX, this result corresponds to linux %Z. So modifying code to use %Z instead of %z. Now using GMT, "date -u '+%Y-%m-%d %H:%M:%S %Z'" in order to everything work well in all cases.
  • SSH.once(Option[SSHOptions]) fix linked to Option type result not at the right place
  • New test source file : ExternalSSHAPITest.scala => Testing the API from an external package
  • Fixed : minor problem with script when invoking jajmx.SSH... or fr.janalyse.sh.SSH... without imports...

Latest changes for 0.9.1

  • SSH tunneling fix, cleanup, and scaladocumented
  • Intricated SSH tunneling test added (self intrication, to simplify test case)

Latest changes for 0.9.0:

  • now using sbt-assembly 0.8.1
  • now using scalatest 0.8
  • new helper methods (For the list of already available helper methods):
    • test
    • exists
    • isFile
    • isDirectory
    • isExecutable
  • findAfterDate & date helper fix !!
    Shell.date -> remote system time zone is now taken into account
  • Test cases fixes :
    Forcing parallelism to 6 ! for test case "Simultaenous SSH operations"
  • Code factorization :
    • ShellOperations trait added. Inherited by SSH and SSHShell.
    • TransferOperations trait added. Inherited by SSH and SSHFtp.
  • SCP supported, for no-persistent transferts sessions, SCP is now used by default (instead of SFTP)
    (e.g. : SSH class transfert operation is now using SCP by default).
  • noneCipher switch added to SSHOptions for higher performance SCP transfert (true by default)
    (http://www.psc.edu/index.php/hpn-ssh)
  • transfert (receive) tests added
    Reference time on a local system: 500Mb using 5 SCP command (100Mb/cmd) takes on the same system 8.7s (~62Mo/s by file)
    file transfert performances (with content loaded in memory)
    • Bytes rate : 38,6Mb/s 500Mb in 12,9s for 5 files - byterates using SCP
    • Bytes rate : 44,9Mb/s 500Mb in 11,1s for 5 files - byterates using SCP (with none cipher)
    • Bytes rate : 38,5Mb/s 500Mb in 13,0s for 5 files - byterates using SFTP
    • Bytes rate : 46,0Mb/s 500Mb in 10,9s for 5 files - byterates using SFTP (with none cipher)
    • Bytes rate : 39,5Mb/s 500Mb in 12,7s for 5 files - byterates using SFTP (session reused
    • Bytes rate : 46,7Mb/s 500Mb in 10,7s for 5 files - byterates using SFTP (session reused, with none cipher)
    • Bytes rate : 29,5Mb/s 500Mb in 16,9s for 500 files - byterates using SCP
    • Bytes rate : 32,1Mb/s 500Mb in 15,6s for 500 files - byterates using SCP (with none cipher)
    • Bytes rate : 26,7Mb/s 500Mb in 18,7s for 500 files - byterates using SFTP
    • Bytes rate : 29,5Mb/s 500Mb in 16,9s for 500 files - byterates using SFTP (with none cipher)
    • Bytes rate : 37,7Mb/s 500Mb in 13,3s for 500 files - byterates using SFTP (session reused)
    • Bytes rate : 43,7Mb/s 500Mb in 11,4s for 500 files - byterates using SFTP (session reused, with none cipher)
  • Code cleanup & Scaladocumenting
  • SSH compression now supported
  • For easier SSH Tunneling, new methods are now available :
    • def remote2Local(rport:Int, lhost:String, lport:Int)
    • def local2Remote(lport:Int, rhost:String, rport:Int)
  • SSHCommand, SSHBatch methods ! renamed to §§

CAVEATS :

  • ssh persisted shell session operations must be executed within the same thread, do not span a persisted shell session across several threads => it may generate exception
    So be careful when using REPL with default config, as each "evaluation" is done within a new thread !
    Workaround : Start the interpreter (REPL) with the "-Yrepl-sync" option.
    No problem with SBT as a scala console started from SBT will execute all its entries in the same thread !
    No problem in scala scripts.
  • SCP operations can't retrieve special file such as /proc/cpuinfo, because their size are not known !
    Workarounds : use SFTP OR use a command such as "cat /proc/cpuinfo". (The last one is the "best workaround", will work in all cases)

hello scala script

#!/bin/sh
exec java -jar jassh.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#

print(jassh.SSH.shell("localhost", "test", "testtest") {_ execute "echo Hello `hostname`" } )
  
  

remote vmstat scala script

(The following script assume that the current user has already automatic access to the given remote host using SSH public key authentication.)
#!/bin/sh
exec java -jar jassh.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#
val host=if (args.size>0) args(0) else "localhost"
val user=if (args.size>1) args(1) else util.Properties.userName
val freq=if (args.size>2) args(2) else ""
val numb=if (args.size>3) args(3) else ""
val vmstatcmd="vmstat %s %s".format(freq, numb)

jassh.SSH.once(host, user)  {
  _.run(vmstatcmd, _.foreach(println _)).waitForEnd
}

smallest script : print remote system name (uname)

#!/bin/sh
exec java -jar jassh.jar -nocompdaemon -usejavacp -savecompiled "$0" "$@"
!#

println(jassh.SSH.shell("localhost", "test", "testtest") { _.uname})