SockJS alternative image

This summer I started working on improving the pseudo web terminal, which emulates a bash Terminal on the web, built by Codelearn team. It was built on top of Reel server with a Rails proxy app. The details are on the Codelearn blog post.

The Terminal app is a realtime app. The command output should come to the user promptly. The Codelearn Terminal had a huge latency (around 1-2 sec) for the command output to appear. My first task was to do end to end profiling to find out the major bottlenecks. Surprisingly, out of 2 sec end to end processing delay, 1 sec was spent establishing HTTP connection from the browser to the server when user executed a command.

Since the original code was in Ruby, we first redid the app using Faye. Faye client uses websockets which brought down connection delay significantly. Results were good with total latency reduced to around 600 msec. But the sad part was bursts of 2-4 sec delay. I could not explain the phenomenon, but it was a good enough reason for us to ditch the implementation.

This time, we decided to ditch Ruby & go for a technology that has a true evented server & uses websockets for communication.

About SockJS

We found out that there existed two libraries which would make our work easier.

SockJS is a javaScript library for real-time web applications. It has two parts. A client-side library that runs in the browser & connects to the server via websocket or any other ‘real-time’ communication protocol depending on the browser. The second part is a server-side library which provides an evented server that can be extended with the application logic. The latter is made available in different technologies like Python (SockJS-tornado), Node.js(SockJS-node) etc. There is a Ruby port too (SockJS-ruby) but it seems to be in alpha stage only.

At first, SockJS client tries to use native WebSockets. If that fails, it has a variety of fallback transport protocols which are browser-specific. This diagram shows the downgrade mechanism for different protocols

SockJS Protocol Fallback Diagram

Why SockJS and not Socket.IO ?

I found that there were some problems with Socket.IO demonstrated on baudehlo blog. Also benchmarking results on pythonanywhere blog and mrjoes blog on github showed that SockJS had better performance then Socket.IO. I did not do any performance evaluation myself & decided to stick with SockJS.

For server side implementation, I decided to go with Node.js based sockjs-node over Python’s sockjs-tornado. Node.js provided callback mechanism which the Terminal app could actually put to good use.

Installing Node.js & SockJS

  • Node.js is easy to install.The Node.js installation page explains how to go about it. I develop on Ubuntu. The following commands did the trick for me

    sudo add-apt-repository ppa:chris-lea/node.js
    sudo apt-get update
    sudo apt-get install nodejs
    
  • Like Ruby comes with RubyGems, npm (Node Packaged Modules) is the official package manager for Node.js. npm is like gem command in Ruby. Installing Node.js also installs npm. To check, simply type

    npm -v
    
  • I installed SockJS using npm through the following command

    npm install sockjs
    

    npm will automatically download and install the SockJS module. It take cares of any dependencies and installs them too.

  • To install dependencies for your application npm uses package.json file. Like Gemfile lists the Gems in a Ruby app, package.json lists the modules required for your Node.js app. npm install installs the listed modules in package.json file.

App Architecture in SockJS

The first step – using SockJS, I developed a pseudo bash terminal with just some basic functionality. A command is sent to the sockjs-node server from a form in the browser, the server processes the command & sends back the output.

Here is screenshot of browser on page load Screenshot of Browser

On the server side a new terminal process is created for every user. The server logs the input (highlighted in red) and the output (highlighted in green).

Server Screen-shot of SockJS App

Screenshot of the browser window after receiving output of ‘ls’ command from the server:

Browser Output Screen-shot

This part was easy & although I did not have any Node.js experience, I was able to figure out most of the things pretty easily. The code looked pretty small & beautiful.

Below is the code for the server

var http = require('http');
var sockjs = require('sockjs');
var pty = require('pty.js');

var echo = sockjs.createServer();
echo.on('connection', function(conn) {

        var term = pty.spawn('bash', [], {
            name: 'xterm-color',
            cols: 80,
            rows: 30,
            cwd: process.env.HOME,
            env: process.env
        });

        term.on('data', function(data) {
            conn.write(data);
        });

        conn.on('data',function(message) {
            term.write(message+'\r');
        });

        conn.on('close', function() {
            term.kill();                    
        });
});

var server = http.createServer();
echo.installHandlers(server, {prefix:'/echo'});
server.listen(3000, '0.0.0.0')

‘echo’ is instance of SockJS server encapsulated inside ‘server’ instance which is a Node HTTP server.

On receiving a new connection, ‘echo’ creates a new bash terminal using pty module in Node.js through the snippet var term = pty.spawn(...).

When the HTTP server receives data, it is sent to the terminal

conn.on('data',function(message) {
  term.write(message+'\r');
});

When the Terminal has the data ready, it sends it to the HTTP server which in turn sends the data back to the browser.

 term.on('data', function(data) {
    conn.write(data);
 });

On the client, the output of the command from the server is shown inside pre with id ‘output’.

 <pre id="output" style="height:300px;overflow:auto"></pre>

I put a form that lets the user enter a command.

 <form id="myForm">
    <input type="text" name="command" />
    <button id="execute" class="btn">Go!</button>   
 </form>

Once the document loaded, I created a new SockJS instance named socket. I attached callbacks to onopen, onmessage & onclose which were fired when the SockJS client established a connection to the server, received the input from the server & when the instances terminated respectively.

            var socket = new SockJS('http://www.codelearn.org:3000/echo');
            socket.onopen = function() {
                    console.log('open');
            };
            socket.onmessage = function(e) {
                    $('#output').append(e.data);
            };
            socket.onclose = function() {
                    console.log('close');
            };

Also, I simply sent the command in the input field to the server when the ‘Go’ button was clicked

            $("#execute").click(function(){
                    command = $("input[name='command']").val();        
                    socket.send(command);
                    $("input[name='command']").val('');
            }); 

You can view the complete code on the github project

Now that the app is ready, it was time to add more features like ‘Kill’ & ‘Reset’ that were present in the Ruby Terminal app. ‘Kill’ killed the last running process while ‘Reset’ reset the Terminal. There were some more features in pipeline like ‘multiple Terminals per user’ & some corner case testing like what happens if user opens up multiple Terminals. BDD (behavior driven development) was the way to go from here on.

BDD in SockJS

Coming from a Ruby on Rails background, I began looking for a tool like Capybara which is good for BDD. A quick google search revealed two resources for BDD in Node.js:

  1. Zombie.js
  2. PhantomJS

Using Zombie.js

Looking at a Zombie example, I found out that the API looked quite intuitive and it could easily be integrated into Mocha , a testing framework for Node.js.

I wrote a test script below.

var Browser = require('zombie');

var browser = new Browser({site:'http://localhost:1134',debug: true,silent : false});

//visit page on localhost
browser.visit('/test_client.html',step1);

//Fill input command and click 'Go!'
function step1(){
  browser.fill('input[name="command"]','ls');
  browser.pressButton('#execute',function(){
    step2();
  });
};

//Check the Output
function step2(){
  console.log(browser.html('#output'));
};

It simply visits the page, fill the input box and clicks the ‘Go!’ button. After that it logs the output.

I fired-up the server on localhost.

npm start

I started the Zombie.js script in another Terminal.

node zombie_test.js

But things did not proceed smoothly.

  • The server terminal window showed that it had received an input command (highlighted in red).

  • Also, it had sent the command’s output to the client (highlighted in green).

Server Output Screenshot when SockJS is using websocket

  • But Zombie.js script did not receive input from the server.

Script Output Screenshot when SockJS is using Websocket

In the yellow outlined box, the only thing written is the input command: ‘ps’. There is no output. I concluded that Zombie.js is unable to receive response from the server. Searching some more, I found that Zombie.js had some issues with websockets.

Since, SockJS provided an easy way to switch protocols by specifying options on either the Client or the Server, I began testing for each and every protocol. I found that none of them were opening the connection to the server, let alone send a command. But the last protocol, ‘jsonp-polling’, has a slightly different story.

The Zombie.js script did show some output, but it was just the prompt which server sent when the client establishes the connection. It implied, that the connection was established with json-polling but either the client did not send the command or server did not receive it.

The screenshot below shows the incomplete output, highlighted in yellow.

Script Output Screenshot when SockJS is using Polling

Looking into the server script output, it was clear that server did not receive the command.

Server Output Screenshot when SockJS is using Polling

This meant that Zombie.js was not able to send the input to the server through the ‘jsonp-polling’ protocol.

Using PhantomJS

This time I decided to switch to PhantomJS since it was a headless WebKit and not a simulated environment like Zombie.js.

The PhantomJS test script is longer than Zombie.js script & hence I am not reproducing it here.

Unfortunately, with PhantomJS, the issues are exactly the same.

  • WebSocket: No output received by the script.
  • Jsonp-Polling: No input sent to the server.
  • Others: No connection to the server.

Using Selenium

By facing the same problems with both Zombie.js and Phatom, it appeared that, I needed an actual browser to do the testing. Being a true Rubyist, my love for Capybara resurfaced. I decided to use Selenium. Selenium has a WebDriver API which can drive different browsers and thus, help in automating browser tasks.

Though the WebDriver API is available in many languages, I decided to go with Ruby (isnt that obvious :] ). The documentation for the Ruby bindings of the webdriver helped me get started.

Here is the Selenuium test script that I coded. Same as before I fired-up the server on localhost.

npm start

After that I installed the Selenium-Webdriver gem.

gem install selenium-webdriver

With the gem installation complete, I could run the test script.

ruby selenium-test/test.rb

Here is a screenshot of the tests passing

Screenshot of Selenium-Tests

Finally the tests worked & it kind of looked cool looking at the browser open up, tests running & browser closing; the whole process require around 15 sec for three behavioral tests I had written.

It worked but I think using Ruby for testing Node.js is an ugly hack. Add to it ,the delay of 15 sec. I happened to write some test scripts in Mocha, but I realized I am just testing the server by creating my own websocket client in the script. It was not really BDD but just testing the server features.

I am still experimenting with various different libraries in Mocha to see if I can do BDD. Probably jsdom library can help me. Stay tuned for my next blog.

Learn Android by creating a twitter app. Visit http://www.codelearn.org/android-tutorial/.
  • http://captnemo.in/ Abhay Rana

    As someone who works quite often with node.js, I’d highly recommend using something like nvm for installing node.js. The packages in ubuntu are not kept up to date, and version mismatch in production and development environment is something everyone should avoid. Hence, nvm.

  • rohitpaulk

    Very well written, and exactly what I was looking for :)