permalink

36

Building Real-time CoffeeScript Web Applications With SocketStream

 

Our engineering team at Aol Europe have been working on a number of exciting new projects over the past few months including a brand new high-performance real-time web framework called SocketStream.We're going to take a first look at how you can start building applications with it today.  

The brain-child of software engineer Owen Barnes, SocketStream is built on top of node.js, resolves around the popular single-page application (SPA) paradigm and utilizes HTML5 WebSockets, Socket.IO, Redis and other techologies to provide an extremely responsive experience for the web.

It's effectively a complete-stack for all of your client and server development needs.

The team launched SS at Hacker News London this week and we were humbled by the positive reaction on both GitHub (where we very surprisingly became the trending repo of the week) and Twitter including some interest from Jeremy Ashkenas, the developer behind Backbone.js, Underscore.js and of course, CoffeeScript.

Why do we believe the industry needs SocketStream?

Last year, we took a look at the current state of web development and noticed that in a lot of cases developers were having to write similar logic on both the server-side and client-side repeatedly.

This is something that many of us have now become used to, managing MVC-style architecture on the server in a language like PHP or Rails and then another MVC-like instance in JavaScript on the client. Although a means to an end, no developer should ideally be doubling their efforts on a codebase unless completely necessary. It makes web development feel broken as a process.

Not only this, but in a lot of situations the best experiences being offered to end-users only provided a snapshot of data from the past (neither Ajax requests nor long-polling give you a truly optimal, real-time level of data syncronization between the front and backend), so Owen decided to come up with a solution that would solve these problems. Imagine being able to change data on the backend and have it update in real-time on the user's screen.

Rather than constantly requiring a page-refresh or long-polled call to the server to maintain syncronization, why not simply pass all of the data needed back and forth between the client and server through WebSockets and packets containing updates in JSON-form instead?. This is effectively what SocketStream does, however it also does a lot more.

Let's take a brief look at WebSockets before we continue.

Understanding WebSockets

If you haven't played around with the WebSocket API before it's effectively a means for providing low-complexity, low-latency, bi-directional communicational channels over a single TCP socket and is relatively straight-forward to use. 

The protocol works by initiating an HTTP-like handshake and provides developers a simple way to send packets of data back and forth between the client and server.

Whilst a complete tutorial on WebSockets is beyond the scope of this article, a basic demonstration of how they function can be seen below:

/*Create a new WebSocket, connect to the server at test.com 
specifying a customized protocol*/
var socket = new WebSocket("http://www.test.com/my-socket-server", "a-protocol");
 

/*Once the connection has been established, we can begin 
transmitting data to the server by using the WebSocket 
objects .send() method:*/
socket.send("lolwat?");
 

/*As an event-driven API , a message event is delievered to an 
onmessage function when messages are received. 
We can easily listen for data that arrives as follows:*/
socket.onmessage(function( e ){
	console.log('Data arrived:' + e.data);
}

/*Connections can then be closed using the WebSocket's 
.close() method*/
socket.close(); //easy!

The low level of latency is one of the primary values of using WebSockets because it effectively enables bi-directional communication without requiring a large overhead.

As mentioned on the Mozilla blog as a comparitive example: Google Wave attempted to bring the world real-time communication with keystrokes but had a several-kilobyte overhead for each keystroke due to the TCP start-up, teardown and HTTP headers involved where realistically, only a few bytes should have been sent down the tubes.WebSockets would have allowed them to do this and this is where their power truly shines. (In case you're interested, there are some notes about the Wave protocol vs. WebSockets here along with suggestions of using them together).

Although the WebSocket API is currently still being standardized by the W3C, developers are already able to take advantage of them in browsers such as Safari 5, mobile Safari, FireFox 5 and Chrome 4+.

This is why Owen thought it was time to start using them – they will be supported by all browsers in the future but in the mean-time for browsers that don't support WebSockets natively, a polyfill such as FlashSockets can be used instead.

The rest of the SocketStream stack

  • Data storage – SocketStream uses Redis which been described as many things, but it's generally considered a multi-purpose swiss army knife. It's a key-value store, a data-structure server, a non-blocking event bus and a lot more. It's very difficult to precisely categorize Redis, however it tends to fall under the NoSQL brand most of the time.
  • SocketStream was built with a number of design patterns in mind, including Pub/Sub (which you can read more about here). Scalable pub/sub is baked right into the framework so you can easily keep the rest of your application decoupled from the get go (as long as you remember to use it!).
  • Namespacing is one of the areas of large-scale application development that is often neglected (but as we know is very-much required) – this is why SocketStream offers the concept of API trees. These are basically a way of representing your entire application's folder structure in a JavaScript object so that code can be easily namespaced, accessed and executed on both the client and server side.
  • You don't have to worry about handling your own authentication or bolting on your own HTTPS module onto the framework – both of these are built-in modules with their own APIs.
  • Intelligent tools for the modern-developer – SocketStream comes with Stylus (for dynamic, expressive CSS), Jade (a node-based templating engine) for static HTML files and JavaScript templating for any dynamic templating needs. The jQuery templates plugin is provided by default, however this can be easily switched out for Mustache, Handlebars or any other client-side templating solution you might prefer.

How does SocketStream work?

As a front-end developer, build processes as essential for making projects production-ready. This includes everything from concatonating files to minification and beyond. SocketStream automatically handles all of this for you (via UglifyJS) and sends all of this data through to users the first time they connect to your site.

After this, any additional data is transmitted as serialized JSON over WebSockets instantaneously establishing when the user first connections and reestablishing if the connection drops for any reason. The end-result of this is that we no longer need worry about issues like latency or header overhead – simply streaming data between clients and servers.

Writing a real-time group chat client using SocketStream

To help you get started with SocketStream, Paul and I have written a simple application for bi-directional messaging between the client and server (effectively a simple chat app). In this section of the post, we'll take you how to install and create your first SocketStream application in next to no-time.

 

First, select a language – CoffeeScript or JavaScript?

In case you haven't tried it out yet, CoffeeScript is a new abstracted way of writing code that compiles to JavaScript in a concise, elegant fashion inspired by some of the syntatic sugar found in Rails and Python. Discussions on the pros and cons of it have been ongoing in the JavaScript community however SocketStream doesn't make any discrimination – you can easily code applications using whichever you prefer.

Simply use the .js or .coffee extension for new files in your application directory and the SocketStream build process will take care of the rest.

Our team however love using CS and encourage developers to give it a try as it doesn't have a large learning curve. We believe that EcmaScript.next may well incorporate aspects of CS as a part of it's specs so it may well be the future. In case you're interested in giving it a go, Arcturo wrote a free book on CS that you can read here. Alternatively, if you're a JavaScript developer just looking for a simpler way to port your code over to CS, I recommend trying out js2coffee.

Step 1 – Getting Started: Installation

Now that you've decided which language to use, let's get on to the installation of SocketStream, it's dependencies and the sources to the SocketChat application.

Before you can run SocketStream, you will need to install the following software:

Instructions on how to install those items of software are included below for reference, and are taken from the home pages of those software libraries.

Note: You may also need to install the following packages:

yum install gettext-devel expat-devel curl-devel zlib-devel openssl-devel

  
Step 1.1 – Install Node.js if you don't currently have it on your system.

Firstly, Check you have the dependencies for Node.js installed first (see here). If you satisfy them, then follow the command line instructions below:

wget http://nodejs.org/dist/node-v0.4.8.tar.gz     # (check this is the latest version)
tar xzf node-v0.4.8.tar.gz
cd node-v0.4.8
./configure
make      
sudo make install

    
Step 1.2 – Install Node Package Manager (npm)

curl http://npmjs.org/install.sh | sh

 

Step 1.3 – Install Redis

wget http://redis.googlecode.com/files/redis-2.2.7.tar.gz
tar xzf redis-2.2.7.tar.gz
cd redis-2.2.7
make

    
Step 1.4 – Install SocketStream

There are two ways you can do this. Either by using a standard git clone as follows:  

git clone git://github.com/socketstream/socketstream.git
cd socketstream/
npm install -g

or by directly using NPM:

sudo npm install socketstream -g

To test that SocketStream is correctly installed, create and run a test project:

socketstream new test 
cd test 
socketstream start

 

Step 1.5 – Install SocketChat

 

git clone https://github.com/addyosmani/socketchat
cd socketchat/

npm link

 

The complete installation process should take less than 20 minutes including dependencies. 

 

Step 2 – Understanding folder structure and API trees

Generating a new SocketStream project is then as simple as typing in:

socketstream new project_name_here

This generates a number of project directories which are covered in detail on our repo's ReadMe. These directories are:

  • /app/client (CoffeeScript or JavaScript files containing application logic for the client)
  • /app/server (effectively controllers in traditional MVC, logic for the server-side)
  • /app/shared (code shared between the client and server)
  • /app/css (the Stylus files containing your style/CSS definitions)
  • /app/views (Jade views and templates)
  • /lib (libraries used such as jQuery)
  • /public (for static files. Used by SS for compiling your project)
  • /static (for warnings)
  • /vendor (optional)

You can structure the sub-folders in these directories as needed but remember that this entire API Tree can also be accessed as a JavaScript object. Effectively, we can call SS.client.app<export name> on either the browser console or on the server to get the same application logic executed.

Eg. if sum was an export defined in /app/server/, you could easily call SS.client.app.sum(25,34) to add the two numbers on either console. Once again, for information on exports, I recommend reading our official documentation.

 

Step 3 – Creating our chat application views

We'll be using Jade to create all of our application views. If you haven't used it before, Jade's a great templating engine that's quite heavily influenced by Haml. The basic idea is that you're able to structure your mark-up without the need for semantic tags which simplifies the amount of work needed to define your layout. As a part of our build-process, Jade templates are later compiled to actual HTML that can be rendered in the browser.

Below, we're defining a layout with a simple div messageContainer and a form (sendMessage) for submitting new messages to other users. Jade is very readable so even if its new to you, you should be able to read the code:

/app/views/app.jade – Main template

!!! 5
html(lang:"en")
  head
    meta(http-equiv="Content-Type", content="text/html;charset=UTF-8")
    meta(name: "apple-mobile-web-app-capable", content: "yes")
    meta(name: "apple-mobile-web-app-status-bar-style", content: "black-translucent")
    meta(name: "viewport", content: "width = 1024, initial-scale = 1, user-scalable = no")
    != SocketStream
    title SocketChat
  body
    #wrapper
      #main.views.hidden
        #header
          #name SocketChat
        #left.column
          #content
            #messageContainer
              #messages

            form#sendMessage
              input(id='newMessage', type='text', name='message', autocomplete='off')
              input(id='sendButton', type='submit', value='Send')
              
            #footer

      form(onsubmit: 'return false')#signIn.views.hidden
        h1 Please enter a username to begin:
        input(type='text')

The last view we need for this single-page application is a template for new messages that are sent and received. We've going to keep this simple as it's a basic chat client but you can easily extend this to support gravatar profile images or a more rich template. The user and body variables used in the below will be dynamically populated at run-time.

/app/views/templates/message.jade – Messages template

.message
  .user
    {{= user}}
  .body
    {{= body}}

 

Step 4 – Posting messages from the client to the server

Broadcasting messages using SocketStream is actually quite straight-forward. We'll be using the server-side .sendMessage() function defined in Step 5 that utilises Pub/Sub to publish a message centrally.

The client-side code then listens out for an event called newMessage to check (subscribe) to any new messages being published by other users that are also connected.

The best part about all of this is that we don't in any way need to worry about which server instance users are connected to. Messages are always passed on to the correct server because every SS server subscribes to the same overall instance of Redis.

Walking through the code below, we check to see if the current user has correctedly been initialized/signed-in and display a sign-in form if not. Assuming they have signed in correctly, they're able to send new messages to all of the other users that are logged in. Because SocketStream comes with built-in authentication, this process is extremely simplified.

The .renderMessage() function is used to render new messages using the jQuery templates plugin to our messages list and jQuery is also used for a minor level of DOM-manipulation and effects such as fading in login forms and new messages.

/app/client/app.coffee – Client side code

 # This function is called automatically once the websocket is setup
exports.init = ->
  $('.message.template').hide()
  SS.server.app.init (user) ->
    if user then $('#main').show() else displaySignInForm()
    
  # Bind to Submit button
  $('form#sendMessage').submit ->
    newMessage = $('#newMessage').val()
    SS.server.app.sendMessage newMessage, (response) ->
      if response.error then alert(response.error) else $('#newMessage').val('')
    false
    
  # Bind to new incoming message event
  SS.events.on 'newMessage', renderMessage

 
# Display the user sign-in form
displaySignInForm = ->
  $('#signIn').show().submit ->
    SS.server.app.signIn $('#signIn').find('input').val(), (response) ->
      $('#signInError').remove()
      displayMainScreen()
    false
    
displayMainScreen = ->
  $('#signIn').fadeOut(230) and $('#main').show()

renderMessage = (params) ->
  $('#templates-message').tmpl(params).appendTo('#messages')
  SS.client.scroll.down('#messages', 450)

 

 

Step 5 – Broadcasting messages received to all of the clients currently connected

As mentioned in Step 4, our server-side code primarily handles broadcasting new mesages but also takes care of user sign-in and session handling. As you can see, the amount of code required to get this type of application done is very minimal.

/app/server/app.coffee – Server-side code

exports.actions =

  init: (cb) ->  
    if @session.user_id
      R.get "user:#{@session.user_id}", (err, data) =>
        if data then cb data else cb false
    else
      cb false

  sendMessage: (message, cb) ->
    SS.publish.broadcast 'newMessage', {user: @session.user_id, body: message}
    cb true
  
  signIn: (user, cb) ->
    @session.setUserId(user)
    cb user

 

Fork or download SocketChat

You can download, fork or check out a demo of SocketChat below. Please note that we recommend running SocketStream demos in a WebKit browser such as Chrome or Safari as those currently have stable WebWorker implementations. SS is definitely going to target all modern browsers including FF5, however support is currently being expanded beyond those recommended.

 

 

 

From multi-user chat to multi-user gaming

SocketChat is of course just a drop in the ocean in terms of what you can create with the framework.

Another of our team-mates, Alan Milford, (who had never used CoffeeScript, Jade or SS before) built a complete multi-player game called SocketRacer using CSS3 and SocketStream in under a week (see below).

He built this on top of the basic bi-directional communication examples to syncronize telemetary data of cars that were racing but it also still supports multi-user chat. You can play the game here if you're interested in checking it out.

Click to play

There are already a number of other demo apps in the pipeline but at the moment sky really is the limit with what's possible.

SocketStream and the road forward

SS is currently in it's very early alpha stages of development and there's still quite a lot of work to get done. The first area that we absolutely need to cover is unit testing, which we intend on getting into the project as soon as possible. Next, we want to focus on getting models into a release and we've been preliminarily looking at Mongoose to assist with this.

Remember that this is really a preview release and there will be bugs, so we don't advise deploying sensitive applications to the web using SS just yet.

We do however encourage developers to start playing around with SocketStream right away as we're more than happy to take on any feedback during these early stages of development. Your feedback is valued and may be considered for future releases. Have fun building some SocketStream apps!

36 Comments

    • You're very welcome, Mike. Please feel free to hook us up to a github link to your demo :) Glad you enjoyed playing around with SS.

  1. Thank you Addy, SocketStream let dreams come true. I've tried various ways to explore the Realtime / WebSocket / node.js universe and find the perfect and modern all-in-one solution. SocketStream is near to that. :D

  2. Great article. I've been trying to run through your code (OS X Snow Leopard) and am encountering some small errors.
    ———-
    sudo socketstream start
    29 Jun 11:08:06 – Starting SocketStream server…
    29 Jun 11:08:07 – Generating essential asset files to get you started…
    29 Jun 11:08:07 – Error: ENOENT, No such file or directory './public/assets'

    node.js:134
    throw e; // process.nextTick error, or 'error' event on first tick
    ^
    Error: Unable to generate client assets libraries. Please ensure you have the latest version of SocketStream and try again
    ———-

    Any thoughts?

    Thanks!

  3. installation problem solved… had to do a little tweak in configuring redis

    This is really a great article, and thanks for putting the code online!!

  4. Pingback: Node Roundup: Porting Node to Windows, socketstream, EventEmitter2

  5. I'm getting an "Incompatible Browser" error message with Firefox 5. Does it not support the minimal requirements?

    • Hi Jake

      Socket.IO 0.6 is a little temperamental with Firefox, so developers can chose to disable 'flashsockets' support and only support browsers with native websockets. This is the SS.config.browser_check.strict setting which I believe was enabled for SocketChat.

      A new unstable version of SocketStream with Socket.IO 0.7 is nearly ready. This will have much better support for Firefox and other browsers – but we still want developers to have the final say in which browsers they wish to support (as it would be difficult to support fast real-time gaming on some of the fallback transports Socket.IO provides).

      Owen

  6. Pingback: SocketStream: A WebSocket Web Framework | FunctionSource Development

  7. Hey guys! Just a few quick tips for anyone trying to get the chat demo working:

    1. Please make sure you have the latest version of SocketStream, Redis and Node installed.
    2. For the time being, it's best to run SocketStream in WebKit browsers. It *should* work fine in FF5 and Opera (latest) with the WebSockets flags turned on (as per our docs), but you'll get the best experience in Chrome and Safari for the time being.
    3. If you receive further error messages, please open a ticket on the repo's 'issues' section on GitHub here as either I or another team member will instantly be alerted to help patch whatever behaviour is broken – https://github.com/addyosmani/socketchat

    Cheers!
    Addy

  8. Great article! I'm trying to get it running on latest node 0.5.2-pre but get this error:

    /usr/local/lib/node_modules/socketstream/node_modules/stylus/lib/visitor/evaluator.js:539
    Evaluator.prototype.visitImport = function(import){
    ^^^^^^

    node.js:189
    throw e; // process.nextTick error, or 'error' event on first tick
    ^
    Error: Unable to start SocketStream as we're missing stylus.
    Please install with 'npm install stylus'. Please also check the version number if you're using a version of npm below 1.0

    But I do have stylus installed. Any ideas?

  9. Websocket API kicks ass! Iamjuststarting to learn it, it makes everything so much quicker! Love your site and all these step-by-stap examples! More of it please!

  10. your tutorials are very good, we have created our own Javascript Framework that can either work individually or you can combine it with Jquery, Prototype or any Javascript Framework. our Framework also comes with 13 Modules that ease your work and handle everything for you. Many People have Downloaded it and using it.
    it is fully documented so you can easily understand everything.

    hope you can check it on your own. http://www.zincksoft.com/javascriptframework.html

  11. Man, I wish I stumbled upon SocketStream sooner. I basically built almost the same thing with almost the exact same dir structure but in a much more rapid and less organized way. Anyway, I plan on switching to it and use at our startup and maybe contribute back.

    REALLY GOOD WORK!

    • Hey Mario! I’m glad you’re interested in giving SocketStream a try. If you run into any issues with it at all feel free to reach out and we’ll do our best to help.

  12. Thanks so much for the post and example app. Socketstream feels like it might be a fit for a simple multiplayer game I’m trying to build. Are you planning to update the example code to be 0.3.x compliant?

    • I don’t have plans on updating this specific example but will be building a new app using SocketStream 0.3.x that will be accompanied with it’s own tutorial at some point in the near future :)

      • Fantastic. Thanks for putting so much of yourself out there for the rest of us. Congratulations on the move to Google.

  13. Pingback: Node Roundup: Porting Node to Windows, socketstream, EventEmitter2 - Jobbook News

Leave a Reply

Required fields are marked *.