Matthew Daly's Blog

I'm a web developer in Norfolk. This is my blog...

31st December 2014 2:10 pm

Building a Chat Server With Node.js and Redis

One of the more interesting capabilities Redis offers is its support for Pub/Sub. This allows you to subscribe to a specific channel, and then react when some content is published to that channel. In this tutorial, we’ll build a very simple web-based chat system that demonstrates Redis’s Pub/Sub support in action. Chat systems are pretty much synonymous with Node.js - it’s widely considered the “Hello, World!” of Node.js. Since we already used Node with the prior Redis tutorial, then it also makes sense to stick with it for this project too.

Installing Node.js

Since the last tutorial, I’ve discovered NVM, and if you’re using any flavour of Unix, I highly recommend using it. It’s not an option if you’re using Windows, however Redis doesn’t officially support Windows anyway, so if you want to follow along on a Windows machine I’d recommend using a VM.

If you followed the URL shortener tutorial, you should already have everything you need, though I’d still recommend switching to NVM as it’s very convenient. We’ll be using Grunt again, so you’ll need to make sure you have grunt-cli installed with the following command:

$ npm install -g grunt-cli

This assumes you used NVM to install Node - if it’s installed globally, you may need to use sudo.

Installing dependencies

As usual with a Node.js project, our first step is to create our package.json file:

$ npm init

Answer the questions so you end up with something like this (or just paste this into package.json and amend it as you see fit):

{
  "name": "babblr",
  "version": "1.0.0",
  "description": "Chat client",
  "main": "index.js",
  "scripts": {
    "test": "grunt test --verbose"
  },
  "keywords": [
    "chat"
  ],
  "author": "Matthew Daly <matthew@matthewdaly.co.uk> (http://matthewdaly.co.uk/)",
  "license": "GPLv2"
}

Now let’s install our dependencies:

$ npm install express hbs redis hiredis socket.io socket.io-client --save
$ npm install chai grunt grunt-contrib-jshint grunt-coveralls grunt-mocha-istanbul istanbul mocha request --save-dev

These two commands will install our dependencies.

Now, if you followed on with the URL shortener tutorial, you’ll notice that we aren’t using Jade - instead we’re going to use Handlebars. Jade is quite a nice templating system, but I find it gets in the way for larger projects - you spend too much time looking up the syntax for things you already know in HTML. Handlebars is closer to HTML so we will use that. We’ll also use Socket.IO extensively on this project.

Support files

As before, we’ll also use Mocha for our unit tests and Istanbul to generate coverage stats. We’ll need a Grunt configuration for that, so here it is:

module.exports = function (grunt) {
    'use strict';
    grunt.initConfig({
        jshint: {
            all: [
                'test/*.js',
                'index.js'
            ]
        },
        mocha_istanbul: {
            coverage: {
                src: 'test', // the folder, not the files,
                options: {
                    mask: '*.js',
                    reportFormats: ['cobertura', 'html', 'lcovonly']
                }
            }
        },
        coveralls: {
            options: {
                src: 'coverage/lcov.info',
                force: false
            },
            app: {
                src: 'coverage/lcov.info'
            }
        }
    });
    // Load tasks
    grunt.loadNpmTasks('grunt-contrib-jshint');
    grunt.loadNpmTasks('grunt-coveralls');
    grunt.loadNpmTasks('grunt-mocha-istanbul');
    // Register tasks
    grunt.registerTask('test', ['jshint', 'mocha_istanbul:coverage', 'coveralls']);
};

We also need a .bowerrc:

{
    "directory": "static/bower_components"
}

And a bower.json:

{
  "name": "babblr",
  "main": "index.js",
  "version": "1.0.0",
  "authors": [
    "Matthew Daly <matthewbdaly@gmail.com>"
  ],
  "description": "A simple chat server",
  "moduleType": [
    "node"
  ],
  "keywords": [
    "chat"
  ],
  "license": "GPLv2",
  "homepage": "http://matthewdaly.co.uk",
  "private": true,
  "ignore": [
    "**/.*",
    "node_modules",
    "bower_components",
    "test",
    "tests"
  ],
  "dependencies": {
    "html5-boilerplate": "~4.3.0",
    "jquery": "~2.1.1",
    "bootstrap": "~3.3.1"
  }
}

Then install the Bower dependencies:

$ bower install

We also need a Procfile so we can run it on Heroku:

web: node index.js

Now, let’s create the main file:

$ touch index.js

And our test file:

$ mkdir test
$ touch test/test.js

Implementing the chat server

Next, let’s implement our first test. First of all, we’ll verify that the index route works:

/*jslint node: true */
/*global describe: false, before: false, after: false, it: false */
"use strict";
// Declare the variables used
var expect = require('chai').expect,
    request = require('request'),
    server = require('../index'),
    redis = require('redis'),
    io = require('socket.io-client'),
    client;
client = redis.createClient();
// Server tasks
describe('server', function () {
    // Beforehand, start the server
    before(function (done) {
        console.log('Starting the server');
        done();
    });
    // Afterwards, stop the server and empty the database
    after(function (done) {
        console.log('Stopping the server');
        client.flushdb();
        done();
    });
    // Test the index route
    describe('Test the index route', function () {
        it('should return a page with the title Babblr', function (done) {
            request.get({ url: 'http://localhost:5000/' }, function (error, response, body) {
                expect(body).to.include('Babblr');
                expect(response.statusCode).to.equal(200);
                expect(response.headers['content-type']).to.equal('text/html; charset=utf-8');
                done();
            });
        });
    });
});

Note that this is very similar to the first test for the URL shortener, because it’s doing basically the same thing.

Now, run the test and make sure it fails:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
  server
Starting the server
    Test the index route
      1) should return a page with the title Babblr
Stopping the server
  0 passing (873ms)
  1 failing
  1) server Test the index route should return a page with the title Babblr:
     Uncaught AssertionError: expected undefined to include 'Babblr'
      at Request._callback (/Users/matthewdaly/Projects/babblr/test/test.js:34:33)
      at self.callback (/Users/matthewdaly/Projects/babblr/node_modules/request/request.js:373:22)
      at Request.emit (events.js:95:17)
      at Request.onRequestError (/Users/matthewdaly/Projects/babblr/node_modules/request/request.js:971:8)
      at ClientRequest.emit (events.js:95:17)
      at Socket.socketErrorListener (http.js:1552:9)
      at Socket.emit (events.js:95:17)
      at net.js:441:14
      at process._tickCallback (node.js:442:13)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/babblr/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/babblr/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements   : 100% ( 0/0 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 0/0 )
Lines        : 100% ( 0/0 )
================================================================================
>>
Warning: Task "mocha_istanbul:coverage" failed. Use --force to continue.
Aborted due to warnings.

With that confirmed, we can start writing code to make the test pass:

/*jslint node: true */
'use strict';
// Declare variables used
var app, base_url, client, express, hbs, io, port, rtg, subscribe;
// Define values
express = require('express');
app = express();
port = process.env.PORT || 5000;
base_url = process.env.BASE_URL || 'http://localhost:5000';
hbs = require('hbs');
// Set up connection to Redis
/* istanbul ignore if */
if (process.env.REDISTOGO_URL) {
    rtg  = require("url").parse(process.env.REDISTOGO_URL);
    client = require("redis").createClient(rtg.port, rtg.hostname);
    subscribe = require("redis").createClient(rtg.port, rtg.hostname);
    client.auth(rtg.auth.split(":")[1]);
    subscribe.auth(rtg.auth.split(":")[1]);
} else {
    client = require('redis').createClient();
    subscribe = require('redis').createClient();
}
// Set up templating
app.set('views', __dirname + '/views');
app.set('view engine', "hbs");
app.engine('hbs', require('hbs').__express);
// Register partials
hbs.registerPartials(__dirname + '/views/partials');
// Set URL
app.set('base_url', base_url);
// Define index route
app.get('/', function (req, res) {
    res.render('index');
});
// Serve static files
app.use(express.static(__dirname + '/static'));
// Listen
io = require('socket.io')({
}).listen(app.listen(port));
console.log("Listening on port " + port);

If you compare this to the code for the URL shortener, you’ll notice a few fairly substantial differences. For one thing, we set up two Redis connections, not one - that’s because we need to do so when using Pub/Sub with Redis. You’ll also notice that we register Handlebars (hbs) rather than Jade, and define not just a directory for views, but another directory inside it for partials. Finally, setting it up to listen at the end is a bit more involved because we’ll be using Socket.IO.

Now, you can run your tests again at this point, but they won’t pass because we haven’t created our views. So let’s do that. Create the directory views and the subdirectory partials inside it. Then add the following content to views/index.hbs:

{{> header }}
        <div class="container">
            <div class="row">
                <div class="col-md-8">
                    <div class="conversation">
                    </div>
                </div>
                <div class="col-md-4">
                    <form>
                        <div class="form-group">
                            <label for="message">Message</label>
                            <textarea class="form-control" id="message" rows="20"></textarea>
                            <a id="submitbutton" class="btn btn-primary form-control">Submit</a>
                        <div>
                    </form>
                </div>
            </div>
        </div>
{{> footer }}

Add this to views/partials/header.hbs:

<!DOCTYPE html>
<!--[if lt IE 7]>      <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]>         <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]>         <html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <title>Babblr</title>
        <meta name="description" content="">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <!-- Place favicon.ico and apple-touch-icon.png in the root directory -->
        <link rel="stylesheet" href="/bower_components/bootstrap/dist/css/bootstrap.min.css">
        <link rel="stylesheet" href="/bower_components/bootstrap/dist/css/bootstrap-theme.min.css">
        <link rel="stylesheet" href="/css/style.css">
    </head>
    <body>
        <!--[if lt IE 7]>
            <p class="browsehappy">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p>
        <![endif]-->
        <nav class="navbar navbar-inverse navbar-static-top" role="navigation">
            <div class="container">
                <div class="navbar-header">
                    <a class="navbar-brand" href="#">Babblr</a>
                </div>
            </div>
        </nav>

And add this to views/partials/footer.hbs:

        <script src="/bower_components/jquery/dist/jquery.min.js"></script>
        <script src="/bower_components/bootstrap/dist/js/bootstrap.min.js"></script>
        <script src="/socket.io/socket.io.js"></script>
        <script src="/js/main.js"></script>
    </body>
</html>

You’ll also want to create placeholder CSS and JavaScript files:

$ mkdir static/js
$ mkdir static/css
$ touch static/js/main.js
$ touch static/css/style.css

The test should now pass:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
Listening on port 5000
  server
Starting the server
    Test the index route
      ✓ should return a page with the title Babblr (41ms)
Stopping the server
  1 passing (54ms)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/babblr/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/babblr/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements   : 100% ( 24/24 ), 5 ignored
Branches     : 100% ( 6/6 ), 1 ignored
Functions    : 100% ( 1/1 )
Lines        : 100% ( 24/24 )
================================================================================
>> Done. Check coverage folder.
Running "coveralls:app" (coveralls) task
>> Failed to submit 'coverage/lcov.info' to coveralls: Bad response: 422 {"message":"Couldn't find a repository matching this job.","error":true}
>> Failed to submit coverage results to coveralls
Warning: Task "coveralls:app" failed. Use --force to continue.
Aborted due to warnings.

Don’t worry about the coveralls task failing, as that only needs to pass when it runs on Travis CI.

So we now have our main route in place. The next step is to actually implement the chat functionality. Add this code to the test file:

    // Test sending a message
    describe('Test sending a message', function () {
        it("should return 'Message received'", function (done) {
            // Connect to server
            var socket = io.connect('http://localhost:5000', {
                'reconnection delay' : 0,
                'reopen delay' : 0,
                'force new connection' : true
            });
            // Handle the message being received
            socket.on('message', function (data) {
                expect(data).to.include('Message received');
                socket.disconnect();
                done();
            });
            // Send the message
            socket.emit('send', { message: 'Message received' });
        });
    });

This code should be fairly straightforward to understand. First, we connect to the server. Then, we set up a handler to verify the content of the message when it gets sent. Finally, we send the message. Let’s run the tests to make sure we get the expected result:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
Listening on port 5000
  server
Starting the server
    Test the index route
      ✓ should return a page with the title Babblr (337ms)
    Test sending a message
      1) should return 'Message received'
Stopping the server
  1 passing (2s)
  1 failing
  1) server Test sending a message should return 'Message received':
     Error: timeout of 2000ms exceeded
      at null.<anonymous> (/Users/matthewdaly/Projects/babblr/node_modules/mocha/lib/runnable.js:159:19)
      at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/babblr/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/babblr/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements   : 100% ( 24/24 ), 5 ignored
Branches     : 100% ( 6/6 ), 1 ignored
Functions    : 100% ( 1/1 )
Lines        : 100% ( 24/24 )
================================================================================
>>
Warning: Task "mocha_istanbul:coverage" failed. Use --force to continue.
Aborted due to warnings.

Now, let’s implement this functionality. Add this at the end of index.js:

// Handle new messages
io.sockets.on('connection', function (socket) {
    // Subscribe to the Redis channel
    subscribe.subscribe('ChatChannel');
    // Handle incoming messages
    socket.on('send', function (data) {
        // Publish it
        client.publish('ChatChannel', data.message);
    });
    // Handle receiving messages
    var callback = function (channel, data) {
        socket.emit('message', data);
    };
    subscribe.on('message', callback);
    // Handle disconnect
    socket.on('disconnect', function () {
        subscribe.removeListener('message', callback);
    });
});

We’ll go through this. First, we create a callback for when a new connection is received. Inside the callback, we then subscribe to a Pub/Sub channel in Redis called ChatChannel.

Then, we define another callback so that on a send event from Socket.IO, we get the message and publish it to ChatChannel. After that, we define another callback to handle receiving messages, and set it to run when a new message is published to ChatChannel. Finally, we set up a callback to handle removing the listener when a user disconnects.

Note the two different connections to Redis - client and subscribe. As mentioned earlier, you need to use two connections to Redis when using Pub/Sub. This is because a client subscribed to one or more channels should not issue commands, so we use subscribe as a dedicated connection to handle subscriptions, and use client to publish new messages.

We’ll also need a bit of client-side JavaScript to handle sending and receiving messages. Amend main.js as follows:

$(document).ready(function () {
    'use strict';
    // Set up the connection
    var field, socket, output;
    socket = io.connect(window.location.href);
    // Get a reference to the input
    field = $('textarea#message');
    // Get a reference to the output
    output = $('div.conversation');
    // Handle message submit
    $('a#submitbutton').on('click', function () {
        // Create the message
        var msg;
        msg = field.val();
        socket.emit('send', { message: msg });
        field.val('');
    });
    // Handle incoming messages
    socket.on('message', function (data) {
        // Insert the message
        output.append('<p>Anonymous Coward : ' + data + '</p>');
    });
});

Here we have one callback that handles sending messages, and another that handles receiving messages. Note that every message will be preceded with Anonymous Coward - we won’t implement user names at this point (though I plan it for a future instalment).

We’ll also add a little bit of additional styling:

div.conversation {
    height: 500px;
    overflow-y: scroll;
    border: 1px solid #000;
    padding: 10px;
}

Now, if you run your tests, they should pass:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
Listening on port 5000
  server
Starting the server
    Test the index route
      ✓ should return a page with the title Babblr (40ms)
    Test sending a message
      ✓ should return 'Message received' (45ms)
Stopping the server
  2 passing (101ms)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/babblr/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/babblr/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements   : 100% ( 33/33 ), 5 ignored
Branches     : 100% ( 6/6 ), 1 ignored
Functions    : 100% ( 5/5 )
Lines        : 100% ( 33/33 )
================================================================================
>> Done. Check coverage folder.
Running "coveralls:app" (coveralls) task
>> Failed to submit 'coverage/lcov.info' to coveralls: Bad response: 422 {"message":"Couldn't find a repository matching this job.","error":true}
>> Failed to submit coverage results to coveralls
Warning: Task "coveralls:app" failed. Use --force to continue.
Aborted due to warnings.

If you now run the following command:

$ node index.js

Then visit http://localhost:5000, you should be able to create new messages. If you then open it up in a second tab, you can see messages added in one tab appear in another. Deploying to Heroku using Redis To Go will be straightforward, and you can then access it from multiple devices and see new chat messages appear in real time.

Wrapping up

This illustrates just how straightforward it is to use Redis’s Pub/Sub capability. The chat system is still quite limited, so in a future instalment we’ll develop it further. You can get the source code from the Github repository - just switch to the lesson-1 tag.

28th December 2014 5:04 pm

My First Grunt Plugin

A while back, I mentioned that I’d written a Yeoman generator for creating a flat HTML blog, called generator-simple-static-blog. For this, I’d used the first Grunt plugin I could find for the purpose, which was grunt-markdown-blog. This worked, but I wasn’t really very happy with it.

The ideal Grunt plugin I had in mind was as follows:

  • Used Handlebars for templating
  • Generated posts from Markdown files
  • Saved files in named folders with a single index.html file in each one (like Octopress does) so that no file extension is visible on a page
  • Generated index pages, rather than just showing the latest post as the first page

Unfortunately, grunt-markdown-blog only fulfilled the second criteria, so it was never going to be something I stuck with long-term. However, I couldn’t find anything else that would do the trick, so it looked like my only option was to write a suitable plugin myself.

I started a new Git repository a while back, but didn’t make much progress. Then, on Christmas Eve, I suddenly got the urge to start working on this again, and in a matter of a few hours I’d gotten a working Grunt plugin that ticked all of these boxes. I had to delay getting it integrated into the generator due to Christmas day, and then an unfortunate bout of flu, but I’ve now published it as grunt-blogbuilder and amended the Yeoman generator to use it instead.

I’m really pleased with the outcome, and while I’m still not yet ready to migrate over to it from Octopress, it’s a massive step forward, and building a Grunt plugin has been an interesting experience.

9th November 2014 5:13 pm

Building a URL Shortener With Node.js and Redis

The NoSQL movement is an exciting one for web developers. While relational databases such as MySQL are applicable to solving a wide range of problems, they aren’t the best solution for every problem. Sometimes you may find yourself dealing with a problem where an alternative data store may make more sense.

Redis is one of the data stores that have appeared as part of this movement, and is arguably one of the more generally useful ones. Since it solves different problems to a relational database, it’s not generally useful as an alternative to them - instead it is often used alongside them.

What is Redis?

Redis is described as follows on the website:

“Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs”.

In other words, its core functionality is that it allows you to store a value by a key, and later retrieve that data using the key. It also allows you to set an optional expiry time for that key-value pair. It’s quite similar to Memcached in that respect, and indeed one obvious use case for Redis is as an alternative to Memcached. However, it offers a number of additional benefits - for one thing, it supports more data types, and for another, it allows you to persist your data to disk (unlike Memcached, which only retains the data in memory, meaning it’s lost on restart or if Memcached crashes). The latter means that for some very simple web applications, Redis can be used as your sole data store.

In this tutorial, we’ll build a simple URL shortener, using Redis as the sole data store. A URL shortener only really requires two fields:

  • A string to identify the correct URL
  • The URL

That makes Redis a good fit for this use case since all we need to do is generate an ID for each URL, then when a link is followed, look up the URL for that key, and redirect the user to it. As long as this is all our application needs to do, we can quite happily use Redis for this rather than a relational database, and it will be significantly faster than a relational database would be for this use case.

Getting started

We’re more interested in the fundamentals of using Redis in our application than a specific language here. As JavaScript is pretty much required to be a web developer, I think it’s a fairly safe bet to use Node.js rather than PHP or Python, since that way, even if your only experience of JavaScript is client-side, you shouldn’t have too much trouble following along.

You’ll need to have Node.js installed, and I’ll leave the details of installing this to you. You’ll also need the Grunt CLI - install this globally as follows:

$ sudo npm install -g grunt-cli

Finally, you’ll want to have Redis itself installed. You might also want to install hiredis, which is a faster Redis client that gets used automatically where available.

Now, let’s create our package.json file:

$ npm init

You’ll see a number of questions. Your generated package.json file should look something like this:

{
  "name": "url-shortener",
  "version": "1.0.0",
  "description": "A URL shortener",
  "main": "index.js",
  "scripts": {
    "test": "grunt test"
  },
  "keywords": [
    "URL",
    "shortener"
  ],
  "author": "Matthew Daly <matthew@matthewdaly.co.uk> (http://matthewdaly.co.uk/)",
  "license": "GPLv2"
}

Note in particular that we set our test command to grunt test.

Next, we install our required Node.js modules. First, install, the normal dependencies:

$ npm install --save body-parser express redis hiredis jade shortid

Next, install the development dependencies:

$ npm install --save-dev grunt grunt-contrib-jshint grunt-mocha-istanbul istanbul mocha chai request

Now, we’re going to use Grunt to run our tests, so that we can easily lint the JavaScript and generate code coverage details. Here’s the Gruntfile:

module.exports = function (grunt) {
    'use strict';
    grunt.initConfig({
        jshint: {
            all: [
                'test/*.js',
                'index.js'
            ]
        },
        mocha_istanbul: {
            coverage: {
                src: 'test', // the folder, not the files,
                options: {
                    mask: '*.js',
                    reportFormats: ['cobertura', 'html']
                }
            }
        }
    });
    // Load tasks
    grunt.loadNpmTasks('grunt-contrib-jshint');
    grunt.loadNpmTasks('grunt-mocha-istanbul');
    // Register tasks
    grunt.registerTask('test', ['jshint', 'mocha_istanbul:coverage']);
};

We’ll also create a Procfile in anticipation of deploying the app to Heroku:

web: node index.js

Creating the views

For this application we’ll be using the Express framework and the Jade templating system. We need three templates:

  • Submission form
  • Output form
  • 404 page

Create the folder views under the application directory and add the files views/index.jade:

doctype html
html(lang="en")
    head
        title="Shortbread"
    body
        div.container
            div.row
                h1 Shortbread
            div.row
                form(action="/", method="POST")
                    input(type="url", name="url")
                    input(type="submit", value="Submit")

Also views/output.jade:

doctype html
html(lang="en")
    head
        title=Shortbread
    body
        div.container
            div.row
                h1 Shortbread
                p
                    | Your shortened URL is
                    a(href=base_url+'/'+id) #{base_url}/#{id}

and views/error.jade:

doctype html
html(lang="en")
    head
        title="Shortbread"
    body
        div.container
            div.row
                h1 Shortbread
                p Link not found

Writing our first test

We’re going to use Mocha for our tests, together with the Chai assertion library. Create a folder called test, and put the following in test/test.js:

/*jslint node: true */
/*global describe: false, before: false, after: false, it: false */
"use strict";
// Declare the variables used
var expect = require('chai').expect,
    request = require('request'),
    server = require('../index'),
    redis = require('redis'),
    client;
client = redis.createClient();
// Server tasks
describe('server', function () {
    // Beforehand, start the server
    before(function (done) {
        console.log('Starting the server');
        done();
    });
    // Afterwards, stop the server and empty the database
    after(function (done) {
        console.log('Stopping the server');
        client.flushdb();
        done();
    });
    // Test the index route
    describe('Test the index route', function () {
        it('should return a page with the title Shortbread', function (done) {
            request.get({ url: 'http://localhost:5000' }, function (error, response, body) {
                expect(body).to.include('Shortbread');
                expect(response.statusCode).to.equal(200);
                expect(response.headers['content-type']).to.equal('text/html; charset=utf-8');
                done();
            });
        });
    });
});

This code bears a little explanation. First, we import the required modules, as well as our index.js file (which we have yet to add). Then we create a callback to contain our tests.

Inside the callback, we call the before() and after() functions, which let us set up and tear down our tests. As part of the teardown process, we flush the Redis database.

Finally, we fetch our home page and verify that it returns a 200 status code and a content type of text/html, as well as including the name or our application.

We’ll need to create our index.js file to avoid a nasty error, but we won’t populate it just yet:

$ touch index.js

Let’s run our tests:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
  server
Starting the server
    Test the index route
      1) should return a page with the title Shortbread
Stopping the server
  0 passing (152ms)
  1 failing
  1) server Test the index route should return a page with the title Shortbread:
     Uncaught AssertionError: expected undefined to include 'Shortbread'
      at Request._callback (/Users/matthewdaly/Projects/url-shortener/test/test.js:33:33)
      at self.callback (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:372:22)
      at Request.emit (events.js:95:17)
      at Request.onRequestError (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:963:8)
      at ClientRequest.emit (events.js:95:17)
      at Socket.socketErrorListener (http.js:1551:9)
      at Socket.emit (events.js:95:17)
      at net.js:440:14
      at process._tickCallback (node.js:419:13)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/url-shortener/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/url-shortener/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements   : 100% ( 0/0 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 0/0 )
Lines        : 100% ( 0/0 )
================================================================================
>>
Warning: Task "mocha_istanbul:coverage" failed. Use --force to continue.
Aborted due to warnings.

Now that we have a failing test, we can start work on our app proper. Open up index.js and add the following code:

/*jslint node: true */
'use strict';
// Declare variables used
var app, base_url, bodyParser, client, express, port, rtg, shortid;
// Define values
express = require('express');
app = express();
port = process.env.PORT || 5000;
shortid = require('shortid');
bodyParser = require('body-parser');
base_url = process.env.BASE_URL || 'http://localhost:5000';
// Set up connection to Redis
if (process.env.REDISTOGO_URL) {
  rtg  = require("url").parse(process.env.REDISTOGO_URL);
  client = require("redis").createClient(rtg.port, rtg.hostname);
  client.auth(rtg.auth.split(":")[1]);
} else {
  client = require('redis').createClient();
}
// Set up templating
app.set('views', __dirname + '/views');
app.set('view engine', "jade");
app.engine('jade', require('jade').__express);
// Set URL
app.set('base_url', base_url);
// Handle POST data
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({
  extended: true
}));
// Define index route
app.get('/', function (req, res) {
  res.render('index');
});
// Serve static files
app.use(express.static(__dirname + '/static'));
// Listen
app.listen(port);
console.log('Listening on port ' + port);

Let’s go through this code. First we confirm that linting tools should treat this as a Node app, and use strict mode (I recommend always using strict mode in JavaScript).

Then we declare our variables and import the required modules. Note here that we set the port to 5000, but can also set it based on the PORT environment variable, which is used by Heroku. We also define a base URL, which again can be overriden from an environment variable when hosted on Heroku.

We then set up our connection to our Redis instance. When we push the code up to Heroku, we’ll use the Redis To Go addon, so we check for an environment variable containing the Redis URL. If it’s set, we use that to connect. Otherwise, we just connect as normal.

We then set up templating using Jade, and define the folder containing our views, and store the base URL within the app. Then we set up bodyParser so that Express can handle POST data.

Next, we define our index route to just render the index.jade file. Finally, we set up our static folder and set the app to listen on the correct port.

Let’s run our test:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
Listening on port 5000
  server
Starting the server
    Test the index route
      ✓ should return a page with the title Shortbread (116ms)
Stopping the server
  1 passing (128ms)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/url-shortener/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/url-shortener/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements   : 86.96% ( 20/23 )
Branches     : 83.33% ( 5/6 )
Functions    : 100% ( 1/1 )
Lines        : 86.96% ( 20/23 )
================================================================================
>> Done. Check coverage folder.
Done, without errors.

Note that Istanbul will have generated a nice HTML coverage report, which will be at coverage/index.html, but this won’t show 100% test coverage due to the Heroku-specific Redis section. To fix this, amend that section as follows:

/* istanbul ignore if */
if (process.env.REDISTOGO_URL) {
    rtg  = require("url").parse(process.env.REDISTOGO_URL);
    client = require("redis").createClient(rtg.port, rtg.hostname);
    client.auth(rtg.auth.split(":")[1]);
} else {
    client = require('redis').createClient();
}

Telling Istanbul to ignore the if clause resolves that problem nicely.

Submitting a URL

Next, let’s add the ability to add a URL. First, add the following test, after the one for the index:

    // Test submitting a URL
    describe('Test submitting a URL', function () {
        it('should return the shortened URL', function (done) {
            request.post('http://localhost:5000', {form: {url: 'http://www.google.co.uk'}}, function (error, response, body) {
                expect(body).to.include('Your shortened URL is');
                expect(response.statusCode).to.equal(200);
                expect(response.headers['content-type']).to.equal('text/html; charset=utf-8');
                done();
            });
        });
    });

This test submits a URL via POST, and checks to see that the response view gets returned. Now, let’s run our tests again:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
Listening on port 5000
  server
Starting the server
    Test the index route
      ✓ should return a page with the title Shortbread (223ms)
    Test submitting a URL
      1) should return the shortened URL
Stopping the server
  1 passing (318ms)
  1 failing
  1) server Test submitting a URL should return the shortened URL:
     Uncaught AssertionError: expected 'Cannot POST /\n' to include 'Your shortened URL is'
      at Request._callback (/Users/matthewdaly/Projects/url-shortener/test/test.js:45:33)
      at Request.self.callback (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:372:22)
      at Request.emit (events.js:98:17)
      at Request.<anonymous> (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:1310:14)
      at Request.emit (events.js:117:20)
      at IncomingMessage.<anonymous> (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:1258:12)
      at IncomingMessage.emit (events.js:117:20)
      at _stream_readable.js:943:16
      at process._tickCallback (node.js:419:13)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/url-shortener/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/url-shortener/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements   : 100% ( 23/23 ), 3 ignored
Branches     : 100% ( 6/6 ), 1 ignored
Functions    : 100% ( 1/1 )
Lines        : 100% ( 23/23 )
================================================================================
>>
Warning: Task "mocha_istanbul:coverage" failed. Use --force to continue.
Aborted due to warnings.

We have a failing test, so let’s make it pass. Add the following route after the index one:

// Define submit route
app.post('/', function (req, res) {
    // Declare variables
    var url, id;
    // Get URL
    url = req.body.url;
    // Create a hashed short version
    id = shortid.generate();
    // Store them in Redis
    client.set(id, url, function () {
        // Display the response
        res.render('output', { id: id, base_url: base_url });
    });
});

This route is fairly simple. It handles POST requests to the index route, and first of all it gets the URL from the POST request. Then it randomly generates a hash to use as the key.

The next part is where we see Redis in action. We create a new key-value pair, with the key set to the newly generated ID, and the value set to the URL. Once Redis confirms that has been done, the callback is fired, which renders the output.jade view with the ID and base URL passed through, so that we can see our shortened URL.

With that done, our test should pass:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
Listening on port 5000
  server
Starting the server
    Test the index route
      ✓ should return a page with the title Shortbread (89ms)
    Test submitting a URL
      ✓ should return the shortened URL (65ms)
Stopping the server
  2 passing (167ms)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/url-shortener/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/url-shortener/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements   : 100% ( 29/29 ), 3 ignored
Branches     : 100% ( 6/6 ), 1 ignored
Functions    : 100% ( 3/3 )
Lines        : 100% ( 29/29 )
================================================================================
>> Done. Check coverage folder.
Done, without errors.

Our final task is to implement the short URL handling. We want to check to see if a short URL exists. If it does, we redirect the user to the destination. If it doesn’t, we raise a 404 error. For that we need two more tests. Here they are:

    // Test following a URL
    describe('Test following a URL', function () {
        it('should redirect the user to the shortened URL', function (done) {
            // Create the URL
            client.set('testurl', 'http://www.google.com', function () {
                // Follow the link
                request.get({
                    url: 'http://localhost:5000/testurl',
                    followRedirect: false
                }, function (error, response, body) {
                    expect(response.headers.location).to.equal('http://www.google.com');
                    expect(response.statusCode).to.equal(301);
                    done();
                });
            });
        });
    });
    // Test non-existent link
    describe('Test following a non-existent-link', function () {
        it('should return a 404 error', function (done) {
            // Follow the link
            request.get({
                url: 'http://localhost:5000/nonexistenturl',
                followRedirect: false
            }, function (error, response, body) {
                expect(response.statusCode).to.equal(404);
                expect(body).to.include('Link not found');
                done();
            });
        });
    });

The first test creates a URL for testing purposes. It then navigates to that URL. Note that we set followRedirect to true - this is because, by default, request will follow any redirect, so we need to prevent it from doing so to ensure that the headers to redirect the user are set correctly.

Once the response has been received, we then check that the status code is 301 (Moved Permanently), and that the Location header is set to the correct destination. When a real browser visits this page, it will be redirected accordingly.

The second test tries to fetch a non-existent URL, and checks that the status code is 404, and the response contains the words Link not found.

If we run our tests, they should now fail:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
Listening on port 5000
  server
Starting the server
    Test the index route
      ✓ should return a page with the title Shortbread (252ms)
    Test submitting a URL
      ✓ should return the shortened URL (47ms)
    Test following a URL
      1) should redirect the user to the shortened URL
    Test following a non-existent-link
      2) should return a 404 error
Stopping the server
  2 passing (322ms)
  2 failing
  1) server Test following a URL should redirect the user to the shortened URL:
     Uncaught AssertionError: expected undefined to equal 'http://www.google.com'
      at Request._callback (/Users/matthewdaly/Projects/url-shortener/test/test.js:63:58)
      at Request.self.callback (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:372:22)
      at Request.emit (events.js:98:17)
      at Request.<anonymous> (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:1310:14)
      at Request.emit (events.js:117:20)
      at IncomingMessage.<anonymous> (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:1258:12)
      at IncomingMessage.emit (events.js:117:20)
      at _stream_readable.js:943:16
      at process._tickCallback (node.js:419:13)
  2) server Test following a non-existent-link should return a 404 error:
     Uncaught AssertionError: expected 'Cannot GET /nonexistenturl\n' to include 'Link not found'
      at Request._callback (/Users/matthewdaly/Projects/url-shortener/test/test.js:80:33)
      at Request.self.callback (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:372:22)
      at Request.emit (events.js:98:17)
      at Request.<anonymous> (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:1310:14)
      at Request.emit (events.js:117:20)
      at IncomingMessage.<anonymous> (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:1258:12)
      at IncomingMessage.emit (events.js:117:20)
      at _stream_readable.js:943:16
      at process._tickCallback (node.js:419:13)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/url-shortener/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/url-shortener/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements   : 100% ( 29/29 ), 3 ignored
Branches     : 100% ( 6/6 ), 1 ignored
Functions    : 100% ( 3/3 )
Lines        : 100% ( 29/29 )
================================================================================
>>
Warning: Task "mocha_istanbul:coverage" failed. Use --force to continue.
Aborted due to warnings.

Now, let’s add our final route:

// Define link route
app.route('/:id').all(function (req, res) {
    // Get ID
    var id = req.params.id.trim();
    // Look up the URL
    client.get(id, function (err, reply) {
        if (!err && reply) {
            // Redirect user to it
            res.status(301);
            res.set('Location', reply);
            res.send();
        } else {
            // Confirm no such link in database
            res.status(404);
            res.render('error');
        }
    });
});

We accept the ID as a parameter in the URL. We trim off any whitespace around it, and then we query Redis for a URL with that ID. If we find one, we set the status code to 301, and the location to the URL, and send the response. Otherwise, we set the status to 404 and render the error view.

Now, let’s check it passes:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
Listening on port 5000
  server
Starting the server
    Test the index route
      ✓ should return a page with the title Shortbread (90ms)
    Test submitting a URL
      ✓ should return the shortened URL (47ms)
    Test following a URL
      ✓ should redirect the user to the shortened URL
    Test following a non-existent-link
      ✓ should return a 404 error
Stopping the server
  4 passing (191ms)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/url-shortener/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/url-shortener/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements   : 100% ( 38/38 ), 3 ignored
Branches     : 100% ( 10/10 ), 1 ignored
Functions    : 100% ( 5/5 )
Lines        : 100% ( 38/38 )
================================================================================
>> Done. Check coverage folder.
Done, without errors.

Excellent! Our URL shortener is now complete. From here, deploying it to Heroku is straightforward - you’ll need to install the Redis to Go addon, and refer to Heroku’s documentation on deploying Node.js applications for more details.

Wrapping up

You’ll find the source for this application here and a demo here.

I hope you’ve enjoyed this brief introduction to Redis, and that it’s opened your eyes to at least one of the alternatives out there to a relational database. I’ll hopefully be able to follow this up with examples of some other problems Redis is ideal for solving.

19th October 2014 7:52 pm

My Django Web Server Setup

This isn’t really part of my Django tutorial series (that has now definitely concluded!), but I thought I’d share the setup I generally use for deploying Django applications, partly for my own reference, and partly because it is quite complex, and those readers who don’t wish to deploy to Heroku may want some guidance on how to deploy their new blogs to a VPS.

Operating system

This isn’t actually that much of a big deal, but while I prefer Ubuntu on desktops, I generally use Debian Stable on servers, since it’s fanatically stable.

Database server

For my first commercial Django app, I used MySQL. However, South had one or two issues with MySQL, and I figured that since using an ORM and migrations meant that I wouldn’t need to write much SQL anyway, I might as well jump to PostgreSQL for the Django app I’m currently in the process of deploying at work. So far I haven’t had any problems with it.

Web server

It’s customary to use two web servers with Django. One handles the static content, and reverse proxies everything else to a different port, where another web server serves the dynamic content.

For serving the static files, I use Nginx - it’s generally considered to be faster than Apache for this use case. Here’s a typical Nginx config file:

server {
    listen 80;
    server_name example.com;
    client_max_body_size 50M;
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;
    location /static {
        root /var/www/mysite;
    }
    location /media {
        root /var/www/mysite;
    }
    location / {
        proxy_pass http://127.0.0.1:8000;
    }
}

For the application server, I use Gunicorn. You can install this with pip install gunicorn, then add it to INSTALLED_APPS. Then add the following config file to the root of your project, as gunicorn.conf.py:

bind = "127.0.0.1:8000"
logfile = "/var/log/gunicorn.log"
loglevel = "debug"
workers = 3

You should normally set the number of workers to 2 times the number of cores on your machine, plus one.

In order to keep Gunicorn running, I use Supervisor. As the installation commands will depend on your OS, I won’t give details here - your package manager of choice should have a suitable package available. Here’s a typical Supervisor config file I might use for running Gunicorn for a Django app, named mysite-supervisor.conf:

[program:mysite]
command=/var/www/mysite/venv/bin/gunicorn myapp.wsgi:application --workers=3
directory=/var/www/mysite/
user=nobody
autostart=true
autorestart=true

Once that’s in place, you can easily add the new app:

$ sudo supervisorctl reread
$ sudo supervisorctl update

Then to start it:

$ sudo supervisorctl start mysite

Or stop it with:

$ sudo supervisorctl stop mysite

Or restart it:

$ sudo supervisorctl restart mysite

Celery

So far, both of the web apps I’ve built professionally have been ones where it made sense to use Celery for some tasks. For the uninitiated, Celery lets you pass a task to a queue to be handled, rather than handling it within the context of the same HTTP request. This offers the following advantages:

  • The user doesn’t need to wait for the task to be completed before getting a response, improving performance
  • It’s more robust, since if the task fails, it can be automatically retried
  • The task queue can be moved to another server if desired, making it easier to scale
  • Scheduling tasks

I’ve used it in cases where I needed to send an email or a push notification, since these don’t have to be done within the context of the same HTTP request, but need to be reliable.

I generally use RabbitMQ as my message queue. I’ll leave setting this up as an exercise for the reader since it’s covered pretty well in the Celery documentation, but like with Gunicorn, I use Supervisor to run the Celery worker. Here’s a typical config file, which might be called celery-supervisor.conf:

[program:celeryd]
command=/var/www/mysite/venv/bin/python manage.py celery worker
directory=/var/www/mysite/
user=nobody
autostart=true
autorestart=true

Then start it up:

$ sudo supervisorctl reread
$ sudo supervisorctl update
$ sudo supervisorctl start celeryd

I make no claims about how good this setup is, but it works well for me. I haven’t yet had the occasion to deploy a Django app to anywhere other than Heroku that really benefited from caching, so I haven’t got any tips to share about that, but if I were building a content-driven web app, I would use Memcached since it’s well-supported.

5th October 2014 7:56 pm

Introducing Generator-simple-static-blog

I’m a big fan of static site generators. I ditched WordPress for Octopress over two years ago because it was free to host on GitHub Pages and much faster, had much better syntax highlighting, and I liked being able to write posts in Vim, and I’ve never looked back since.

That said, Octopress is written in Ruby, a language I’ve never been that keen on. Ideally I’d prefer to use Python or JavaScript, but none of the solutions I’ve found have been to my liking. Recently I’ve been using Grunt and Yeoman to some extent, and I’ve wondered about the idea of creating a Yeoman generator to build a static blogging engine. After discovering grunt-markdown-blog, I took the plunge and have built a simple blog generator called generator-simple-static-blog.

I’ve published it to NPM, so feel free to check it out. It includes code highlighting with the Zenburn colour scheme by default (although highlight.js includes many other themes, so just switch to another one if you want), and it should be easy to edit the templates. I’ve also included the ability to deploy automatically to GitHub Pages using Grunt.

I don’t anticipate moving over to this from Octopress for the foreseeable future, but it’s been an interesting project for the weekend.

Recent Posts

Adding React to a Legacy Project

Do You Still Need Jquery?

An Approach to Writing Golden Master Tests for PHP Web Applications

Understanding the Pipeline Pattern

Replacing Switch Statements With Polymorphism in PHP

About me

I'm a web and mobile app developer based in Norfolk. My skillset includes Python, PHP and Javascript, and I have extensive experience working with CodeIgniter, Laravel, Django, Phonegap and Angular.js.