Matthew Daly's Blog

I'm a web developer in Norfolk. This is my blog...

9th November 2014 5:13 pm

Building a URL Shortener With Node.js and Redis

The NoSQL movement is an exciting one for web developers. While relational databases such as MySQL are applicable to solving a wide range of problems, they aren’t the best solution for every problem. Sometimes you may find yourself dealing with a problem where an alternative data store may make more sense.

Redis is one of the data stores that have appeared as part of this movement, and is arguably one of the more generally useful ones. Since it solves different problems to a relational database, it’s not generally useful as an alternative to them - instead it is often used alongside them.

What is Redis?

Redis is described as follows on the website:

“Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs”.

In other words, its core functionality is that it allows you to store a value by a key, and later retrieve that data using the key. It also allows you to set an optional expiry time for that key-value pair. It’s quite similar to Memcached in that respect, and indeed one obvious use case for Redis is as an alternative to Memcached. However, it offers a number of additional benefits - for one thing, it supports more data types, and for another, it allows you to persist your data to disk (unlike Memcached, which only retains the data in memory, meaning it’s lost on restart or if Memcached crashes). The latter means that for some very simple web applications, Redis can be used as your sole data store.

In this tutorial, we’ll build a simple URL shortener, using Redis as the sole data store. A URL shortener only really requires two fields:

  • A string to identify the correct URL
  • The URL

That makes Redis a good fit for this use case since all we need to do is generate an ID for each URL, then when a link is followed, look up the URL for that key, and redirect the user to it. As long as this is all our application needs to do, we can quite happily use Redis for this rather than a relational database, and it will be significantly faster than a relational database would be for this use case.

Getting started

We’re more interested in the fundamentals of using Redis in our application than a specific language here. As JavaScript is pretty much required to be a web developer, I think it’s a fairly safe bet to use Node.js rather than PHP or Python, since that way, even if your only experience of JavaScript is client-side, you shouldn’t have too much trouble following along.

You’ll need to have Node.js installed, and I’ll leave the details of installing this to you. You’ll also need the Grunt CLI - install this globally as follows:

$ sudo npm install -g grunt-cli

Finally, you’ll want to have Redis itself installed. You might also want to install hiredis, which is a faster Redis client that gets used automatically where available.

Now, let’s create our package.json file:

$ npm init

You’ll see a number of questions. Your generated package.json file should look something like this:

{
"name": "url-shortener",
"version": "1.0.0",
"description": "A URL shortener",
"main": "index.js",
"scripts": {
"test": "grunt test"
},
"keywords": [
"URL",
"shortener"
],
"author": "Matthew Daly <matthew@matthewdaly.co.uk> (http://matthewdaly.co.uk/)",
"license": "GPLv2"
}

Note in particular that we set our test command to grunt test.

Next, we install our required Node.js modules. First, install, the normal dependencies:

$ npm install --save body-parser express redis hiredis jade shortid

Next, install the development dependencies:

$ npm install --save-dev grunt grunt-contrib-jshint grunt-mocha-istanbul istanbul mocha chai request

Now, we’re going to use Grunt to run our tests, so that we can easily lint the JavaScript and generate code coverage details. Here’s the Gruntfile:

module.exports = function (grunt) {
'use strict';
grunt.initConfig({
jshint: {
all: [
'test/*.js',
'index.js'
]
},
mocha_istanbul: {
coverage: {
src: 'test', // the folder, not the files,
options: {
mask: '*.js',
reportFormats: ['cobertura', 'html']
}
}
}
});
// Load tasks
grunt.loadNpmTasks('grunt-contrib-jshint');
grunt.loadNpmTasks('grunt-mocha-istanbul');
// Register tasks
grunt.registerTask('test', ['jshint', 'mocha_istanbul:coverage']);
};

We’ll also create a Procfile in anticipation of deploying the app to Heroku:

web: node index.js

Creating the views

For this application we’ll be using the Express framework and the Jade templating system. We need three templates:

  • Submission form
  • Output form
  • 404 page

Create the folder views under the application directory and add the files views/index.jade:

doctype html
html(lang="en")
head
title="Shortbread"
body
div.container
div.row
h1 Shortbread
div.row
form(action="/", method="POST")
input(type="url", name="url")
input(type="submit", value="Submit")

Also views/output.jade:

doctype html
html(lang="en")
head
title=Shortbread
body
div.container
div.row
h1 Shortbread
p
| Your shortened URL is
a(href=base_url+'/'+id) #{base_url}/#{id}

and views/error.jade:

doctype html
html(lang="en")
head
title="Shortbread"
body
div.container
div.row
h1 Shortbread
p Link not found

Writing our first test

We’re going to use Mocha for our tests, together with the Chai assertion library. Create a folder called test, and put the following in test/test.js:

/*jslint node: true */
/*global describe: false, before: false, after: false, it: false */
"use strict";
// Declare the variables used
var expect = require('chai').expect,
request = require('request'),
server = require('../index'),
redis = require('redis'),
client;
client = redis.createClient();
// Server tasks
describe('server', function () {
// Beforehand, start the server
before(function (done) {
console.log('Starting the server');
done();
});
// Afterwards, stop the server and empty the database
after(function (done) {
console.log('Stopping the server');
client.flushdb();
done();
});
// Test the index route
describe('Test the index route', function () {
it('should return a page with the title Shortbread', function (done) {
request.get({ url: 'http://localhost:5000' }, function (error, response, body) {
expect(body).to.include('Shortbread');
expect(response.statusCode).to.equal(200);
expect(response.headers['content-type']).to.equal('text/html; charset=utf-8');
done();
});
});
});
});

This code bears a little explanation. First, we import the required modules, as well as our index.js file (which we have yet to add). Then we create a callback to contain our tests.

Inside the callback, we call the before() and after() functions, which let us set up and tear down our tests. As part of the teardown process, we flush the Redis database.

Finally, we fetch our home page and verify that it returns a 200 status code and a content type of text/html, as well as including the name or our application.

We’ll need to create our index.js file to avoid a nasty error, but we won’t populate it just yet:

$ touch index.js

Let’s run our tests:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
server
Starting the server
Test the index route
1) should return a page with the title Shortbread
Stopping the server
0 passing (152ms)
1 failing
1) server Test the index route should return a page with the title Shortbread:
Uncaught AssertionError: expected undefined to include 'Shortbread'
at Request._callback (/Users/matthewdaly/Projects/url-shortener/test/test.js:33:33)
at self.callback (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:372:22)
at Request.emit (events.js:95:17)
at Request.onRequestError (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:963:8)
at ClientRequest.emit (events.js:95:17)
at Socket.socketErrorListener (http.js:1551:9)
at Socket.emit (events.js:95:17)
at net.js:440:14
at process._tickCallback (node.js:419:13)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/url-shortener/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/url-shortener/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements : 100% ( 0/0 )
Branches : 100% ( 0/0 )
Functions : 100% ( 0/0 )
Lines : 100% ( 0/0 )
================================================================================
>>
Warning: Task "mocha_istanbul:coverage" failed. Use --force to continue.
Aborted due to warnings.

Now that we have a failing test, we can start work on our app proper. Open up index.js and add the following code:

/*jslint node: true */
'use strict';
// Declare variables used
var app, base_url, bodyParser, client, express, port, rtg, shortid;
// Define values
express = require('express');
app = express();
port = process.env.PORT || 5000;
shortid = require('shortid');
bodyParser = require('body-parser');
base_url = process.env.BASE_URL || 'http://localhost:5000';
// Set up connection to Redis
if (process.env.REDISTOGO_URL) {
rtg = require("url").parse(process.env.REDISTOGO_URL);
client = require("redis").createClient(rtg.port, rtg.hostname);
client.auth(rtg.auth.split(":")[1]);
} else {
client = require('redis').createClient();
}
// Set up templating
app.set('views', __dirname + '/views');
app.set('view engine', "jade");
app.engine('jade', require('jade').__express);
// Set URL
app.set('base_url', base_url);
// Handle POST data
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({
extended: true
}));
// Define index route
app.get('/', function (req, res) {
res.render('index');
});
// Serve static files
app.use(express.static(__dirname + '/static'));
// Listen
app.listen(port);
console.log('Listening on port ' + port);

Let’s go through this code. First we confirm that linting tools should treat this as a Node app, and use strict mode (I recommend always using strict mode in JavaScript).

Then we declare our variables and import the required modules. Note here that we set the port to 5000, but can also set it based on the PORT environment variable, which is used by Heroku. We also define a base URL, which again can be overriden from an environment variable when hosted on Heroku.

We then set up our connection to our Redis instance. When we push the code up to Heroku, we’ll use the Redis To Go addon, so we check for an environment variable containing the Redis URL. If it’s set, we use that to connect. Otherwise, we just connect as normal.

We then set up templating using Jade, and define the folder containing our views, and store the base URL within the app. Then we set up bodyParser so that Express can handle POST data.

Next, we define our index route to just render the index.jade file. Finally, we set up our static folder and set the app to listen on the correct port.

Let’s run our test:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
Listening on port 5000
server
Starting the server
Test the index route
✓ should return a page with the title Shortbread (116ms)
Stopping the server
1 passing (128ms)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/url-shortener/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/url-shortener/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements : 86.96% ( 20/23 )
Branches : 83.33% ( 5/6 )
Functions : 100% ( 1/1 )
Lines : 86.96% ( 20/23 )
================================================================================
>> Done. Check coverage folder.
Done, without errors.

Note that Istanbul will have generated a nice HTML coverage report, which will be at coverage/index.html, but this won’t show 100% test coverage due to the Heroku-specific Redis section. To fix this, amend that section as follows:

/* istanbul ignore if */
if (process.env.REDISTOGO_URL) {
rtg = require("url").parse(process.env.REDISTOGO_URL);
client = require("redis").createClient(rtg.port, rtg.hostname);
client.auth(rtg.auth.split(":")[1]);
} else {
client = require('redis').createClient();
}

Telling Istanbul to ignore the if clause resolves that problem nicely.

Submitting a URL

Next, let’s add the ability to add a URL. First, add the following test, after the one for the index:

// Test submitting a URL
describe('Test submitting a URL', function () {
it('should return the shortened URL', function (done) {
request.post('http://localhost:5000', {form: {url: 'http://www.google.co.uk'}}, function (error, response, body) {
expect(body).to.include('Your shortened URL is');
expect(response.statusCode).to.equal(200);
expect(response.headers['content-type']).to.equal('text/html; charset=utf-8');
done();
});
});
});

This test submits a URL via POST, and checks to see that the response view gets returned. Now, let’s run our tests again:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
Listening on port 5000
server
Starting the server
Test the index route
✓ should return a page with the title Shortbread (223ms)
Test submitting a URL
1) should return the shortened URL
Stopping the server
1 passing (318ms)
1 failing
1) server Test submitting a URL should return the shortened URL:
Uncaught AssertionError: expected 'Cannot POST /\n' to include 'Your shortened URL is'
at Request._callback (/Users/matthewdaly/Projects/url-shortener/test/test.js:45:33)
at Request.self.callback (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:372:22)
at Request.emit (events.js:98:17)
at Request.<anonymous> (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:1310:14)
at Request.emit (events.js:117:20)
at IncomingMessage.<anonymous> (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:1258:12)
at IncomingMessage.emit (events.js:117:20)
at _stream_readable.js:943:16
at process._tickCallback (node.js:419:13)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/url-shortener/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/url-shortener/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements : 100% ( 23/23 ), 3 ignored
Branches : 100% ( 6/6 ), 1 ignored
Functions : 100% ( 1/1 )
Lines : 100% ( 23/23 )
================================================================================
>>
Warning: Task "mocha_istanbul:coverage" failed. Use --force to continue.
Aborted due to warnings.

We have a failing test, so let’s make it pass. Add the following route after the index one:

// Define submit route
app.post('/', function (req, res) {
// Declare variables
var url, id;
// Get URL
url = req.body.url;
// Create a hashed short version
id = shortid.generate();
// Store them in Redis
client.set(id, url, function () {
// Display the response
res.render('output', { id: id, base_url: base_url });
});
});

This route is fairly simple. It handles POST requests to the index route, and first of all it gets the URL from the POST request. Then it randomly generates a hash to use as the key.

The next part is where we see Redis in action. We create a new key-value pair, with the key set to the newly generated ID, and the value set to the URL. Once Redis confirms that has been done, the callback is fired, which renders the output.jade view with the ID and base URL passed through, so that we can see our shortened URL.

With that done, our test should pass:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
Listening on port 5000
server
Starting the server
Test the index route
✓ should return a page with the title Shortbread (89ms)
Test submitting a URL
✓ should return the shortened URL (65ms)
Stopping the server
2 passing (167ms)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/url-shortener/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/url-shortener/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements : 100% ( 29/29 ), 3 ignored
Branches : 100% ( 6/6 ), 1 ignored
Functions : 100% ( 3/3 )
Lines : 100% ( 29/29 )
================================================================================
>> Done. Check coverage folder.
Done, without errors.

Our final task is to implement the short URL handling. We want to check to see if a short URL exists. If it does, we redirect the user to the destination. If it doesn’t, we raise a 404 error. For that we need two more tests. Here they are:

// Test following a URL
describe('Test following a URL', function () {
it('should redirect the user to the shortened URL', function (done) {
// Create the URL
client.set('testurl', 'http://www.google.com', function () {
// Follow the link
request.get({
url: 'http://localhost:5000/testurl',
followRedirect: false
}, function (error, response, body) {
expect(response.headers.location).to.equal('http://www.google.com');
expect(response.statusCode).to.equal(301);
done();
});
});
});
});
// Test non-existent link
describe('Test following a non-existent-link', function () {
it('should return a 404 error', function (done) {
// Follow the link
request.get({
url: 'http://localhost:5000/nonexistenturl',
followRedirect: false
}, function (error, response, body) {
expect(response.statusCode).to.equal(404);
expect(body).to.include('Link not found');
done();
});
});
});

The first test creates a URL for testing purposes. It then navigates to that URL. Note that we set followRedirect to true - this is because, by default, request will follow any redirect, so we need to prevent it from doing so to ensure that the headers to redirect the user are set correctly.

Once the response has been received, we then check that the status code is 301 (Moved Permanently), and that the Location header is set to the correct destination. When a real browser visits this page, it will be redirected accordingly.

The second test tries to fetch a non-existent URL, and checks that the status code is 404, and the response contains the words Link not found.

If we run our tests, they should now fail:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
Listening on port 5000
server
Starting the server
Test the index route
✓ should return a page with the title Shortbread (252ms)
Test submitting a URL
✓ should return the shortened URL (47ms)
Test following a URL
1) should redirect the user to the shortened URL
Test following a non-existent-link
2) should return a 404 error
Stopping the server
2 passing (322ms)
2 failing
1) server Test following a URL should redirect the user to the shortened URL:
Uncaught AssertionError: expected undefined to equal 'http://www.google.com'
at Request._callback (/Users/matthewdaly/Projects/url-shortener/test/test.js:63:58)
at Request.self.callback (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:372:22)
at Request.emit (events.js:98:17)
at Request.<anonymous> (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:1310:14)
at Request.emit (events.js:117:20)
at IncomingMessage.<anonymous> (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:1258:12)
at IncomingMessage.emit (events.js:117:20)
at _stream_readable.js:943:16
at process._tickCallback (node.js:419:13)
2) server Test following a non-existent-link should return a 404 error:
Uncaught AssertionError: expected 'Cannot GET /nonexistenturl\n' to include 'Link not found'
at Request._callback (/Users/matthewdaly/Projects/url-shortener/test/test.js:80:33)
at Request.self.callback (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:372:22)
at Request.emit (events.js:98:17)
at Request.<anonymous> (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:1310:14)
at Request.emit (events.js:117:20)
at IncomingMessage.<anonymous> (/Users/matthewdaly/Projects/url-shortener/node_modules/request/request.js:1258:12)
at IncomingMessage.emit (events.js:117:20)
at _stream_readable.js:943:16
at process._tickCallback (node.js:419:13)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/url-shortener/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/url-shortener/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements : 100% ( 29/29 ), 3 ignored
Branches : 100% ( 6/6 ), 1 ignored
Functions : 100% ( 3/3 )
Lines : 100% ( 29/29 )
================================================================================
>>
Warning: Task "mocha_istanbul:coverage" failed. Use --force to continue.
Aborted due to warnings.

Now, let’s add our final route:

// Define link route
app.route('/:id').all(function (req, res) {
// Get ID
var id = req.params.id.trim();
// Look up the URL
client.get(id, function (err, reply) {
if (!err && reply) {
// Redirect user to it
res.status(301);
res.set('Location', reply);
res.send();
} else {
// Confirm no such link in database
res.status(404);
res.render('error');
}
});
});

We accept the ID as a parameter in the URL. We trim off any whitespace around it, and then we query Redis for a URL with that ID. If we find one, we set the status code to 301, and the location to the URL, and send the response. Otherwise, we set the status to 404 and render the error view.

Now, let’s check it passes:

$ grunt test
Running "jshint:all" (jshint) task
>> 2 files lint free.
Running "mocha_istanbul:coverage" (mocha_istanbul) task
Listening on port 5000
server
Starting the server
Test the index route
✓ should return a page with the title Shortbread (90ms)
Test submitting a URL
✓ should return the shortened URL (47ms)
Test following a URL
✓ should redirect the user to the shortened URL
Test following a non-existent-link
✓ should return a 404 error
Stopping the server
4 passing (191ms)
=============================================================================
Writing coverage object [/Users/matthewdaly/Projects/url-shortener/coverage/coverage.json]
Writing coverage reports at [/Users/matthewdaly/Projects/url-shortener/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements : 100% ( 38/38 ), 3 ignored
Branches : 100% ( 10/10 ), 1 ignored
Functions : 100% ( 5/5 )
Lines : 100% ( 38/38 )
================================================================================
>> Done. Check coverage folder.
Done, without errors.

Excellent! Our URL shortener is now complete. From here, deploying it to Heroku is straightforward - you’ll need to install the Redis to Go addon, and refer to Heroku’s documentation on deploying Node.js applications for more details.

Wrapping up

You’ll find the source for this application here and a demo here.

I hope you’ve enjoyed this brief introduction to Redis, and that it’s opened your eyes to at least one of the alternatives out there to a relational database. I’ll hopefully be able to follow this up with examples of some other problems Redis is ideal for solving.

19th October 2014 7:52 pm

My Django Web Server Setup

This isn’t really part of my Django tutorial series (that has now definitely concluded!), but I thought I’d share the setup I generally use for deploying Django applications, partly for my own reference, and partly because it is quite complex, and those readers who don’t wish to deploy to Heroku may want some guidance on how to deploy their new blogs to a VPS.

Operating system

This isn’t actually that much of a big deal, but while I prefer Ubuntu on desktops, I generally use Debian Stable on servers, since it’s fanatically stable.

Database server

For my first commercial Django app, I used MySQL. However, South had one or two issues with MySQL, and I figured that since using an ORM and migrations meant that I wouldn’t need to write much SQL anyway, I might as well jump to PostgreSQL for the Django app I’m currently in the process of deploying at work. So far I haven’t had any problems with it.

Web server

It’s customary to use two web servers with Django. One handles the static content, and reverse proxies everything else to a different port, where another web server serves the dynamic content.

For serving the static files, I use Nginx - it’s generally considered to be faster than Apache for this use case. Here’s a typical Nginx config file:

server {
listen 80;
server_name example.com;
client_max_body_size 50M;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
location /static {
root /var/www/mysite;
}
location /media {
root /var/www/mysite;
}
location / {
proxy_pass http://127.0.0.1:8000;
}
}

For the application server, I use Gunicorn. You can install this with pip install gunicorn, then add it to INSTALLED_APPS. Then add the following config file to the root of your project, as gunicorn.conf.py:

bind = "127.0.0.1:8000"
logfile = "/var/log/gunicorn.log"
loglevel = "debug"
workers = 3

You should normally set the number of workers to 2 times the number of cores on your machine, plus one.

In order to keep Gunicorn running, I use Supervisor. As the installation commands will depend on your OS, I won’t give details here - your package manager of choice should have a suitable package available. Here’s a typical Supervisor config file I might use for running Gunicorn for a Django app, named mysite-supervisor.conf:

[program:mysite]
command=/var/www/mysite/venv/bin/gunicorn myapp.wsgi:application --workers=3
directory=/var/www/mysite/
user=nobody
autostart=true
autorestart=true

Once that’s in place, you can easily add the new app:

$ sudo supervisorctl reread
$ sudo supervisorctl update

Then to start it:

$ sudo supervisorctl start mysite

Or stop it with:

$ sudo supervisorctl stop mysite

Or restart it:

$ sudo supervisorctl restart mysite

Celery

So far, both of the web apps I’ve built professionally have been ones where it made sense to use Celery for some tasks. For the uninitiated, Celery lets you pass a task to a queue to be handled, rather than handling it within the context of the same HTTP request. This offers the following advantages:

  • The user doesn’t need to wait for the task to be completed before getting a response, improving performance
  • It’s more robust, since if the task fails, it can be automatically retried
  • The task queue can be moved to another server if desired, making it easier to scale
  • Scheduling tasks

I’ve used it in cases where I needed to send an email or a push notification, since these don’t have to be done within the context of the same HTTP request, but need to be reliable.

I generally use RabbitMQ as my message queue. I’ll leave setting this up as an exercise for the reader since it’s covered pretty well in the Celery documentation, but like with Gunicorn, I use Supervisor to run the Celery worker. Here’s a typical config file, which might be called celery-supervisor.conf:

[program:celeryd]
command=/var/www/mysite/venv/bin/python manage.py celery worker
directory=/var/www/mysite/
user=nobody
autostart=true
autorestart=true

Then start it up:

$ sudo supervisorctl reread
$ sudo supervisorctl update
$ sudo supervisorctl start celeryd

I make no claims about how good this setup is, but it works well for me. I haven’t yet had the occasion to deploy a Django app to anywhere other than Heroku that really benefited from caching, so I haven’t got any tips to share about that, but if I were building a content-driven web app, I would use Memcached since it’s well-supported.

5th October 2014 7:56 pm

Introducing Generator-simple-static-blog

I’m a big fan of static site generators. I ditched WordPress for Octopress over two years ago because it was free to host on GitHub Pages and much faster, had much better syntax highlighting, and I liked being able to write posts in Vim, and I’ve never looked back since.

That said, Octopress is written in Ruby, a language I’ve never been that keen on. Ideally I’d prefer to use Python or JavaScript, but none of the solutions I’ve found have been to my liking. Recently I’ve been using Grunt and Yeoman to some extent, and I’ve wondered about the idea of creating a Yeoman generator to build a static blogging engine. After discovering grunt-markdown-blog, I took the plunge and have built a simple blog generator called generator-simple-static-blog.

I’ve published it to NPM, so feel free to check it out. It includes code highlighting with the Zenburn colour scheme by default (although highlight.js includes many other themes, so just switch to another one if you want), and it should be easy to edit the templates. I’ve also included the ability to deploy automatically to GitHub Pages using Grunt.

I don’t anticipate moving over to this from Octopress for the foreseeable future, but it’s been an interesting project for the weekend.

28th September 2014 8:51 pm

Django Blog Tutorial - the Next Generation - Part 9

Yes, I know the eight instalment was meant to be the last one! Within 24 hours of that post going live, Django 1.7 was released, so naturally I’d like to show you how to upgrade to it.

The biggest change is that Django 1.7 introduces its own migration system, which means South is now surplus to requirements. We therefore need to switch from South to Django’s native migrations. Fortunately, this is fairly straightforward.

First of all, activate your virtualenv:

$ virtualenv venv

Then make sure your migrations are up to date:

$ python manage.py syncdb
$ python manage.py migrate

Then, upgrade your Django version and uninstall South:

$ pip install Django --upgrade
$ pip uninstall South
$ pip freeze > requirements.txt

Next, remove South from INSTALLED_APPS in django_tutorial_blog_ng/settings.py.

You now need to delete all of the numbered migration files in blogengine/migrations/, and the relevant .pyc files, but NOT the directory or the __init__.py file. You can do so with this command on Linux or OS X:

$ rm blogengine/migrations/00*

Next, we recreate our migrations with the following command:

$ python manage.py makemigrations
Migrations for 'blogengine':
0001_initial.py:
- Create model Category
- Create model Post
- Create model Tag
- Add field tags to post

Then we run the migrations:

$ python manage.py migrate
Operations to perform:
Synchronize unmigrated apps: sitemaps, django_jenkins, debug_toolbar
Apply all migrations: sessions, admin, sites, flatpages, contenttypes, auth, blogengine
Synchronizing apps without migrations:
Creating tables...
Installing custom SQL...
Installing indexes...
Running migrations:
Applying contenttypes.0001_initial... FAKED
Applying auth.0001_initial... FAKED
Applying admin.0001_initial... FAKED
Applying sites.0001_initial... FAKED
Applying blogengine.0001_initial... FAKED
Applying flatpages.0001_initial... FAKED
Applying sessions.0001_initial... FAKED

Don’t worry too much if the output doesn’t look exactly the same as this - as long as it works, that’s the main thing.

Let’s run our test suite to ensure it works:

$ python manage.py jenkins
Creating test database for alias 'default'...
....FF.F.FFFFFF..............
======================================================================
FAIL: test_create_post (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 385, in test_create_post
self.assertTrue('added successfully' in response.content)
AssertionError: False is not true
======================================================================
FAIL: test_create_post_without_tag (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 417, in test_create_post_without_tag
self.assertTrue('added successfully' in response.content)
AssertionError: False is not true
======================================================================
FAIL: test_delete_category (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 278, in test_delete_category
self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200
======================================================================
FAIL: test_delete_tag (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 346, in test_delete_tag
self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200
======================================================================
FAIL: test_edit_category (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 255, in test_edit_category
self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200
======================================================================
FAIL: test_edit_post (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 447, in test_edit_post
self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200
======================================================================
FAIL: test_edit_tag (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 323, in test_edit_tag
self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200
======================================================================
FAIL: test_login (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 183, in test_login
self.assertEquals(response.status_code, 200)
AssertionError: 302 != 200
======================================================================
FAIL: test_logout (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 214, in test_logout
self.assertEquals(response.status_code, 200)
AssertionError: 302 != 200
----------------------------------------------------------------------
Ran 29 tests in 7.383s
FAILED (failures=9)
Destroying test database for alias 'default'...

We have an issue here. A load of the tests for the admin interface now fail. If we now try running the dev server, we see this error:

$ python manage.py runserver
Performing system checks...
System check identified no issues (0 silenced).
September 28, 2014 - 20:16:47
Django version 1.7, using settings 'django_tutorial_blog_ng.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Unhandled exception in thread started by <function wrapper at 0x1024a5ed8>
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/utils/autoreload.py", line 222, in wrapper
fn(*args, **kwargs)
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 132, in inner_run
handler = self.get_handler(*args, **options)
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/runserver.py", line 25, in get_handler
handler = super(Command, self).get_handler(*args, **options)
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 48, in get_handler
return get_internal_wsgi_application()
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 66, in get_internal_wsgi_application
sys.exc_info()[2])
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 56, in get_internal_wsgi_application
return import_string(app_path)
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/utils/module_loading.py", line 26, in import_string
module = import_module(module_path)
File "/usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/django_tutorial_blog_ng/wsgi.py", line 14, in <module>
from dj_static import Cling
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/dj_static.py", line 7, in <module>
from django.core.handlers.base import get_path_info
django.core.exceptions.ImproperlyConfigured: WSGI application 'django_tutorial_blog_ng.wsgi.application' could not be loaded; Error importing module: 'cannot import name get_path_info'

Fortunately, the error above is easy to fix by upgrading dj_static:

$ pip install dj_static --upgrade
$ pip freeze > requirements.txt

That resolves the error in serving static files, but not the error with the admin. If you run the dev server, you’ll be able to see that the admin actually works fine. The problem is caused by the test client not following redirects in the admin. We can easily run just the admin tests with the following command:

$ python manage.py test blogengine.tests.AdminTest
Creating test database for alias 'default'...
.FF.F.FFFFFF
======================================================================
FAIL: test_create_post (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 385, in test_create_post
self.assertTrue('added successfully' in response.content)
AssertionError: False is not true
======================================================================
FAIL: test_create_post_without_tag (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 417, in test_create_post_without_tag
self.assertTrue('added successfully' in response.content)
AssertionError: False is not true
======================================================================
FAIL: test_delete_category (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 278, in test_delete_category
self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200
======================================================================
FAIL: test_delete_tag (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 346, in test_delete_tag
self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200
======================================================================
FAIL: test_edit_category (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 255, in test_edit_category
self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200
======================================================================
FAIL: test_edit_post (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 447, in test_edit_post
self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200
======================================================================
FAIL: test_edit_tag (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 323, in test_edit_tag
self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200
======================================================================
FAIL: test_login (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 183, in test_login
self.assertEquals(response.status_code, 200)
AssertionError: 302 != 200
======================================================================
FAIL: test_logout (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 214, in test_logout
self.assertEquals(response.status_code, 200)
AssertionError: 302 != 200
----------------------------------------------------------------------
Ran 12 tests in 3.283s
FAILED (failures=9)
Destroying test database for alias 'default'...

Let’s commit our changes so far first:

$ git add django_tutorial_blog_ng/ requirements.txt blogengine/
$ git commit -m 'Upgraded to Django 1.7'

Now let’s fix our tests. Here’s the amended version of the AdminTest class:

class AdminTest(BaseAcceptanceTest):
fixtures = ['users.json']
def test_login(self):
# Get login page
response = self.client.get('/admin/', follow=True)
# Check response code
self.assertEquals(response.status_code, 200)
# Check 'Log in' in response
self.assertTrue('Log in' in response.content)
# Log the user in
self.client.login(username='bobsmith', password="password")
# Check response code
response = self.client.get('/admin/')
self.assertEquals(response.status_code, 200)
# Check 'Log out' in response
self.assertTrue('Log out' in response.content)
def test_logout(self):
# Log in
self.client.login(username='bobsmith', password="password")
# Check response code
response = self.client.get('/admin/')
self.assertEquals(response.status_code, 200)
# Check 'Log out' in response
self.assertTrue('Log out' in response.content)
# Log out
self.client.logout()
# Check response code
response = self.client.get('/admin/', follow=True)
self.assertEquals(response.status_code, 200)
# Check 'Log in' in response
self.assertTrue('Log in' in response.content)
def test_create_category(self):
# Log in
self.client.login(username='bobsmith', password="password")
# Check response code
response = self.client.get('/admin/blogengine/category/add/')
self.assertEquals(response.status_code, 200)
# Create the new category
response = self.client.post('/admin/blogengine/category/add/', {
'name': 'python',
'description': 'The Python programming language'
},
follow=True
)
self.assertEquals(response.status_code, 200)
# Check added successfully
self.assertTrue('added successfully' in response.content)
# Check new category now in database
all_categories = Category.objects.all()
self.assertEquals(len(all_categories), 1)
def test_edit_category(self):
# Create the category
category = CategoryFactory()
# Log in
self.client.login(username='bobsmith', password="password")
# Edit the category
response = self.client.post('/admin/blogengine/category/' + str(category.pk) + '/', {
'name': 'perl',
'description': 'The Perl programming language'
}, follow=True)
self.assertEquals(response.status_code, 200)
# Check changed successfully
self.assertTrue('changed successfully' in response.content)
# Check category amended
all_categories = Category.objects.all()
self.assertEquals(len(all_categories), 1)
only_category = all_categories[0]
self.assertEquals(only_category.name, 'perl')
self.assertEquals(only_category.description, 'The Perl programming language')
def test_delete_category(self):
# Create the category
category = CategoryFactory()
# Log in
self.client.login(username='bobsmith', password="password")
# Delete the category
response = self.client.post('/admin/blogengine/category/' + str(category.pk) + '/delete/', {
'post': 'yes'
}, follow=True)
self.assertEquals(response.status_code, 200)
# Check deleted successfully
self.assertTrue('deleted successfully' in response.content)
# Check category deleted
all_categories = Category.objects.all()
self.assertEquals(len(all_categories), 0)
def test_create_tag(self):
# Log in
self.client.login(username='bobsmith', password="password")
# Check response code
response = self.client.get('/admin/blogengine/tag/add/')
self.assertEquals(response.status_code, 200)
# Create the new tag
response = self.client.post('/admin/blogengine/tag/add/', {
'name': 'python',
'description': 'The Python programming language'
},
follow=True
)
self.assertEquals(response.status_code, 200)
# Check added successfully
self.assertTrue('added successfully' in response.content)
# Check new tag now in database
all_tags = Tag.objects.all()
self.assertEquals(len(all_tags), 1)
def test_edit_tag(self):
# Create the tag
tag = TagFactory()
# Log in
self.client.login(username='bobsmith', password="password")
# Edit the tag
response = self.client.post('/admin/blogengine/tag/' + str(tag.pk) + '/', {
'name': 'perl',
'description': 'The Perl programming language'
}, follow=True)
self.assertEquals(response.status_code, 200)
# Check changed successfully
self.assertTrue('changed successfully' in response.content)
# Check tag amended
all_tags = Tag.objects.all()
self.assertEquals(len(all_tags), 1)
only_tag = all_tags[0]
self.assertEquals(only_tag.name, 'perl')
self.assertEquals(only_tag.description, 'The Perl programming language')
def test_delete_tag(self):
# Create the tag
tag = TagFactory()
# Log in
self.client.login(username='bobsmith', password="password")
# Delete the tag
response = self.client.post('/admin/blogengine/tag/' + str(tag.pk) + '/delete/', {
'post': 'yes'
}, follow=True)
self.assertEquals(response.status_code, 200)
# Check deleted successfully
self.assertTrue('deleted successfully' in response.content)
# Check tag deleted
all_tags = Tag.objects.all()
self.assertEquals(len(all_tags), 0)
def test_create_post(self):
# Create the category
category = CategoryFactory()
# Create the tag
tag = TagFactory()
# Log in
self.client.login(username='bobsmith', password="password")
# Check response code
response = self.client.get('/admin/blogengine/post/add/')
self.assertEquals(response.status_code, 200)
# Create the new post
response = self.client.post('/admin/blogengine/post/add/', {
'title': 'My first post',
'text': 'This is my first post',
'pub_date_0': '2013-12-28',
'pub_date_1': '22:00:04',
'slug': 'my-first-post',
'site': '1',
'category': str(category.pk),
'tags': str(tag.pk)
},
follow=True
)
self.assertEquals(response.status_code, 200)
# Check added successfully
self.assertTrue('added successfully' in response.content)
# Check new post now in database
all_posts = Post.objects.all()
self.assertEquals(len(all_posts), 1)
def test_create_post_without_tag(self):
# Create the category
category = CategoryFactory()
# Log in
self.client.login(username='bobsmith', password="password")
# Check response code
response = self.client.get('/admin/blogengine/post/add/')
self.assertEquals(response.status_code, 200)
# Create the new post
response = self.client.post('/admin/blogengine/post/add/', {
'title': 'My first post',
'text': 'This is my first post',
'pub_date_0': '2013-12-28',
'pub_date_1': '22:00:04',
'slug': 'my-first-post',
'site': '1',
'category': str(category.pk)
},
follow=True
)
self.assertEquals(response.status_code, 200)
# Check added successfully
self.assertTrue('added successfully' in response.content)
# Check new post now in database
all_posts = Post.objects.all()
self.assertEquals(len(all_posts), 1)
def test_edit_post(self):
# Create the post
post = PostFactory()
# Create the category
category = CategoryFactory()
# Create the tag
tag = TagFactory()
post.tags.add(tag)
# Log in
self.client.login(username='bobsmith', password="password")
# Edit the post
response = self.client.post('/admin/blogengine/post/' + str(post.pk) + '/', {
'title': 'My second post',
'text': 'This is my second blog post',
'pub_date_0': '2013-12-28',
'pub_date_1': '22:00:04',
'slug': 'my-second-post',
'site': '1',
'category': str(category.pk),
'tags': str(tag.pk)
},
follow=True
)
self.assertEquals(response.status_code, 200)
# Check changed successfully
self.assertTrue('changed successfully' in response.content)
# Check post amended
all_posts = Post.objects.all()
self.assertEquals(len(all_posts), 1)
only_post = all_posts[0]
self.assertEquals(only_post.title, 'My second post')
self.assertEquals(only_post.text, 'This is my second blog post')
def test_delete_post(self):
# Create the post
post = PostFactory()
# Create the tag
tag = TagFactory()
post.tags.add(tag)
# Check new post saved
all_posts = Post.objects.all()
self.assertEquals(len(all_posts), 1)
# Log in
self.client.login(username='bobsmith', password="password")
# Delete the post
response = self.client.post('/admin/blogengine/post/' + str(post.pk) + '/delete/', {
'post': 'yes'
}, follow=True)
self.assertEquals(response.status_code, 200)
# Check deleted successfully
self.assertTrue('deleted successfully' in response.content)
# Check post deleted
all_posts = Post.objects.all()
self.assertEquals(len(all_posts), 0)

There are two main issues here. The first is that when we try to edit or delete an existing item, or refer to it when creating something else, we can no longer rely on the number representing the primary key being set to 1. So we need to specifically obtain this, rather than hard-coding it to 1. Therefore, whenever we pass through a number to represent an item (with the exception of the site, but including tags, categories and posts), we need to instead fetch its primary key and return it. So, above where we try to delete a post, we replace 1 with str(post.pk). This will solve a lot of the problems. As there’s a lot of them, I won’t go through each one, but you can see the entire class above for reference, and if you’ve followed along so far, you shouldn’t have any problems.

The other issue we need to fix is the login and logout tests. We simply add follow=True to these to ensure that the test client follows the redirects.

Let’s run our tests to make sure they pass:

$ python manage.py jenkins
Creating test database for alias 'default'...
.............................
----------------------------------------------------------------------
Ran 29 tests in 8.210s
OK
Destroying test database for alias 'default'...

With that done, you can commit your changes:

$ git add blogengine/tests.py
$ git commit -m 'Fixed broken tests'

Don’t forget to deploy your changes:

$ fab deploy

Our blog has now been happily migrated over to Django 1.7!

28th September 2014 7:53 pm

Changing Date Format from DD/MM/YYYY to YYYY-MM-DD in Vim

Recently I had the occasion to reformat a load of dates in Vim from DD/MM/YYYY to YYYY-MM-DD. In Vim, this is quite simple:

:%s/\(\d\{2}\)\/\(\d\{2}\)\/\(\d\{4}\)/\3-\2-\1/g

This should be easy to adapt to reformatting other date formats.

Recent Posts

Powering Up Git Bisect With the Run Command

Writing Golden Master Tests for Laravel Applications

How Much Difference Does Adding An Index to a Database Table Make?

Searching Content With Fuse.js

Higher-order Components in React

About me

I'm a web and mobile app developer based in Norfolk. My skillset includes Python, PHP and Javascript, and I have extensive experience working with CodeIgniter, Laravel, Zend Framework, Django, Phonegap and React.js.