5th June 2016 4:32 pm

Using Jenkins Pipelines

I use Jenkins as my main continuous integration solution at work, largely for two reasons:

• It generally works out cheaper to host it ourselves than to use one of the paid CI solutions for closed-source projects
• The size of the plugin ecosystem

However, we also use Travis CI for testing one or two open-source projects, and one distinct advantage Travis has is the way you can configure it using a single text file.

With the Pipeline plugin, it’s possible to define the steps required to run your tests in a Jenkinsfile and then set up a Pipeline job which reads that file from the version control system and runs it accordingly. Here’s a sample Jenkinsfile for a Laravel project:

node {
   // Mark the code checkout 'stage'....
   stage 'Checkout'
   // Get some code from a Bitbucket repository
   git credentialsId: '5239c33e-10ab-4c1b-a4a0-91b96a07955e', url: 'git@bitbucket.org:matthewbdaly/my-app.git'
   // Install dependencies
   stage 'Install dependencies'
   // Run Composer
   sh 'composer install'
   // Test stage
   stage 'Test'
   // Run the tests
   sh "vendor/bin/phpunit"
}

Note the steps it’s broken down into:

• stage defines the start of a new stage in the build
• git defines a point where we check out the code from the repository
• sh defines a point where we run a shell command

Using these three commands it’s straightforward to define a fairly simple build process for your application in a way that’s more easily repeatable when creating new projects - for instance, you can copy this over to a new project and change the source repository URL and you’re pretty much ready to go.

Unfortunately, support for the Pipeline plugin is missing from a lot of Jenkins plugins - for instance, I can’t publish the XML coverage reports. This is something of a deal-breaker for most of my projects as I use these kind of report plugins a lot - it’s one of the reasons I chose Jenkins over Travis. Still, this is definitely a big step forward, and if you don’t need this kind of reporting then there’s no reason not to consider using the Pipeline plugin for your Jenkins jobs. Hopefully in future more plugins will be amended to work with Pipeline so that it’s more widely usable.

22nd May 2016 11:29 pm

Adding Google AMP Support to My Site

You may have heard of Google’s AMP Project, which allows you to create mobile-optimized pages using a subset of HTML. After seeing the sheer speed at which you can load an AMP page (practically instantaneous in many cases), I was eager to see if I could apply it to my own site.

I still wanted to retain the existing functionality for my site, such as comments and search, so I elected not to rewrite the whole thing to make it AMP-compliant. Instead, I opted to create AMP versions of every blog post, and link to them from the original. This preserves the advantages of AMP since search engines will be able to discover it from the header of the original, while allowing those wanting a richer experience to view the original, where the comments are hosted. You can now view the AMP version of any post by appending amp/ to its URL.

The biggest problem was the images in the post body, as the <img> tag needs to be replaced by the <amp-img> tag, which also requires an explicit height and width. I wound up amending the renderer for AMP pages to render an image tag as an empty string, since I have only ever used one image in the post body and I think I can live without them.

It’s also a bit of a pain styling it as it will be awkward to use Bootstrap. I’ve therefore opted to skip Bootstrap for now and write my own fairly basic theme for the AMP pages instead.

It’ll be interesting to see what effect having the AMP versions of the pages available will have on my site in terms of search results. It obviously takes some time before the page gets crawled, and until then the AMP version won’t be served from the CDN used by AMP, so I really can’t guess what effect it will have right now.

14th May 2016 9:00 pm

Broadcasting Events With Laravel and Socket.io

PHP frameworks like Laravel aren’t really set up to handle real-time events properly, so if you want to build a real-time app, you’re generally better off with another platform, such as Node.js. However, if that only forms a small part of your application, you may still prefer to work with PHP. Fortunately it’s fairly straightforward to hand off the real-time aspects of your application to a dedicated microservice written using Node.js and still use Laravel to handle the rest of the functionality.

Here I’ll show you how I built a Laravel app that uses a separate Node.js script to handle sending real-time updates to the user.

Events in Laravel

In this case, I was building a REST API to serve as the back end for a Phonegap app that allowed users to message each other. The API includes an endpoint that allows users to create and fetch messages. Now, in theory, we could just repeatedly poll the endpoint for new messages, but that would be inefficient. What we needed was a way to notify users of new messages in real time, which seemed like the perfect opportunity to use Socket.io.

Laravel comes with a simple, but robust system that allows you to broadcast events to a Redis server. Another service can then listen for these events and carry out jobs on them, and there is no reason why this service has to be written in PHP. This makes it easy to decouple your application into smaller parts. In essence the functionality we wanted was as follows:

• Receive message
• Push message to Redis
• Have a separate service pick up message on Redis
• Push message to clients

First off, we need to define an event in our Laravel app. You can create a boilerplate with the following Artisan command:

$ php artisan make:event NewMessage

This will create the file app/Events/NewMessage.php. You can then customise this as follows:

<?php
namespace App\Events;
use App\Events\Event;
use App\Message;
use Illuminate\Queue\SerializesModels;
use Illuminate\Contracts\Broadcasting\ShouldBroadcast;
class NewMessage extends Event implements ShouldBroadcast
{
    use SerializesModels;
    public $message;
    /**
     * Create a new event instance.
     *
     * @return void
     */
    public function __construct(Message $message)
    {
        // Get message
        $this->message = $message;
    }
    /**
     * Get the channels the event should be broadcast on.
     *
     * @return array
     */
    public function broadcastOn()
    {
        return ['room_'.$this->message->room_id];
    }
}

This particular event is a class that accepts a single argument, which is an instance of the Message model. This model includes an attribute of room_id that is used to determine which room the message is posted to - note that this is returned in the broadcastOn() method.

When we want to trigger our new event, we can do so as follows:

use App\Events\NewMessage;
Event::fire(new NewMessage($message));

Here, $message is the saved Eloquent object containing the message. Note the use of SerializesModels - this means that the Eloquent model is serialized into JSON when broadcasting the event.

We also need to make sure Redis is set as our broadcast driver. Ensure the Composer package predis/predis is installed, and set BROADCAST_DRIVER=redis in your .env file. Also, please note that I found that setting QUEUE_DRIVER=redis in .env as well broke the broadcasting system, so it looks like you can’t use Redis as both a queue and a broadcasting system unless you set up multiple connections.

Next, we need another server-side script to handle processing the received events and pushing the messages out. In my case, this was complicated by the fact that we were using HTTPS, courtesy of Let’s Encrypt. I installed the required dependencies for the Node.js script as follows:

$ npm install socket.io socket.io-client ioredis --save-dev

Here’s an example Node.js script for processing the events:

var fs = require('fs');
var pkey = fs.readFileSync('/etc/letsencrypt/live/example.com/privkey.pem');
var pcert = fs.readFileSync('/etc/letsencrypt/live/example.com/fullchain.pem')
var options = {
  key: pkey,
  cert: pcert
};
var app = require('https').createServer(options);
var io = require('socket.io')(app);
var Redis = require('ioredis');
var redis = new Redis();
app.listen(9000, function() {
    console.log('Server is running!');
});
function handler(req, res) {
    res.setHeader('Access-Control-Allow-Origin', '*');
    res.writeHead(200);
    res.end('');
}
io.on('connection', function(socket) {
    //
});
redis.psubscribe('*', function(err, count) {
    //
});
redis.on('pmessage', function(subscribed, channel, message) {
    message = JSON.parse(message);
    console.log('Channel is ' + channel + ' and message is ' + message);
    io.emit(channel, message.data);
});

Note we use the https module instead of the http one, and we pass the key and certificate as options to the server. This server runs on port 9000, but feel free to move it to any arbitrary port you wish. In production, you’d normally use something like Supervisor or systemd to run a script like this as a service.

Next, we need a client-side script to connect to the Socket.io instance and handle any incoming messages. Here’s a very basic example that just dumps them to the browser console:

var url = window.location.protocol + '//' + window.location.hostname;
var socket = io(url, {
  'secure': true,
  'reconnect': true,
  'reconnection delay': 500,
  'max reconnection attempts': 10
});
var chosenEvent = 'room_' + room.id;
socket.on(chosenEvent, function (data) {
  console.log(data);
});

Finally, we need to configure our web server. I’m using Nginx with PHP-FPM and PHP 7, and this is how I configured it:

upstream websocket {
    server 127.0.0.1:9000;
}
server {
    listen         80;
    server_name example.com;
    return 301 https://$host$request_uri;
}
server {
    listen 443 ssl;
    server_name example.com;
    ssl on;
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Real-IP $remote_addr;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
    client_max_body_size 50M;
    server_tokens off;
    add_header X-Frame-Options SAMEORIGIN;
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    root /var/www/public;
    index index.php index.html index.htm;
    location / {
        try_files $uri $uri/ /index.php?$query_string;
        gzip on;
        gzip_proxied any;
        gzip_types text/plain text/css application/javascript application/x-javascript text/xml application/xml application/xml-rss text/javascript text/js application/json;
        expires 1y;
        charset utf-8;
    }
    location ~ \.php$ {
        try_files $uri /index.php =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
    }
    location ~ /.well-known {
        root /var/www/public;
        allow all;
    }
    location /socket.io {
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_pass https://websocket;
    }
}

Any requests to /socket.io are proxied to port 9000, where our chat handling script is listening. Note that we allow the HTTPS connection to be upgraded to a WebSocket one.

Once that’s done, you just need to restart your PHP application and Nginx, and start running your chat script, and everything should be working fine. If it isn’t, the command redis-cli monitor is invaluable in verifying that the event is being published correctly.

Summary

Getting this all working together did take quite a bit of trial and error, but that was mostly a matter of configuration. Actually implementing this is pretty straightforward, and it’s an easy way to add some basic real-time functionality to an existing Laravel application.

4th April 2016 8:55 pm

Writing Faster Laravel Tests

Nowadays, Laravel tends to be my go-to PHP framework, to the point that we use it as our default framework at work. A big part of this is that Laravel is relatively easy to test, making practicing TDD a lot easier.

Out of the box running Laravel tests can be quite slow, which is a big issue - if your test suite takes several minutes to run, that’s a huge disruption. Also, Laravel doesn’t create a dedicated test database - instead it runs the tests against the same database you’re using normally, which is almost always not what you want. I’ll show you how to set up a dedicated test database, and how to use an in-memory SQLite database for faster tests. This results in cleaner and easier-to-maintain tests, since you can be sure the test database is restored to a clean state at the end of every test.

Setup

Our first step is to make sure that when a new test begins, the following should happen:

• We should create a new transaction
• We should empty and migrate our database

Then, at the end of each test:

• We should roll back our transaction to restore the database to its prior state

To do so, we can create custom setUp() and tearDown() methods for our base TestCase class. Save this in tests/TestCase.php:

<?php
class TestCase extends Illuminate\Foundation\Testing\TestCase
{
    /**
     * The base URL to use while testing the application.
     *
     * @var string
     */
    protected $baseUrl = 'http://localhost';
    /**
     * Creates the application.
     *
     * @return \Illuminate\Foundation\Application
     */
    public function createApplication()
    {
        $app = require __DIR__.'/../bootstrap/app.php';
        $app->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();
        return $app;
    }
    public function setUp()
    {
        parent::setUp();
        DB::beginTransaction();
        Artisan::call('migrate:refresh');
    }
    public function tearDown()
    {
        DB::rollBack();
        parent::tearDown();
    }
}

That takes care of building up and tearing down our database for each test.

EDIT: Turns out there’s actually a much easier way of doing this already included in Laravel. Just import and add either use DatabaseMigrations; or use DatabaseTransactions; to the TestCase class. The first will roll back the database and migrate it again after each test, while the second wraps each test in a transaction.

Using an in-memory SQLite database for testing purposes

It’s not always practical to do this, especially if you rely on database features in PostgreSQL that aren’t available in SQLite, but if it is, it’s probably worth using an in-memory SQLite database for your tests. If you want to do so, here’s some example settings you might want to use in phpunit.xml:

        <env name="APP_ENV" value="testing"/>
        <env name="CACHE_DRIVER" value="array"/>
        <env name="DB_CONNECTION" value="sqlite"/>
        <env name="DB_DATABASE" value=":memory:"/>

This can result in a very significant speed boost.

I would still recommend that you test against your production database, but this can be easily handed off to a continuous integration server such as Jenkins, since that way it won’t disrupt your workflow.

During TDD, you’ll typically run your tests several times for any change you make, so if they’re too slow it can have a disastrous effect on your productivity. But with a few simple changes like this, you can ensure your tests run as quickly as possible. This approach should also be viable for Lumen apps.

26th March 2016 9:30 pm

Building a Location Aware Web App With Geodjango

PostgreSQL has excellent support for geographical data thanks to the PostGIS extension, and Django allows you to take full advantage of it thanks to GeoDjango. In this tutorial, I’ll show you how to use GeoDjango to build a web app that allows users to search for gigs and events near them.

Requirements

I’ve made the jump to Python 3, and if you haven’t done so yet, I highly recommend it - it’s not hard, and there’s very few modules left that haven’t been ported across. As such, this tutorial assumes you’re using Python 3. You’ll also need to have Git, PostgreSQL and PostGIS installed - I’ll leave the details of doing so up to you as it varies by platform, but you can generally do so easily with a package manager on most Linux distros. On Mac OS X I recommend using Homebrew. If you’re on Windows I think your best bet is probably to use a Vagrant VM.

We’ll be using Django 1.9 - if by the time you read this a newer version of Django is out, it’s quite possible that some things may have changed and you’ll need to work around any problems caused. Generally search engines are the best place to look for this, and I’ll endeavour to keep the resulting Github repository as up to date as I can, so try those if you get stuck.

Getting started

First of all, let’s create our database. Make sure you’re running as a user that has the required privileges to create users and databases for PostgreSQL and run the following command:

$ createdb gigfinder

This creates the database. Next, we create the user:

$ createuser -s giguser -P

You’ll be prompted to enter a password for the new user. Next, we want to use the psql command-line client to interact with our new database:

$ psql gigfinder

This connects to the database. Run these commands to set up access to the database and install the PostGIS extension:

# GRANT ALL PRIVILEGES ON DATABASE gigfinder TO giguser;
# CREATE EXTENSION postgis;
# \q

With our database set up, it’s time to start work on our project. Let’s create our virtualenv in a new folder:

$ pyvenv venv

Then activate it:

$ source venv/bin/activate

Then we install Django, along with a few other production dependencies:

$ pip install django-toolbelt

And record our dependencies:

$ pip freeze > requirements.txt

Next, we create our application skeleton:

$ django-admin.py startproject gigfinder .

We’ll also create a .gitignore file:

venv/
.DS_Store
*.swp
node_modules/
*.pyc

Let’s commit our changes:

$ git init
$ git add .gitignore requirements.txt manage.py gigfinder
$ git commit -m 'Initial commit'

Next, let’s create our first app, which we will call gigs:

$ python manage.py startapp gigs

We need to add our new app to the INSTALLED_APPS setting. While we’re there we’ll also add GIS support and set up the database connection. First, add the required apps to INSTALLED_APPS:

INSTALLED_APPS = [
    ...
    'django.contrib.gis',
    'gigs',
]

Next, configure the database:

DATABASES = {
    'default': {
         'ENGINE': 'django.contrib.gis.db.backends.postgis',
         'NAME': 'gigfinder',
         'USER': 'giguser',
         'PASSWORD': 'password',
    },
}

Let’s run the migrations:

$ python manage.py migrate
Operations to perform:
  Apply all migrations: sessions, contenttypes, admin, auth
Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying sessions.0001_initial... OK

And create our superuser account:

$ python manage.py createsuperuser

Now, we’ll commit our changes:

$ git add gigfinder/ gigs/
$ git commit -m 'Created gigs app'
[master e72a846] Created gigs app
 8 files changed, 24 insertions(+), 3 deletions(-)
 create mode 100644 gigs/__init__.py
 create mode 100644 gigs/admin.py
 create mode 100644 gigs/apps.py
 create mode 100644 gigs/migrations/__init__.py
 create mode 100644 gigs/models.py
 create mode 100644 gigs/tests.py
 create mode 100644 gigs/views.py

Our first model

At this point, it’s worth thinking about the models we plan for our app to have. First we’ll have a Venue model that contains details of an individual venue, which will include a name and a geographical location. We’ll also have an Event model that will represent an individual gig or event at a venue, and will include a name, date/time and a venue as a foreign key.

Before we start writing our first model, we need to write a test for it, but we also need to be able to create objects easily in our tests. We also want to be able to easily examine our objects, so we’ll install iPDB and Factory Boy:

$ pip install ipdb factory-boy
$ pip freeze > requirements.txt

Next, we write a test for the Venue model:

from django.test import TestCase
from gigs.models import Venue
from factory.fuzzy import BaseFuzzyAttribute
from django.contrib.gis.geos import Point
import factory.django, random
class FuzzyPoint(BaseFuzzyAttribute):
    def fuzz(self):
        return Point(random.uniform(-180.0, 180.0),
                     random.uniform(-90.0, 90.0))
# Factories for tests
class VenueFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Venue
        django_get_or_create = (
            'name',
            'location'
        )
    name = 'Wembley Arena'
    location = FuzzyPoint()
class VenueTest(TestCase):
    def test_create_venue(self):
        # Create the venue
        venue = VenueFactory()
        # Check we can find it
        all_venues = Venue.objects.all()
        self.assertEqual(len(all_venues), 1)
        only_venue = all_venues[0]
        self.assertEqual(only_venue, venue)
        # Check attributes
        self.assertEqual(only_venue.name, 'Wembley Arena')

Note that we randomly generate our location - this is done as suggested in this Stack Overflow post.

Now, running our tests brings up an expected error:

$ python manage.py test gigs
Creating test database for alias 'default'...
E
======================================================================
ERROR: gigs.tests (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: gigs.tests
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 2, in <module>
    from gigs.models import Venue
ImportError: cannot import name 'Venue'
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (errors=1)
Destroying test database for alias 'default'...

Let’s create our Venue model in gigs/models.py:

from django.contrib.gis.db import models
class Venue(models.Model):
    """
    Model for a venue
    """
    pass

For now, we’re just creating a simple dummy model. Note that we import models from django.contrib.gis.db instead of the usual place - this gives us access to the additional geographical fields.

If we run our tests again we get an error:

$ python manage.py test gigs
Creating test database for alias 'default'...
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: relation "gigs_venue" does not exist
LINE 1: SELECT "gigs_venue"."id" FROM "gigs_venue" ORDER BY "gigs_ve...
                                      ^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/commands/test.py", line 30, in run_from_argv
    super(Command, self).run_from_argv(argv)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/commands/test.py", line 74, in execute
    super(Command, self).execute(*args, **options)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/commands/test.py", line 90, in handle
    failures = test_runner.run_tests(test_labels)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/test/runner.py", line 532, in run_tests
    old_config = self.setup_databases()
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/test/runner.py", line 482, in setup_databases
    self.parallel, **kwargs
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/test/runner.py", line 726, in setup_databases
    serialize=connection.settings_dict.get("TEST", {}).get("SERIALIZE", True),
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 78, in create_test_db
    self.connection._test_serialized_contents = self.serialize_db_to_string()
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 122, in serialize_db_to_string
    serializers.serialize("json", get_objects(), indent=None, stream=out)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/serializers/__init__.py", line 129, in serialize
    s.serialize(queryset, **options)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/serializers/base.py", line 79, in serialize
    for count, obj in enumerate(queryset, start=1):
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 118, in get_objects
    for obj in queryset.iterator():
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/models/query.py", line 52, in __iter__
    results = compiler.execute_sql()
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 848, in execute_sql
    cursor.execute(sql, params)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/utils.py", line 95, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "gigs_venue" does not exist
LINE 1: SELECT "gigs_venue"."id" FROM "gigs_venue" ORDER BY "gigs_ve...

Let’s update our model:

from django.contrib.gis.db import models
class Venue(models.Model):
    """
    Model for a venue
    """
    name = models.CharField(max_length=200)
    location = models.PointField()

Then create our migration:

$ python manage.py makemigrations
Migrations for 'gigs':
  0001_initial.py:
    - Create model Venue

And run it:

$ python manage.py migrate
Operations to perform:
  Apply all migrations: gigs, sessions, contenttypes, auth, admin
Running migrations:
  Rendering model states... DONE
  Applying gigs.0001_initial... OK

Then if we run our tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
.
----------------------------------------------------------------------
Ran 1 test in 0.362s
OK
Destroying test database for alias 'default'...

They should pass. Note that Django may complain about needing to delete the test database before running the tests, but this should not cause any problems. Let’s commit our changes:

$ git add requirements.txt gigs/
$ git commit -m 'Venue model in place'

With our venue done, let’s turn to our Event model. Amend gigs/tests.py as follows:

from django.test import TestCase
from gigs.models import Venue, Event
from factory.fuzzy import BaseFuzzyAttribute
from django.contrib.gis.geos import Point
import factory.django, random
from django.utils import timezone
class FuzzyPoint(BaseFuzzyAttribute):
    def fuzz(self):
        return Point(random.uniform(-180.0, 180.0),
                     random.uniform(-90.0, 90.0))
# Factories for tests
class VenueFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Venue
        django_get_or_create = (
            'name',
            'location'
        )
    name = 'Wembley Arena'
    location = FuzzyPoint()
class EventFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Event
        django_get_or_create = (
            'name',
            'venue',
            'datetime'
        )
    name = 'Queens of the Stone Age'
    datetime = timezone.now()
class VenueTest(TestCase):
    def test_create_venue(self):
        # Create the venue
        venue = VenueFactory()
        # Check we can find it
        all_venues = Venue.objects.all()
        self.assertEqual(len(all_venues), 1)
        only_venue = all_venues[0]
        self.assertEqual(only_venue, venue)
        # Check attributes
        self.assertEqual(only_venue.name, 'Wembley Arena')
class EventTest(TestCase):
    def test_create_event(self):
        # Create the venue
        venue = VenueFactory()
        # Create the event
        event = EventFactory(venue=venue)
        # Check we can find it
        all_events = Event.objects.all()
        self.assertEqual(len(all_events), 1)
        only_event = all_events[0]
        self.assertEqual(only_event, event)
        # Check attributes
        self.assertEqual(only_event.name, 'Queens of the Stone Age')
        self.assertEqual(only_event.venue.name, 'Wembley Arena')

Then we run our tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
E
======================================================================
ERROR: gigs.tests (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: gigs.tests
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 2, in <module>
    from gigs.models import Venue, Event
ImportError: cannot import name 'Event'
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (errors=1)
Destroying test database for alias 'default'...

As expected, this fails, so create an empty Event model in gigs/models.py:

class Event(models.Model):
    """
    Model for an event
    """
    pass

Running the tests now will raise an error due to the table not existing:

$ python manage.py test gigs
Creating test database for alias 'default'...
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: relation "gigs_event" does not exist
LINE 1: SELECT "gigs_event"."id" FROM "gigs_event" ORDER BY "gigs_ev...
                                      ^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/commands/test.py", line 30, in run_from_argv
    super(Command, self).run_from_argv(argv)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/commands/test.py", line 74, in execute
    super(Command, self).execute(*args, **options)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/commands/test.py", line 90, in handle
    failures = test_runner.run_tests(test_labels)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/test/runner.py", line 532, in run_tests
    old_config = self.setup_databases()
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/test/runner.py", line 482, in setup_databases
    self.parallel, **kwargs
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/test/runner.py", line 726, in setup_databases
    serialize=connection.settings_dict.get("TEST", {}).get("SERIALIZE", True),
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 78, in create_test_db
    self.connection._test_serialized_contents = self.serialize_db_to_string()
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 122, in serialize_db_to_string
    serializers.serialize("json", get_objects(), indent=None, stream=out)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/serializers/__init__.py", line 129, in serialize
    s.serialize(queryset, **options)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/serializers/base.py", line 79, in serialize
    for count, obj in enumerate(queryset, start=1):
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 118, in get_objects
    for obj in queryset.iterator():
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/models/query.py", line 52, in __iter__
    results = compiler.execute_sql()
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 848, in execute_sql
    cursor.execute(sql, params)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/utils.py", line 95, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "gigs_event" does not exist
LINE 1: SELECT "gigs_event"."id" FROM "gigs_event" ORDER BY "gigs_ev...

So let’s populate our model:

class Event(models.Model):
    """
    Model for an event
    """
    name = models.CharField(max_length=200)
    datetime = models.DateTimeField()
    venue = models.ForeignKey(Venue)

And create our migration:

$ python manage.py makemigrations
Migrations for 'gigs':
  0002_event.py:
    - Create model Event

And run it:

$ python manage.py migrate
Operations to perform:
  Apply all migrations: auth, admin, sessions, contenttypes, gigs
Running migrations:
  Rendering model states... DONE
  Applying gigs.0002_event... OK

And run our tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
..
----------------------------------------------------------------------
Ran 2 tests in 0.033s
OK
Destroying test database for alias 'default'...

Again, you may be prompted to delete the test database, but this should not be an issue.

With this done, let’s commit our changes:

$ git add gigs
$ git commit -m 'Added Event model'
[master 47ba686] Added Event model
 3 files changed, 67 insertions(+), 1 deletion(-)
 create mode 100644 gigs/migrations/0002_event.py

Setting up the admin

For an application like this, you’d expect the curators of the site to maintain the gigs and venues stored in the database, and that’s an obvious use case for the Django admin. So let’s set our models up to be available in the admin. Open up gigs/admin.py and amend it as follows:

from django.contrib import admin
from gigs.models import Venue, Event
admin.site.register(Venue)
admin.site.register(Event)

Now, if you start up the dev server as usual with python manage.py runserver and visit http://127.0.0.1:8000/admin/, you can see that our Event and Venue models are now available. However, the string representations of them are pretty useless. Let’s fix that. First, we amend our tests:

from django.test import TestCase
from gigs.models import Venue, Event
from factory.fuzzy import BaseFuzzyAttribute
from django.contrib.gis.geos import Point
import factory.django, random
from django.utils import timezone
class FuzzyPoint(BaseFuzzyAttribute):
    def fuzz(self):
        return Point(random.uniform(-180.0, 180.0),
                     random.uniform(-90.0, 90.0))
# Factories for tests
class VenueFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Venue
        django_get_or_create = (
            'name',
            'location'
        )
    name = 'Wembley Arena'
    location = FuzzyPoint()
class EventFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Event
        django_get_or_create = (
            'name',
            'venue',
            'datetime'
        )
    name = 'Queens of the Stone Age'
    datetime = timezone.now()
class VenueTest(TestCase):
    def test_create_venue(self):
        # Create the venue
        venue = VenueFactory()
        # Check we can find it
        all_venues = Venue.objects.all()
        self.assertEqual(len(all_venues), 1)
        only_venue = all_venues[0]
        self.assertEqual(only_venue, venue)
        # Check attributes
        self.assertEqual(only_venue.name, 'Wembley Arena')
        # Check string representation
        self.assertEqual(only_venue.__str__(), 'Wembley Arena')
class EventTest(TestCase):
    def test_create_event(self):
        # Create the venue
        venue = VenueFactory()
        # Create the event
        event = EventFactory(venue=venue)
        # Check we can find it
        all_events = Event.objects.all()
        self.assertEqual(len(all_events), 1)
        only_event = all_events[0]
        self.assertEqual(only_event, event)
        # Check attributes
        self.assertEqual(only_event.name, 'Queens of the Stone Age')
        self.assertEqual(only_event.venue.name, 'Wembley Arena')
        # Check string representation
        self.assertEqual(only_event.__str__(), 'Queens of the Stone Age - Wembley Arena')

Next, we run our tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
FF
======================================================================
FAIL: test_create_event (gigs.tests.EventTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 74, in test_create_event
    self.assertEqual(only_event.__str__(), 'Queens of the Stone Age - Wembley Arena')
AssertionError: 'Event object' != 'Queens of the Stone Age - Wembley Arena'
- Event object
+ Queens of the Stone Age - Wembley Arena
======================================================================
FAIL: test_create_venue (gigs.tests.VenueTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 52, in test_create_venue
    self.assertEqual(only_venue.__str__(), 'Wembley Arena')
AssertionError: 'Venue object' != 'Wembley Arena'
- Venue object
+ Wembley Arena
----------------------------------------------------------------------
Ran 2 tests in 0.059s
FAILED (failures=2)
Destroying test database for alias 'default'...

They fail as expected. So let’s update gigs/models.py:

from django.contrib.gis.db import models
class Venue(models.Model):
    """
    Model for a venue
    """
    name = models.CharField(max_length=200)
    location = models.PointField()
    def __str__(self):
        return self.name
class Event(models.Model):
    """
    Model for an event
    """
    name = models.CharField(max_length=200)
    datetime = models.DateTimeField()
    venue = models.ForeignKey(Venue)
    def __str__(self):
        return "%s - %s" % (self.name, self.venue.name)

For the venue, we just use the name. For the event, we use the event name and the venue name.

Now, we run our tests again:

$ python manage.py test gigs
Creating test database for alias 'default'...
..
----------------------------------------------------------------------
Ran 2 tests in 0.048s
OK
Destroying test database for alias 'default'...

Time to commit our changes:

$ git add gigs
$ git commit -m 'Added models to admin'
[master 65d051f] Added models to admin
 3 files changed, 15 insertions(+), 1 deletion(-)

Our models are now in place, so you may want to log into the admin and create a few venues and events so you can see it in action. Note that the location field for the Venue model creates a map widget that allows you to select a geographical location. It is a bit basic, however, so let’s make it better. Let’s install django-floppyforms:

$ pip install django-floppyforms

And add it to our requirements:

$ pip freeze -r requirements.txt

Then add it to INSTALLED_APPS in gigfinder/setttings.py:

INSTALLED_APPS = [
    ...
    'django.contrib.gis',
    'gigs',
    'floppyforms',
]

Now we create a custom point widget for our admin, a custom form for the venues, and a custom venue admin:

from django.contrib import admin
from gigs.models import Venue, Event
from django.forms import ModelForm
from floppyforms.gis import PointWidget, BaseGMapWidget
class CustomPointWidget(PointWidget, BaseGMapWidget):
    class Media:
        js = ('/static/floppyforms/js/MapWidget.js',)
class VenueAdminForm(ModelForm):
    class Meta:
        model = Venue
        fields = ['name', 'location']
        widgets = {
            'location': CustomPointWidget()
        }
class VenueAdmin(admin.ModelAdmin):
    form = VenueAdminForm
admin.site.register(Venue, VenueAdmin)
admin.site.register(Event)

Note in particular that we define the media for our widget so we can include some required Javascript. If you run the dev server again, you should see that the map widget in the admin is now provided by Google Maps, making it much easier to identify the correct location of the venue.

Time to commit our changes:

$ git add gigfinder/ gigs/ requirements.txt
$ git commit -m 'Customised location widget'

With our admin ready, it’s time to move on to the user-facing part of the web app.

Creating our views

We will keep the front end for this app as simple as possible for the purposes of this tutorial, but of course you should feel free to expand upon this as you see fit. What we’ll do is create a form that uses HTML5 geolocation to get the user’s current geographical coordinates. It will then return events in the next week, ordered by how close the venue is. Please note that there are plans afoot in some browsers to prevent HTML5 geolocation from working unless content is server over HTTPS, so that may complicate things.

How do we query the database to get this data? It’s not too difficult, as shown in this example:

$ python manage.py shell
Python 3.5.1 (default, Mar 25 2016, 00:17:15)
Type "copyright", "credits" or "license" for more information.
IPython 4.1.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
In [1]: from gigs.models import *
In [2]: from django.contrib.gis.geos import Point
In [3]: from django.contrib.gis.db.models.functions import Distance
In [4]: location = Point(52.3749159, 1.1067473, srid=4326)
In [5]: Venue.objects.all().annotate(distance=Distance('location', location)).order_by('distance')
Out[5]: [<Venue: Diss Corn Hall>, <Venue: Waterfront Norwich>, <Venue: UEA Norwich>, <Venue: Wembley Arena>]

I’ve set up a number of venues using the admin, one round the corner, two in Norwich, and one in London. I then imported the models, the Point class, and the Distance function, and created a Point object. Note that the Point is passed three fields - the first and second are the latitude and longitude, respectively, while the srid field takes a value of 4326. This field represents the Spatial Reference System Identifier used for this query - we’ve gone for WGS 84, which is a common choice and is referred to with the SRID 4326.

Now, we want the user to be able to submit the form and get the 5 nearest events in the next week. We can get the date for this time next week as follows:

In [6]: next_week = timezone.now() + timezone.timedelta(weeks=1)

Then we can get the events we want, sorted by distance, like this:

In [7]: Event.objects.filter(datetime__gte=timezone.now()).filter(datetime__lte=next_week).annotate(distance=Distance('venue__location', location)).order_by('distance')[0:5]
Out[7]: [<Event: Primal Scream - UEA Norwich>, <Event: Queens of the Stone Age - Wembley Arena>]

With that in mind, let’s write the test for our view. The view should contain a single form that accepts a user’s geographical coordinates - for convenience we’ll autocomplete this with HTML5 geolocation. On submit, the user should see a list of the five closest events in the next week.

First, let’s test the GET request. Amend gigs/tests.py as follows:

from django.test import TestCase
from gigs.models import Venue, Event
from factory.fuzzy import BaseFuzzyAttribute
from django.contrib.gis.geos import Point
import factory.django, random
from django.utils import timezone
from django.test import RequestFactory
from django.core.urlresolvers import reverse
from gigs.views import LookupView
class FuzzyPoint(BaseFuzzyAttribute):
    def fuzz(self):
        return Point(random.uniform(-180.0, 180.0),
                     random.uniform(-90.0, 90.0))
# Factories for tests
class VenueFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Venue
        django_get_or_create = (
            'name',
            'location'
        )
    name = 'Wembley Arena'
    location = FuzzyPoint()
class EventFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Event
        django_get_or_create = (
            'name',
            'venue',
            'datetime'
        )
    name = 'Queens of the Stone Age'
    datetime = timezone.now()
class VenueTest(TestCase):
    def test_create_venue(self):
        # Create the venue
        venue = VenueFactory()
        # Check we can find it
        all_venues = Venue.objects.all()
        self.assertEqual(len(all_venues), 1)
        only_venue = all_venues[0]
        self.assertEqual(only_venue, venue)
        # Check attributes
        self.assertEqual(only_venue.name, 'Wembley Arena')
        # Check string representation
        self.assertEqual(only_venue.__str__(), 'Wembley Arena')
class EventTest(TestCase):
    def test_create_event(self):
        # Create the venue
        venue = VenueFactory()
        # Create the event
        event = EventFactory(venue=venue)
        # Check we can find it
        all_events = Event.objects.all()
        self.assertEqual(len(all_events), 1)
        only_event = all_events[0]
        self.assertEqual(only_event, event)
        # Check attributes
        self.assertEqual(only_event.name, 'Queens of the Stone Age')
        self.assertEqual(only_event.venue.name, 'Wembley Arena')
        # Check string representation
        self.assertEqual(only_event.__str__(), 'Queens of the Stone Age - Wembley Arena')
class LookupViewTest(TestCase):
    """
    Test lookup view
    """
    def setUp(self):
        self.factory = RequestFactory()
    def test_get(self):
        request = self.factory.get(reverse('lookup'))
        response = LookupView.as_view()(request)
        self.assertEqual(response.status_code, 200)
        self.assertTemplateUsed('gigs/lookup.html')

Let’s run our tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
E
======================================================================
ERROR: gigs.tests (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: gigs.tests
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 9, in <module>
    from gigs.views import LookupView
ImportError: cannot import name 'LookupView'
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (errors=1)
Destroying test database for alias 'default'...

Our first issue is that we can’t import the view in the test. Let’s fix that by amending gigs/views.py:

from django.shortcuts import render
from django.views.generic.base import View
class LookupView(View):
    pass

Running the tests again results in the following:

$ python manage.py test gigs
Creating test database for alias 'default'...
.E.
======================================================================
ERROR: test_get (gigs.tests.LookupViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 88, in test_get
    request = self.factory.get(reverse('lookup'))
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/urlresolvers.py", line 600, in reverse
    return force_text(iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs)))
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/urlresolvers.py", line 508, in _reverse_with_prefix
    (lookup_view_s, args, kwargs, len(patterns), patterns))
django.core.urlresolvers.NoReverseMatch: Reverse for 'lookup' with arguments '()' and keyword arguments '{}' not found. 0 pattern(s) tried: []
----------------------------------------------------------------------
Ran 3 tests in 0.154s
FAILED (errors=1)
Destroying test database for alias 'default'...

We can’t resolve the URL for our new view, so we need to add it to our URLconf. First of all, save this as gigs/urls.py:

from django.conf.urls import url
from gigs.views import LookupView
urlpatterns = [
    # Lookup
    url(r'', LookupView.as_view(), name='lookup'),
]

Then amend gigfinder/urls.py as follows:

from django.conf.urls import url, include
from django.contrib import admin
urlpatterns = [
    url(r'^admin/', admin.site.urls),
    # Gig URLs
    url(r'', include('gigs.urls')),
]

Then run the tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
.F.
======================================================================
FAIL: test_get (gigs.tests.LookupViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 90, in test_get
    self.assertEqual(response.status_code, 200)
AssertionError: 405 != 200
----------------------------------------------------------------------
Ran 3 tests in 0.417s
FAILED (failures=1)
Destroying test database for alias 'default'...

We get a 405 response because the view does not accept GET requests. Let’s resolve that:

from django.shortcuts import render_to_response
from django.views.generic.base import View
class LookupView(View):
    def get(self, request):
        return render_to_response('gigs/lookup.html')

If we run our tests now:

$ python manage.py test gigs
Creating test database for alias 'default'...
.E.
======================================================================
ERROR: test_get (gigs.tests.LookupViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 89, in test_get
    response = LookupView.as_view()(request)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/views/generic/base.py", line 88, in dispatch
    return handler(request, *args, **kwargs)
  File "/Users/matthewdaly/Projects/gigfinder/gigs/views.py", line 6, in get
    return render_to_response('gigs/lookup.html')
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/shortcuts.py", line 39, in render_to_response
    content = loader.render_to_string(template_name, context, using=using)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/template/loader.py", line 96, in render_to_string
    template = get_template(template_name, using=using)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/template/loader.py", line 43, in get_template
    raise TemplateDoesNotExist(template_name, chain=chain)
django.template.exceptions.TemplateDoesNotExist: gigs/lookup.html
----------------------------------------------------------------------
Ran 3 tests in 0.409s
FAILED (errors=1)
Destroying test database for alias 'default'...

We see that the template is not defined. Save the following as gigs/templates/gigs/includes/base.html:

<!DOCTYPE html>
<html>
    <head>
        <title>Gig finder</title>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css"></link>
    </head>
    <body>
        <h1>Gig Finder</h1>
        <div class="container">
            <div class="row">
                {% block content %}{% endblock %}
            </div>
        </div>
        <script src="https://code.jquery.com/jquery-2.2.2.min.js" integrity="sha256-36cp2Co+/62rEAAYHLmRCPIych47CvdM+uTBJwSzWjI=" crossorigin="anonymous"></script>
        <script language="javascript" type="text/javascript" src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"></script>
    </body>
</html>

And the following as gigs/templates/gigs/lookup.html:

{% extends "gigs/includes/base.html" %}
{% block content %}
    <form role="form" action="/" method="post">{% csrf_token %}
        <div class="form-group">
            <label for="latitude">Latitude:</label>
            <input id="id_latitude" name="latitude" type="text" class="form-control"></input>
        </div>
        <div class="form-group">
            <label for="longitude">Longitude:</label>
            <input id="id_longitude" name="longitude" type="text" class="form-control"></input>
        </div>
        <input class="btn btn-primary" type="submit" value="Submit" />
    </form>
    <script language="javascript" type="text/javascript">
        navigator.geolocation.getCurrentPosition(function (position) {
            var lat = document.getElementById('id_latitude');
            var lon = document.getElementById('id_longitude');
            lat.value = position.coords.latitude;
            lon.value = position.coords.longitude;
        });
    </script>
{% endblock %}

Note the JavaScript to populate the latitude and longitude. Now, if we run our tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
...
----------------------------------------------------------------------
Ran 3 tests in 1.814s
OK
Destroying test database for alias 'default'...

Success! We now render our form as expected. Time to commit:

$ git add gigs gigfinder
$ git commit -m 'Implemented GET handler'

Handling POST requests

Now we need to be able to handle POST requests and return the appropriate results. First, let’s write a test for it in our existing LookupViewTest class:

    def test_post(self):
        # Create venues to return
        v1 = VenueFactory(name='Venue1')
        v2 = VenueFactory(name='Venue2')
        v3 = VenueFactory(name='Venue3')
        v4 = VenueFactory(name='Venue4')
        v5 = VenueFactory(name='Venue5')
        v6 = VenueFactory(name='Venue6')
        v7 = VenueFactory(name='Venue7')
        v8 = VenueFactory(name='Venue8')
        v9 = VenueFactory(name='Venue9')
        v10 = VenueFactory(name='Venue10')
        # Create events to return
        e1 = EventFactory(name='Event1', venue=v1)
        e2 = EventFactory(name='Event2', venue=v2)
        e3 = EventFactory(name='Event3', venue=v3)
        e4 = EventFactory(name='Event4', venue=v4)
        e5 = EventFactory(name='Event5', venue=v5)
        e6 = EventFactory(name='Event6', venue=v6)
        e7 = EventFactory(name='Event7', venue=v7)
        e8 = EventFactory(name='Event8', venue=v8)
        e9 = EventFactory(name='Event9', venue=v9)
        e10 = EventFactory(name='Event10', venue=v10)
        # Set parameters
        lat = 52.3749159
        lon = 1.1067473
        # Put together request
        data = {
            'latitude': lat,
            'longitude': lon
        }
        request = self.factory.post(reverse('lookup'), data)
        response = LookupView.as_view()(request)
        self.assertEqual(response.status_code, 200)
        self.assertTemplateUsed('gigs/lookupresults.html')

If we now run this test:

$ python manage.py test gigs
Creating test database for alias 'default'...
..F.
======================================================================
FAIL: test_post (gigs.tests.LookupViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 117, in test_post
    self.assertEqual(response.status_code, 200)
AssertionError: 405 != 200
----------------------------------------------------------------------
Ran 4 tests in 1.281s
FAILED (failures=1)
Destroying test database for alias 'default'...

We can see that it fails because the POST method is not supported. Now we can start work on implementing it. First, let’s create a form in gigs/forms.py:

from django.forms import Form, FloatField
class LookupForm(Form):
    latitude = FloatField()
    longitude = FloatField()

Next, edit gigs/views.py:

from django.shortcuts import render_to_response
from django.views.generic.edit import FormView
from gigs.forms import LookupForm
from gigs.models import Event
from django.utils import timezone
from django.contrib.gis.geos import Point
from django.contrib.gis.db.models.functions import Distance
class LookupView(FormView):
    form_class = LookupForm
    def get(self, request):
        return render_to_response('gigs/lookup.html')
    def form_valid(self, form):
        # Get data
        latitude = form.cleaned_data['latitude']
        longitude = form.cleaned_data['longitude']
        # Get today's date
        now = timezone.now()
        # Get next week's date
        next_week = now + timezone.timedelta(weeks=1)
        # Get Point
        location = Point(longitude, latitude, srid=4326)
        # Look up events
        events = Event.objects.filter(datetime__gte=now).filter(datetime__lte=next_week).annotate(distance=Distance('venue__location', location)).order_by('distance')[0:5]
        # Render the template
        return render_to_response('gigs/lookupresults.html', {
            'events': events
            })

Note that we’re switching from a View to a FormView so that it can more easily handle our form. We could render the form using this as well, but as it’s a simple form I decided it wasn’t worth the bother. Also, note that the longitude goes first - this caught me out as I expected the latitude to be the first argument.

Now, if we run our tests, they should complain about our missing template:

$ python manage.py test gigs
Creating test database for alias 'default'...
..E.
======================================================================
ERROR: test_post (gigs.tests.LookupViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 116, in test_post
    response = LookupView.as_view()(request)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/views/generic/base.py", line 88, in dispatch
    return handler(request, *args, **kwargs)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/views/generic/edit.py", line 222, in post
    return self.form_valid(form)
  File "/Users/matthewdaly/Projects/gigfinder/gigs/views.py", line 31, in form_valid
    'events': events
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/shortcuts.py", line 39, in render_to_response
    content = loader.render_to_string(template_name, context, using=using)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/template/loader.py", line 96, in render_to_string
    template = get_template(template_name, using=using)
  File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/template/loader.py", line 43, in get_template
    raise TemplateDoesNotExist(template_name, chain=chain)
django.template.exceptions.TemplateDoesNotExist: gigs/lookupresults.html
----------------------------------------------------------------------
Ran 4 tests in 0.506s
FAILED (errors=1)
Destroying test database for alias 'default'...

So let’s create gigs/templates/gigs/lookupresults.html:

{% extends "gigs/includes/base.html" %}
{% block content %}
    <ul>
    {% for event in events %}
    <li>{{ event.name }} - {{ event.venue.name }}</li>
    {% endfor %}
    </ul>
{% endblock %}

Now, if we run our tests, they should pass:

$ python manage.py test gigs
Creating test database for alias 'default'...
....
----------------------------------------------------------------------
Ran 4 tests in 0.728s
OK
Destroying test database for alias 'default'...

However, if we try actually submitting the form by hand, we get the error CSRF token missing or incorrect. Edit views.py as follows to resolve this:

from django.shortcuts import render_to_response
from django.views.generic.edit import FormView
from gigs.forms import LookupForm
from gigs.models import Event
from django.utils import timezone
from django.contrib.gis.geos import Point
from django.contrib.gis.db.models.functions import Distance
from django.template import RequestContext
class LookupView(FormView):
    form_class = LookupForm
    def get(self, request):
        return render_to_response('gigs/lookup.html', RequestContext(request))
    def form_valid(self, form):
        # Get data
        latitude = form.cleaned_data['latitude']
        longitude = form.cleaned_data['longitude']
        # Get today's date
        now = timezone.now()
        # Get next week's date
        next_week = now + timezone.timedelta(weeks=1)
        # Get Point
        location = Point(longitude, latitude, srid=4326)
        # Look up events
        events = Event.objects.filter(datetime__gte=now).filter(datetime__lte=next_week).annotate(distance=Distance('venue__location', location)).order_by('distance')[0:5]
        # Render the template
        return render_to_response('gigs/lookupresults.html', {
            'events': events
            })

Here we’re adding the request context so that the CSRF token is available.

If you run the dev server, add a few events and venues via the admin, and submit a search, you’ll see that you’re returning events closest to you first.

Now that we can submit searches, we’re ready to commit:

$ git add gigs/
$ git commit -m 'Can now retrieve search results'

And we’re done! Of course, you may want to expand on this by plotting each gig venue on a map, or something like that, in which case there’s plenty of methods of doing so - you can retrieve the latitude and longitude in the template and use Google Maps to display them. I’ll leave doing so as an exercise for the reader.

I can’t say that working with GeoDjango isn’t a bit of a struggle at times, but being able to make spatial queries in this fashion is very useful. With more and more people carrying smartphones, you’re more likely than ever to be asked to build applications that return data based on someone’s geographical location, and GeoDjango is a great way to do this with a Django application. You can find the source on Github.