Matthew Daly's Blog

I'm a web developer in Norfolk. This is my blog...

26th March 2016 9:30 pm

Building a Location Aware Web App With Geodjango

PostgreSQL has excellent support for geographical data thanks to the PostGIS extension, and Django allows you to take full advantage of it thanks to GeoDjango. In this tutorial, I’ll show you how to use GeoDjango to build a web app that allows users to search for gigs and events near them.

Requirements

I’ve made the jump to Python 3, and if you haven’t done so yet, I highly recommend it - it’s not hard, and there’s very few modules left that haven’t been ported across. As such, this tutorial assumes you’re using Python 3. You’ll also need to have Git, PostgreSQL and PostGIS installed - I’ll leave the details of doing so up to you as it varies by platform, but you can generally do so easily with a package manager on most Linux distros. On Mac OS X I recommend using Homebrew. If you’re on Windows I think your best bet is probably to use a Vagrant VM.

We’ll be using Django 1.9 - if by the time you read this a newer version of Django is out, it’s quite possible that some things may have changed and you’ll need to work around any problems caused. Generally search engines are the best place to look for this, and I’ll endeavour to keep the resulting Github repository as up to date as I can, so try those if you get stuck.

Getting started

First of all, let’s create our database. Make sure you’re running as a user that has the required privileges to create users and databases for PostgreSQL and run the following command:

$ createdb gigfinder

This creates the database. Next, we create the user:

$ createuser -s giguser -P

You’ll be prompted to enter a password for the new user. Next, we want to use the psql command-line client to interact with our new database:

$ psql gigfinder

This connects to the database. Run these commands to set up access to the database and install the PostGIS extension:

# GRANT ALL PRIVILEGES ON DATABASE gigfinder TO giguser;
# CREATE EXTENSION postgis;
# \q

With our database set up, it’s time to start work on our project. Let’s create our virtualenv in a new folder:

$ pyvenv venv

Then activate it:

$ source venv/bin/activate

Then we install Django, along with a few other production dependencies:

$ pip install django-toolbelt

And record our dependencies:

$ pip freeze > requirements.txt

Next, we create our application skeleton:

$ django-admin.py startproject gigfinder .

We’ll also create a .gitignore file:

venv/
.DS_Store
*.swp
node_modules/
*.pyc

Let’s commit our changes:

$ git init
$ git add .gitignore requirements.txt manage.py gigfinder
$ git commit -m 'Initial commit'

Next, let’s create our first app, which we will call gigs:

$ python manage.py startapp gigs

We need to add our new app to the INSTALLED_APPS setting. While we’re there we’ll also add GIS support and set up the database connection. First, add the required apps to INSTALLED_APPS:

INSTALLED_APPS = [
...
'django.contrib.gis',
'gigs',
]

Next, configure the database:

DATABASES = {
'default': {
'ENGINE': 'django.contrib.gis.db.backends.postgis',
'NAME': 'gigfinder',
'USER': 'giguser',
'PASSWORD': 'password',
},
}

Let’s run the migrations:

$ python manage.py migrate
Operations to perform:
Apply all migrations: sessions, contenttypes, admin, auth
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying sessions.0001_initial... OK

And create our superuser account:

$ python manage.py createsuperuser

Now, we’ll commit our changes:

$ git add gigfinder/ gigs/
$ git commit -m 'Created gigs app'
[master e72a846] Created gigs app
8 files changed, 24 insertions(+), 3 deletions(-)
create mode 100644 gigs/__init__.py
create mode 100644 gigs/admin.py
create mode 100644 gigs/apps.py
create mode 100644 gigs/migrations/__init__.py
create mode 100644 gigs/models.py
create mode 100644 gigs/tests.py
create mode 100644 gigs/views.py

Our first model

At this point, it’s worth thinking about the models we plan for our app to have. First we’ll have a Venue model that contains details of an individual venue, which will include a name and a geographical location. We’ll also have an Event model that will represent an individual gig or event at a venue, and will include a name, date/time and a venue as a foreign key.

Before we start writing our first model, we need to write a test for it, but we also need to be able to create objects easily in our tests. We also want to be able to easily examine our objects, so we’ll install iPDB and Factory Boy:

$ pip install ipdb factory-boy
$ pip freeze > requirements.txt

Next, we write a test for the Venue model:

from django.test import TestCase
from gigs.models import Venue
from factory.fuzzy import BaseFuzzyAttribute
from django.contrib.gis.geos import Point
import factory.django, random
class FuzzyPoint(BaseFuzzyAttribute):
def fuzz(self):
return Point(random.uniform(-180.0, 180.0),
random.uniform(-90.0, 90.0))
# Factories for tests
class VenueFactory(factory.django.DjangoModelFactory):
class Meta:
model = Venue
django_get_or_create = (
'name',
'location'
)
name = 'Wembley Arena'
location = FuzzyPoint()
class VenueTest(TestCase):
def test_create_venue(self):
# Create the venue
venue = VenueFactory()
# Check we can find it
all_venues = Venue.objects.all()
self.assertEqual(len(all_venues), 1)
only_venue = all_venues[0]
self.assertEqual(only_venue, venue)
# Check attributes
self.assertEqual(only_venue.name, 'Wembley Arena')

Note that we randomly generate our location - this is done as suggested in this Stack Overflow post.

Now, running our tests brings up an expected error:

$ python manage.py test gigs
Creating test database for alias 'default'...
E
======================================================================
ERROR: gigs.tests (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: gigs.tests
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 2, in <module>
from gigs.models import Venue
ImportError: cannot import name 'Venue'
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (errors=1)
Destroying test database for alias 'default'...

Let’s create our Venue model in gigs/models.py:

from django.contrib.gis.db import models
class Venue(models.Model):
"""
Model for a venue
"""
pass

For now, we’re just creating a simple dummy model. Note that we import models from django.contrib.gis.db instead of the usual place - this gives us access to the additional geographical fields.

If we run our tests again we get an error:

$ python manage.py test gigs
Creating test database for alias 'default'...
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: relation "gigs_venue" does not exist
LINE 1: SELECT "gigs_venue"."id" FROM "gigs_venue" ORDER BY "gigs_ve...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/commands/test.py", line 30, in run_from_argv
super(Command, self).run_from_argv(argv)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/commands/test.py", line 74, in execute
super(Command, self).execute(*args, **options)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/commands/test.py", line 90, in handle
failures = test_runner.run_tests(test_labels)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/test/runner.py", line 532, in run_tests
old_config = self.setup_databases()
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/test/runner.py", line 482, in setup_databases
self.parallel, **kwargs
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/test/runner.py", line 726, in setup_databases
serialize=connection.settings_dict.get("TEST", {}).get("SERIALIZE", True),
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 78, in create_test_db
self.connection._test_serialized_contents = self.serialize_db_to_string()
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 122, in serialize_db_to_string
serializers.serialize("json", get_objects(), indent=None, stream=out)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/serializers/__init__.py", line 129, in serialize
s.serialize(queryset, **options)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/serializers/base.py", line 79, in serialize
for count, obj in enumerate(queryset, start=1):
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 118, in get_objects
for obj in queryset.iterator():
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/models/query.py", line 52, in __iter__
results = compiler.execute_sql()
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 848, in execute_sql
cursor.execute(sql, params)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/utils.py", line 95, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "gigs_venue" does not exist
LINE 1: SELECT "gigs_venue"."id" FROM "gigs_venue" ORDER BY "gigs_ve...

Let’s update our model:

from django.contrib.gis.db import models
class Venue(models.Model):
"""
Model for a venue
"""
name = models.CharField(max_length=200)
location = models.PointField()

Then create our migration:

$ python manage.py makemigrations
Migrations for 'gigs':
0001_initial.py:
- Create model Venue

And run it:

$ python manage.py migrate
Operations to perform:
Apply all migrations: gigs, sessions, contenttypes, auth, admin
Running migrations:
Rendering model states... DONE
Applying gigs.0001_initial... OK

Then if we run our tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
.
----------------------------------------------------------------------
Ran 1 test in 0.362s
OK
Destroying test database for alias 'default'...

They should pass. Note that Django may complain about needing to delete the test database before running the tests, but this should not cause any problems. Let’s commit our changes:

$ git add requirements.txt gigs/
$ git commit -m 'Venue model in place'

With our venue done, let’s turn to our Event model. Amend gigs/tests.py as follows:

from django.test import TestCase
from gigs.models import Venue, Event
from factory.fuzzy import BaseFuzzyAttribute
from django.contrib.gis.geos import Point
import factory.django, random
from django.utils import timezone
class FuzzyPoint(BaseFuzzyAttribute):
def fuzz(self):
return Point(random.uniform(-180.0, 180.0),
random.uniform(-90.0, 90.0))
# Factories for tests
class VenueFactory(factory.django.DjangoModelFactory):
class Meta:
model = Venue
django_get_or_create = (
'name',
'location'
)
name = 'Wembley Arena'
location = FuzzyPoint()
class EventFactory(factory.django.DjangoModelFactory):
class Meta:
model = Event
django_get_or_create = (
'name',
'venue',
'datetime'
)
name = 'Queens of the Stone Age'
datetime = timezone.now()
class VenueTest(TestCase):
def test_create_venue(self):
# Create the venue
venue = VenueFactory()
# Check we can find it
all_venues = Venue.objects.all()
self.assertEqual(len(all_venues), 1)
only_venue = all_venues[0]
self.assertEqual(only_venue, venue)
# Check attributes
self.assertEqual(only_venue.name, 'Wembley Arena')
class EventTest(TestCase):
def test_create_event(self):
# Create the venue
venue = VenueFactory()
# Create the event
event = EventFactory(venue=venue)
# Check we can find it
all_events = Event.objects.all()
self.assertEqual(len(all_events), 1)
only_event = all_events[0]
self.assertEqual(only_event, event)
# Check attributes
self.assertEqual(only_event.name, 'Queens of the Stone Age')
self.assertEqual(only_event.venue.name, 'Wembley Arena')

Then we run our tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
E
======================================================================
ERROR: gigs.tests (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: gigs.tests
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 2, in <module>
from gigs.models import Venue, Event
ImportError: cannot import name 'Event'
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (errors=1)
Destroying test database for alias 'default'...

As expected, this fails, so create an empty Event model in gigs/models.py:

class Event(models.Model):
"""
Model for an event
"""
pass

Running the tests now will raise an error due to the table not existing:

$ python manage.py test gigs
Creating test database for alias 'default'...
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: relation "gigs_event" does not exist
LINE 1: SELECT "gigs_event"."id" FROM "gigs_event" ORDER BY "gigs_ev...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/commands/test.py", line 30, in run_from_argv
super(Command, self).run_from_argv(argv)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/commands/test.py", line 74, in execute
super(Command, self).execute(*args, **options)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/management/commands/test.py", line 90, in handle
failures = test_runner.run_tests(test_labels)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/test/runner.py", line 532, in run_tests
old_config = self.setup_databases()
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/test/runner.py", line 482, in setup_databases
self.parallel, **kwargs
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/test/runner.py", line 726, in setup_databases
serialize=connection.settings_dict.get("TEST", {}).get("SERIALIZE", True),
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 78, in create_test_db
self.connection._test_serialized_contents = self.serialize_db_to_string()
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 122, in serialize_db_to_string
serializers.serialize("json", get_objects(), indent=None, stream=out)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/serializers/__init__.py", line 129, in serialize
s.serialize(queryset, **options)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/serializers/base.py", line 79, in serialize
for count, obj in enumerate(queryset, start=1):
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 118, in get_objects
for obj in queryset.iterator():
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/models/query.py", line 52, in __iter__
results = compiler.execute_sql()
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 848, in execute_sql
cursor.execute(sql, params)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/utils.py", line 95, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "gigs_event" does not exist
LINE 1: SELECT "gigs_event"."id" FROM "gigs_event" ORDER BY "gigs_ev...

So let’s populate our model:

class Event(models.Model):
"""
Model for an event
"""
name = models.CharField(max_length=200)
datetime = models.DateTimeField()
venue = models.ForeignKey(Venue)

And create our migration:

$ python manage.py makemigrations
Migrations for 'gigs':
0002_event.py:
- Create model Event

And run it:

$ python manage.py migrate
Operations to perform:
Apply all migrations: auth, admin, sessions, contenttypes, gigs
Running migrations:
Rendering model states... DONE
Applying gigs.0002_event... OK

And run our tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
..
----------------------------------------------------------------------
Ran 2 tests in 0.033s
OK
Destroying test database for alias 'default'...

Again, you may be prompted to delete the test database, but this should not be an issue.

With this done, let’s commit our changes:

$ git add gigs
$ git commit -m 'Added Event model'
[master 47ba686] Added Event model
3 files changed, 67 insertions(+), 1 deletion(-)
create mode 100644 gigs/migrations/0002_event.py

Setting up the admin

For an application like this, you’d expect the curators of the site to maintain the gigs and venues stored in the database, and that’s an obvious use case for the Django admin. So let’s set our models up to be available in the admin. Open up gigs/admin.py and amend it as follows:

from django.contrib import admin
from gigs.models import Venue, Event
admin.site.register(Venue)
admin.site.register(Event)

Now, if you start up the dev server as usual with python manage.py runserver and visit http://127.0.0.1:8000/admin/, you can see that our Event and Venue models are now available. However, the string representations of them are pretty useless. Let’s fix that. First, we amend our tests:

from django.test import TestCase
from gigs.models import Venue, Event
from factory.fuzzy import BaseFuzzyAttribute
from django.contrib.gis.geos import Point
import factory.django, random
from django.utils import timezone
class FuzzyPoint(BaseFuzzyAttribute):
def fuzz(self):
return Point(random.uniform(-180.0, 180.0),
random.uniform(-90.0, 90.0))
# Factories for tests
class VenueFactory(factory.django.DjangoModelFactory):
class Meta:
model = Venue
django_get_or_create = (
'name',
'location'
)
name = 'Wembley Arena'
location = FuzzyPoint()
class EventFactory(factory.django.DjangoModelFactory):
class Meta:
model = Event
django_get_or_create = (
'name',
'venue',
'datetime'
)
name = 'Queens of the Stone Age'
datetime = timezone.now()
class VenueTest(TestCase):
def test_create_venue(self):
# Create the venue
venue = VenueFactory()
# Check we can find it
all_venues = Venue.objects.all()
self.assertEqual(len(all_venues), 1)
only_venue = all_venues[0]
self.assertEqual(only_venue, venue)
# Check attributes
self.assertEqual(only_venue.name, 'Wembley Arena')
# Check string representation
self.assertEqual(only_venue.__str__(), 'Wembley Arena')
class EventTest(TestCase):
def test_create_event(self):
# Create the venue
venue = VenueFactory()
# Create the event
event = EventFactory(venue=venue)
# Check we can find it
all_events = Event.objects.all()
self.assertEqual(len(all_events), 1)
only_event = all_events[0]
self.assertEqual(only_event, event)
# Check attributes
self.assertEqual(only_event.name, 'Queens of the Stone Age')
self.assertEqual(only_event.venue.name, 'Wembley Arena')
# Check string representation
self.assertEqual(only_event.__str__(), 'Queens of the Stone Age - Wembley Arena')

Next, we run our tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
FF
======================================================================
FAIL: test_create_event (gigs.tests.EventTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 74, in test_create_event
self.assertEqual(only_event.__str__(), 'Queens of the Stone Age - Wembley Arena')
AssertionError: 'Event object' != 'Queens of the Stone Age - Wembley Arena'
- Event object
+ Queens of the Stone Age - Wembley Arena
======================================================================
FAIL: test_create_venue (gigs.tests.VenueTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 52, in test_create_venue
self.assertEqual(only_venue.__str__(), 'Wembley Arena')
AssertionError: 'Venue object' != 'Wembley Arena'
- Venue object
+ Wembley Arena
----------------------------------------------------------------------
Ran 2 tests in 0.059s
FAILED (failures=2)
Destroying test database for alias 'default'...

They fail as expected. So let’s update gigs/models.py:

from django.contrib.gis.db import models
class Venue(models.Model):
"""
Model for a venue
"""
name = models.CharField(max_length=200)
location = models.PointField()
def __str__(self):
return self.name
class Event(models.Model):
"""
Model for an event
"""
name = models.CharField(max_length=200)
datetime = models.DateTimeField()
venue = models.ForeignKey(Venue)
def __str__(self):
return "%s - %s" % (self.name, self.venue.name)

For the venue, we just use the name. For the event, we use the event name and the venue name.

Now, we run our tests again:

$ python manage.py test gigs
Creating test database for alias 'default'...
..
----------------------------------------------------------------------
Ran 2 tests in 0.048s
OK
Destroying test database for alias 'default'...

Time to commit our changes:

$ git add gigs
$ git commit -m 'Added models to admin'
[master 65d051f] Added models to admin
3 files changed, 15 insertions(+), 1 deletion(-)

Our models are now in place, so you may want to log into the admin and create a few venues and events so you can see it in action. Note that the location field for the Venue model creates a map widget that allows you to select a geographical location. It is a bit basic, however, so let’s make it better. Let’s install django-floppyforms:

$ pip install django-floppyforms

And add it to our requirements:

$ pip freeze -r requirements.txt

Then add it to INSTALLED_APPS in gigfinder/setttings.py:

INSTALLED_APPS = [
...
'django.contrib.gis',
'gigs',
'floppyforms',
]

Now we create a custom point widget for our admin, a custom form for the venues, and a custom venue admin:

from django.contrib import admin
from gigs.models import Venue, Event
from django.forms import ModelForm
from floppyforms.gis import PointWidget, BaseGMapWidget
class CustomPointWidget(PointWidget, BaseGMapWidget):
class Media:
js = ('/static/floppyforms/js/MapWidget.js',)
class VenueAdminForm(ModelForm):
class Meta:
model = Venue
fields = ['name', 'location']
widgets = {
'location': CustomPointWidget()
}
class VenueAdmin(admin.ModelAdmin):
form = VenueAdminForm
admin.site.register(Venue, VenueAdmin)
admin.site.register(Event)

Note in particular that we define the media for our widget so we can include some required Javascript. If you run the dev server again, you should see that the map widget in the admin is now provided by Google Maps, making it much easier to identify the correct location of the venue.

Time to commit our changes:

$ git add gigfinder/ gigs/ requirements.txt
$ git commit -m 'Customised location widget'

With our admin ready, it’s time to move on to the user-facing part of the web app.

Creating our views

We will keep the front end for this app as simple as possible for the purposes of this tutorial, but of course you should feel free to expand upon this as you see fit. What we’ll do is create a form that uses HTML5 geolocation to get the user’s current geographical coordinates. It will then return events in the next week, ordered by how close the venue is. Please note that there are plans afoot in some browsers to prevent HTML5 geolocation from working unless content is server over HTTPS, so that may complicate things.

How do we query the database to get this data? It’s not too difficult, as shown in this example:

$ python manage.py shell
Python 3.5.1 (default, Mar 25 2016, 00:17:15)
Type "copyright", "credits" or "license" for more information.
IPython 4.1.2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: from gigs.models import *
In [2]: from django.contrib.gis.geos import Point
In [3]: from django.contrib.gis.db.models.functions import Distance
In [4]: location = Point(52.3749159, 1.1067473, srid=4326)
In [5]: Venue.objects.all().annotate(distance=Distance('location', location)).order_by('distance')
Out[5]: [<Venue: Diss Corn Hall>, <Venue: Waterfront Norwich>, <Venue: UEA Norwich>, <Venue: Wembley Arena>]

I’ve set up a number of venues using the admin, one round the corner, two in Norwich, and one in London. I then imported the models, the Point class, and the Distance function, and created a Point object. Note that the Point is passed three fields - the first and second are the latitude and longitude, respectively, while the srid field takes a value of 4326. This field represents the Spatial Reference System Identifier used for this query - we’ve gone for WGS 84, which is a common choice and is referred to with the SRID 4326.

Now, we want the user to be able to submit the form and get the 5 nearest events in the next week. We can get the date for this time next week as follows:

In [6]: next_week = timezone.now() + timezone.timedelta(weeks=1)

Then we can get the events we want, sorted by distance, like this:

In [7]: Event.objects.filter(datetime__gte=timezone.now()).filter(datetime__lte=next_week).annotate(distance=Distance('venue__location', location)).order_by('distance')[0:5]
Out[7]: [<Event: Primal Scream - UEA Norwich>, <Event: Queens of the Stone Age - Wembley Arena>]

With that in mind, let’s write the test for our view. The view should contain a single form that accepts a user’s geographical coordinates - for convenience we’ll autocomplete this with HTML5 geolocation. On submit, the user should see a list of the five closest events in the next week.

First, let’s test the GET request. Amend gigs/tests.py as follows:

from django.test import TestCase
from gigs.models import Venue, Event
from factory.fuzzy import BaseFuzzyAttribute
from django.contrib.gis.geos import Point
import factory.django, random
from django.utils import timezone
from django.test import RequestFactory
from django.core.urlresolvers import reverse
from gigs.views import LookupView
class FuzzyPoint(BaseFuzzyAttribute):
def fuzz(self):
return Point(random.uniform(-180.0, 180.0),
random.uniform(-90.0, 90.0))
# Factories for tests
class VenueFactory(factory.django.DjangoModelFactory):
class Meta:
model = Venue
django_get_or_create = (
'name',
'location'
)
name = 'Wembley Arena'
location = FuzzyPoint()
class EventFactory(factory.django.DjangoModelFactory):
class Meta:
model = Event
django_get_or_create = (
'name',
'venue',
'datetime'
)
name = 'Queens of the Stone Age'
datetime = timezone.now()
class VenueTest(TestCase):
def test_create_venue(self):
# Create the venue
venue = VenueFactory()
# Check we can find it
all_venues = Venue.objects.all()
self.assertEqual(len(all_venues), 1)
only_venue = all_venues[0]
self.assertEqual(only_venue, venue)
# Check attributes
self.assertEqual(only_venue.name, 'Wembley Arena')
# Check string representation
self.assertEqual(only_venue.__str__(), 'Wembley Arena')
class EventTest(TestCase):
def test_create_event(self):
# Create the venue
venue = VenueFactory()
# Create the event
event = EventFactory(venue=venue)
# Check we can find it
all_events = Event.objects.all()
self.assertEqual(len(all_events), 1)
only_event = all_events[0]
self.assertEqual(only_event, event)
# Check attributes
self.assertEqual(only_event.name, 'Queens of the Stone Age')
self.assertEqual(only_event.venue.name, 'Wembley Arena')
# Check string representation
self.assertEqual(only_event.__str__(), 'Queens of the Stone Age - Wembley Arena')
class LookupViewTest(TestCase):
"""
Test lookup view
"""
def setUp(self):
self.factory = RequestFactory()
def test_get(self):
request = self.factory.get(reverse('lookup'))
response = LookupView.as_view()(request)
self.assertEqual(response.status_code, 200)
self.assertTemplateUsed('gigs/lookup.html')

Let’s run our tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
E
======================================================================
ERROR: gigs.tests (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: gigs.tests
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 9, in <module>
from gigs.views import LookupView
ImportError: cannot import name 'LookupView'
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (errors=1)
Destroying test database for alias 'default'...

Our first issue is that we can’t import the view in the test. Let’s fix that by amending gigs/views.py:

from django.shortcuts import render
from django.views.generic.base import View
class LookupView(View):
pass

Running the tests again results in the following:

$ python manage.py test gigs
Creating test database for alias 'default'...
.E.
======================================================================
ERROR: test_get (gigs.tests.LookupViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 88, in test_get
request = self.factory.get(reverse('lookup'))
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/urlresolvers.py", line 600, in reverse
return force_text(iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs)))
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/core/urlresolvers.py", line 508, in _reverse_with_prefix
(lookup_view_s, args, kwargs, len(patterns), patterns))
django.core.urlresolvers.NoReverseMatch: Reverse for 'lookup' with arguments '()' and keyword arguments '{}' not found. 0 pattern(s) tried: []
----------------------------------------------------------------------
Ran 3 tests in 0.154s
FAILED (errors=1)
Destroying test database for alias 'default'...

We can’t resolve the URL for our new view, so we need to add it to our URLconf. First of all, save this as gigs/urls.py:

from django.conf.urls import url
from gigs.views import LookupView
urlpatterns = [
# Lookup
url(r'', LookupView.as_view(), name='lookup'),
]

Then amend gigfinder/urls.py as follows:

from django.conf.urls import url, include
from django.contrib import admin
urlpatterns = [
url(r'^admin/', admin.site.urls),
# Gig URLs
url(r'', include('gigs.urls')),
]

Then run the tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
.F.
======================================================================
FAIL: test_get (gigs.tests.LookupViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 90, in test_get
self.assertEqual(response.status_code, 200)
AssertionError: 405 != 200
----------------------------------------------------------------------
Ran 3 tests in 0.417s
FAILED (failures=1)
Destroying test database for alias 'default'...

We get a 405 response because the view does not accept GET requests. Let’s resolve that:

from django.shortcuts import render_to_response
from django.views.generic.base import View
class LookupView(View):
def get(self, request):
return render_to_response('gigs/lookup.html')

If we run our tests now:

$ python manage.py test gigs
Creating test database for alias 'default'...
.E.
======================================================================
ERROR: test_get (gigs.tests.LookupViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 89, in test_get
response = LookupView.as_view()(request)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/views/generic/base.py", line 88, in dispatch
return handler(request, *args, **kwargs)
File "/Users/matthewdaly/Projects/gigfinder/gigs/views.py", line 6, in get
return render_to_response('gigs/lookup.html')
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/shortcuts.py", line 39, in render_to_response
content = loader.render_to_string(template_name, context, using=using)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/template/loader.py", line 96, in render_to_string
template = get_template(template_name, using=using)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/template/loader.py", line 43, in get_template
raise TemplateDoesNotExist(template_name, chain=chain)
django.template.exceptions.TemplateDoesNotExist: gigs/lookup.html
----------------------------------------------------------------------
Ran 3 tests in 0.409s
FAILED (errors=1)
Destroying test database for alias 'default'...

We see that the template is not defined. Save the following as gigs/templates/gigs/includes/base.html:

<!DOCTYPE html>
<html>
<head>
<title>Gig finder</title>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css"></link>
</head>
<body>
<h1>Gig Finder</h1>
<div class="container">
<div class="row">
{% block content %}{% endblock %}
</div>
</div>
<script src="https://code.jquery.com/jquery-2.2.2.min.js" integrity="sha256-36cp2Co+/62rEAAYHLmRCPIych47CvdM+uTBJwSzWjI=" crossorigin="anonymous"></script>
<script language="javascript" type="text/javascript" src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"></script>
</body>
</html>

And the following as gigs/templates/gigs/lookup.html:

{% extends "gigs/includes/base.html" %}
{% block content %}
<form role="form" action="/" method="post">{% csrf_token %}
<div class="form-group">
<label for="latitude">Latitude:</label>
<input id="id_latitude" name="latitude" type="text" class="form-control"></input>
</div>
<div class="form-group">
<label for="longitude">Longitude:</label>
<input id="id_longitude" name="longitude" type="text" class="form-control"></input>
</div>
<input class="btn btn-primary" type="submit" value="Submit" />
</form>
<script language="javascript" type="text/javascript">
navigator.geolocation.getCurrentPosition(function (position) {
var lat = document.getElementById('id_latitude');
var lon = document.getElementById('id_longitude');
lat.value = position.coords.latitude;
lon.value = position.coords.longitude;
});
</script>
{% endblock %}

Note the JavaScript to populate the latitude and longitude. Now, if we run our tests:

$ python manage.py test gigs
Creating test database for alias 'default'...
...
----------------------------------------------------------------------
Ran 3 tests in 1.814s
OK
Destroying test database for alias 'default'...

Success! We now render our form as expected. Time to commit:

$ git add gigs gigfinder
$ git commit -m 'Implemented GET handler'

Handling POST requests

Now we need to be able to handle POST requests and return the appropriate results. First, let’s write a test for it in our existing LookupViewTest class:

def test_post(self):
# Create venues to return
v1 = VenueFactory(name='Venue1')
v2 = VenueFactory(name='Venue2')
v3 = VenueFactory(name='Venue3')
v4 = VenueFactory(name='Venue4')
v5 = VenueFactory(name='Venue5')
v6 = VenueFactory(name='Venue6')
v7 = VenueFactory(name='Venue7')
v8 = VenueFactory(name='Venue8')
v9 = VenueFactory(name='Venue9')
v10 = VenueFactory(name='Venue10')
# Create events to return
e1 = EventFactory(name='Event1', venue=v1)
e2 = EventFactory(name='Event2', venue=v2)
e3 = EventFactory(name='Event3', venue=v3)
e4 = EventFactory(name='Event4', venue=v4)
e5 = EventFactory(name='Event5', venue=v5)
e6 = EventFactory(name='Event6', venue=v6)
e7 = EventFactory(name='Event7', venue=v7)
e8 = EventFactory(name='Event8', venue=v8)
e9 = EventFactory(name='Event9', venue=v9)
e10 = EventFactory(name='Event10', venue=v10)
# Set parameters
lat = 52.3749159
lon = 1.1067473
# Put together request
data = {
'latitude': lat,
'longitude': lon
}
request = self.factory.post(reverse('lookup'), data)
response = LookupView.as_view()(request)
self.assertEqual(response.status_code, 200)
self.assertTemplateUsed('gigs/lookupresults.html')

If we now run this test:

$ python manage.py test gigs
Creating test database for alias 'default'...
..F.
======================================================================
FAIL: test_post (gigs.tests.LookupViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 117, in test_post
self.assertEqual(response.status_code, 200)
AssertionError: 405 != 200
----------------------------------------------------------------------
Ran 4 tests in 1.281s
FAILED (failures=1)
Destroying test database for alias 'default'...

We can see that it fails because the POST method is not supported. Now we can start work on implementing it. First, let’s create a form in gigs/forms.py:

from django.forms import Form, FloatField
class LookupForm(Form):
latitude = FloatField()
longitude = FloatField()

Next, edit gigs/views.py:

from django.shortcuts import render_to_response
from django.views.generic.edit import FormView
from gigs.forms import LookupForm
from gigs.models import Event
from django.utils import timezone
from django.contrib.gis.geos import Point
from django.contrib.gis.db.models.functions import Distance
class LookupView(FormView):
form_class = LookupForm
def get(self, request):
return render_to_response('gigs/lookup.html')
def form_valid(self, form):
# Get data
latitude = form.cleaned_data['latitude']
longitude = form.cleaned_data['longitude']
# Get today's date
now = timezone.now()
# Get next week's date
next_week = now + timezone.timedelta(weeks=1)
# Get Point
location = Point(longitude, latitude, srid=4326)
# Look up events
events = Event.objects.filter(datetime__gte=now).filter(datetime__lte=next_week).annotate(distance=Distance('venue__location', location)).order_by('distance')[0:5]
# Render the template
return render_to_response('gigs/lookupresults.html', {
'events': events
})

Note that we’re switching from a View to a FormView so that it can more easily handle our form. We could render the form using this as well, but as it’s a simple form I decided it wasn’t worth the bother. Also, note that the longitude goes first - this caught me out as I expected the latitude to be the first argument.

Now, if we run our tests, they should complain about our missing template:

$ python manage.py test gigs
Creating test database for alias 'default'...
..E.
======================================================================
ERROR: test_post (gigs.tests.LookupViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/matthewdaly/Projects/gigfinder/gigs/tests.py", line 116, in test_post
response = LookupView.as_view()(request)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/views/generic/base.py", line 88, in dispatch
return handler(request, *args, **kwargs)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/views/generic/edit.py", line 222, in post
return self.form_valid(form)
File "/Users/matthewdaly/Projects/gigfinder/gigs/views.py", line 31, in form_valid
'events': events
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/shortcuts.py", line 39, in render_to_response
content = loader.render_to_string(template_name, context, using=using)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/template/loader.py", line 96, in render_to_string
template = get_template(template_name, using=using)
File "/Users/matthewdaly/Projects/gigfinder/venv/lib/python3.5/site-packages/django/template/loader.py", line 43, in get_template
raise TemplateDoesNotExist(template_name, chain=chain)
django.template.exceptions.TemplateDoesNotExist: gigs/lookupresults.html
----------------------------------------------------------------------
Ran 4 tests in 0.506s
FAILED (errors=1)
Destroying test database for alias 'default'...

So let’s create gigs/templates/gigs/lookupresults.html:

{% extends "gigs/includes/base.html" %}
{% block content %}
<ul>
{% for event in events %}
<li>{{ event.name }} - {{ event.venue.name }}</li>
{% endfor %}
</ul>
{% endblock %}

Now, if we run our tests, they should pass:

$ python manage.py test gigs
Creating test database for alias 'default'...
....
----------------------------------------------------------------------
Ran 4 tests in 0.728s
OK
Destroying test database for alias 'default'...

However, if we try actually submitting the form by hand, we get the error CSRF token missing or incorrect. Edit views.py as follows to resolve this:

from django.shortcuts import render_to_response
from django.views.generic.edit import FormView
from gigs.forms import LookupForm
from gigs.models import Event
from django.utils import timezone
from django.contrib.gis.geos import Point
from django.contrib.gis.db.models.functions import Distance
from django.template import RequestContext
class LookupView(FormView):
form_class = LookupForm
def get(self, request):
return render_to_response('gigs/lookup.html', RequestContext(request))
def form_valid(self, form):
# Get data
latitude = form.cleaned_data['latitude']
longitude = form.cleaned_data['longitude']
# Get today's date
now = timezone.now()
# Get next week's date
next_week = now + timezone.timedelta(weeks=1)
# Get Point
location = Point(longitude, latitude, srid=4326)
# Look up events
events = Event.objects.filter(datetime__gte=now).filter(datetime__lte=next_week).annotate(distance=Distance('venue__location', location)).order_by('distance')[0:5]
# Render the template
return render_to_response('gigs/lookupresults.html', {
'events': events
})

Here we’re adding the request context so that the CSRF token is available.

If you run the dev server, add a few events and venues via the admin, and submit a search, you’ll see that you’re returning events closest to you first.

Now that we can submit searches, we’re ready to commit:

$ git add gigs/
$ git commit -m 'Can now retrieve search results'

And we’re done! Of course, you may want to expand on this by plotting each gig venue on a map, or something like that, in which case there’s plenty of methods of doing so - you can retrieve the latitude and longitude in the template and use Google Maps to display them. I’ll leave doing so as an exercise for the reader.

I can’t say that working with GeoDjango isn’t a bit of a struggle at times, but being able to make spatial queries in this fashion is very useful. With more and more people carrying smartphones, you’re more likely than ever to be asked to build applications that return data based on someone’s geographical location, and GeoDjango is a great way to do this with a Django application. You can find the source on Github.

18th March 2016 7:42 pm

My Experience Using PHP 7 in Production

In the last couple of weeks I’ve been working on a PHP web app. Nothing unusual there, except this was the first time we’d used PHP 7 in production. We discussed the possibility a while back, and eventually decided that for certain projects we’d use PHP 7 without waiting another year or so (or maybe longer) for a version of Debian stable with it by default. I wanted to talk about how our experience has been using it in production.

Background

We’ve never really had a fixed stack that we work with at work before until recently - it was largely based on personal preferences and experience. For many jobs, especially content-based sites, we generally used WordPress - it has its issues, but it does fine for a lot of work. For more complex websites, I tended to use CodeIgniter because I’d learned it during my previous job and knew it fairly well, but I was not terribly happy with it - it’s a bit too basic and simplistic, as well as being somewhat behind the times, and I only really kept using it through inertia. For mobile app backends, I tended to use Django, partly for the admin interface, and partly because Django REST Framework makes it easy to build a REST API quickly and easily in a way that wasn’t viable with CodeIgniter.

This state of affairs couldn’t really continue. I love Python and Django, but I was the only one at work who had ever used Python, so in the event I got hit by a bus there would have been no-one who could have taken over from me. As for CodeIgniter, it was clearly falling further and further behind the curve, and I was sick of it and looking to replace it. Ideally we needed a PHP framework as both myself and my colleague knew it.

I’d also been playing around with Laravel on a few little projects, but I didn’t get the chance to use it for a new web app until autumn last year. Around the same time, we hired a third developer, who also had some experience using Laravel. In addition, the presence of Lumen meant that we could use that for smaller apps or services that were too small to use Laravel. We therefore decided to adopt Laravel as our default framework - in future we’d only use something else if there was a particular justification for it. I was rather sad to have to abandon Django for work, but pleased to have something more modern than CodeIgniter for PHP projects.

This also enabled us to standardize our new server builds. Over the last year or so I’ve been pushing to automate what we can of our server setup using Ansible. We now have two standard stacks that we plan to use for future projects. One is for WordPress sites and consists of:

  • Debian stable
  • Apache
  • MySQL
  • PHP 5.6
  • Memcached
  • Varnish

The other is for Laravel or Lumen web apps or APIs and consists of:

  • Debian stable
  • Nginx
  • PHP 7
  • PostgreSQL
  • Redis

It took some time to decide what we wanted to settle on, and indeed we had a mobile app backend that went up around Christmas time that we wrote with Laravel, but deployed to Apache with PHP 5.6 because when we first pushed it up PHP 7 wasn’t out yet. However, given that Laravel 5 already had good support for PHP 7, we decided we’d consider it for the next app. I tend to use PostgreSQL rather than MySQL these days because it has a lot of nifty features like JSON fields and full text search, and using an ORM minimises the learning curve in switching, and Redis is much more versatile than Memcached, so they were vital parts of our stack.

Our first PHP 7 app

As it happened, we had a Laravel app in the pipeline that was ideal. In the summer of last year, we were hired to make an existing site responsive. In the end, it turned out not to be viable - it was built with Zend Framework, which none of us had ever touched before, and the front end used a lot of custom widgets and fields tied together with RequireJS. The whole thing was rather unwieldy and extremely difficult to maintain and develop. In the end, we decided to tell the client it wasn’t worth developing further and offer to rewrite the whole thing from scratch using Laravel and AngularJS, with Browserify used to handle JavaScript modules - the basic idea was quite simple, it was just the implementation that was overly complex, and AngularJS made it possible to do the same kind of thing with a fraction of the code, so a rewrite in only a few weeks was perfectly viable.

I’d already built a simple prototype to demonstrate the viability of a from-scratch rewrite using Laravel and Angular, and once the client had agreed to the rewrite, we were able to work on this further. As the web app was going to be particularly useful on mobile devices, I wanted to ensure that the performance was as good as I could possibly make it. By the time we were looking at deploying it to a server, three months had passed since PHP 7 had been first released, and I figured that was long enough for the most serious issues to be resolved, and we could definitely do with the very significant speed boost we’d get from using PHP 7 for this app.

I use Jenkins to run my unit tests, and so I decided to try installing PHP 7 on the Jenkins server and using that to run the tests. The results were encouraging - nothing broke as a result of the switch. So we therefore decided that when we deployed it, we’d try it with PHP 7, and if it failed, we’d switch to PHP 5.6.

I opted to use FPM with Nginx rather than Apache and mod_php as since the web app was purely custom we didn’t really need things like .htaccess, and while the amount of static content was limited, Nginx might well perform better for this use case. The results are fairly encouraging - the document for the home page is typically being returned in under 40ms, with the uncached homepage taking around 1.5s in total to load, despite having to load several external fonts. In its current state, the web app scores a solid 93% on YSlow, which I’m very happy with. I don’t know how much of that is down to using PHP 7, but choosing to use it was definitely a good call. I have had absolutely zero issues with it during that time.

Summary

As always, you should bear in mind that your needs may not be the same as mine, and it could well be that you need something that PHP 7 doesn’t yet provide. However, I have had a very good experience with PHP 7 in production. I may have had to jump through a few more hoops to get it up and running, and there may be some level of risk associated with using PHP 7 when it’s only been available for three months, but it’s more than justified by the speed we get from our web app. Using a configuration management system like Ansible means that even if you do have to jump through some extra hoops, it’s relatively easy to automate that process so it’s not as much of an issue as you might think. For me, using PHP 7 with a Laravel app has worked as well as I could have possibly hoped.

26th January 2016 11:40 pm

Mocking External Apis in Python

It’s quite common to have to integrate an external API into your web app for some of your functionality. However, it’s a really bad idea to have requests be sent to the remote API when running your tests. At best, it means your tests may fail due to unexpected circumstances, such as a network outage. At worst, you could wind up making requests to paid services that will cost you money, or sending push notifications to clients. It’s therefore a good idea to mock these requests in some way, but it can be fiddly.

In this post I’ll show you several ways you can mock an external API so as to prevent requests being sent when running your test suite. I’m sure there are many others, but these have worked for me recently.

Mocking the client library

Nowadays many third-party services realise that providing developers with client libraries in a variety of languages is a good idea, so it’s quite common to find a library for interfacing with a third-party service. Under these circumstances, the library itself is usually already thoroughly tested, so there’s no point in you writing additional tests for that functionality. Instead, you can just mock the client library so that the request is never sent, and if you need a response, then you can specify one that will remain constant.

I recently had to integrate Stripe with a mobile app backend, and I used their client library. I needed to ensure that I got the right result back. In this case I only needed to use the Token object’s create() method. I therefore created a new MockToken class that inherited from Token, and overrode its create() method so that it only accepted one card number and returned a hard-coded response for it:

from stripe.resource import Token, convert_to_stripe_object
from stripe.error import CardError
class MockToken(Token):
@classmethod
def create(cls, api_key=None, idempotency_key=None,
stripe_account=None, **params):
if params['card']['number'] != '4242424242424242':
raise CardError('Invalid card number', None, 402)
response = {
"card": {
"address_city": None,
"address_country": None,
"address_line1": None,
"address_line1_check": None,
"address_line2": None,
"address_state": None,
"address_zip": None,
"address_zip_check": None,
"brand": "Visa",
"country": "US",
"cvc_check": "unchecked",
"dynamic_last4": None,
"exp_month": 12,
"exp_year": 2017,
"fingerprint": "49gS1c4YhLaGEQbj",
"funding": "credit",
"id": "card_17XXdZGzvyST06Z022EiG1zt",
"last4": "4242",
"metadata": {},
"name": None,
"object": "card",
"tokenization_method": None
},
"client_ip": "192.168.1.1",
"created": 1453817861,
"id": "tok_42XXdZGzvyST06Z0LA6h5gJp",
"livemode": False,
"object": "token",
"type": "card",
"used": False
}
return convert_to_stripe_object(response, api_key, stripe_account)

Much of this was lifted straight from the source code for the library. I then wrote a test for the payment endpoint and patched the Token class:

class PaymentTest(TestCase):
@mock.patch('stripe.Token', MockToken)
def test_payments(self):
data = {
"number": '1111111111111111',
"exp_month": 12,
"exp_year": 2017,
"cvc": '123'
}
response = self.client.post(reverse('payments'), data=data, format='json')
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)

This replaced stripe.Token with MockToken so that in this test, the response from the client library was always going to be the expected one.

If the response doesn’t matter and all you need to do is be sure that the right method would have been called, this is easier. You can just mock the method in question using MagicMock and assert that it has been called afterwards, as in this example:

class ReminderTest(TestCase):
def test_send_reminder(self):
# Mock PushService.create_message()
PushService.create_message = mock.MagicMock(name="create_message")
# Call reminder task
send_reminder()
# Check user would have received a push notification
PushService.create_message.assert_called_with([{'text': 'My push', 'conditions': ['UserID', 'EQ', 1]}])

Mocking lower-level requests

Sometimes, no client library is available, or it’s not worth using one as you only have to make one or two requests. Under these circumstances, there are ways to mock the actual request to the external API. If you’re using the requests module, then there’s a responses module that’s ideal for mocking the API request.

Suppose we have the following code:

import json, requests
def send_request_to_api(data):
# Put together the request
params = {
'auth': settings.AUTH_KEY,
'data': data
}
response = requests.post(settings.API_URL, data={'params': json.dumps(params)})
return response

Using responses we can mock the response from the server in our test:

class APITest(TestCase):
@responses.activate
def test_send_request(self):
# Mock the API
responses.add(responses.POST,
settings.API_URL,
status=200,
content_type="application/json",
body='{"item_id": "12345678"}')
# Call function
data = {
"surname": "Smith",
"location": "London"
}
send_request_to_api(data)
# Check request went to correct URL
assert responses.calls[0].request.url == settings.API_URL

Note the use of the @responses.activate decorator. We use responses.add() to set up each URL we want to be able to mock, and pass through details of the response we want to return. We then make the request, and check that it was made as expected.

You can find more details of the responses module here.

Summary

I’m pretty certain that there are other ways you can mock an external API in Python, but these ones have worked for me recently. If you use another method, please feel free to share it in the comments.

18th November 2015 7:52 pm

Learning More About React.js and Flux

Udemy have very kindly provided some vouchers for free access to their course, “Build Web Apps with ReactJS and Flux” for me to give away to subscribers. To redeem them, follow the link above and use the voucher code MatthewDalysBlog.

There’s only 50 in total, and they are on a first-come, first-serve basis, so I suggest you redeem them sooner rather than later.

28th September 2015 8:00 pm

Building a Real-time Twitter Stream With Node.js, React.js and Redis

In the last year or so, React.js has taken the world of web development by storm. A major reason for this is that it makes it possible to build isomorphic web applications - web apps where the same code can run on the client and the server. Using React.js, you can create a template that will be executed on the server when the page first loads, and then the same template can be used to re-render the content when it’s updated, whether that’s via AJAX, WebSockets or another method entirely.

In this tutorial, I’ll show you how to build a simple Twitter streaming app using Node.js. I’m actually not the only person to have built this to demonstrate React.js, but this is my own particular take on this idea, since it’s such an obvious use case for React.

What is React.js?

A lot of people get rather confused over this issue. It’s not correct to compare React.js with frameworks like Angular.js or Backbone.js. It’s often described as being just the V in MVC - it represents only the view layer. If you’re familiar with Backbone.js, I think it’s reasonable to compare it to Backbone’s views, albeit with it’s own templating syntax. It does not provide the following functionality like Angular and Backbone do:

  • Support for models
  • Any kind of helpers for AJAX requests
  • Routing

If you want any of this functionality, you need to look elsewhere. There are other libraries around that offer this kind of functionality, so if you want to use React as part of some kind of MVC structure, you can do so - they’re just not a part of the library itself.

React.js uses a so-called “virtual DOM” - rather than re-rendering the view from scratch when the state changes, it instead retains a virtual representation of the DOM in memory, updates that, then figures out what changes are required to update the existing DOM and applies them. This means it only needs to change what actually changes, making it faster than other client-side templating systems. Combined with the ability to render on the server side, React allows you to build high-performance apps that combine the initial speed and SEO advantages of conventional web apps with the responsiveness of single-page web apps.

To create components with React, it’s common to use an XML-like syntax called JSX. It’s not mandatory, but I highly recommend you do so as it’s much more intuitive than creating elements with Javascript.

Getting started

You’ll need a Twitter account, and you’ll need to create a new Twitter app and obtain the security credentials to let you access the Twitter Streaming API. You’ll also need to have Node.js installed (ideally using nvm) - at this time, however, you can’t use Node 4.0 because of issues with Redis. You will also need to install Redis and hiredis - if you’ve worked through my previous Redis tutorials you’ll have these already.

We’ll be using Gulp.js as our build system, and Bower to install some client-side packages, so they need to be installed globally:

$ npm install -g gulp bower

We’ll also be using Compass to help with our stylesheets:

$ sudo gem install compass

With that all done, let’s start work on our app. First, run the following command to create your package.json:

$ npm init

I’m assuming you’re well-acquainted enough with Node.js to know what this does, and can answer the questions without difficulty. I won’t cover writing tests in this tutorial as, but set your test command to gulp test and you should be fine.

Next, we need to install our dependencies:

$ npm install --save babel compression express hbs hiredis lodash morgan react redis socket.io socket.io-client twitter
$ npm install --save-dev browserify chai gulp gulp-compass gulp-coveralls gulp-istanbul gulp-jshint gulp-mocha gulp-uglify jshint-stylish reactify request vinyl-buffer vinyl-source-stream

Planning our app

Now, it’s worth taking a few minutes to plan the architecture of our app. We want to have the app listen to the Twitter Streaming API and filter for messages with any arbitrary string in them - in this case we’ll be searching for “javascript”, but you can set it to anything you like. That means that that part needs to be listening all the time, not just when someone is using the app. Also, it doesn’t fit neatly into the usual request-response cycle - if several people visit the site at once, we could end up with multiple connections to fetch the same data, which is really not efficient, and could cause problems with duplicate tweets showing up.

Instead, we’ll have a separate worker.js file which runs constantly. This will listen for any matching messages on Twitter. When one appears, rather than returning it itself, it will publish it to a Redis channel, as well as persisting it. Then, the web app, which will be the index.js file, will be subscribed to the same channel, and will receive the tweet and push it to all current users using Socket.io.

This is a good example of a message queue, and it’s a common pattern. It allows you to create dedicated sections of your app for different tasks, and means that they will generally be more robust. In this case, if the worker goes down, users will still be able to see some tweets, and if the server goes down, the tweets will still be persisted to Redis. In theory, this would also allow you to scale your app more easily by allowing movement of different tasks to different servers, and several app servers could interface with a single worker process. The only downside I can think of is that on a platform like Heroku you’d need to have a separate dyno for the worker process - however, with Heroku’s pricing model changing recently, since this needs to be listening all the time it won’t be suitable for the free tier anyway.

First let’s create our gulpfile.js:

var gulp = require('gulp');
var jshint = require('gulp-jshint');
var source = require('vinyl-source-stream');
var buffer = require('vinyl-buffer');
var browserify = require('browserify');
var reactify = require('reactify');
var mocha = require('gulp-mocha');
var istanbul = require('gulp-istanbul');
var coveralls = require('gulp-coveralls');
var compass = require('gulp-compass');
var uglify = require('gulp-uglify');
var paths = {
scripts: ['components/*.jsx'],
styles: ['src/sass/*.scss']
};
gulp.task('lint', function () {
return gulp.src([
'index.js',
'components/*.js'
])
.pipe(jshint())
.pipe(jshint.reporter('jshint-stylish'));
});
gulp.task('compass', function() {
gulp.src('src/sass/*.scss')
.pipe(compass({
css: 'static/css',
sass: 'src/sass'
}))
.pipe(gulp.dest('static/css'));
});;
gulp.task('test', function () {
gulp.src('index.js')
.pipe(istanbul())
.pipe(istanbul.hookRequire())
.on('finish', function () {
gulp.src('test/test.js', {read: false})
.pipe(mocha({ reporter: 'spec' }))
.pipe(istanbul.writeReports({
reporters: [
'lcovonly',
'cobertura',
'html'
]
}))
.pipe(istanbul.enforceThresholds({ thresholds: { global: 90 } }))
.once('error', function () {
process.exit(0);
})
.once('end', function () {
process.exit(0);
});
});
});
gulp.task('coveralls', function () {
gulp.src('coverage/lcov.info')
.pipe(coveralls());
});
gulp.task('react', function () {
return browserify({ entries: ['components/index.jsx'], debug: true })
.transform(reactify)
.bundle()
.pipe(source('bundle.js'))
.pipe(buffer())
.pipe(uglify())
.pipe(gulp.dest('static/jsx/'));
});
gulp.task('default', function () {
gulp.watch(paths.scripts, ['react']);
gulp.watch(paths.styles, ['compass']);
});

I’ve added tasks for the tests and JSHint if you choose to implement them, but the only ones I’ve actually used are the compass and react tasks. The compass task compiles our Sass files into CSS, while the react task uses Browserify to take our React components and various modules installed using NPM and build them for use in the browser, as well as minifying them. Note that we installed React and lodash with NPM? We’re going to be able to use them in the browser and on the server, thanks to Browserify.

Next, let’s create our worker.js file:

/*jslint node: true */
'use strict';
// Get dependencies
var Twitter = require('twitter');
// Set up Twitter client
var client = new Twitter({
consumer_key: process.env.TWITTER_CONSUMER_KEY,
consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
access_token_key: process.env.TWITTER_ACCESS_TOKEN_KEY,
access_token_secret: process.env.TWITTER_ACCESS_TOKEN_SECRET
});
// Set up connection to Redis
var redis;
if (process.env.REDIS_URL) {
redis = require('redis').createClient(process.env.REDIS_URL);
} else {
redis = require('redis').createClient();
}
client.stream('statuses/filter', {track: 'javascript', lang: 'en'}, function(stream) {
stream.on('data', function(tweet) {
// Log it to console
console.log(tweet);
// Publish it
redis.publish('tweets', JSON.stringify(tweet));
// Persist it to a Redis list
redis.rpush('stream:tweets', JSON.stringify(tweet));
});
// Handle errors
stream.on('error', function (error) {
console.log(error);
});
});

Most of this file should be fairly straightforward. We set up our connection to Twitter (you’ll need to set the various environment variables listed here using the appropriate method for your operating system), and a connection to Redis.

We then stream the Twitter statuses that match our filter. When we receive a tweet, we log it to the console (feel free to comment this out in production if desired), publish it to a Redis channel called tweets, and push it to the end of a Redis list called stream:tweets. When an error occurs, we output it to the console.

Let’s use Bootstrap to style the app. Create the following .bowerrc file:

{
"directory": "static/bower_components"
}

Then run bower init to create your bower.json file, and install Bootstrap with bower install --save sass-bootstrap.

With that done, create the file src/sass/style.scss and enter the following:

@import "compass/css3/user-interface";
@import "compass/css3";
@import "../../static/bower_components/sass-bootstrap/lib/bootstrap.scss";

This includes some dependencies from Compass, as well as Bootstrap. We won’t be using any of the Javascript features of Bootstrap, so we don’t need to worry too much about that.

Next, we need to create our view files. As React will be used to render the main part of the page, these will be very basic, with just the header, footer, and a section where the content can be rendered. First, create views/index.hbs:

{{> header }}
<div class="container">
<div class="row">
<div class="col-md-12">
<div id='view'>{{{ markup }}}</div>
</div>
</div>
</div>
<script id="initial-state" type="application/json">{{{state}}}</script>
{{> footer }}

As promised, this a very basic layout. Note the markup variable, which is where the markup generated by React will be inserted when rendered on the server, and the state variable, which will contain the JSON representation of the data used to generate that markup. By passing that data through, you can ensure that the instance of React on the client has access to the same raw data as was passed through to the view on the server side, so that when the data needs to be re-rendered, it can be done so correctly.

We’ll also define partials for the header and footer. The header should be in views/partials/header.hbs:

<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Tweet Stream</title>
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<!-- Place favicon.ico and apple-touch-icon.png in the root directory -->
<link rel="stylesheet" type="text/css" href="/css/style.css">
</head>
<body>
<!--[if lt IE 7]>
<p class="browsehappy">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p>
<![endif]-->
<nav class="navbar navbar-inverse navbar-static-top" role="navigation">
<div class="container-fluid">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#header-nav">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="/">Tweet Stream</a>
<div class="collapse navbar-collapse navbar-right" id="header-nav">
</div>
</div>
</div>
</nav>

The footer should be in views/partials/footer.hbs:

<script src="/jsx/bundle.js"></script>
</body>
</html>

Note that we load the Javascript file /jsx/bundle.js - this is the output from the command gulp react.

Creating the back end

The next step is to implement the back end of the website. Add the following code as index.js:

/*jslint node: true */
'use strict';
require('babel/register');
// Get dependencies
var express = require('express');
var app = express();
var compression = require('compression');
var port = process.env.PORT || 5000;
var base_url = process.env.BASE_URL || 'http://localhost:5000';
var hbs = require('hbs');
var morgan = require('morgan');
var React = require('react');
var Tweets = React.createFactory(require('./components/tweets.jsx'));
// Set up connection to Redis
var redis, subscribe;
if (process.env.REDIS_URL) {
redis = require('redis').createClient(process.env.REDIS_URL);
subscribe = require('redis').createClient(process.env.REDIS_URL);
} else {
redis = require('redis').createClient();
subscribe = require('redis').createClient();
}
// Set up templating
app.set('views', __dirname + '/views');
app.set('view engine', "hbs");
app.engine('hbs', require('hbs').__express);
// Register partials
hbs.registerPartials(__dirname + '/views/partials');
// Set up logging
app.use(morgan('combined'));
// Compress responses
app.use(compression());
// Set URL
app.set('base_url', base_url);
// Serve static files
app.use(express.static(__dirname + '/static'));
// Render main view
app.get('/', function (req, res) {
// Get tweets
redis.lrange('stream:tweets', 0, -1, function (err, tweets) {
if (err) {
console.log(err);
} else {
// Get tweets
var tweet_list = [];
tweets.forEach(function (tweet, i) {
tweet_list.push(JSON.parse(tweet));
});
// Render page
var markup = React.renderToString(Tweets({ data: tweet_list.reverse() }));
res.render('index', {
markup: markup,
state: JSON.stringify(tweet_list)
});
}
});
});
// Listen
var io = require('socket.io')({
}).listen(app.listen(port));
console.log("Listening on port " + port);
// Handle connections
io.sockets.on('connection', function (socket) {
// Subscribe to the Redis channel
subscribe.subscribe('tweets');
// Handle receiving messages
var callback = function (channel, data) {
socket.emit('message', data);
};
subscribe.on('message', callback);
// Handle disconnect
socket.on('disconnect', function () {
subscribe.removeListener('message', callback);
});
});

Let’s go through this bit by bit:

/*jslint node: true */
'use strict';
require('babel/register');

Here we’re using Babel, which is a library that allows you to use new features in Javascript even if the interpreter doesn’t support it. It also includes support for JSX, allowing us to require JSX files in the same way we would require Javascript files.

// Get dependencies
var express = require('express');
var app = express();
var compression = require('compression');
var port = process.env.PORT || 5000;
var base_url = process.env.BASE_URL || 'http://localhost:5000';
var hbs = require('hbs');
var morgan = require('morgan');
var React = require('react');
var Tweets = React.createFactory(require('./components/tweets.jsx'));

Here we include our dependencies. Most of this will be familiar if you’ve used Express before, but we also use React to create a factory for a React component called Tweets.

// Set up connection to Redis
var redis, subscribe;
if (process.env.REDIS_URL) {
redis = require('redis').createClient(process.env.REDIS_URL);
subscribe = require('redis').createClient(process.env.REDIS_URL);
} else {
redis = require('redis').createClient();
subscribe = require('redis').createClient();
}
// Set up templating
app.set('views', __dirname + '/views');
app.set('view engine', "hbs");
app.engine('hbs', require('hbs').__express);
// Register partials
hbs.registerPartials(__dirname + '/views/partials');
// Set up logging
app.use(morgan('combined'));
// Compress responses
app.use(compression());
// Set URL
app.set('base_url', base_url);
// Serve static files
app.use(express.static(__dirname + '/static'));

This section sets up the various dependencies of our app. We set up two connections to Redis - one for handling subscriptions, the other for reading from Redis in order to populate the view.

We also set up our views, logging, compression of the HTTP response, a base URL, and serving static files.

// Render main view
app.get('/', function (req, res) {
// Get tweets
redis.lrange('stream:tweets', 0, -1, function (err, tweets) {
if (err) {
console.log(err);
} else {
// Get tweets
var tweet_list = [];
tweets.forEach(function (tweet, i) {
tweet_list.push(JSON.parse(tweet));
});
// Render page
var markup = React.renderToString(Tweets({ data: tweet_list.reverse() }));
res.render('index', {
markup: markup,
state: JSON.stringify(tweet_list)
});
}
});
});

Our app only has a single view. When the root is loaded, we first of all fetch all of the tweets stored in the stream:tweets list. We then convert them into an array of objects.

Next, we render the Tweets component to a string, passing through our list of tweets, and store the resulting markup. We then pass through this markup and the string representation of the list of tweets to the template.

// Listen
var io = require('socket.io')({
}).listen(app.listen(port));
console.log("Listening on port " + port);
// Handle connections
io.sockets.on('connection', function (socket) {
// Subscribe to the Redis channel
subscribe.subscribe('tweets');
// Handle receiving messages
var callback = function (channel, data) {
socket.emit('message', data);
};
subscribe.on('message', callback);
// Handle disconnect
socket.on('disconnect', function () {
subscribe.removeListener('message', callback);
});
});

Finally, we set up Socket.io. On a connection, we subscribe to the Redis channel tweets. When we receive a tweet from Redis, we emit that tweet so that it can be rendered on the client side. We also handle disconnections by removing our Redis subscription.

Creating our React components

Now it’s time to create our first React component. We’ll create a folder called components to hold all of our component files. Our first file is components/index.jsx:

var React = require('react');
var Tweets = require('./tweets.jsx');
var initialState = JSON.parse(document.getElementById('initial-state').innerHTML);
React.render(
<Tweets data={initialState} />,
document.getElementById('view')
);

First of all, we include React and the same Tweets component we require on the server side (note that we need to specify the .jsx extension). Then we fetch the initial state from the script tag we created earlier. Finally we render the Tweets components, passing through the initial state, and specify that it should be inserted into the element with an id of view. Note that we store the initial state in data - inside the component, this can be accessed as this.props.data.

This particular component is only ever used on the client side - when we render on the server side, we don’t need any of this functionality since we insert the markup into the view element anyway, and we don’t need to specify the initial data in the same way.

Next, we define the Tweets component in components/tweets.jsx:

var React = require('react');
var io = require('socket.io-client');
var TweetList = require('./tweetlist.jsx');
var _ = require('lodash');
var Tweets = React.createClass({
componentDidMount: function () {
// Get reference to this item
var that = this;
// Set up the connection
var socket = io.connect(window.location.href);
// Handle incoming messages
socket.on('message', function (data) {
// Insert the message
var tweets = that.props.data;
tweets.push(JSON.parse(data));
tweets = _.sortBy(tweets, function (item) {
return item.created_at;
}).reverse();
that.setProps({data: tweets});
});
},
getInitialState: function () {
return {data: this.props.data};
},
render: function () {
return (
<div>
<h1>Tweets</h1>
<TweetList data={this.props.data} />
</div>
)
}
});
module.exports = Tweets;

Let’s work our way through each section in turn:

var React = require('react');
var io = require('socket.io-client');
var TweetList = require('./tweetlist.jsx');
var _ = require('lodash');

Here we include React and the Socket.io client, as well as Lodash and our TweetList component. With React.js, it’s recommend that you break up each individual part of your interface into a single component - here Tweets is a wrapper for the tweets that includes a heading. TweetList will be a list of tweets, and TweetItem will be an individual tweet.

var Tweets = React.createClass({
componentDidMount: function () {
// Get reference to this item
var that = this;
// Set up the connection
var socket = io.connect(window.location.href);
// Handle incoming messages
socket.on('message', function (data) {
// Insert the message
var tweets = that.props.data;
tweets.push(JSON.parse(data));
tweets = _.sortBy(tweets, function (item) {
return item.created_at;
}).reverse();
that.setProps({data: tweets});
});
},

Note the use of the componentDidMount method - this fires when a component has been rendered on the client side for the first time. You can therefore use it to set up events. Here, we’re setting up a callback so that when a new tweet is received, we get the existing tweets (stored in this.props.data, although we copy this to that so it works inside the callback), push the tweet to this list, sort it by the time created, and set this.props.data to the new value. This will result in the tweets being re-rendered.

getInitialState: function () {
return {data: this.props.data};
},

Here we set the initial state of the component - it sets the value of this.state to the object passed through. In this case, we pass through an object with the attribute data defined as the value of this.props.data, meaning that this.state.data is the same as this.props.data.

render: function () {
return (
<div>
<h1>Tweets</h1>
<TweetList data={this.props.data} />
</div>
)
}
});
module.exports = Tweets;

Here we define our render function. This can be thought of as our template. Note that we include TweetList inside our template and pass through the data. Afterwards, we export Tweets so it can be used elsewhere.

Next, let’s create components/tweetlist.jsx:

var React = require('react');
var TweetItem = require('./tweetitem.jsx');
var TweetList = React.createClass({
render: function () {
var that = this;
var tweetNodes = this.props.data.map(function (item, index) {
return (
<TweetItem key={index} text={item.text}></TweetItem>
);
});
return (
<ul className="tweets list-group">
{tweetNodes}
</ul>
)
}
});
module.exports = TweetList;

This component is much simpler - it only has a render method. First, we get our individual tweets and for each one define a TweetItem component. Then we create an unordered list and insert the tweet items into it. We then export it as TweetList.

Our final component is the TweetItem component. Create the following file at components/tweetitem.jsx:

var React = require('react');
var TweetItem = React.createClass({
render: function () {
return (
<li className="list-group-item">{this.props.text}</li>
);
}
});
module.exports = TweetItem;

This component is quite simple. It’s just a single list item with the text set to the value of the tweet’s text attribute.

That should be all of our components done. Time to compile our Sass and run Browserify:

$ gulp compass
$ gulp react

Now, if you make sure you have set the appropriate environment variables, and then run node worker.js in one terminal, and node index.js in another, and visit http://localhost:5000/, you should see your Twitter stream in all its glory! You can also try it with Javascript disabled, or in a text-mode browser such as Lynx, to demonstrate that it still renders the page without having to do anything on the client side - you’re only missing the constant updates.

Wrapping up

I hope this gives you some idea of how you can easily use React.js on both the client and server side to make web apps that are fast and search-engine friendly while also being easy to update dynamically. You can find the source code on GitHub.

Hopefully I’ll be able to publish some later tutorials that build on this to show you how to build more substantial web apps with React.

Recent Posts

Skipping Environment Specific Phpunit Tests

Powering Up Git Bisect With the Run Command

Writing Golden Master Tests for Laravel Applications

How Much Difference Does Adding An Index to a Database Table Make?

Searching Content With Fuse.js

About me

I'm a web and mobile app developer based in Norfolk. My skillset includes Python, PHP and Javascript, and I have extensive experience working with CodeIgniter, Laravel, Zend Framework, Django, Phonegap and React.js.