Matthew Daly's Blog

I'm a web developer in Norfolk. This is my blog...

17th June 2015 8:34 pm

Getting Django-behave and Celery to Work Together

I ran into a small issue today. I’m working on a Django app which uses Celery to handle certain tasks that don’t need to return a response within the context of the HTTP request. I also wanted to use django_behave for running BDD tests. The trouble is that both django_behave and Celery provide their own custom test runners that extend the default Django test runner, and so it looked like I might have to choose between the two.

However, it turned out that the Celery one was actually very simple, with only a handful of changes needing to be made to the default test runner to make it work with Celery. I was therefore able to create my own custom test runner that inherited from DjangoBehaveTestSuiteRunner and applied the changes necessary to get Celery working with it. Here is the test runner I wrote, which was saved as myproject/runner.py:

from django.conf import settings
from djcelery.contrib.test_runner import _set_eager
from django_behave.runner import DjangoBehaveTestSuiteRunner
class CeleryAndBehaveRunner(DjangoBehaveTestSuiteRunner):
def setup_test_environment(self, **kwargs):
_set_eager()
settings.BROKER_BACKEND = 'memory'
super(CeleryAndBehaveRunner, self).setup_test_environment(**kwargs)

To use it, you need to set the test runner in settings.py

TEST_RUNNER = 'myproject.runner.CeleryAndBehaveRunner'

Once that was done, my tests worked flawlessly with Celery, and the Behave tests ran as expected.

14th June 2015 9:29 pm

Setting Etags in Laravel 5

Although I’d prefer to use Python or Node.js, there are some times when circumstances dictate that I need to use PHP for a project at work. In the past, I used CodeIgniter, but that was through nothing more than inertia. For some time I’d been planning to switch to Laravel, largely because of the baked-in PHPUnit support, but events conspired against me - one big project that came along had a lot in common with an earlier one, so I forked it rather than starting over.

Recently I built a REST API for a mobile app, and I decided to use that to try out Laravel (if it had been available at the time, I’d have gone for Lumen instead). I was very pleased with the results - I was able to quickly put together the back end I wanted, with good test coverage, and the tinker command in particular was useful in debugging. The end result is fast and efficient, with query caching in place using Memcached to improve response times.

I also implemented a simple middleware to add ETags to HTTP responses and compare them on incoming requests, returning a 304 Not Modified status code if they are the same, which is given below:

<?php namespace App\Http\Middleware;
use Closure;
class ETagMiddleware {
/**
* Implement Etag support
*
* @param \Illuminate\Http\Request $request
* @param \Closure $next
* @return mixed
*/
public function handle($request, Closure $next)
{
// Get response
$response = $next($request);
// If this was a GET request...
if ($request->isMethod('get')) {
// Generate Etag
$etag = md5($response->getContent());
$requestEtag = str_replace('"', '', $request->getETags());
// Check to see if Etag has changed
if($requestEtag && $requestEtag[0] == $etag) {
$response->setNotModified();
}
// Set Etag
$response->setEtag($etag);
}
// Send response
return $response;
}
}

This is based on a solution for Laravel 4 by Nick Verwymeren, but implemented as Laravel 5 middleware, not a Laravel 4 filter. To use this with Laravel 5, save this as app/Http/Middleware/ETagMiddleware.php. Then add this to the $middleware array in app/Http/Kernel.php:

        'App\Http\Middleware\ETagMiddleware',

It’s quite simple to write this kind of middleware with Laravel, and using something like this is a no-brainer for most web apps considering the bandwidth it will likely save your users.

18th April 2015 3:05 pm

How I Added Search to My Site With Lunr.js

As I mentioned a while back, I recently switched the search on my site from Google’s site-specific search to Lunr.js. Since my site is built with a static site generator, I can’t implement search using database queries, and I was keen to have an integrated search method that would be fast and not require server-side scripting, and Lunr.js seemed to fit the bill.

The first task in implementing it was to generate the index. As I wrote the Grunt task that generates the blog, I amended that task to generate an index at the same time as I generated the posts. I installed Lunr.js with the following command:

npm install lunr --save

I then imported it in the task, and set up the field names:

var lunr = require('lunr');
searchIndex = lunr(function () {
this.field('title', { boost: 10 });
this.field('body');
this.ref('href');
});

This defined fields for the title, body, and hyperlink, and set the hyperlink as the reference. The variable searchIndex represents the Lunr index.

Next, I looped through the posts, and passed the appropriate details to be added to the index:

for (post in post_items) {
var doc = {
'title': post_items[post].meta.title,
'body': post_items[post].post.rawcontent,
'href': post_items[post].path
};
store[doc.href] = {
'title': doc.title
};
searchIndex.add(doc);
}

At this point, post_items represents an array of objects, with each object representing a blog post. Note that the body field is set to the value of the item’s attribute post.rawcontent, which represents the raw Markdown rather than the compiled HTML.

I then store the title in the store object, so that it can be accessed using the href field as a key.

I then do the same thing when generating the pages:

// Add them to the index
var doc = {
'title': data.meta.title,
'body': data.post.rawcontent,
'href': permalink + '/'
};
store[doc.href] = {
'title': data.meta.title
};
searchIndex.add(doc);

Note that this is already inside the loop that generates the pages, so I don’t include that.

We then write the index to a file:

// Write index
grunt.file.write(options.www.dest + '/lunr.json', JSON.stringify({
index: searchIndex.toJSON(),
store: store
}));

That takes care of generating our index, but we need to implement some client-side code to handle the search. We need to include Lunr.js on the client side as well, (I recommend using Bower to do so), alongside jQuery. If you include both, the following code should do the trick:

$(document).ready(function () {
'use strict';
// Set up search
var index, store;
$.getJSON('/lunr.json', function (response) {
// Create index
index = lunr.Index.load(response.index);
// Create store
store = response.store;
// Handle search
$('input#search').on('keyup', function () {
// Get query
var query = $(this).val();
// Search for it
var result = index.search(query);
// Output it
var resultdiv = $('ul.searchresults');
if (result.length === 0) {
// Hide results
resultdiv.hide();
} else {
// Show results
resultdiv.empty();
for (var item in result) {
var ref = result[item].ref;
var searchitem = '<li><a href="' + ref + '">' + store[ref].title + '</a></li>';
resultdiv.append(searchitem);
}
resultdiv.show();
}
});
});
});

This should be easy to understand. On load, we fetch and parse the lunr.json file from the server, and load the index. We then set up an event handler for the keyup event on an input with the ID of search. We get the value of the input, and query our index, and we loop through our results and display them.

I was pleased with how straightforward it was to implement search with Lunr.js, and it works well. It’s also a lot faster than any server-side solution since the index is generated during the build process, and is loaded with the rest of the site, so the only factor in the speed of the response is how quick your browser executes JavaScript. You could probably also use it with a Node.js application by generating the index dynamically, although you’d probably want to cache it to some extent.

4th April 2015 1:47 am

Adding a New Search Engine to My Site

I’ve just finished implementing a new search engine for this site. Obviously, with it using a static site generator, searching a relational database isn’t an option. For a long while I’d just been getting by with Google’s site-specific search, which worked, but meant leaving the site to view the search results.

Now, I’ve implemented a client-side search system using Lunr.js. It wasn’t too time consuming, and as the index is generated with the rest of the site and loaded with the page, the response is almost instantaneous. I may write a future blog post on how to integrate Lunr.js with your site, as it’s very handy and is an ideal solution for implementing search on a static site.

Recent Posts

Mutation Testing With Infection

Switching from Vim to Neovim

Better Strings in PHP

Forcing SSL in Codeigniter

Logging to the ELK Stack With Laravel

About me

I'm a web and mobile app developer based in Norfolk. My skillset includes Python, PHP and Javascript, and I have extensive experience working with CodeIgniter, Laravel, Django, Phonegap and Angular.js.