Matthew Daly's Blog

I'm a web developer in Norfolk. This is my blog...

29th October 2017 7:31 pm

An Azure Filesystem Integration for Laravel

My earlier post about integrating Laravel and Azure storage seems to have become something of a go-to resource on this subject (I suspect this is because very few developers actually use Laravel and Azure together). Unfortunately it hasn’t really aged terribly well - changes to the namespace and to Guzzle mean that it needs some work to integrate it.

I’ve therefore created a package for it. That way, it’s easier to keep it up to date as if someone finds and fixes an issue with it, they can submit their changes back.

20th October 2017 10:55 pm

Using Phpiredis With Laravel

Laravel has support out of the box for using Redis. However, by default it uses a Redis client written in PHP, which will always be a little slower than one written in C. If you’re making heavy use of Redis, it may be worth using the phpiredis extension to squeeze a little more performance out of it.

I’m using PHP 7.0 on Ubuntu Zesty and I installed the dependencies with the following command:

$ sudo apt-get install libhiredis-dev php-redis php7.0-dev

Then I installed phpiredis as follows:

git clone https://github.com/nrk/phpiredis.git && \
cd phpiredis && \
phpize && \
./configure --enable-phpiredis && \
make && \
sudo make install

Finally, I configured Redis to use phpiredis in the redis section of config/database.php for a Laravel app:

'redis' => [
'cluster' => false,
'default' => [
'host' => env('REDIS_HOST', 'localhost'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
'options' => [
'connections' => [
'tcp' => 'Predis\Connection\PhpiredisStreamConnection', // PHP streams
'unix' => 'Predis\Connection\PhpiredisSocketConnection', // ext-socket
],
]
],
],

Now, I’m going to be honest - in a casual comparison I couldn’t see much difference in terms of speed. I would probably only bother with setting this up on a site where high Redis performance was absolutely necessary. If you just want a quicker cache response it might make more sense to put Varnish in front of the site instead. However, in cases where Redis gets used heavily, it’s probably worth doing.

3rd October 2017 11:56 pm

Simple Fuzzy Search With Laravel and Postgresql

When implementing fuzzy search, many developers reach straight for specialised tools like Elasticsearch. However, for simple implementations, this is often overkill. PostgreSQL, my relational database of choice, can natively handle fuzzy search quite easily if you know how. Here’s how you might use this with Laravel.

Suppose we have the following migration to create a locations table, storing towns, cities and villages:

<?php
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class CreateLocations extends Migration
{
/**
* Run the migrations.
*
* @return void
*/
public function up()
{
// Create locations table
Schema::create('locations', function (Blueprint $table) {
$table->increments('id')->unsigned();
$table->string('name');
$table->timestamps();
});
}
/**
* Reverse the migrations.
*
* @return void
*/
public function down()
{
// Drop locations table
Schema::drop('locations');
}
}

The key to this implementation of fuzzy search is trigrams. A trigram is a group of three consecutive characters taken from a string. Using the pg_trgm module, which comes with PostgreSQL, we can break a string into as many trigrams as possible, and then return the strings with the most matching trigrams.

We can ensure that pg_trgm is set up on the database by creating a migration:

<?php
use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class AddTrgmExtension extends Migration
{
/**
* Run the migrations.
*
* @return void
*/
public function up()
{
DB::statement('CREATE EXTENSION IF NOT EXISTS pg_trgm');
}
/**
* Reverse the migrations.
*
* @return void
*/
public function down()
{
DB::statement('DROP EXTENSION IF EXISTS pg_trgm');
}
}

Make sure you run the migration as well. Once that is done, we can make a raw fuzzy query against the name field as follows:

SELECT * FROM locations WHERE 'burgh' % name;

Translating that to work with the Eloquent ORM, we can perform fuzzy queries against the name field as follows:

$location = Location::whereRaw("'burgh' % name")->get();

This query might match both Aldeburgh and Edinburgh. It’s also able to handle slight misspellings, as in this example:

$location = Location::whereRaw("'hendrad' % name")->get();

This query will match East Hendred or West Hendred successfully. As you can see, we can match strings at any point in the name string, and handle slight mis-spellings without any problems.

In practice, rather than using whereRaw() every time, you’ll probably want to create a local scope that accepts the name you want to match against. You’ll also want to use query parameters to prevent SQL injection:

$location = Location::whereRaw("? % name", [$name])->get();

Improving performance with an index

The performance of these queries isn’t that great out of the box. We can improve them by creating an index:

<?php
use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class AddTrgmExtension extends Migration
{
/**
* Run the migrations.
*
* @return void
*/
public function up()
{
DB::statement('CREATE EXTENSION IF NOT EXISTS pg_trgm');
DB::statement('CREATE INDEX locations_name_trigram ON locations USING gist(name gist_trgm_ops);');
}
/**
* Reverse the migrations.
*
* @return void
*/
public function down()
{
DB::statement('DROP INDEX IF EXISTS locations_name_trigram');
DB::statement('DROP EXTENSION IF EXISTS pg_trgm');
}
}

Adding an index should produce a noticeable improvement in the response time.

Final thoughts

PostgreSQL’s pg_trgm module is a fairly straightforward way of implementing fuzzy search. It’s not much more involved than a LIKE or ILIKE clause in your query, and for many use cases, it’s more than sufficient. If you don’t have a huge number of records, it’s probably a more appropriate choice than something like Elasticsearch, and has the advantage of a simpler stack. However, if you have a larger dataset, you may be better off with a dedicated search solution. As always, if you’re unsure it’s a good idea to try both and see what works best for that particular use case.

25th September 2017 10:18 pm

A Generic PHP SMS Library

This weekend I published sms-client, a generic PHP library for sending SMS notifications. It’s intended to offer a consistent interface when sending SMS notifications by using swappable drivers. That way, if your SMS service provider suddenly goes out of business or bumps up their prices, it’s easy to switch to a new one.

Out of the box it comes with drivers for the following services:

  • Nexmo
  • ClockworkSMS

In addition, it provides the following test drivers:

  • Null
  • Log
  • RequestBin

Here’s an example of how you might use it with the ClockworkSMS driver:

use GuzzleHttp\Client as GuzzleClient;
use GuzzleHttp\Psr7\Response;
use Matthewbdaly\SMS\Drivers\Clockwork;
use Matthewbdaly\SMS\Client;
$guzzle = new GuzzleClient;
$resp = new Response;
$driver = new Clockwork($guzzle, $resp, [
'api_key' => 'MY_CLOCKWORK_API_KEY',
]);
$client = new Client($driver);
$msg = [
'to' => '+44 01234 567890',
'content' => 'Just testing',
];
$client->send($msg);

If you want to roll your own driver for it, it should be easy - just create a class that implements the Matthewbdaly\SMS\Contracts\Driver interface. Most of the existing drivers work using Guzzle to send HTTP requests to an API, but you don’t necessarily have to do that - for instance, you could create a driver for a mail-to-SMS gateway by using Swiftmailer or the PHP mail class. If you create a driver for it, please feel free to submit a pull request so I can add it to the repository.

For Laravel or Lumen users, there’s an integration package that should make it easier to use. For users of other frameworks, it should still be fairly straightforward to integrate.

8th September 2017 10:05 pm

Installing Nginx Unit on Ubuntu

Recently Nginx announced the release of the first beta of Unit, an application server that supports Python, PHP and Go, with support coming for Java, Node.js and Ruby.

The really interesting part is that not only does it support more than one language, but Unit can be configured by making HTTP requests, rather than by editing config files. This makes it potentially very interesting to web developers like myself who have worked in multiple languages - I could use it to serve a Python or PHP web app, simply by making different requests during the setup process. I can see this being a boon for SaaS providers - you could pick up the language from a file, much like the runtime.txt used by Heroku, and set up the application on the fly.

It’s currently in public beta, and there are packages for Ubuntu, so I decided to try it out. I’ve created the Ansible role below to set up Unit on an Ubuntu 16.04 server or VM:

---
- name: Install keys
apt_key: url=http://nginx.org/keys/nginx_signing.key state=present
- name: Setup main repo
apt_repository: repo='deb http://nginx.org/packages/mainline/ubuntu/ xenial nginx' state=present
- name: Setup source rep
apt_repository: repo='deb-src http://nginx.org/packages/mainline/ubuntu/ xenial nginx' state=present
- name: Update system
apt: upgrade=full update_cache=yes
- name: Install dependencies
apt: name={{ item }} state=present
with_items:
- nginx
- unit
- golang
- php-dev
- php7.0-dev
- libphp-embed
- libphp7.0-embed
- python-dev
- python3
- python3-dev
- php7.0-cli
- php7.0-mcrypt
- php7.0-pgsql
- php7.0-sqlite3
- php7.0-opcache
- php7.0-curl
- php7.0-mbstring
- php7.0-dom
- php7.0-xml
- php7.0-zip
- php7.0-bcmath
- name: Copy over Nginx configuration
copy: src=nginx.conf dest=/etc/nginx/sites-available/default owner=root group=root mode=0644

Note the section that copies over the Nginx config file. Here is that file:

upstream unit_backend {
server 127.0.0.1:8300;
}
server {
listen 80 default_server;
listen [::]:80 default_server ipv6only=on;
fastcgi_param HTTP_PROXY "";
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
root /var/www/public;
index index.php index.html index.htm;
server_name server_domain_or_IP;
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location ~ \.php$ {
try_files $uri /index.php =404;
proxy_pass http://unit_backend;
proxy_set_header Host $host;
}
}

This setup proxies all dynamic requests to the Unit backend in a similar fashion to how it would normally pass it to PHP-FPM.

There were still a few little issues. It doesn’t exactly help that the Nginx package provided with this repository isn’t quite the same as the one in Ubuntu by default - not only is it the unstable version, but it doesn’t set up the sites-available and sites-enabled folders, so I had to do that manually. I also had an issue with Systemd starting the process (at /run/control.unit.sock) with permissions that didn’t allow Nginx to access it. I’m not that familiar with Systemd so I wound up just setting the permissions of the file manually, but that doesn’t persist between restarts. I expect this issue isn’t that big a deal to someone more familiar with Systemd, but I haven’t been able to resolve it yet.

I decided to try it out with a Laravel application. I created a new Laravel app and set it up with the web root at /var/www. I then saved the following configuration for it as app.json:

{
"listeners": {
"*:8300": {
"application": "myapp"
}
},
"applications": {
"myapp": {
"type": "php",
"workers": 20,
"user": "www-data",
"group": "www-data",
"root": "/var/www/public",
"index": "index.php"
}
}
}

This is fairly basic, but a good example of how you configure an application with Unit. The listener section maps a port to an application, while the applications section defines an application called myapp. In this case, we specify that the type should be php. Note that each platform has slightly different options - for instance, the Python type doesn’t have the index or root options, instead having the path option, which specifies the path to the wsgi.py file.

I then ran the following command to upload the file:

$ curl -X PUT -d @app.json --unix-socket /run/control.unit.sock http://localhost

Note that we send it direct to the Unix socket file - this way we don’t have to expose the API to the outside. After this was done, the Laravel app began working as expected.

We can then make a GET request to view the configured applications:

$ curl --unix-socket /run/control.unit.sock http://localhost/
{
"listeners": {
"*:8300": {
"application": "saas"
}
},
"applications": {
"saas": {
"type": "php",
"workers": 20,
"user": "www-data",
"group": "www-data",
"root": "/var/www/public",
"index": "index.php"
}
}
}

It’s also possible to update and delete existing applications via the API using PUT and DELETE requests.

Final thoughts

This is way too early to be seriously considering using Unit in production. It’s only just been released as a public beta, and it’s a bit fiddly to set up. However, it has an enormous amount of promise.

One thing I can’t really see right now is whether it’s possible to use a virtualenv with it for Python applications. In the Python community it’s standard practice to use Virtualenv to isolate the dependencies for individual applications, and it’s not clear how I’d be able to go about using this, if it is possible. For deploying Python applications, lack of virtualenv support would be a deal breaker, and I hope this gets clarified soon.

I’d also be curious to see benchmarks of how it compares to something like PHP-FPM. It’s quite possible that it may be less performant than other solutions. However, I will keep a close eye on this in future.

Recent Posts

Powering Up Git Bisect With the Run Command

Writing Golden Master Tests for Laravel Applications

How Much Difference Does Adding An Index to a Database Table Make?

Searching Content With Fuse.js

Higher-order Components in React

About me

I'm a web and mobile app developer based in Norfolk. My skillset includes Python, PHP and Javascript, and I have extensive experience working with CodeIgniter, Laravel, Zend Framework, Django, Phonegap and React.js.