Matthew Daly's Blog

I'm a web developer in Norfolk. This is my blog...

20th October 2017 10:55 pm

Using Phpiredis With Laravel

Laravel has support out of the box for using Redis. However, by default it uses a Redis client written in PHP, which will always be a little slower than one written in C. If you’re making heavy use of Redis, it may be worth using the phpiredis extension to squeeze a little more performance out of it.

I’m using PHP 7.0 on Ubuntu Zesty and I installed the dependencies with the following command:

$ sudo apt-get install libhiredis-dev php-redis php7.0-dev

Then I installed phpiredis as follows:

git clone https://github.com/nrk/phpiredis.git && \
cd phpiredis && \
phpize && \
./configure --enable-phpiredis && \
make && \
sudo make install

Finally, I configured Redis to use phpiredis in the redis section of config/database.php for a Laravel app:

'redis' => [
'cluster' => false,
'default' => [
'host' => env('REDIS_HOST', 'localhost'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
'options' => [
'connections' => [
'tcp' => 'Predis\Connection\PhpiredisStreamConnection', // PHP streams
'unix' => 'Predis\Connection\PhpiredisSocketConnection', // ext-socket
],
]
],
],

Now, I’m going to be honest - in a casual comparison I couldn’t see much difference in terms of speed. I would probably only bother with setting this up on a site where high Redis performance was absolutely necessary. If you just want a quicker cache response it might make more sense to put Varnish in front of the site instead. However, in cases where Redis gets used heavily, it’s probably worth doing.

3rd October 2017 11:56 pm

Simple Fuzzy Search With Laravel and Postgresql

When implementing fuzzy search, many developers reach straight for specialised tools like Elasticsearch. However, for simple implementations, this is often overkill. PostgreSQL, my relational database of choice, can natively handle fuzzy search quite easily if you know how. Here’s how you might use this with Laravel.

Suppose we have the following migration to create a locations table, storing towns, cities and villages:

<?php
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class CreateLocations extends Migration
{
/**
* Run the migrations.
*
* @return void
*/
public function up()
{
// Create locations table
Schema::create('locations', function (Blueprint $table) {
$table->increments('id')->unsigned();
$table->string('name');
$table->timestamps();
});
}
/**
* Reverse the migrations.
*
* @return void
*/
public function down()
{
// Drop locations table
Schema::drop('locations');
}
}

The key to this implementation of fuzzy search is trigrams. A trigram is a group of three consecutive characters taken from a string. Using the pg_trgm module, which comes with PostgreSQL, we can break a string into as many trigrams as possible, and then return the strings with the most matching trigrams.

We can ensure that pg_trgm is set up on the database by creating a migration:

<?php
use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class AddTrgmExtension extends Migration
{
/**
* Run the migrations.
*
* @return void
*/
public function up()
{
DB::statement('CREATE EXTENSION IF NOT EXISTS pg_trgm');
}
/**
* Reverse the migrations.
*
* @return void
*/
public function down()
{
DB::statement('DROP EXTENSION IF EXISTS pg_trgm');
}
}

Make sure you run the migration as well. Once that is done, we can make a raw fuzzy query against the name field as follows:

SELECT * FROM locations WHERE 'burgh' % name;

Translating that to work with the Eloquent ORM, we can perform fuzzy queries against the name field as follows:

$location = Location::whereRaw("'burgh' % name")->get();

This query might match both Aldeburgh and Edinburgh. It’s also able to handle slight misspellings, as in this example:

$location = Location::whereRaw("'hendrad' % name")->get();

This query will match East Hendred or West Hendred successfully. As you can see, we can match strings at any point in the name string, and handle slight mis-spellings without any problems.

In practice, rather than using whereRaw() every time, you’ll probably want to create a local scope that accepts the name you want to match against.

Improving performance with an index

The performance of these queries isn’t that great out of the box. We can improve them by creating an index:

<?php
use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class AddTrgmExtension extends Migration
{
/**
* Run the migrations.
*
* @return void
*/
public function up()
{
DB::statement('CREATE EXTENSION IF NOT EXISTS pg_trgm');
DB::statement('CREATE INDEX locations_name_trigram ON locations USING gist(name gist_trgm_ops);');
}
/**
* Reverse the migrations.
*
* @return void
*/
public function down()
{
DB::statement('DROP INDEX IF EXISTS locations_name_trigram');
DB::statement('DROP EXTENSION IF EXISTS pg_trgm');
}
}

Adding an index should produce a noticeable improvement in the response time.

Final thoughts

PostgreSQL’s pg_trgm module is a fairly straightforward way of implementing fuzzy search. It’s not much more involved than a LIKE or ILIKE clause in your query, and for many use cases, it’s more than sufficient. If you don’t have a huge number of records, it’s probably a more appropriate choice than something like Elasticsearch, and has the advantage of a simpler stack. However, if you have a larger dataset, you may be better off with a dedicated search solution. As always, if you’re unsure it’s a good idea to try both and see what works best for that particular use case.

25th September 2017 10:18 pm

A Generic PHP SMS Library

This weekend I published sms-client, a generic PHP library for sending SMS notifications. It’s intended to offer a consistent interface when sending SMS notifications by using swappable drivers. That way, if your SMS service provider suddenly goes out of business or bumps up their prices, it’s easy to switch to a new one.

Out of the box it comes with drivers for the following services:

  • Nexmo
  • ClockworkSMS

In addition, it provides the following test drivers:

  • Null
  • Log
  • RequestBin

Here’s an example of how you might use it with the ClockworkSMS driver:

use GuzzleHttp\Client as GuzzleClient;
use GuzzleHttp\Psr7\Response;
use Matthewbdaly\SMS\Drivers\Clockwork;
use Matthewbdaly\SMS\Client;
$guzzle = new GuzzleClient;
$resp = new Response;
$driver = new Clockwork($guzzle, $resp, [
'api_key' => 'MY_CLOCKWORK_API_KEY',
]);
$client = new Client($driver);
$msg = [
'to' => '+44 01234 567890',
'content' => 'Just testing',
];
$client->send($msg);

If you want to roll your own driver for it, it should be easy - just create a class that implements the Matthewbdaly\SMS\Contracts\Driver interface. Most of the existing drivers work using Guzzle to send HTTP requests to an API, but you don’t necessarily have to do that - for instance, you could create a driver for a mail-to-SMS gateway by using Swiftmailer or the PHP mail class. If you create a driver for it, please feel free to submit a pull request so I can add it to the repository.

For Laravel or Lumen users, there’s an integration package that should make it easier to use. For users of other frameworks, it should still be fairly straightforward to integrate.

8th September 2017 10:05 pm

Installing Nginx Unit on Ubuntu

Recently Nginx announced the release of the first beta of Unit, an application server that supports Python, PHP and Go, with support coming for Java, Node.js and Ruby.

The really interesting part is that not only does it support more than one language, but Unit can be configured by making HTTP requests, rather than by editing config files. This makes it potentially very interesting to web developers like myself who have worked in multiple languages - I could use it to serve a Python or PHP web app, simply by making different requests during the setup process. I can see this being a boon for SaaS providers - you could pick up the language from a file, much like the runtime.txt used by Heroku, and set up the application on the fly.

It’s currently in public beta, and there are packages for Ubuntu, so I decided to try it out. I’ve created the Ansible role below to set up Unit on an Ubuntu 16.04 server or VM:

---
- name: Install keys
apt_key: url=http://nginx.org/keys/nginx_signing.key state=present
- name: Setup main repo
apt_repository: repo='deb http://nginx.org/packages/mainline/ubuntu/ xenial nginx' state=present
- name: Setup source rep
apt_repository: repo='deb-src http://nginx.org/packages/mainline/ubuntu/ xenial nginx' state=present
- name: Update system
apt: upgrade=full update_cache=yes
- name: Install dependencies
apt: name={{ item }} state=present
with_items:
- nginx
- unit
- golang
- php-dev
- php7.0-dev
- libphp-embed
- libphp7.0-embed
- python-dev
- python3
- python3-dev
- php7.0-cli
- php7.0-mcrypt
- php7.0-pgsql
- php7.0-sqlite3
- php7.0-opcache
- php7.0-curl
- php7.0-mbstring
- php7.0-dom
- php7.0-xml
- php7.0-zip
- php7.0-bcmath
- name: Copy over Nginx configuration
copy: src=nginx.conf dest=/etc/nginx/sites-available/default owner=root group=root mode=0644

Note the section that copies over the Nginx config file. Here is that file:

upstream unit_backend {
server 127.0.0.1:8300;
}
server {
listen 80 default_server;
listen [::]:80 default_server ipv6only=on;
fastcgi_param HTTP_PROXY "";
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
root /var/www/public;
index index.php index.html index.htm;
server_name server_domain_or_IP;
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location ~ \.php$ {
try_files $uri /index.php =404;
proxy_pass http://unit_backend;
proxy_set_header Host $host;
}
}

This setup proxies all dynamic requests to the Unit backed in a similar fashion to how it would normally pass it to PHP-FPM.

There were still a few little issues. It doesn’t exactly help that the Nginx package provided with this repository isn’t quite the same as the one in Ubuntu by default - not only is it the unstable version, but it doesn’t set up the sites-available and sites-enabled folders, so I had to do that manually. I also had an issue with Systemd starting the process (at /run/control.unit.sock) with permissions that didn’t allow Nginx to access it. I’m not that familiar with Systemd so I wound up just setting the permissions of the file manually, but that doesn’t persist between restarts. I expect this issue isn’t that big a deal to someone more familiar with Systemd, but I haven’t been able to resolve it yet.

I decided to try it out with a Laravel application. I created a new Laravel app and set it up with the web root at /var/www. I then saved the following configuration for it as app.json:

{
"listeners": {
"*:8300": {
"application": "myapp"
}
},
"applications": {
"myapp": {
"type": "php",
"workers": 20,
"user": "www-data",
"group": "www-data",
"root": "/var/www/public",
"index": "index.php"
}
}
}

This is fairly basic, but a good example of how you configure an application with Unit. The listener section maps a port to an application, while the applications section defines an application called myapp. In this case, we specify that the type should be php. Note that each platform has slightly different options - for instance, the Python type doesn’t have the index or root options, instead having the path option, which specifies the path to the wsgi.py file.

I then ran the following command to upload the file:

$ curl -X PUT -d @app.json --unix-socket /run/control.unit.sock http://localhost

Note that we send it direct to the Unix socket file - this way we don’t have to expose the API to the outside. After this was done, the Laravel app began working as expected.

We can then make a GET request to view the configured applications:

$ curl --unix-socket /run/control.unit.sock http://localhost/
{
"listeners": {
"*:8300": {
"application": "saas"
}
},
"applications": {
"saas": {
"type": "php",
"workers": 20,
"user": "www-data",
"group": "www-data",
"root": "/var/www/public",
"index": "index.php"
}
}
}

It’s also possible to update and delete existing applications via the API using PUT and DELETE requests.

Final thoughts

This is way too early to be seriously considering using Unit in production. It’s only just been released as a public beta, and it’s a bit fiddly to set up. However, it has an enormous amount of promise.

One thing I can’t really see right now is whether it’s possible to use a virtualenv with it for Python applications. In the Python community it’s standard practice to use Virtualenv to isolate the dependencies for individual applications, and it’s not clear how I’d be able to go about using this, if it is possible. For deploying Python applications, lack of virtualenv support would be a deal breaker, and I hope this gets clarified soon.

I’d also be curious to see benchmarks of how it compares to something like PHP-FPM. It’s quite possible that it may be less performant than other solutions. However, I will keep a close eye on this in future.

2nd September 2017 2:45 pm

Making Internal Requests With Laravel

Recently I’ve been working on a Phonegap app that needs to work offline. The nature of relational databases can often make this tricky if you’re dealing with related objects and you’re trying to retrofit it to something that wasn’t built with this use case in mind.

Originally my plan was to push each request that would have been made to a queue in WebSQL, and then on reconnect, make every request individually. It quickly became apparent, however, that this approach had a few problems:

  • If one request failed, the remaining requests had to be stopped from executing
  • It didn’t allow for storing the failed transactions in a way that made them easy to retrieve

Instead, I decided to create a single sync endpoint for the API that would accept an object containing all the requests that would be made, and then step through each one. If it failed, it would get the failed request and all subsequent ones in the object, and store them in the database. That way, even if the data didn’t sync correctly, it wasn’t lost, and if necessary it could be resolved manually.

Since the necessary API endpoints already existed, and were thoroughly tested, it was not a good idea to start duplicating that functionality. Instead, I implemented the functionality to carry out internal requests, and I thought I’d share how you can do this.

For any service you may build for your Laravel applications, it’s a good idea to create an interface for it first:

<?php
namespace App\Contracts;
interface MakesInternalRequests
{
/**
* Make an internal request
*
* @param string $action The HTTP verb to use.
* @param string $resource The API resource to look up.
* @param array $data The request body.
* @return \Illuminate\Http\Response
*/
public function request(string $action, string $resource, array $data = []);
}

That way you can resolve the service using dependency injection, making it trivial to replace it with a mock when testing.

Now, actually making an internal request is pretty easy. You get the app instance (you can do so by resolving it using dependency injection as I do below, or call the app() helper). Then you put together the request you want to make and pass it as an argument to the app’s handle() method:

<?php
namespace App\Services;
use Illuminate\Http\Request;
use App\Contracts\MakesInternalRequests;
use Illuminate\Foundation\Application;
use App\Exceptions\FailedInternalRequestException;
/**
* Internal request service
*/
class InternalRequest implements MakesInternalRequests
{
/**
* The app instance
*
* @var $app
*/
protected $app;
/**
* Constructor
*
* @param Application $app The app instance.
* @return void
*/
public function __construct(Application $app)
{
$this->app = $app;
}
/**
* Make an internal request
*
* @param string $action The HTTP verb to use.
* @param string $resource The API resource to look up.
* @param array $data The request body.
* @throws FailedInternalRequestException Request could not be synced.
* @return \Illuminate\Http\Response
*/
public function request(string $action, string $resource, array $data = [])
{
// Create request
$request = Request::create('/api/' . $resource, $action, $data, [], [], [
'HTTP_Accept' => 'application/json',
]);
// Get response
$response = $this->app->handle($request);
if ($response->getStatusCode() >= 400) {
throw new FailedInternalRequestException($request, $response);
}
// Dispatch the request
return $response;
}
}

Also note that I’ve created a custom exception, called FailedInternalRequestException. This is fired if the status code returned from the internal requests is greater than or equal to 400 (thus denoting an error):

<?php
namespace App\Exceptions;
use Illuminate\Http\Request;
use Illuminate\Http\Response;
/**
* Exception for when a bulk sync job fails
*/
class FailedInternalRequestException extends \Exception
{
/**
* Request instance
*
* @var $request
*/
protected $request;
/**
* Response instance
*
* @var $response
*/
protected $response;
/**
* Constructor
*
* @param Request $request The request object.
* @param Response $response The response object.
* @return void
*/
public function __construct(Request $request, Response $response)
{
parent::__construct();
$this->request = $request;
$this->response = $response;
}
/**
* Get request object
*
* @return Request
*/
public function getRequest()
{
return $this->request;
}
/**
* Get response object
*
* @return Response
*/
public function getResponse()
{
return $this->response;
}
}

You can catch this exception in an appropriate place and handle it as you wish. Now, if you import the internal request class as $dispatcher, you can just call $dispatcher->request($action, $resource, $data), where $action is the HTTP verb, $resource is the API resource to send to, and $data is the data to send.

It’s actually quite rare to have to do this. In this case, because this was a REST API and every request made to it was changing the state of the application (there were no GET requests, only POST, PUT, PATCH and DELETE), it made sense to just break down the request body and do internal requests against the existing API, since otherwise I’d have to duplicate the existing functionality. I would not recommend this approach for something like fetching data to render a page on the server side, as there are more efficient ways of accomplishing it. In all honesty I can’t think of any other scenario where this would genuinely be the best option. However, it worked well for my use case and allowed me to implement this functionality quickly and simply.

Recent Posts

Using Phpiredis With Laravel

Simple Fuzzy Search With Laravel and Postgresql

A Generic PHP SMS Library

Installing Nginx Unit on Ubuntu

Making Internal Requests With Laravel

About me

I'm a web and mobile app developer based in Norfolk. My skillset includes Python, PHP and Javascript, and I have extensive experience working with CodeIgniter, Laravel, Django, Phonegap and Angular.js.