Matthew Daly's Blog

I'm a web developer in Norfolk. This is my blog...

9th September 2018 1:40 pm

Switching from Vim to Neovim

I honestly thought it would never happen. I’ve been using Vim since 2008, and every other editor I’ve tried (including VSCode, Emacs, Sublime Text and Atom) hasn’t come up to scratch. There were a few useful features in PHPStorm, to be fair, but nothing that justified the bother of moving. Also, I suffer from a degree of RSI from my prior career as an insurance clerk (years of using crap keyboards and mice on Windows XP took its toll…), and Vim has always been the most RSI-friendly editor I found.

Yet I have actually gone ahead and migrated away… to Neovim. Of course, the fact that the workflow is essentially identical helps in the migration process, as does the fact that it supports most of the same plugins.

My workflow has always been strongly CLI-based. I use GNU Screen and Byobu together to run multiple “tabs” in the terminal, so the lack of GUI support in Neovim doesn’t bother me in the slightest. The only change I really made was to my .bash_aliases so that the Vim command ran screen -t Vim nvim, so that it would open up Neovim rather than Vim in a new Screen tab.

Initially I switched straight over to using the same settings and plugins I had with Vim, and they worked seamlessly. However, after a while I decided to use the opportunity to completely overhaul the plugins and settings I used and largely start over - cull the ones I no longer needed, add some new ones, and comment it properly.

Loading plugins

I used to use Pathogen to manage my Vim plugins, but it didn’t actually import the plugins itself, and just provided a structure for them. This meant that the only practical way I found to pull in third-party plugins was to set them up as Git submodules, meaning I had to store my configuration in version control and clone it recursively onto a new machine. It also made updating cumbersome.

Now I’ve switched to vim-plug, which makes things much easier. I can define my dependencies in my .config/nvim/init.vim and pull them in with PlugInstall. If I want to update them, I run PlugUpdate, or if I need to add something else, I merely add it in the file and run PlugInstall again. Nice and easy.

The first section of my configuration file loads the dependencies:

call plug#begin()
" NERDTree
Plug 'scrooloose/nerdtree'
" Git integration
Plug 'tpope/vim-fugitive'
Plug 'airblade/vim-gitgutter'
" Linting
Plug 'neomake/neomake'
Plug 'w0rp/ale'
" PHP-specific integration
Plug 'phpactor/phpactor' , {'do': 'composer install', 'for': 'php'}
Plug 'ncm2/ncm2'
Plug 'roxma/nvim-yarp'
Plug 'phpactor/ncm2-phpactor'
" Snippets
Plug 'SirVer/ultisnips'
Plug 'honza/vim-snippets'
" Comments
Plug 'tpope/vim-commentary'
" Search
Plug 'ctrlpvim/ctrlp.vim'
" Syntax
Plug 'sheerun/vim-polyglot'
Plug 'matthewbdaly/vim-filetype-settings'
" Themes
Plug 'nanotech/jellybeans.vim' , {'as': 'jellybeans'}
call plug#end()

As always, it’s a good idea to comment your config and try to group things logically. Note that I have one plugin of my own listed here - this is just a collection of settings for different filetypes, such as making Javascript files use 2 spaces for indentation, and it’s easier to keep that in a repository and pull it in as a dependency.

Completion

The next part of the config deals with configuration. Most of the time the default omnicompletion is pretty good, but in the process of building out this config, I discovered PHPActor, which has massively improved my development experience with PHP - it finally provides completion as good as most IDE’s, and also provides similar refactoring tools. My config for completion currently looks like this:

"Completion
autocmd FileType * setlocal formatoptions-=c formatoptions-=r formatoptions-=o
set ofu=syntaxcomplete#Complete
autocmd FileType php setlocal omnifunc=phpactor#Complete
let g:phpactorOmniError = v:true
autocmd BufEnter * call ncm2#enable_for_buffer()
set completeopt=noinsert,menuone,noselect

General config

This is a set of standard settings for the general behaviour of the application, such as setting the colorscheme and default indentation levels. I also routinely disable the mouse because it bugs me.

"General
syntax on
colorscheme jellybeans
set nu
filetype plugin indent on
set nocp
set ruler
set wildmenu
set mouse-=a
set t_Co=256
"Code folding
set foldmethod=manual
"Tabs and spacing
set autoindent
set cindent
set tabstop=4
set expandtab
set shiftwidth=4
set smarttab
"Search
set hlsearch
set incsearch
set ignorecase
set smartcase
set diffopt +=iwhite

Markdown configuration

This section sets the file type for Markdown. It disables the Markdown plugin included in vim-polyglot as I had problems with it, and sets the languages that will be highlighted in fenced code blocks. I may at some point migrate this to the filetype repository.

"Syntax highlighting in Markdown
au BufNewFile,BufReadPost *.md set filetype=markdown
let g:polyglot_disabled = ['markdown']
let g:markdown_fenced_languages = ['bash=sh', 'css', 'django', 'javascript', 'js=javascript', 'json=javascript', 'perl', 'php', 'python', 'ruby', 'sass', 'xml', 'html', 'vim']

Neomake

I used to use Syntastic for checking my code for errors, but I’ve always found it problematic - it was slow and would often block the editor for some time. Neovim does have support for asynchronous jobs (as does Vim 8), but Syntastic doesn’t use it, so I decided to look elsewhere.

Neomake seemed a lot better, so I migrated over to it. It doesn’t require much configuration, and it’s really fast - unlike Syntastic, it supports asynchronous jobs. This part of the config sets it up to run on changes with no delay in writing, so I get near-instant feedback if a syntax error creeps in, and it doesn’t block the editor the way Syntastic used to.

" Neomake config
" Full config: when writing or reading a buffer, and on changes in insert and
" normal mode (after 1s; no delay when writing).
call neomake#configure#automake('nrwi', 500)

PHPActor

As mentioned above, PHPActor has dramatically improved my experience when coding in PHP by providing access to features normally found only in full IDE’s. Here’s the fairly standard config I use for the refactoring functionality:

" PHPActor config
" Include use statement
nmap <Leader>u :call phpactor#UseAdd()<CR>
" Invoke the context menu
nmap <Leader>mm :call phpactor#ContextMenu()<CR>
" Invoke the navigation menu
nmap <Leader>nn :call phpactor#Navigate()<CR>
" Goto definition of class or class member under the cursor
nmap <Leader>o :call phpactor#GotoDefinition()<CR>
" Transform the classes in the current file
nmap <Leader>tt :call phpactor#Transform()<CR>
" Generate a new class (replacing the current file)
nmap <Leader>cc :call phpactor#ClassNew()<CR>
" Extract expression (normal mode)
nmap <silent><Leader>ee :call phpactor#ExtractExpression(v:false)<CR>
" Extract expression from selection
vmap <silent><Leader>ee :<C-U>call phpactor#ExtractExpression(v:true)<CR>
" Extract method from selection
vmap <silent><Leader>em :<C-U>call phpactor#ExtractMethod()<CR>

Summary

Vim or Neovim configuration files are never static. Your needs are always changing, and you’re constantly discovering new plugins and new settings to try out, and keeping ones that prove useful. It’s been helpful to start over and ditch some plugins I no longer needed, pull in some new ones, and organise my configuration a bit better.

Now that I can set the dependencies in a text file rather than pulling them in as Git submodules, it makes more sense to keep my config in a Github Gist rather than a Git repository, and that’s where I plan to retain it for now. Feel free to fork or cannibalize it for your own purposes if you wish.

25th July 2018 10:25 pm

Better Strings in PHP

One of the weaknesses of PHP as a programming language is the limitations of some of the fundamental types. For instance, a string in PHP is a simple value, rather than an object, and doesn’t have any methods associated with it. Instead, to manipulate a string, you have to call all manner of functions. By comparison, in Python, not only can you call methods on a string, and receive a new string as the response, making them easily chainable, but you can also iterate through a string, as in this example:

>>> a = 'foo'
>>> a.upper()
'FOO'
>>> a.lower()
'foo'
>>> for letter in a:
... print(letter)
...
f
o
o

A little while back, I read Adam Wathan’s excellent book Refactoring to Collections, which describes how you can use a collection implementation (such as the one included with Laravel) to replace convoluted array manipulation with simpler, chainable calls to a collection object. Using this approach, you can turn something like this:

$result = array_filter(
array_map(function ($item) {
return $item->get('foo');
}, $items),
function ($item) {
return $item->bar == true;
});

Or, even worse, this:

$result1 = array_map(function ($item) {
return $item->get('foo');
}, $items);
$result2 = array_filter($result1, function ($item) {
return $item->bar == true;
});

Into this:

$result = Collection::make($items)
->map(function ($item) {
return $item->get('foo');
})->filter(function ($item) {
return $item->bar == true;
})->toArray();

Much cleaner, more elegant, and far easier to understand.

A while back, after some frustration with PHP’s native strings, I started wondering how practical it would be to produce a string implementation that was more like the string objects in languages like Python and Javascript, with inspiration from collection implementations such as that used by Laravel. I soon discovered that it was very practical, and with a bit of work it’s not hard to produce your own, more elegant string class.

The most fundamental functionality required is to be able to create a string object, either by passing a string to the constructor or calling a static method. Our string class should be able to do both:

<?php
class Str
{
protected $string;
public function __construct(string $string = '')
{
$this->string = $string;
}
public static function make(string $string)
{
return new static($string);
}
}

Making it iterable

To be able to get the length of a string, it needs to implement the Countable interface:

use Countable;
class Str implements Countable
{
...
public function count()
{
return strlen($this->string);
}
}

To access it as an array, it needs to implement the ArrayAccess interface:

...
use ArrayAccess;
class Str implements Countable, ArrayAccess
{
...
public function offsetExists($offset)
{
return isset($this->string[$offset]);
}
public function offsetGet($offset)
{
return isset($this->string[$offset]) ? $this->string[$offset] : null;
}
public function offsetSet($offset, $value)
{
if (is_null($offset)) {
$this->string[] = $value;
} else {
$this->string[$offset] = $value;
}
}
public function offsetUnset($offset)
{
$this->string = substr_replace($this->string, '', $offset, 1);
}
}

And to make it iterable, it needs to implement the Iterator interface:

use Iterator;
class Str implements Countable, ArrayAccess, Iterator
{
...
public function current()
{
return $this->string[$this->position];
}
public function key()
{
return $this->position;
}
public function next()
{
++$this->position;
}
public function rewind()
{
$this->position = 0;
}
public function valid()
{
return isset($this->string[$this->position]);
}
}

Making it work as a string

To be useful, it also needs to be possible to actually use it as a string - for instance, you should be able to do this:

$foo = Str::make('I am the very model of a modern major general');
echo $foo;

Fortunately, the __toString() magic method allows this:

public function __toString()
{
return $this->string;
}

Adding methods

With that functionality in place, you can then start adding support for the methods you need in your string objects. If you’re looking to be able to use the same functionality as existing PHP methods, you can call those functions inside your methods. However, be sure to return a new instance of your string object from each method - that way, you can continually chain them:

public function replace($find, $replace)
{
return new static(str_replace($find, $replace, $this->string));
}
public function toUpper()
{
return new static(strtoupper($this->string));
}
public function toLower()
{
return new static(strtolower($this->string));
}
public function trim()
{
return new static(trim($this->string));
}
public function ltrim()
{
return new static(ltrim($this->string));
}
public function rtrim()
{
return new static(rtrim($this->string));
}

Now, you can write something like this:

return Str::make('I am the very model of a modern major general ')
->trim()
->replace('modern major general', 'scientist Salarian')
->toLower();

While you could do this with PHP’s native string functions alone, it would be a lot less elegant. In addition, if you have other, more complex string manipulations that you often do in a particular application, it may make sense to write a method for that so that your string objects can encapsulate that functionality for easier reuse.

As our string objects are iterable, we can also do this:

>>> $foo = Str::make('foo');
>>> foreach ($foo as $letter) { echo "$letter\n"; }
f
o
o

If you have an application that does some complex string manipulation, having a string utility class like this can make for much more expressive, elegant and easy-to-comprehend code than PHP’s native string functions. If you want to see a working implementation for this, check out my proof of concept collection and string utility library Proper.

23rd June 2018 1:03 pm

Forcing SSL in Codeigniter

I haven’t started a new CodeIgniter project since 2014, and don’t intend to, but on occasion I’ve been asked to do maintenance work on legacy CodeIgniter projects. This week I was asked to help out with a situation where a CodeIgniter site was being migrated to HTTPS and there were issues resulting from the migration.

Back in 2012, when working on my first solo project, I’d built a website using CodeIgniter that used HTTPS, but also needed to support an affiliate marketing system that did not support it, so certain pages had to force HTTP, and others had to force HTTPS, so I’d used the hook system to create hooks to enforce this. This kind of requirement is unlikely to reoccur now because HTTPS is becoming more prevalent, but sometimes it may be easier to enforce HTTPS at application level than in the web server configuration or using htaccess. It’s relatively straightforward to do that in CodeIgniter.

The first step is to create the hook. Save this as application/hooks/ssl.php:

<?php
function force_ssl()
{
$CI =& get_instance();
$CI->config->config['base_url'] = str_replace('http://', 'https://', $CI->config->config['base_url']);
if ($_SERVER['SERVER_PORT'] != 443) redirect($CI->uri->uri_string());
}
?>

Next, we register the hook. Update application/configs/hooks.php as follows:

<?php if ( ! defined('BASEPATH')) exit('No direct script access allowed');
/*
| -------------------------------------------------------------------------
| Hooks
| -------------------------------------------------------------------------
| This file lets you define "hooks" to extend CI without hacking the core
| files. Please see the user guide for info:
|
| http://codeigniter.com/user_guide/general/hooks.html
|
*/
$hook['post_controller_constructor'][] = array(
'function' => 'force_ssl',
'filename' => 'ssl.php',
'filepath' => 'hooks'
);
/* End of file hooks.php */
/* Location: ./application/config/hooks.php */

This tells CodeIgniter that it should looks in the application/hooks directory for a file called ssl.php, and return the function force_ssl.

Finally, we enable hooks. Update application/config/config.php:

$config['enable_hooks'] = TRUE;

If you only want to force SSL in production, not development, you may want to amend the ssl.php file to only perform the redirect in non-development environments, perhaps by using an environment variable via DotEnv.

3rd June 2018 4:30 pm

Logging to the ELK Stack With Laravel

Logging to text files is the simplest and most common logging setup for web apps, and it works fine for relatively small and simple applications. However, it does have some downsides:

  • It’s difficult to make the log files accessible - normally users have to SSH in to read them.
  • The tools used to filter and analyse log files have a fairly high technical barrier to access - grep and sed are not exactly easy for non-programmers to pick up, so business information can be hard to get.
  • It’s hard to visually identify trends in the data.
  • Log files don’t let you know immediately when something urgent happens
  • You can’t access logs for different applications through the same interface.

For rare, urgent issues where you need to be informed immediately they occur, it’s straightforward to log to an instant messaging solution such as Slack or Hipchat. However, these aren’t easily searchable, and can only be used for the most important errors (otherwise, there’s a risk that important data will be lost in the noise). There are third-party services that allow you to search and filter your logs, but they can be prohibitively expensive.

The ELK stack has recently gained a lot of attention as a sophisticated solution for logging application data. It consists of:

  • Logstash for processing log data
  • Elasticsearch as a searchable storage backend
  • Kibana as a web interface

By making the log data available using a powerful web interface, you can easily expose it to non-technical users. Kibana also comes with powerful tools to aggregate and filter the data. In addition, you can run your own instance, giving you a greater degree of control (as well as possibly being more cost-effective) compared to using a third-party service.

In this post I’ll show you how to configure a Laravel application to log to an instance of the ELK stack. Fortunately, Laravel uses the popular Monolog logging library by default, which is relatively easy to get to work with the ELK stack. First, we need to install support for the GELF logging format:

$ composer require graylog2/gelf-php

Then, we create a custom logger class:

<?php
namespace App\Logging;
use Monolog\Logger;
use Monolog\Handler\GelfHandler;
use Gelf\Publisher;
use Gelf\Transport\UdpTransport;
class GelfLogger
{
/**
* Create a custom Monolog instance.
*
* @param array $config
* @return \Monolog\Logger
*/
public function __invoke(array $config)
{
$handler = new GelfHandler(new Publisher(new UdpTransport($config['host'], $config['port'])));
return new Logger('main', [$handler]);
}
}

Finally, we configure our application to use this as our custom driver and specify the host and port in config/logging.php:

'custom' => [
'driver' => 'custom',
'via' => App\Logging\GelfLogger::class,
'host' => '127.0.0.1',
'port' => 12201,
],

You can then set up whatever logging channels you need for your application, and specify whatever log level you feel is appropriate.

Please note that this requires at least Laravel 5.6 - this file doesn’t exist in Laravel 5.5 and earlier, so you may have more work on your hands to integrate it with older versions.

If you already have an instance of the ELK stack set up on a remote server that’s already set up to accept input as GELF, then you should be able to point it at that and you’ll be ready to go. If you just want to try it out, I’ve been using a Docker-based project that makes it straightforward to run the whole stack locally. However, you will need to amend logstash/pipeline/logstash.conf as follows to allow it to accept log data:

input {
tcp {
port => 5000
}
gelf {
port => 12201
type => gelf
codec => "json"
}
}
## Add your filters / logstash plugins configuration here
output {
elasticsearch {
hosts => "elasticsearch:9200"
}
}

Then you can start it up using the instructions in the repository and it should be ready to go. Now, if you run the following command from Tinker:

Log::info('Just testing');

Then if you access the web interface, you should be able to find that log message without any difficulty.

Now, this only covers the Laravel application logs. You may well want to pass other logs through to Logstash, such as Apache, Nginx or MySQL logs, and a quick Google should be sufficient to find ideas on how you might log for these services. Creating visualisations with Kibana is a huge subject, and the existing documentation covers that quite well, so if you’re interested in learning more about that I’d recommend reading the documentation and having a play with the dashboard.

13th May 2018 2:55 pm

Full-text Search With Mariadb

Recently I had the occasion to check out MariaDB’s implementation of full-text search. As it’s a relatively recent arrival in MySQL and MariaDB, it doesn’t seem to get all that much attention. In this post I’ll show you how to use it, with a few Laravel-specific pointers. We’ll be using the default User model in a new Laravel installation, which has columns for name and email.

Our first task is to create the fulltext index, which is necessary to perform the query. Run the following command:

ALTER TABLE users ADD FULLTEXT (name, email);

As you can see, we can specify multiple columns in our table to index.

If you’re using Laravel, you’ll want to create the following migration for this:

<?php
use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class AddFulltextIndexForUsers extends Migration
{
/**
* Run the migrations.
*
* @return void
*/
public function up()
{
DB::statement('ALTER TABLE users ADD FULLTEXT(name, email)');
}
/**
* Reverse the migrations.
*
* @return void
*/
public function down()
{
DB::statement('ALTER TABLE users DROP INDEX IF EXISTS name');
}
}

Note that the index is named after the first field passed to it, so when we drop it we refer to it as name. Then, to actually query the index, you should run a command something like this:

SELECT * FROM users WHERE MATCH(name, email) AGAINST ('jeff' IN NATURAL LANGUAGE MODE);

Note that NATURAL LANGUAGE MODE is actually the default, so you can leave it off if you wish. We also have to specify the columns to match against.

If you’re using Laravel, you may want to create a reusable local scope for it:

public function scopeSearch($query, $search)
{
if (!$search) {
return $query;
}
return $query->whereRaw('MATCH(name, email) AGAINST (?)', [$search]);
}

Then you can call it as follows:

User::search('jeff')->get();

I personally have noticed that the query using the MATCH keywords seems to be far more performant, with the response time being between five and ten times less than a similar command using LIKE, however this observation isn’t very scientific (plus, we are talking about queries that still run in a fraction of a second). However, if you’re doing a particularly expensive query that currently uses a LIKE statement, it’s possible you may get better results by switching to a MATCH statement. Full-text search probably isn’t all that useful in this context - it’s only once we’re talking about longer text, such as blog posts, that some of the advantages like support for stopwords comes into play.

From what I’ve seen this implementation of full-text search is a lot simpler than in PostgreSQL, which has ups and downs. On the one hand, it’s a lot easier to implement, but conversely it’s less useful - there’s no obvious way to perform a full-text search against joined tables. However, it does seem to be superior to using a LIKE statement, so it’s probably a good fit for smaller sites where something like Elasticsearch would be overkill.

Recent Posts

Decorating Service Classes

Simplify Your Tests With Anonymous Classes

Adding React to a Legacy Project

Do You Still Need Jquery?

An Approach to Writing Golden Master Tests for PHP Web Applications

About me

I'm a web and mobile app developer based in Norfolk. My skillset includes Python, PHP and Javascript, and I have extensive experience working with CodeIgniter, Laravel, Django, Phonegap and Angular.js.