Building a postcode lookup client with HTTPlug and PHPSpec
Published by Matthew Daly at 28th November 2017 11:40 am
While PHPUnit is my normal go-to PHP testing framework, for some applications I find PHPSpec superior, in particular REST API clients. I've found that it makes for a better flow when doing test-driven development, because it makes it very natural to write a test first, then run it, then make the test pass.
In this tutorial I'll show you how to build a lookup API client for UK postcodes. In the process of doing so, we'll use PHPSpec to drive our development process. We'll also use HTTPlug as our underlying HTTP library. The advantage of this over using something like Guzzle is that we give library users the freedom to choose the HTTP library they feel is most appropriate to their situation.
Background
If you're unfamiliar with it, the UK postcode system is our equivalent of a zip code in the US, but with two major differences:
- The codes themselves are alphanumeric instead of numeric, with the first part including one or two letters usually (but not always) derived from the nearest large town or city (eg L for Liverpool, B for Birmingham, OX for Oxford), or for London, based on the part of the city (eg NW for the north-west of London)
- A full postcode is in two parts (eg NW1 8TQ), and the first part narrows the location down to a similar area to a US-style zip code, while the second part usually narrows it down to a street (although sometimes large organisations that receive a lot of mail will have a postcode to themselves).
This means that if you have someone's postcode and house name or address, you can use those details to look up the rest of their address details. This obviously makes it easier for users to fill out a form, such as when placing an order on an e-commerce site - you can just request those two details and then autofill the rest from them.
Unfortunately, it's not quite that simple. The data is owned by Royal Mail, and they charge through the nose for access to the raw data, which places this data well outside the budgets of many web app developers. Fortunately, Ideal Postcodes offer a REST API for querying this data. It's not free, but at 2.5p per request it's not going to break the bank unless used excessively, and they offer some dummy postcodes that are free to query, which is perfectly fine for testing.
For those of you outside the UK, this may not be of much immediate use, but the underlying principles will still be useful, and you can probably build a similar client for your own nation's postal code system. For instance, there's a Zipcode API that those of you in the US can use, and if you understand what's going on here it shouldn't be hard to adapt it to work with that. If you do produce a similar client for your country's postal code system, submit a pull request to update the README with a link to it and I'll include it.
Setting up
First we'll create a composer.json to specify our dependencies:
1{2 "name": "matthewbdaly/postcode-client",3 "description": "A postcode lookup client.",4 "type": "library",5 "keywords": ["postcode"],6 "require": {7 "psr/http-message": "^1.0",8 "php-http/client-implementation": "^1.0",9 "php-http/httplug": "^1.0",10 "php-http/message-factory": "^1.0",11 "php-http/discovery": "^1.0"12 },13 "require-dev": {14 "psy/psysh": "^0.8.0",15 "phpspec/phpspec": "^3.2",16 "squizlabs/php_codesniffer": "^2.7",17 "php-http/mock-client": "^1.0",18 "php-http/message": "^1.0",19 "guzzlehttp/psr7": "^1.0"20 },21 "license": "MIT",22 "authors": [23 {24 "name": "Matthew Daly",25 "email": "matthewbdaly@gmail.com"26 }27 ],28 "autoload": {29 "psr-4": {30 "Matthewbdaly\\Postcode\\": "src/"31 }32 }33}
Note that we don't install an actual HTTPlug client, other than the mock one, which is only useful for testing. This is deliberate - we're giving developers working with this library the choice of working with whatever HTTP client they see fit. We do use the Guzzle PSR7 library, but that's just for the PSR7 library.
Then we install our dependencies:
$ composer install
We also need to tell PHPSpec what our namespace will be. Save this as phpspec.yml:
1suites:2 test_suite:3 namespace: Matthewbdaly\Postcode4 psr4_prefix: Matthewbdaly\Postcode
Don't forget to update the namespace in both files to whatever you're using, which should have a vendor name and a package name.
With that done, it's time to introduce the next component.
Introducing HTTPlug
In the past I've usually used either Curl or Guzzle to carry out HTTP requests. However, the problem with this approach is that you're forcing whoever uses your library to use whatever HTTP client, and whatever version of that client, that you deem appropriate. If they're also using another library that someone else has written and they made different choices, you could have problems.
HTTPlug is an excellent way of solving this problem. By requiring only an interface and not a concrete implementation, using HTTPlug means that you can specify that the consumer of the library must provide a suitable implementation of that library, but leave the choice of implementation up to them. This means that they can choose whatever implementation best fits their use case. There are adapters for many different clients, so it's unlikely that they won't be able to find one that meets their needs.
In addition, HTTPlug provides the means to automatically determine what HTTP client to use, so that if one is not explicitly provided, it can be resolved without any action on the part of the developer. As long as a suitable HTTP adapter is installed, it will be used.
Getting started
One advantage of PHPSpec is that it will automatically generate much of the boilerplate for our client and specs. To create our client spec, run this command:
1$ vendor/bin/phpspec desc Matthewbdaly/Postcode/Client2Specification for Matthewbdaly\Postcode\Client created in /home/matthew/Projects/postcode-client/spec/ClientSpec.php.
Now that we have a spec for our client, we can generate the client itself:
1$ vendor/bin/phpspec run2Matthewbdaly/Postcode/Client3 11 - it is initializable4 class Matthewbdaly\Postcode\Client does not exist.56 100% 171 specs81 example (1 broken)914ms101112 Do you want me to create `Matthewbdaly\Postcode\Client` for you?13 [Y/n]14y15Class Matthewbdaly\Postcode\Client created in /home/matthew/Projects/postcode-client/src/Client.php.1617 100% 1181 specs191 example (1 passed)2016ms
You will need to enter Y when prompted. We now have an empty class for our client.
Next, we need to make sure that the constructor for our client accepts two parameters:
- The HTTP client
- A message factory instance, which is used to create the request
Amend spec/ClientSpec.php as follows:
1<?php23namespace spec\Matthewbdaly\Postcode;45use Matthewbdaly\Postcode\Client;6use PhpSpec\ObjectBehavior;7use Prophecy\Argument;8use Http\Client\HttpClient;9use Http\Message\MessageFactory;1011class ClientSpec extends ObjectBehavior12{13 function let (HttpClient $client, MessageFactory $messageFactory)14 {15 $this->beConstructedWith($client, $messageFactory);16 }1718 function it_is_initializable()19 {20 $this->shouldHaveType(Client::class);21 }22}
Note the use of the let() method here. This lets us specify how the object is constructed, with the beConstructedWith() method. Also, note that $this refers not to the test, but to the object being tested - this takes a bit of getting used to if you're used to working with PHPUnit.
Also, note that the objects passed through are not actual instances of those objects - instead they are mocks created automatically by PHPSpec. This makes mocking extremely easy, and you can easily set up your own expectations on those mock objects in the test. If you want to use a real object, you can instantiate it in the spec as usual. If we need any other mocks, we can typehint them in our method in exactly the same way.
If we once again use vendor/bin/phpspec run we can now generate a constructor:
1$ vendor/bin/phpspec run2Matthewbdaly/Postcode/Client3 18 - it is initializable4 method Matthewbdaly\Postcode\Client::__construct not found.56 100% 171 specs81 example (1 broken)9281ms101112 Do you want me to create `Matthewbdaly\Postcode\Client::__construct()` for13 you?14 [Y/n]15y16 Method Matthewbdaly\Postcode\Client::__construct() has been created.1718 100% 1191 specs201 example (1 passed)2150ms
This will only create a placeholder for the constructor. You need to populate it yourself, so update src/Client.php as follows:
1<?php23namespace Matthewbdaly\Postcode;45use Http\Client\HttpClient;6use Http\Discovery\HttpClientDiscovery;7use Http\Message\MessageFactory;8use Http\Discovery\MessageFactoryDiscovery;910class Client11{12 public function __construct(HttpClient $client = null, MessageFactory $messageFactory = null)13 {14 $this->client = $client ?: HttpClientDiscovery::find();15 $this->messageFactory = $messageFactory ?: MessageFactoryDiscovery::find();16 }17}
A little explanation is called for here. We need two arguments in our construct:
- An instance of
Http\Client\HttpClientto send the request - An instance of
Http\Message\MessageFactoryto create the request
However, we don't want to force the user to create one. Therefore if they are not set, we use Http\Discovery\HttpClientDiscovery and Http\Discovery\MessageFactoryDiscovery to create them for us.
If we re-run PHPSpec, it should now pass:
1$ vendor/bin/phpspec run2 100% 131 specs41 example (1 passed)531ms
Next, we want to have a method for retrieving the endpoint. Add the following method to spec/ClientSpec.php:
1 function it_can_retrieve_the_base_url()2 {3 $this->getBaseUrl()->shouldReturn('https://api.ideal-postcodes.co.uk/v1/postcodes/');4 }
Here we're asserting that fetching the base URL returns the given result. Note how much simpler and more intuitive this syntax is than PHPUnit would be:
$this->assertEquals('https://api.ideal-postcodes.co.uk/v1/postcodes/', $client->getBaseUrl());
Running the tests again should prompt us to create the boilerplate for the new method:
1$ vendor/bin/phpspec run2Matthewbdaly/Postcode/Client3 23 - it can retrieve the base url4 method Matthewbdaly\Postcode\Client::getBaseUrl not found.56 50% 50% 271 specs82 examples (1 passed, 1 broken)940ms101112 Do you want me to create `Matthewbdaly\Postcode\Client::getBaseUrl()` for13 you?14 [Y/n]15y16 Method Matthewbdaly\Postcode\Client::getBaseUrl() has been created.1718Matthewbdaly/Postcode/Client19 23 - it can retrieve the base url20 expected "https://api.ideal-postcod...", but got null.2122 50% 50% 2231 specs242 examples (1 passed, 1 failed)2572ms
Now we need to update that method to work as expected:
1 protected $baseUrl = 'https://api.ideal-postcodes.co.uk/v1/postcodes/';23 ...45 public function getBaseUrl()6 {7 return $this->baseUrl;8 }
This should make the tests pass:
1$ vendor/bin/phpspec run2 100% 231 specs42 examples (2 passed)534ms
Next, we need to be able to get and set the API key. Add the following to spec/ClientSpec.php:
1 function it_can_get_and_set_the_key()2 {3 $this->getKey()->shouldReturn(null);4 $this->setKey('foo')->shouldReturn($this);5 $this->getKey()->shouldReturn('foo');6 }
Note that we expect $this->setKey('foo') to return $this. This is an example of a fluent interface - by returning an instance of the object, it enables methods to be chained, eg $client->setKey('foo')->get(). Obviously it won't work for anything that has to return a value, but it's a useful way of making your classes more intuitive to use.
Next, run the tests again and agree to the prompts as before:
1$ vendor/bin/phpspec run2Matthewbdaly/Postcode/Client3 28 - it can get and set the key4 method Matthewbdaly\Postcode\Client::getKey not found.56 66% 33% 371 specs83 examples (2 passed, 1 broken)951ms101112 Do you want me to create `Matthewbdaly\Postcode\Client::getKey()` for you?13 [Y/n]14y15 Method Matthewbdaly\Postcode\Client::getKey() has been created.1617Matthewbdaly/Postcode/Client18 28 - it can get and set the key19 method Matthewbdaly\Postcode\Client::setKey not found.2021 66% 33% 3221 specs233 examples (2 passed, 1 broken)2443ms252627 Do you want me to create `Matthewbdaly\Postcode\Client::setKey()` for you?28 [Y/n]29y30 Method Matthewbdaly\Postcode\Client::setKey() has been created.3132Matthewbdaly/Postcode/Client33 28 - it can get and set the key34 expected [obj:Matthewbdaly\Postcode\Client], but got null.3536 66% 33% 3371 specs383 examples (2 passed, 1 failed)3952ms
Next, add our getter and setter for the key, as well as declaring the property $key:
1 protected $key;23 public function getKey()4 {5 return $this->key;6 }78 public function setKey(string $key)9 {10 $this->key = $key;11 return $this;12 }
That should make the tests pass:
1$ vendor/bin/phpspec run2 100% 331 specs43 examples (3 passed)538ms
With that done, our final task is to be able to handle sending requests. Add the following imports at the top of spec/ClientSpec.php:
1use Psr\Http\Message\RequestInterface;2use Psr\Http\Message\ResponseInterface;3use Psr\Http\Message\StreamInterface;
And add the following method at the bottom of the same file:
1 function it_can_send_the_request(HttpClient $client, MessageFactory $messageFactory, RequestInterface $request, ResponseInterface $response, StreamInterface $stream)2 {3 $this->beConstructedWith($client, $messageFactory);4 $this->setKey('foo');5 $data = json_encode([6 'result' => [7 "postcode" => "SW1A 2AA",8 "postcode_inward" => "2AA",9 "postcode_outward" => "SW1A",10 "post_town" => "LONDON",11 "dependant_locality" => "",12 "double_dependant_locality" => "",13 "thoroughfare" => "Downing Street",14 "dependant_thoroughfare" => "",15 "building_number" => "10",16 "building_name" => "",17 "sub_building_name" => "",18 "po_box" => "",19 "department_name" => "",20 "organisation_name" => "Prime Minister & First Lord Of The Treasury",21 "udprn" => 23747771,22 "umprn" => "",23 "postcode_type" => "L",24 "su_organisation_indicator" => "",25 "delivery_point_suffix" => "1A",26 "line_1" => "Prime Minister & First Lord Of The Treasury",27 "line_2" => "10 Downing Street",28 "line_3" => "",29 "premise" => "10",30 "longitude" => -0.127695242183412,31 "latitude" => 51.5035398826274,32 "eastings" => 530047,33 "northings" => 179951,34 "country" => "England",35 "traditional_county" => "Greater London",36 "administrative_county" => "",37 "postal_county" => "London",38 "county" => "London",39 ]40 ]);41 $messageFactory->createRequest('GET', 'https://api.ideal-postcodes.co.uk/v1/postcodes/SW1A%202AA?api_key=foo', [], null, '1.1')->willReturn($request);42 $client->sendRequest($request)->willReturn($response);43 $response->getStatusCode()->willReturn(200);44 $response->getBody()->willReturn($stream);45 $stream->getContents()->willReturn($data);46 $this->get('SW1A 2AA')->shouldBeLike(json_decode($data, true));47 }
This test is by far the biggest so far, so it merits some degree of explanation.
Note that we don't make a real HTTP request against the API. This may sound strange, but bear with me. We have no control whatsoever over that API, and it could in theory become inaccessible or be subject to breaking changes at any time. We also don't want to be shelling out for a paid service just to test our API client works. All we can do is test that our implementation will send the request we expect it to send - we don't want our test suite reporting a bug when the API goes down.
We therefore typehint not just the dependencies for the constructor, but a request, response and stream instance. We mock our our responses from those instances using the willReturn() method, so we have complete control over what we pass to our client. That way we can return any appropriate response or throw any exception we deem fit to test the behaviour under those circumstances. For the message factory, we specify what arguments it should receive to create the request, and return our mocked-out request object.
Also, note we use shouldBeLike() to verify the response - this is effectively using the == operator, whereas shouldBe() uses the === operator, making it stricter.
Let's run the tests, and don't forget the prompt:
1$ vendor/bin/phpspec run2Matthewbdaly/Postcode/Client3 38 - it can send the request4 method Matthewbdaly\Postcode\Client::get not found.56 75% 25% 471 specs84 examples (3 passed, 1 broken)955ms101112 Do you want me to create `Matthewbdaly\Postcode\Client::get()` for you?13 [Y/n]14y15 Method Matthewbdaly\Postcode\Client::get() has been created.1617Matthewbdaly/Postcode/Client18 38 - it can send the request19 expected [array:1], but got null.2021 75% 25% 4221 specs234 examples (3 passed, 1 failed)2456ms
Now we can implement the get() method:
1 public function get(string $postcode)2 {3 $url = $this->getBaseUrl() . rawurlencode($postcode) . '?' . http_build_query([4 'api_key' => $this->getKey()5 ]);6 $request = $this->messageFactory->createRequest(7 'GET',8 $url,9 [],10 null,11 '1.1'12 );13 $response = $this->client->sendRequest($request);14 $data = json_decode($response->getBody()->getContents(), true);15 return $data;16 }
We first build up our URL, before using the message factory to create a request object. We then pass the built request to our client to send, before decoding the response into the format we want.
This should make our tests pass:
1$ vendor/bin/phpspec run2 100% 431 specs44 examples (4 passed)5307ms
Our client now works, but there are a couple of situations we need to account for. First, the API will raise a 402 if you make a request for a real postcode without having paid. We need to catch this and throw an exception. Add this to spec/ClientSpec.php:
1use Matthewbdaly\Postcode\Exceptions\PaymentRequired;23 ...45 function it_throws_an_exception_if_payment_required(HttpClient $client, MessageFactory $messageFactory, RequestInterface $request, ResponseInterface $response, StreamInterface $stream)6 {7 $this->beConstructedWith($client, $messageFactory);8 $this->setKey('foo');9 $messageFactory->createRequest('GET', 'https://api.ideal-postcodes.co.uk/v1/postcodes/SW1A%202AA?api_key=foo', [], null, '1.1')->willReturn($request);10 $client->sendRequest($request)->willReturn($response);11 $response->getStatusCode()->willReturn(402);12 $this->shouldThrow(PaymentRequired::class)->duringGet('SW1A 2AA');13 }
With that done, run the tests again:
1$ vendor/bin/phpspec run2Matthewbdaly/Postcode/Client3 87 - it throws an exception if payment required4 expected exception of class "Matthewbdaly\Postcode\Exc...", but got5 [exc:Prophecy\Exception\Call\UnexpectedCallException("Method call:6 - getBody()7 on Double\ResponseInterface\P15 was not expected, expected calls were:8 - getStatusCode()")].910 80% 20% 5111 specs125 examples (4 passed, 1 failed)13130ms
Let's amend the client to throw this exception:
1use Matthewbdaly\Postcode\Exceptions\PaymentRequired;23 ...45 public function get(string $postcode)6 {7 $url = $this->getBaseUrl() . rawurlencode($postcode) . '?' . http_build_query([8 'api_key' => $this->getKey()9 ]);10 $request = $this->messageFactory->createRequest(11 'GET',12 $url,13 [],14 null,15 '1.1'16 );17 $response = $this->client->sendRequest($request);18 if ($response->getStatusCode() == 402) {19 throw new PaymentRequired;20 }21 $data = json_decode($response->getBody()->getContents(), true);22 return $data;23 }
And let's re-run the tests:
1$ vendor/bin/phpspec run2Matthewbdaly/Postcode/Client3 87 - it throws an exception if payment required4 expected exception of class "Matthewbdaly\Postcode\Exc...", but got [obj:Error] with the5 message: "Class 'Matthewbdaly\Postcode\Exceptions\PaymentRequired' not found"67 80% 20% 581 specs95 examples (4 passed, 1 failed)10389ms
It fails now because the exception doesn't exist. Let's create it at src/Exceptions/PaymentRequired.php:
1<?php23namespace Matthewbdaly\Postcode\Exceptions;45class PaymentRequired extends \Exception6{7}
That should be enough to make our tests pass:
1$ vendor/bin/phpspec run2 100% 531 specs45 examples (5 passed)589ms
We also need to raise an exception when the postcode is not found, which raises a 404 error. Add the following spec:
1use Matthewbdaly\Postcode\Exceptions\PostcodeNotFound;2 ...3 function it_throws_an_exception_if_postcode_not_found(HttpClient $client, MessageFactory $messageFactory, RequestInterface $request, ResponseInterface $response, StreamInterface $stream)4 {5 $this->beConstructedWith($client, $messageFactory);6 $this->setKey('foo');7 $messageFactory->createRequest('GET', 'https://api.ideal-postcodes.co.uk/v1/postcodes/SW1A%202AA?api_key=foo', [], null, '1.1')->willReturn($request);8 $client->sendRequest($request)->willReturn($response);9 $response->getStatusCode()->willReturn(404);10 $this->shouldThrow(PostcodeNotFound::class)->duringGet('SW1A 2AA');11 }
Run the tests:
1$ vendor/bin/phpspec run2Matthewbdaly/Postcode/Client3 98 - it throws an exception if postcode not found4 expected exception of class "Matthewbdaly\Postcode\Exc...", but got5 [exc:Prophecy\Exception\Call\UnexpectedCallException("Method call:6 - getBody()7 on Double\ResponseInterface\P20 was not expected, expected calls were:8 - getStatusCode()")].910 83% 16% 6111 specs126 examples (5 passed, 1 failed)13538ms
This time we'll create the exception class before updating the client. Create the following class at src/Exceptions/PostcodeNotFound.php:
1<?php23namespace Matthewbdaly\Postcode\Exceptions;45/**6 * Postcode not found exception7 *8 */9class PostcodeNotFound extends \Exception10{11}
And update the client:
1use Matthewbdaly\Postcode\Exceptions\PostcodeNotFound;2 ...3 public function get(string $postcode)4 {5 $url = $this->getBaseUrl() . rawurlencode($postcode) . '?' . http_build_query([6 'api_key' => $this->getKey()7 ]);8 $request = $this->messageFactory->createRequest(9 'GET',10 $url,11 [],12 null,13 '1.1'14 );15 $response = $this->client->sendRequest($request);16 if ($response->getStatusCode() == 402) {17 throw new PaymentRequired;18 }19 if ($response->getStatusCode() == 404) {20 throw new PostcodeNotFound;21 }22 $data = json_decode($response->getBody()->getContents(), true);23 return $data;24 }
Re-run the tests:
1$ vendor/bin/phpspec run2 100% 631 specs46 examples (6 passed)5103ms
And our API client is feature complete! You can find the source code of the finished client here.
Summary
Personally, I find that while PHPSpec isn't appropriate for every use case, it's particularly handy for API clients and it's generally my go-to testing solution for them. It handles producing a lot of the boilerplate for me, and it results in a much better workflow for test-driven development as it makes it very natural to write the test first, then make it pass.
HTTPlug has been a revelation for me. While it takes a bit of getting used to if you're used to something like Guzzle, it means that you're giving consumers of your library the freedom to choose the HTTP client of their choice, meaning they don't have to fight with several different libraries requiring different versions of Guzzle. It also allows for easy resolution of the HTTP client, rather than having to explicitly pass through an instance when instantiating your client. I'm planning to use it extensively in the future.