Writing automated tests is an excellent way of catching bugs during development and maintenance of your application, not to mention the other benefits. However, it's hard to gauge the quality of your tests, particularly when you first start out. Coverage will give you a good idea of what code was actually run during the test, but it won't tell you if the test itself actually tests anything worthwhile.

Infection is a mutation testing framework. The documentation defines mutation testing as follows:

Mutation testing involves modifying a program in small ways. Each mutated version is called a Mutant. To assess the quality of a given test set, these mutants are executed against the input test set to see if the seeded faults can be detected. If mutated program produces failing tests, this is called a killed mutant. If tests are green with mutated code, then we have an escaped mutant.

Infection works by running the test suite, carrying out a series of mutations on the source code in order to try to break the tests, and then collecting the results. The actual mutations carried out are not random - there is a set of mutations that get carried out every time, so results should be consistent. Ideally, all mutants should be killed by your tests - escaped mutants can indicate that either the line of mutated code is not tested, or the tests for that line are not very useful.

I decided to add mutation testing to my Laravel shopping cart package. In order to use Infection, you need to be able to generate code coverage, which means having either XDebug or phpdbg installed. Once Infection is installed (refer to the documentation for this), you can run this command in the project directory to configure it:

$ infection

Infection defaults to using PHPUnit for the tests, but it also supports PHPSpec. If you're using PHPSpec, you will need to specify the testing framework like this:

$ infection --test-framework=phpspec

Since PHPSpec doesn't support code coverage out of the box, you'll need to install a package for that - I used leanphp/phpspec-code-coverage.

On first run, you'll be prompted to create a configuration file. Your source directory should be straightforward to set up, but at the next step, if your project uses interfaces in the source directory, you should exclude them. The rest of the defaults should be fine.

I found that the first run gave a large number of uncovered results, but the second and later ones were more consistent - not sure if it's an issue with my setup or not. Running it gave me this:

1$ infection
2You are running Infection with xdebug enabled.
3    ____      ____          __  _
4   /  _/___  / __/__  _____/ /_(_)___  ____
5   / // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \
6 _/ // / / / __/  __/ /__/ /_/ / /_/ / / / /
7/___/_/ /_/_/  \___/\___/\__/_/\____/_/ /_/
8
9    0 [>---------------------------] < 1 secRunning initial test suite...
10
11PHPUnit version: 6.5.13
12
13   27 [============================] 3 secs
14
15Generate mutants...
16
17Processing source code files: 5/5
18Creating mutated files and processes: 43/43
19.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out
20
21...................MMM...M.......M.........          (43 / 43)
22
2343 mutations were generated:
24      38 mutants were killed
25       0 mutants were not covered by tests
26       5 covered mutants were not detected
27       0 errors were encountered
28       0 time outs were encountered
29
30Metrics:
31         Mutation Score Indicator (MSI): 88%
32         Mutation Code Coverage: 100%
33         Covered Code MSI: 88%
34
35Please note that some mutants will inevitably be harmless (i.e. false positives).
36
37Time: 21s. Memory: 12.00MB

Our test run shows 5 escaped mutants, and the remaining 38 were killed. We can view the results by looking at the generated infection-log.txt:

1Escaped mutants:
2================
3
4
51) /home/matthew/Projects/laravel-cart/src/Services/Cart.php:132    [M] DecrementInteger
6
7--- Original
8+++ New
9@@ @@
10     {
11         $content = Collection::make($this->all())->map(function ($item) use($rowId) {
12             if ($item['row_id'] == $rowId) {
13-                if ($item['qty'] > 0) {
14+                if ($item['qty'] > -1) {
15                     $item['qty'] -= 1;
16                 }
17             }
18
19
202) /home/matthew/Projects/laravel-cart/src/Services/Cart.php:132    [M] OneZeroInteger
21
22--- Original
23+++ New
24@@ @@
25     {
26         $content = Collection::make($this->all())->map(function ($item) use($rowId) {
27             if ($item['row_id'] == $rowId) {
28-                if ($item['qty'] > 0) {
29+                if ($item['qty'] > 1) {
30                     $item['qty'] -= 1;
31                 }
32             }
33
34
353) /home/matthew/Projects/laravel-cart/src/Services/Cart.php:132    [M] GreaterThan
36
37--- Original
38+++ New
39@@ @@
40     {
41         $content = Collection::make($this->all())->map(function ($item) use($rowId) {
42             if ($item['row_id'] == $rowId) {
43-                if ($item['qty'] > 0) {
44+                if ($item['qty'] >= 0) {
45                     $item['qty'] -= 1;
46                 }
47             }
48
49
504) /home/matthew/Projects/laravel-cart/src/Services/Cart.php:133    [M] Assignment
51
52--- Original
53+++ New
54@@ @@
55         $content = Collection::make($this->all())->map(function ($item) use($rowId) {
56             if ($item['row_id'] == $rowId) {
57                 if ($item['qty'] > 0) {
58-                    $item['qty'] -= 1;
59+                    $item['qty'] = 1;
60                 }
61             }
62             return $item;
63
64
655) /home/matthew/Projects/laravel-cart/src/Services/Cart.php:197    [M] OneZeroInteger
66
67--- Original
68+++ New
69@@ @@
70      */
71     private function hasStringKeys(array $items)
72     {
73-        return count(array_filter(array_keys($items), 'is_string')) > 0;
74+        return count(array_filter(array_keys($items), 'is_string')) > 1;
75     }
76     /**
77      * Validate input
78
79Timed Out mutants:
80==================
81
82Not Covered mutants:
83====================

This displays the mutants that escaped, and include a diff of the changed code, so we can see that all of these involve changing the comparison operators.

The last one can be resolved easily because the comparison is superfluous - the result of count() can be evaluated as true or false by itself, so removing the > 0 at the end in the test solves the problem quite neatly.

The other four mutations are somewhat harder. They all amend the decrement method's conditions, showing that a single assertion doesn't really fully check the behaviour. Here's the current test for that method:

1<?php
2
3namespace Tests\Unit\Services;
4
5use Tests\TestCase;
6use Matthewbdaly\LaravelCart\Services\Cart;
7use Mockery as m;
8
9class CartTest extends TestCase
10{
11    /**
12     * @dataProvider arrayProvider
13     */
14    public function testCanDecrementQuantity($data)
15    {
16        $data[0]['row_id'] = 'my_row_id_1';
17        $data[1]['row_id'] = 'my_row_id_2';
18        $newdata = $data;
19        $newdata[1]['qty'] = 1;
20        $session = m::mock('Illuminate\Contracts\Session\Session');
21        $session->shouldReceive('get')->with('Matthewbdaly\LaravelCart\Services\Cart')->once()->andReturn($data);
22        $session->shouldReceive('put')->with('Matthewbdaly\LaravelCart\Services\Cart', $newdata)->once();
23        $uniqid = m::mock('Matthewbdaly\LaravelCart\Contracts\Services\UniqueId');
24        $cart = new Cart($session, $uniqid);
25        $this->assertEquals(null, $cart->decrement('my_row_id_2'));
26    }
27}

It should be possible to decrement it if the quantity is more than zero, but not to go any lower. However, our current test does not catch anything but decrementing it from 2 to 1, which doesn't fully demonstrate this. We therefore need to add a few more assertions to cover taking it down to zero, and then trying to decrement it again. Here's how we might do that.

1<?php
2
3namespace Tests\Unit\Services;
4
5use Tests\TestCase;
6use Matthewbdaly\LaravelCart\Services\Cart;
7use Mockery as m;
8
9class CartTest extends TestCase
10{
11    /**
12     * @dataProvider arrayProvider
13     */
14    public function testCanDecrementQuantity($data)
15    {
16        $data[0]['row_id'] = 'my_row_id_1';
17        $data[1]['row_id'] = 'my_row_id_2';
18        $newdata = $data;
19        $newdata[1]['qty'] = 1;
20        $session = m::mock('Illuminate\Contracts\Session\Session');
21        $session->shouldReceive('get')->with('Matthewbdaly\LaravelCart\Services\Cart')->once()->andReturn($data);
22        $session->shouldReceive('put')->with('Matthewbdaly\LaravelCart\Services\Cart', $newdata)->once();
23        $uniqid = m::mock('Matthewbdaly\LaravelCart\Contracts\Services\UniqueId');
24        $cart = new Cart($session, $uniqid);
25        $this->assertEquals(null, $cart->decrement('my_row_id_2'));
26        $newerdata = $newdata;
27        $newerdata[1]['qty'] = 0;
28        $session->shouldReceive('get')->with('Matthewbdaly\LaravelCart\Services\Cart')->once()->andReturn($newdata);
29        $session->shouldReceive('put')->with('Matthewbdaly\LaravelCart\Services\Cart', $newerdata)->once();
30        $this->assertEquals(null, $cart->decrement('my_row_id_2'));
31        $session->shouldReceive('get')->with('Matthewbdaly\LaravelCart\Services\Cart')->once()->andReturn($newerdata);
32        $session->shouldReceive('put')->with('Matthewbdaly\LaravelCart\Services\Cart', $newerdata)->once();
33        $this->assertEquals(null, $cart->decrement('my_row_id_2'));
34    }
35}

If we re-run Infection, we now get a much better result:

1$ infection
2You are running Infection with xdebug enabled.
3    ____      ____          __  _
4   /  _/___  / __/__  _____/ /_(_)___  ____
5   / // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \
6 _/ // / / / __/  __/ /__/ /_/ / /_/ / / / /
7/___/_/ /_/_/  \___/\___/\__/_/\____/_/ /_/
8
9Running initial test suite...
10
11PHPUnit version: 6.5.13
12
13   22 [============================] 3 secs
14
15Generate mutants...
16
17Processing source code files: 5/5
18Creating mutated files and processes: 41/41
19.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out
20
21.........................................            (41 / 41)
22
2341 mutations were generated:
24      41 mutants were killed
25       0 mutants were not covered by tests
26       0 covered mutants were not detected
27       0 errors were encountered
28       0 time outs were encountered
29
30Metrics:
31         Mutation Score Indicator (MSI): 100%
32         Mutation Code Coverage: 100%
33         Covered Code MSI: 100%
34
35Please note that some mutants will inevitably be harmless (i.e. false positives).
36
37Time: 19s. Memory: 12.00MB

Code coverage only tells you what lines of code are actually executed - it doesn't tell you much about how effectively that line of code is tested. Infection gives you a different insight into the quality of your tests, helping to write better ones. I've so far found it very useful for getting feedback on the quality of my tests. It's interesting that PHPSpec tests seem to have a consistently lower proportion of escaped mutants than PHPUnit ones - perhaps the more natural workflow when writing specs with PHPSpec makes it easier to write good tests.