As I mentioned in an earlier post, I recently had the occasion to use Varnish to improve the performance of a website that otherwise would have been unreliable and unusably slow due to WordPress making an excessive number of queries. The difference it made was nothing short of staggering, and I'm not exaggerating when I say it saved the day. I now use Ansible for provisioning new WordPress sites, and Varnish is now a standard part of my WordPress site setup playbook.

However, Varnish can be quite fiddly to configure, and it was something of a baptism of fire for me to learn how to configure it appropriately for this use case. I did make a few mistakes that caused problems down the line, so I thought I'd share the details of how I got it working for that particular site.

What is Varnish?

From the website:

Varnish Cache is a web application accelerator also known as a caching HTTP reverse proxy. You install it in front of any server that speaks HTTP and configure it to cache the contents. Varnish Cache is really, really fast. It typically speeds up delivery with a factor of 300 - 1000x, depending on your architecture.

In other words, you run it on the usual HTTP or HTTPS port, move your usual web server to a different port, and configure it, and it will cache web pages so they can be served more quickly to subsequent visitors.

Be warned - Varnish is not something where you can generally stick with the default settings. The default behaviour does make a lot of sense, but in practice almost no-one will be able to get away with leaving the configuration unchanged.

Installing Varnish

If you're using Debian or a derivative such as Ubuntu, Varnish is available via apt-get:

$ sudo apt-get install varnish

You may also want to install the documentation:

$ sudo apt-get install varnish-doc

If you're using Apache I'd also recommend installing libapache2-mod-rpaf and enabling it with sudo a2enmod rpaf - without this, Apache will log all incoming requests as coming from the same server.

I'm assuming you already have a normal web server installed. I'll assume you're using Apache, but it shouldn't be hard to adapt these instructions to work with Nginx. I'm also assuming that the site you want to use Varnish for is a WordPress site with WooCommerce and W3 Total Cache installed. However, this is only for example purposes. If you want to use Varnish for a different web app, you'll need to plan your caching strategy around that web app yourself.

Please also note that this is using Varnish 4.0, which is the version available with Debian Jessie. If you're using an older operating system, you may have Varnish 3.0 in the repositories - be warned, the configuration language changed in Varnish 4.0, so the examples here will not work with older versions of Varnish.

By default, Varnish runs on port 6081, which is fine for testing it out, but once you want to go live it's not what you want. When it's time to go live, you'll need to open up /etc/default/varnish and edit the value of DAEMON_OPTS to something like this:

1DAEMON_OPTS="-a :80 \
2             -T localhost:6082 \
3             -f /etc/varnish/default.vcl \
4             -S /etc/varnish/secret \
5             -s malloc,256m"

Note that the -a flag represents the port Varnish is running on.

If you're using an operating system that uses systemd, such as Debian Jessie, this alone won't be sufficient. Create a new file at /etc/systemd/system/varnish.service and enter the following:

1[Unit]
2Description=Varnish HTTP accelerator
3
4[Service]
5Type=forking
6LimitNOFILE=131072
7LimitMEMLOCK=82000
8ExecStartPre=/usr/sbin/varnishd -C -f /etc/varnish/default.vcl
9ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
10ExecReload=/usr/share/varnish/reload-vcl
11
12[Install]
13WantedBy=multi-user.target

Next, we need to move our web server to a different port. We'll use port 8080. Replace the contents of /etc/apache2/ports.conf with this:

1# If you just change the port or add more ports here, you will likely also
2# have to change the VirtualHost statement in
3# /etc/apache2/sites-enabled/000-default
4# This is also true if you have upgraded from before 2.2.9-3 (i.e. from
5# Debian etch). See /usr/share/doc/apache2.2-common/NEWS.Debian.gz and
6# README.Debian.gz
7
8NameVirtualHost *:8080
9Listen 8080
10
11<IfModule mod_ssl.c>
12    # If you add NameVirtualHost *:443 here, you will also have to change
13    # the VirtualHost statement in /etc/apache2/sites-available/default-ssl
14    # to <VirtualHost *:443>
15    # Server Name Indication for SSL named virtual hosts is currently not
16    # supported by MSIE on Windows XP.
17    Listen 443
18</IfModule>
19
20<IfModule mod_gnutls.c>
21    Listen 443
22</IfModule>

You'll also need to change the ports for the individual site files under /etc/apache2/sites-available, as in this example:

1<VirtualHost *:8080>
2    ServerAdmin webmaster@localhost
3
4    DocumentRoot /var/www
5    <Directory />
6        Options FollowSymLinks
7        AllowOverride All
8    </Directory>
9    <Directory /var/www/>
10        Options FollowSymLinks MultiViews
11        AllowOverride All
12        Order allow,deny
13        allow from all
14    </Directory>
15
16    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
17    <Directory "/usr/lib/cgi-bin">
18        AllowOverride None
19        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
20        Order allow,deny
21        Allow from all
22    </Directory>
23
24    ErrorLog ${APACHE_LOG_DIR}/error.log
25
26    # Possible values include: debug, info, notice, warn, error, crit,
27    # alert, emerg.
28    LogLevel warn
29
30    CustomLog ${APACHE_LOG_DIR}/access.log combined
31</VirtualHost>

Writing our VCL file

Next, we come to our Varnish configuration proper, which resides at /etc/varnish/default.vcl. The vcl stands for Varnish Configuration Language, and it has a syntax somewhat reminiscent of C.

The default behaviour for Varnish is as follows:

• It does not cache requests that contain cookie or authorisation headers
• It does not cache requests which the backend HTTP server indicates should not be cached
• It will only cache GET and HEAD requests

This behaviour is unlikely to meet your needs. We'll therefore work through the Varnish config file I wrote for this WordPress site in the hope that it will teach you enough to adapt it to your own needs.

1vcl 4.0;
2
3backend default {
4    .host = "127.0.0.1";
5    .port = "8080";
6}
7
8acl purge {
9    "127.0.0.1";
10    "localhost";
11}
12
13sub vcl_recv {
14
15    # Never cache PUT, PATCH, DELETE or POST requests
16    if (req.method == "PUT" || req.method == "PATCH" || req.method == "DELETE" || req.method == "POST") {
17        return (pass);
18    }
19
20    # Never cache cart, account, checkout or addons
21    if (req.url ~ "^/(cart|my-account|checkout|addons)") {
22        return (pass);
23    }
24
25    # Never cache adding to cart
26    if ( req.url ~ "\?add-to-cart=" ) {
27        return (pass);
28    }
29
30    # Never cache admin or login
31    if ( req.url ~ "^/wp-(admin|login|cron)" ) {
32        return (pass);
33    }
34
35    # Never cache WooCommerce API
36    if ( req.url ~ "wc-api" ) {
37        return (pass);
38    }
39
40    # Remove has_js and CloudFlare/Google Analytics __* cookies and statcounter is_unique
41    set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js|is_unique)=[^;]*", "");
42    # Remove a ";" prefix, if present.
43    set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
44
45    # Remove the wp-settings-1 cookie
46    set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", "");
47
48    # Remove the wp-settings-time-1 cookie
49    set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", "");
50
51    # Remove the wp test cookie
52    set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", "");
53
54    # Static content unique to the theme can be cached (so no user uploaded images)
55    # The reason I don't take the wp-content/uploads is because of cache size on bigger blogs
56    # that would fill up with all those files getting pushed into cache
57    if (req.url ~ "wp-content/themes/" && req.url ~ "\.(css|js|png|gif|jp(e)?g)") {
58        unset req.http.cookie;
59    }
60
61    # Even if no cookies are present, I don't want my "uploads" to be cached due to their potential size
62    if (req.url ~ "/wp-content/uploads/") {
63        return (pass);
64    }
65
66    # any pages with captchas need to be excluded
67    if (req.url ~ "^/contact/")
68    {
69        return(pass);
70    }
71
72    # Check the cookies for wordpress-specific items
73    if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
74        # A wordpress specific cookie has been set
75        return (pass);
76    }
77
78    # allow PURGE from localhost
79    if (req.method == "PURGE") {
80        if (!client.ip ~ purge) {
81            return(synth(405, "Not allowed."));
82        }
83        return (purge);
84    }
85
86    # Force lookup if the request is a no-cache request from the client
87    if (req.http.Cache-Control ~ "no-cache") {
88        return (pass);
89    }
90
91    # Try a cache-lookup
92    return (hash);
93}
94
95sub vcl_backend_response {
96    set beresp.grace = 5m;
97}

Let's take a closer look at the first part of the config:

1vcl 4.0;
2
3backend default {
4    .host = "127.0.0.1";
5    .port = "8080";
6}

Here we define that we're using version 4.0 of VCL, and that the host to use as a back end is port 8080 on the same server. If your normal HTTP server is running on a different port, you will need to set it here. Also, note that you can use a different host as the backend.

1acl purge {
2    "127.0.0.1";
3    "localhost";
4}

We also set which hosts can trigger a purge of the cache, namely localhost and 127.0.0.1. The web app hosted on the server can then make an HTTP PURGE request to a given path, which will clear that path from the cache. In our case, W3 Total Cache supports this - if it's a custom web app, you'll need to implement this functionality yourself to clear the cache when new content is added.

Next, we start the vcl_recv subroutine. This is where we define our rules for deciding whether or not to serve content from the cache. Let's look at our first rule:

1sub vcl_recv {
2
3    # Never cache PUT, PATCH, DELETE or POST requests
4    if (req.method == "PUT" || req.method == "PATCH" || req.method == "DELETE" || req.method == "POST") {
5        return (pass);
6    }

Here, we declare that we should never cache any PUT, PATCH, DELETE or POST requests, on the basis that these change the state of the application. This ensures that things like contact forms will work as expected.

Note that we're getting the value of req.method to determine the HTTP verb used. The req object has many other properties we'll see being used.

1    # Never cache cart, account, checkout or addons
2    if (req.url ~ "^/(cart|my-account|checkout|addons)") {
3        return (pass);
4    }
5
6    # Never cache adding to cart
7    if ( req.url ~ "\?add-to-cart=" ) {
8        return (pass);
9    }
10
11    # Never cache admin or login
12    if ( req.url ~ "^/wp-(admin|login|cron)" ) {
13        return (pass);
14    }
15
16    # Never cache WooCommerce API
17    if ( req.url ~ "wc-api" ) {
18        return (pass);
19    }

Next, we define a series of regular expressions, and if the URL (represented by req.url) matches that regex, then the request is passed straight through to Apache without Varnish getting involved. In this case, we never want to cache the following sections:

• The shopping cart, checkout, addons page or account page
• The Add to cart button
• The WordPress admin and login screen, and cron requests
• The WooCommerce API

You'll need to consider which parts of your site must always serve the latest content and which don't need everything to be fully up to date. Typically admin areas any anything interactive must not be cached, while the front page is usually fine.

1    # Remove has_js and CloudFlare/Google Analytics __* cookies and statcounter is_unique
2    set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js|is_unique)=[^;]*", "");
3    # Remove a ";" prefix, if present.
4    set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
5
6    # Remove the wp-settings-1 cookie
7    set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", "");
8
9    # Remove the wp-settings-time-1 cookie
10    set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", "");
11
12    # Remove the wp test cookie
13    set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", "");
14

Cookies, even ones set on the client side such as those for Google Analytics, can prevent content from being cached. To prevent this, you need to configure Varnish to discard these cookies before passing them on to Apache. In this case, we want to exclude Google Analytics and various WordPress cookies.

1    # Static content unique to the theme can be cached (so no user uploaded images)
2    if (req.url ~ "wp-content/themes/" && req.url ~ "\.(css|js|png|gif|jp(e)?g)") {
3        unset req.http.cookie;
4    }

Here we allow static content that's part of the site theme to be cached since that doesn't change often, so we unset the cookies for that request.

1    # Even if no cookies are present, I don't want my "uploads" to be cached due to their potential size
2    if (req.url ~ "/wp-content/uploads/") {
3        return (pass);
4    }

Here we prevent any user-uploaded content from being cached, since that can change often.

1    # any pages with captchas need to be excluded
2    if (req.url ~ "^/contact/")
3    {
4        return(pass);
5    }

Captchas must obviously never be cached since that will break them. In this case, we assume that the contact form has a captcha, so it gets excluded from the cache.

1    # Check the cookies for wordpress-specific items
2    if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
3        # A wordpress specific cookie has been set
4        return (pass);
5    }

Here we check for remaining WordPress-specific cookies. These would indicate that a user is signed in, in which case we may want to serve them all the latest content rather than displaying content from the cache.

1    # allow PURGE from localhost
2    if (req.method == "PURGE") {
3        if (!client.ip ~ purge) {
4            return(synth(405, "Not allowed."));
5        }
6        return (purge);
7    }

Remember where we allowed the local server to clear the cache? This section actually carries out the purge when it receives a request from an authorised client.

1    # Force lookup if the request is a no-cache request from the client
2    if (req.http.Cache-Control ~ "no-cache") {
3        return (pass);
4    }

Here we check to see if the Cache-Control HTTP header is set to no-cache. If so, we pass it straight through to Apache.

1    # Try a cache-lookup
2    return (hash);
3}

This is the last rule under vcl_recv, because it only reaches this point if the request has got past all the other rules. It tries to fetch the page from the cache. If the page is not in the cache, it passes it on to Apache and will cache the response.

1sub vcl_backend_response {
2    set beresp.grace = 5m;
3}

This is where we set how long responses are cached for. Here we've set it to 5 minutes.

With that done, we should be ready to restart Varnish and Apache. If you are using an operating system with systemd, then the following commands should restart Apache and Varnish:

1$ sudo systemctl reload apache2.service
2$ sudo systemctl reload varnish.service

For those not yet using systemd, try this instead:

1$ sudo service apache2 restart
2$ sudo service varnish restart

If you then visit your site and inspect the HTTP headers using your browser's dev tools, you'll notice the new HTTP header X-Varnish in the response. This tells you that Varnish is up and running. If you make sure you're logged out, you should hopefully see that if you load a page, and then load it again, the second response is noticeably quicker.

Installing and configuring Varnish is a relatively quick and easy way of helping your website scale to be able to serve many more users, and if the site becomes popular all of a sudden, it can make a huge difference as to whether the site can stand up to the load or not. If you need more information on how to configure Varnish for your own needs, I recommend consulting the excellent documentation.