- Unsupervised Learning
- Posts
- A Varnish Primer
A Varnish Primer
Despite the title, this article has nothing to do with painting. Here I’m going to talk a bit about the Varnish Cache — a web application accelerator designed to dramatically improve how quickly your website loads. Varnish works by caching and serving as much content as possible as “static”, including dynamic results, e.g. blog posts from a CMS like WordPress.
Administration
Administration is handled by two main files: /etc/default/varnish and /etc/varnish/default.vcl. The first is a main daemon config file with information about how much memory is used and such, and the second is the configuration file for the website itself.
Main Config : /etc/default/varnish
[ The config being discussed runs on Ubuntu. ]
The main config file is where you configure how the daemon itself runs, including memory, threading, etc. The primary section looks like this:
DAEMON_OPTS="-a :80 -T localhost:1974 -f /etc/varnish/default.vcl -S /etc/varnish/secret -w "250,24000,30" -p thread_pools="4"; -p session_linger="200"; -s malloc,"512M";
Other things that can be set include whether to start the service at boot, etc.
VCL Config : /etc/varnish/default.vcl
VCL is the language of Varnish configuration. Its syntax is C-like, which makes sense since it’s compiled and run as a module when the service is started.
The rules allow you to define how to handle various types of content, e.g. stripping cookies, setting cookies, defining what happens when something misses the cache, etc. Here’s an example exerpt:
backend default {
.host = "localhost";
.port = "8080";
}
acl purge {
"localhost";
"danielmiessler.com";
}
sub vcl_recv {
if (req.request == "PURGE") {
if(!client.ip ~ purge) {
error 405 "Not allowed.";
}
return (lookup);
}
set req.grace = 30s;
if (req.http.Accept-Encoding) {
if (req.url ~ ".(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
# No point in compressing these
remove req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
set req.http.Accept-Encoding = "deflate";
} else {
# unkown algorithm
remove req.http.Accept-Encoding;
}
}
The main thing I do in my config file is strip cookies from blog posts in WordPress, which traditionally prevent Varnish from caching content (Varnish assumes that if it has a cookie then it needs to be refreshed constantly).
I also enable purging so that when WordPress updates a post or creates a new one the Varnish cache for that page will be purged. This will keep the site instantly updated with changes instead of requiring a time-based drop out of the cache, which could take a while.
Administration
Ok, so once you have it up and running, how do you interact with it? Here are some tools
varnishncsavarnishncsa is my preferred way of looking a Varnish logs. It displays them in Apache format, which is how we’re accustomed to looking at web logs. One thing to realize about Varnish logs is that they go, by default, to shared memory and not to disk. So don’t go hunting for your logs in /var/log/; they won’t be there unless you put them there via -w /log/path.
varnishlogvarnishlog is the more verbose way of looking at logs, with a massive amount of information coming in for every hit on the daemon. This includes all the headers and can be pretty overwhelming (hence varnishncsa).
varnishstatvarnishstat is a great tool for looking at the status of your cache. You can see hits, misses, purges, and the overall performance of the Varnish daemon. Most important, perhaps, is how many hits vs. misses you’re seeing. Too many misses means that you may need to adjust your .vcl.
varnishadmvarnishadm allows you to administer the daemon on the fly while the service is running, and without disrupting the serving of requests.
varnishreplayvarnishreplay is used to replay logs through the daemon, often to prime it for real traffic.
varnishsizesvarnishsizes reads varnishd(1) shared memory logs and presents a continuously updated histogram showing the distribution of the last N requests by their processing. The value of N and the vertical scale are displayed in the topleft corner. The horizontal scale is a logarithmic representation of bytes. Hits are marked with a pipe character (“|”), and misses are marked with a hash character (“#”).
varnishtopvarnishtop reads varnishd(1) shared memory logs and presents a continuously updated list of the most commonly occurring log entries. With suitable filtering using the -I, -i, -X and -x options, it can be used to display aranking of requested documents, clients, user agents, or any other information which is recorded in the log.
WordPress Compatibility
The main thing you need to do to make Varnish compatible with WordPress is ensure that Varnish handles certain pieces (in order to get the benefits), while it ignores others (like the Admin page). If you try to have Varnish handle the admin functionality in WordPress you’ll need to strip cookies, which will prevent you from logging in.
Here’s some code to use:
if (!(req.url ~ "wp-(login|admin)")) {
unset req.http.cookie;
}
Unsupervised Learning — Security, Tech, and AI in 10 minutes…
Get a weekly breakdown of what's happening in security and tech—and why it matters.
Purging
In order to purge properly for CMSs like WordPress, you want to use something like this:
First ensure that you have permission to purge:
acl purge {
"localhost";
}
Then purge:
sub vcl_hit{
if (req.request == "PURGE") {
purge;
error 200 "Purged.";
}
}
sub vcl_miss {
if (req.request == "PURGE") {
purge;
error 200 "Purged.";
}
return (fetch);
}
Benefits
So the obvious question is: why do this? The answer is simple: raw, silly, speed. Here are my latest load times from Pingdom. Keep in mind, these aren’t ping times: this is loading a full web page. And not a static one either — this page being loaded is being served from WordPress, and includes an image.
Conclusion
This is a very quick primer designed to make you aware of Varnish-cache and get you started; hit me up if you have any questions and I’ll try to help.