[ 01.01.12: I have since optimized my configuration even further. See this post for my current configuration. ]

I’ve been on a strong optimization kick in recent months and have now hit the sweet spot. Combining wicked site optimization through CSS data URLs, a Varnish front end with an Nginx backend, APC caching, the W3 Total Cache plugin, Amazon as a CDN, and CloudFlare’s performance service (deep breath)…I’ve been able to get my page load times down to wickedly ludicrous.

Let’s look at what we’re talking about:

So, around 300ms to 400ms to load a full WordPress post–with DISQUS comments–from all over the world. And some of the load times are many times faster than that. I think it’s stunning, and if anyone is interested in the details of the various components, feel free to hit me up.

[ 2011-05-31 : I figured out what the problem was. I was blocking some of their servers with netfilter/iptables. That'll do it. I'm super-happy this wasn't their fault. ]

So there’s a new service out called CloudFlare that speeds up your site by man-in-the-middling it via DNS. You move your DNS to point to it and it sees all requests for your site, and can then serve those requests, filter them, etc.

It’s a two-pronged play: performance, and security.

My experience thus far has been mixed: I’ve had tons of down time due to them serving out 500s, which Pingdom didn’t like and sent me 6,000 emails about. That turned out to be some beta feature drama–or at least I thought so until I just got another flurry of alerts as I am writing this.

But load times from around the world plummeted significantly when the service went live, so I think I’m going to try and keep messing with it for a bit longer before I give up.

I also reduced the size of my Linode–an option that’s both possible (because CF takes load off my server) and practical (because I pay $20/month for CF). Doing this actually reduced my overall bill by 25% (from $80 to $60/month).

I’m not sure I’ll stick with it, however, as I need to be able to troubleshoot quickly, I need to be able to update my page and have it show up immediately (blog posts, custom articles, etc.), and generally still want to feel “in the loop” of what’s on my site. If the service keeps forcing Pingdom to send me hate mail and/or I’m unable to stay connected to my content, then I’ll probably dump it soon.

I do hope I can get it to work, though, as it has some interesting reporting and the performance gains (when it’s working properly) have been significant. ::

[ Yeah, that's a consistent <50ms load time. That's wicked fast. ]

So I just further enhanced my site performance by inlining all the core images on the site within CSS via Base64.

The web performance game is largely about reducing the number of queries that need to be made to pull a page, and every image counts against that. So using data URLs, i.e. putting images in CSS, means you actually encode your images in Base64 and place them directly into your CSS.

Your CSS then comes down once, and after that it’s cached. Subsequent page loads all over the site are then rather fast. We’re talking about going from like 9 requests for follow-up pages to like 2.

Right now, however, I’m having an issue with the display of my logo image. It’s all wonky (see the image above) in most browsers due to it showing up multiple times in the markup. It’s quite odd. I have a post over at DSLReports to see if anyone can help. If you know how to fix this I’d like to hear any thoughts.

[ 2011-05-26 : I fixed this issue; it was a broken a tag above the pertinent section. ]

In the meantime, however, let me know if the site seems any faster. Nobody ever responds when I ask that, but I ask anyway. Most pull from RSS I think.

[ Oh, I also trimmed some superfluous content out of my main blog pages (index, single, etc.); that may have helped speed things up as well... ]

::

So I’ve been tweaking my varnish configuration over the last few nights and I made the transition from file-based storage to doing everything in memory. I also tweaked a number of options regarding threading.

The result was that yesterday my server dropped a couple of times due to memory starvation and excessive paging. No bueno.

So I just upgraded my server’s memory. She’s been nice, so it was due. We’re now sporting 2GB instead of 1, and we’ve set aside a healthy chunk just for Varnish.

I’ll be keeping an eye out, but reach out if you feel the site get faster, slower, or see it go sideways.

Thanks,

Daniel

[ 01.01.12: I have since optimized my configuration even further. See this post for my current configuration. ]

So I’ve spent the last couple of weeks tweaking my web server stack. I like my site to pop, and I’ve finally achieved a configuration that I’m somewhat happy with.

In sum, I’m running Varnish as a front-end to Nginx which is running WordPress loaded with the W3-Total-Cache plugin. The W3-Total-Cache plugin is configured to use both memcached as well as Amazon S3 as its CDN. All of this sits on Ubuntu Linux with Linode as my host.

So let’s take those one at a time (or you can jump around using the TOC below).

• Nginx
• Varnish
• W3-Total-Cache
• Benchmarks
• Fin

Nginx

The first thing I did was dump Apache. I love Apache, don’t get me wrong, but I prefer to go with simple and fast if I have the option, and that’s what Nginx offers.

Again, I’m using Ubuntu, so the installations here are pretty clean.

You’re also going to need to install php-fpm to cache PHP within Nginx.

Configuration is handled by /etc/nginx/nginx.conf, where you’ll want to just do a few things:

# Miscellaneous Options
   sendfile        on;
   tcp_nopush     off;
   keepalive_timeout  30;
   tcp_nodelay        on;
   multi_accept     on;
   gzip  on;
   gzip_proxied any;
   gzip_comp_level 2;
   gzip_disable "MSIE [1-6].(?!.*SV1)";
gzip_types text/plain text/css application/x-javascript text/xml
 application/xml application/xml+rss text/javascript;

And then here’s the site config, under /etc/nginx/sites-enabled/default:

[ A couple of options have been omitted, but they should be self-explanatory, such as server name and the port you're listening on. ]

## Default location
    location / {
        root   /var/www/;
        index  index index.php;
        try_files $uri/ $uri /index.php?q=$uri&$args;
        port_in_redirect off;
    }
    location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|xml)$ {
      access_log        off;
      expires           30d;
      root /var/www/;
    }
    location ~ .php$ {
        fastcgi_split_path_info ^(.+.php)(.*)$;
        fastcgi_pass   backend;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  /var/www/$fastcgi_script_name;
        include fastcgi_params;
        fastcgi_param  QUERY_STRING     $query_string;
        fastcgi_param  REQUEST_METHOD   $request_method;
        fastcgi_param  CONTENT_TYPE     $content_type;
        fastcgi_param  CONTENT_LENGTH   $content_length;
        fastcgi_intercept_errors        on;
        fastcgi_ignore_client_abort     off;
        fastcgi_connect_timeout 60;
        fastcgi_send_timeout 180;
        fastcgi_read_timeout 180;
        fastcgi_buffer_size 128k;
        fastcgi_buffers 4 256k;
        fastcgi_busy_buffers_size 256k;
        fastcgi_temp_file_write_size 256k;
    }
    location ~ /.ht {
        deny  all;
    }
    location ~ /.git {
        deny  all;
    }
        location ~ /.svn {
            deny  all;
    }
upstream backend {
            server 127.0.0.1:9000;
    }

[ I've collected all these options from multiple sources--mostly from the official docs when possible--and have tweaked them through experimentation. Hopefully this will save you lots of time getting up and running with a decent config. ]

Varnish

So, Varnish is a wicked fast reverse proxy for serving up content. The idea is that if someone just requested something from your backend (Nginx), and it hasn’t been long, Varnish can serve it much faster than a full web server like Apache or even Nginx.

Varnish is a simple install as well when using Ubuntu.

 backend default {
     .host = "localhost";
     .port = "8080";
}
acl purge {
        "localhost";
}
sub vcl_recv {
        if (req.request == "PURGE") {
                if (!client.ip ~ purge) {
                        error 405 "Not allowed.";
                }
                return(lookup);
        }
if (req.url ~ "^/$") {
               unset req.http.cookie;
            }
}
sub vcl_hit {
        if (req.request == "PURGE") {
                set obj.ttl = 0s;
                error 200 "Purged.";
        }
}
sub vcl_miss {
        if (req.request == "PURGE") {
                error 404 "Not in cache.";
        }
if (!(req.url ~ "wp-(login|admin)")) {
                        unset req.http.cookie;
                }
    if (req.url ~ "^/[^?]+.(jpeg|jpg|png|gif|ico|js|css|txt|gz|
zip|lzma|bz2|tgz|tbz|html|htm)(\?.|)$") {
       unset req.http.cookie;
       set req.url = regsub(req.url, "\?.$", "");
    }
    if (req.url ~ "^/$") {
       unset req.http.cookie;
    }
}
sub vcl_fetch {
        if (req.url ~ "^/$") {
                unset beresp.http.set-cookie;
        }
if (!(req.url ~ "wp-(login|admin)")) {
                        unset beresp.http.set-cookie;
}
}

So a key thing to realize about Varnish is that you need to get rid of cookies to see the benefit from it. If Varnish sees cookies flying back and forth, it’s going to assume there’s some sensitive functionality at play, and it’s not going to interfere. So part of this is saying that if you don’t see wp-login or wp-admin in the URL, strip the cookies.

The key to the main config here at the top is that Varnish sits on port 80 as your public presentation of your website to the world, and your “real” web server (Nginx in our case) sits behind it on another port, e.g. 8080.

Ok, so if we spin both Nginx and Varnish up at this point we’ll have a website, and it’ll be decently fast. What we’ve done so far is:

• In our Nginx config we applied gzip compression, enabled keepalive, and allowed it to handle multiple requests at once (plus a few other settings)
• In Varnish, we’ve stripped cookies from most WordPress requests (non-admin/login), as well as made it so that when we create or update a WordPress post it’ll refresh the Varnish cache appropriately so we don’t have a stale site while waiting for expiration. We’ve also removed the cookies from the front page within Varnish.

Now on to W3-Total-Cache.

W3-Total-Cache

In the mode of clean, tight configs with as little clutter as possible, I try to run as few WordPress plugins as possible. One of them is W3-Total-Cache. It’s simply phenomenal at speeding up either Apache or Nginx.

It gets its speed gains by combining a few techniques: Browser Caching, APC for PHP caching (reducing database lookups), and the use of a CDN. I use all three of those.

First we’ll install memcached on Ubuntu:

Then install W3-Total-Cache within WordPress and perform the following steps:

1. Enable page caching (memcached)
2. Enable the object cache (memcached)
3. Enable the browser cache
4. Do NOT enable the database cache (it’ll slow everything down)
5. If your kung-fu is strong, enable the CDN functionality (you should be serving as much of your site from a CDN anyway–regardless of this plugin. I use Amazon S3)

[ I am actually not using W3 Total Cache now, as I've transfered its functionality to my nginx config and removed the plugin. Remember the principle of simplicity. ]

Benchmarks

Ok, so we now have Varnish acting as the front-end cache to Nginx, with tons of optimization happening at all layers. So the question is: How much did we improve things?

I use a myriad of tools to test web performance, but the two I’ll discuss here are Apache Bench and Which Loads Faster.

[ Don't forget to restart everything before continuing. Here's a script I use ]

Apache Bench (ab)

ab is a tool for testing the performance of web servers. It sends mad requests to sites in the form of many concurrent connections. You can install ab on Ubuntu really easily by installing the Apache Utilities:

You can then run ab against your site like so:

ab allows you to define a port as well, which gives you the option of testing Varnish + Nginx vs. Nginx directly.

The second example here is a test of Nginx without Varnish. Keep in mind that you’ll have to lower your firewall to be able to connect in to a port other than 80. You do run iptables on your web server, right?

Anyway, here is what I get when hitting a default Apache-based WordPress install (still on Ubuntu and Linode, though) using 10 concurrent connections for 1000 hits (with keepalive):

Server Software:        Apache/2.2.16
Server Hostname:        somesite.com
Server Port:            80
Document Path:          /?p=5
Document Length:        8577 bytes
Concurrency Level:      10
Time taken for tests:   57.812 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      8833000 bytes
HTML transferred:       8577000 bytes
Requests per second:    17.30 [#/sec] (mean)
Time per request:       578.119 [ms] (mean)
Time per request:       57.812 [ms] (mean, across all concurrent requests)
Transfer rate:          149.21 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       59   60   0.9     60      88
Processing:   297  517  91.1    521    1055
Waiting:      236  456  90.9    461     995
Total:        357  576  91.2    581    1114
Percentage of the requests served within a certain time (ms)
  50%    581
  66%    616
  75%    637
  80%    651
  90%    689
  95%    722
  98%    753
  99%    777
 100%   1114 (longest request)

This isn’t horrible, with 95% of requests coming in around 755ms, but that’s a default site with virtually no content in it, e.g. no images, plugins, ads, etc.

Now compare that with my pure Nginx performance (without Varnish) with all those things slowing me down:

Server Software:        nginx/0.7.65
Server Hostname:        danielmiessler.com
Server Port:            81
Document Path:          /
Document Length:        0 bytes
Concurrency Level:      10
Time taken for tests:   17.816 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Non-2xx responses:      1000
Keep-Alive requests:    0
Total transferred:      279000 bytes
HTML transferred:       0 bytes
Requests per second:    56.13 [#/sec] (mean)
Time per request:       178.162 [ms] (mean)
Time per request:       17.816 [ms] (mean, across all concurrent requests)
Transfer rate:          15.29 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       59   60   0.5     60      75
Processing:    97  109  12.3    104     172
Waiting:       97  109  12.3    104     172
Total:        157  168  12.3    163     232
Percentage of the requests served within a certain time (ms)
  50%    163
  66%    166
  75%    170
  80%    173
  90%    184
  95%    196
  98%    210
  99%    218
 100%    232 (longest request)

That’s stupid faster.

Now let’s try with Varnish added in (hitting port 80 instead of 8080):

Server Software:        nginx/0.7.65
Server Hostname:        danielmiessler.com
Server Port:            80
Document Path:          /
Document Length:        39435 bytes
Concurrency Level:      10
Time taken for tests:   7.530 seconds
Complete requests:      1000
Failed requests:        992
   (Connect: 0, Receive: 0, Length: 992, Exceptions: 0)
Write errors:           0
Keep-Alive requests:    1000
Total transferred:      39806014 bytes
HTML transferred:       39434008 bytes
Requests per second:    132.80 [#/sec] (mean)
Time per request:       75.301 [ms] (mean)
Time per request:       7.530 [ms] (mean, across all concurrent requests)
Transfer rate:          5162.33 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   5.9      0      60
Processing:    60   74  23.2     61     332
Waiting:       60   70  12.7     61     151
Total:         60   75  28.0     61     391
Percentage of the requests served within a certain time (ms)
  50%     61
  66%     80
  75%     88
  80%     89
  90%     89
  95%     90
  98%    120
  99%    298
 100%    391 (longest request)

Duh-am. That’s 95% of the requests finishing in 90ms or less! Happiness.

Which Site Loads Faster

Just for giggles I like to use a site called whichloadsfaster.com, which graphically and side-by-side tests two sites against each other in terms of load speed. Here’s what I get if I compare my Varnish+Nginx against pure Nginx over 100 pulls (notice the port number in the fields at the top of the screenshot):

So, almost a 2.5x improvement in speed using Varnish + Nginx vs. just Nginx for loading my WordPress front page. And for hitting my non-database PHP content I am at around 25% faster (tests of /study not shown for brevity).

Fin

To sum it, we’re looking at nearly a 10x improvement over a default Apache / WordPress install, and nearly 2.5x over Nginx, et al. without Varnish. That’s for WordPress content that’s touching MySQL, but we still see major improvement with standard PHP as well.

And keep in mind that this was with only ten concurrent connections. This setup gets really brilliant when you start upping that to over 100 concurrent connections. :) I’ll be posting some of those results soon.

So that’s it for now. Happy server tweaking, and feel free to ping me with questions either in the comments or via email. ::

Notes

¹ I also make extensive use of Google Page Speed for in-depth analysis of nagging delays.
² I also do some mysql optimization, but it’s nothing special: I just enable caching with the same config that’s found all over the Internet.
³ I know there are tons of variables involved here–like pulling standard PHP vs. having to go to a database, etc., but I tried to isolate the main ones. And I also know that it’s possible to tweak Apache to get some pretty decent performance.

I’ve been doing a lot of server optimization lately, and thus have been checking out what HTTP stacks the major sites are running. I found Facebook’s HTTP headers to be interesting for a different reason (see screenshot above).

So they actually take the time and resources to send out a no-p3p header to everyone. That’s what I call dedication to a stance. But seriously, who do they think is learning about the Facebook privacy policy via parsed HTTP headers?

Weirdness. ::

[ For those who asked, this screenshot was taken using the HTTP header bookmarklet. ]