Logs from Splunk

CloudFlare is a great service that proxies your site’s traffic in order to offer performance gains and filtering options. It can compress and cache static content such as CSS files, JavaScript, and image files and then geographically optimize how they’re given to your users (think CDN).

One annoying issue, however, is the fact that because it’s a proxy you see incoming requests as coming from CloudFlare servers rather than the original client. So if you’re doing any cool data analytics on your server your source IP information will be borked.

There’s an easy way to fix it, however.

I run Nginx as my main webserver, and Ubuntu’s version of the app includes support for the http-real-ip module, which allows you to specify a set of proxy server IPs and the original IP header within the forwarded traffic so you can map it properly.

So, using Nginx, edit your nginx.conf file and add the following to your http section:

http {
set_real_ip_from   204.93.240.0/24;
set_real_ip_from   204.93.177.0/24;
set_real_ip_from   199.27.128.0/21;
set_real_ip_from   173.245.48.0/20;
set_real_ip_from   103.22.200.0/22;
set_real_ip_from   141.101.64.0/18;
set_real_ip_from   108.162.192.0/18;
real_ip_header     CF-Connecting-IP;

Restart Nginx and you’ll start seeing original IPs in your logs.

Varnish

If you’re like me and you have Varnish in front of Nginx, there’s one more layer of complexity. With Varnish passing to Nginx you’ll get a source IP of 127.0.0.1 for all requests to Nginx, which is the same problem as above with CloudFlare.

Luckily, the fix is just as simple: just add 127.0.0.1 to the list of proxy IPs so that Nginx pulls the CF-Connecting-IP header out of the Varnish requests just like those from CloudFlare.

http {
set_real_ip_from   204.93.240.0/24;
set_real_ip_from   204.93.177.0/24;
set_real_ip_from   199.27.128.0/21;
set_real_ip_from   173.245.48.0/20;
set_real_ip_from   103.22.200.0/22;
set_real_ip_from   141.101.64.0/18;
set_real_ip_from   108.162.192.0/18;
set_real_ip_from   127.0.0.1/32;
real_ip_header     CF-Connecting-IP;

[ Note that the list of CloudFlare IPs changes regularly, so be sure to keep your set of CloudFlare servers updated. ]

::

I run Varnish here on the site, with Nginx as the backend. I’ve written before about my overall setup, and how to improve site performance, so I won’t go into it here.

Here I want to cover a subtlety of putting varnish in front of nginx (or any other web server, really) with respect to redirects. Redirects occur when the client makes a request to the server that cannot (or should not) be served in the manner asked, e.g. asking for /resource instead of /resource/. The trailing slash matters, as those to URLs look the same to humans but are not the same to the web server.

Web servers such as Apache and Nginx handle this naturally by sending clients a redirect for the latter when you ask for the former, i.e. you ask for /resource, you get sent back a redirect (HTTP code 301 or 301) saying that you should be asking for /resource/.

Simple enough. This all happens transparently to the client, as the initial request, the redirect response, and the second request happen so quickly that most people don’t even notice the turnaround.

But things can get a bit weird when you have two web daemons fielding requests, like when you have Varnish in front of Nginx. The problem is that the two daemons have to be listening on different ports. Varnish and Nginx can’t both be on the same exact IP address and port combination (TCP/IP stacks don’t like that), so you have to perform some sort of networking trickery to get them to work.

One common way of doing it is to have Varnish listen out front on port 80, and to have Nginx listen in back on 8080, or 81, or whatever. Then you tell Varnish what port Nginx is on and everything works great.

Until someone asks for /resource.

When that happens, Varnish asks Nginx for the resource and gets a redirect, only the redirect isn’t to a usable URL — it’s to the backend URL. So instead of being sent to site.com/resource/ like you would be if Varnish weren’t there, you get sent to site.com:8080/resource/, which your users shouldn’t be able to connect to (you do have a firewall on your server, right?).

But there’s a simple solution.

TL;DR: Have Varnish listen on 1.2.3.4:80 (your external IP), and have Nginx listen on localhost:80. This way both daemons are convinced they’re authoritative, so when Nginx responds with redirects the won’t be mangled with some alternative port in the URL that external users can’t reach.

Here’s what the localhost config looks like in Nginx:

    server {
        listen  localhost:80;
        server_name  localhost;
    ...

Now all your redirects should look as if Nginx is sitting in front, and you should be good to go. Hope this helps.

[ Note: Don't try to handle this in Varnish itself; this is a backend issue and should be handled by properly situating your daemons, not by rewrite hand-waving on the acceleration server. ]

::

I never stop being fascinated by the fact that web admins have boxes online that anyone in the world can reach out and touch. Visualizing this activity is enthralling to me, as visualization often highlights answers to existing questions while exposing new ones.

Geolocated firewall drops within the last 4 hours

[ Fullsize image. ]

Splunk excels at this task, and if you haven’t used it yet I suggest you get a copy (free) and install it immediately (or sooner). Splunk lets you search pretty much any type of log data in a very Unixy way: you can do basic searches, pipe them to additional filters, and pipe those to various types of output. It’s quite stunning. As an example, the map above was created with the following syntax:

firewall drop | geoip SRC

That command (result shown above) just piped all your firewall drops into a command called geoip, which is used by Google Maps — one of the applications that you can install (free) on Splunk. It takes any IP information as input, runs it through the GeoIP database, and maps it onto, well, a Google Map. If you don’t think that’s über, then stop reading and attack yourself. Using Splunk combined with GeoIP information, you can ask some wicked-cool questions like:

• what countries are hitting the drop rule on my firewall?
• what cities are triggering firewall drops on TCP 1433?
• show me everyone requesting any of the directories denied in robots.txt
• what country hits my time server the most?
• what passwords are people guessing against my SSH server?
• who’s sending me SQL commands in HTTP traffic?
• give me a list of all user-agents that hit me from Australia

Etc, etc. It’s whatever you can come up with, and it depends on the types of things you’re hosting and curious about. So let’s play a bit more. Let’s look at firewall drops, by destination port, coming from China, in the last 24 hours:

Firewall drops from china, by port

[ Fullsize image. ]

Boom: sorted list of ports that they tried to touch. Oh, and you can do another pipe and create a graph on the fly if you wanted to. And then the questions: what are these ports that they’re trying to hit, and why? Should we check some malware lists, perhaps?

Let’s pivot by city (yes, you can pull city data too) and see if there are any universities in those towns.

A list of Chinese cities hitting my firewall

Interesting.

You’re limited only by your creativity with this. Go get a copy of Splunk and load up a couple of apps. Everything I’ve shown here is free — both Splunk itself and the Google Maps application. And they have tons more apps as well: some that very attractively display your Cisco logs, Unix logs, Windows logs, and a myriad of others.

Have fun with it, and hit me up with some cool ideas for search queries.

::

More than ever, the speed of your website is now affecting its perceived quality, and this is true of both users and search engines. Using the techniques  below you will be able to get the most from limited server resources, typically allowing you to handle millions of requests per day from a relatively small VPS.

1. Use a Faster Web Server
   You may have just set up your site and be using its default web server (most likely Apache). Apache is great, but it’s usually bloated to the point of being rather slow. Consider moving to a web server like Nginx or LightHTTPD that focuses on speed and simplicity rather than being able to do anything and everything. This site uses Nginx.

2. Implement a Cloud Proxy Service
   Services like CloudFlare (used here) sit in front of your site and proxy all sorts of content for you, such as CSS, JavaScript, and images. Not only do they serve these things really quickly, but they also serve them from the closest location to the user, so you’ll see much faster load times from across the globe. In addition, this tier can provide minification, script optimization (e.g. Google Analytics), security, and a number of other services. This site uses CloudFlare.

3. Use an HTTP Accelerator
   One of the most massive improvements you can make to your site is putting an accelerator in front of your main web daemon. Caches serve content by storing responses as static files and serving them directly out of memory. As a result, they can often respond in mere milliseconds. This site uses Varnish for that role, which I highly recommend.

1. Ensure Your Content Caching Settings Are Solid
   You need to make sure you’re telling users’ browsers to cache the content you’re serving, so that subsequent requests to your site won’t require them to pull the CSS, images, etc. again. This will improve your site experience enormously. Give your CSS and other long-term items a significant expiration time. I handle all of my settings through <code>nginx</code> configuration combined with CloudFlare’s optimizations.

2. Put Your Images in CSS
   If you’re serious about getting your site to pop you’ll want to reduce the number of images that get requested when your site is pulled. One way to do that is to simply remove images; another is to place your core images into data URLs served by CSS. This way, when a visitor loads your CSS (only once), the main images are now in cache and will display instantly everywhere else on your page.

3. Use a Content Delivery Network (CDN)
   This is a lot like using a cloud-based caching server in that some of your content is coming from a distributed content provider, but it’s different in that you don’t have to put anything between you and your users. Most people store their images here so that they’re served from very fast, locally-placed servers (e.g. Amazon). Keep in mind that Cloud Proxy services such as CloudFlare give much of the same advantage, so you may want to do one or the other. I switched from S3 to CloudFlare.

4. Get Off of Shared Hosting
   If you’re still on a shared host, get off it. Anyone serious about speed should either be on a VPS or a dedicated server. I use Linode, which I have found to be nothing less than excellent over the years.

5. Reduce the Number of Requests Required
   Fundamentally, the game is simple: reduce the number of requests required to serve your page, and make each response come back quickly. You should consider everything here, including how many DNS lookups will be required, how many external scripts are called, etc. Remove as much as you can from your pages; start from scratch and add only what you absolutely must include.

6. Test, Test, Test
   As you start implementing the techniques above you’ll want to see how they affect your site’s performance. You can use the tools located here to do that.

7. Enjoy
   Once you’ve done these steps you should be enjoying some serious performance improvements. My site went from loading in ~2 seconds to loading in approximately ~250ms using these techniques.

Finally, if you have any other techniques that should be added here, do let me know, and don’t hesitate to ping me with questions or ideas.

::

So you’ve replaced with Apache with Nginx, or put Varnish out in front of your main webserver. Or maybe you just spent time converting all your main site images to CSS Data URLs. Either way, you’re now ready to taste the fruits of your work via speed tests. Here are some of the best resources for doing that.

1. Google Page Speed Online | Google
   This is a brand-new service from Google that replicates it’s Page Speed service in online form. It’s focus is not on displaying a visual of the response, but rather on giving recommendations on how to improve speed.

2. Site-perf.com | Site Performance
   This site is very responsive and lets you test from a few different servers — two in the U.S. and two overseas.

3. Websiteoptimization.com | Website Optimization
   This service (and it’s accompanying documentation and guidance around optimization) is one of the best at giving recommendations on what to fix.

4. Whichloadsfaster.com | Which Site Loads Faster?
   This site allows you to compare your site against another to see which loads faster. It also lets you do multiple tests in a row to get a better baseline. Hint: if you load faster than Google’s front page then you’re doing well.

5. tools.pingdom.com/fpt/ | Pingdom Load Time Test
   Pingdom’s test service is perhaps the cleanest in terms of presentation. I find myself using this one quite often.
   

6. Browser Load Time Stopwatch | LifeHacker
   This is also one of my most-used options for testing speed. It’s different than the others because it’s not a web page, but rather a piece of JavaScript that runs. You add the link to your bookmarks and execute it to test the page you’re currently on.

7. Google Chrome Page Speed | Extension
   The Page Speed chrome extension is phenomenal for getting results that look like a local application in terms of detail and presentation.

8. Google Chrome Developer Tools | Built-in Tool
   Many don’t realize that Google Chrome has a wicked-powerful set of tools built in. By opening the tools from the options menu you not only have the classics that let you manipulate JavaScript, see page layout, etc., but you also have a network display that lets you see what took the longest to load. It’s like a mini Page Speed display without the extension.

9. YSlow | Browser Extension
   YSlow is like Google’s Page Speed, only not quite as advanced and available for more platforms. It’s now available for Chrome as well.

10. webpagetest.org | Web Page Test
    This is another very solid web-based test that gives extremely detailed output with regard to what the request looked like from the browser.

11. Google Analytics | Google
    It’s now possible to track load speed through Google Analytics. Simply get the latest version of the tracking script that has the functionality in it, and you’ll see it as an option within your metrics.

12. Google’s Webmaster Tools
    Google’s Webmaster Tools has a Page Speed menu under Labs that shows you how fast your various pages have been loading. The options is very beta, however, as it gives you no control over what to test and not much information on what has been tested.

13. Pingdom’s Web Monitoring Service
    This offering doesn’t test on demand like the others, but rather is an ongoing service that continuously tests pages that you define to let you know how their responsiveness changes over time. It’s very helpful for noticing how various techniques, like CloudFlare or Varnish, improve site load times. And since their servers are located all over the world, you get to see how fast your site loads from different countries as well.

14. Blitz.io. It is a real-deal traffic simulation platform.

If you know of any I should add to this list, let me know.

[ UPDATE: Yes, I know I'm over ten options now...but it's too late to change the title. :) ]

::

I made a couple of changes to the site today and I’m curious as to whether anyone can notice a speed difference — either positive or negative.

I’ve taken Amazon out of the loop and put CloudFlare back inline, so the content is coming from me directly, but being cached by CloudFlare. This is in contrast to CloudFlare fetching my content from Amazon S3 and then caching it.

So it’s basically just taking one step out of the process. Let me know if you see any change.

::

[ No, this isn't in place. But at one point I was angry enough to try it for a while. ]

My web configuration makes use of Varnish as a front-end, and I noticed a bug recently where asking for “/study” was failing, while asking for “/study/” would succeed.

But when I moved Varnish out of the way, Nginx handled this without issue.

So I set out to figure out a recipe for adding the trailing slash within Varnish. The following solution works for me, but if you know a better way do let me know.

# If the requested URL contains $directory and doesn’t contain index.php, add a trailing slash to the end.

sub vcl_recv {
if ((req.url ~ "/directory" ) && (! (req.url ~ "index.php"))){
   set req.url = req.url "/";
}
}

::

I’ve spend significant time optimizing my site for speed. It’s true that Google favors sites that are fast, but I do it because I hate waste, and I find it enjoyable to optimize things.

The yield from this effort is that I now average ~200ms load times for my WordPress content. In short, my site is fast, and I get a lot of questions and comments on how I get this kind of speed, so here’s my setup:

1. I run highly optimized markup. I wrote my theme myself (it’s the same for my WordPress and custom content), and it’s focused on simplicity and speed.
2. I use inline CSS data to serve my core site images. So, once you pull my CSS you have my logo image and every other image you see in my core template. And that’s cached, so you only get it once. Everything pops after that.
3. I run Nginx instead of Apache. It’s attacked less, extremely lightweight, and very fast.
4. I compress and cache the hell out of everything using nginx.
5. I run Varnish as a front-end to Nginx, which gives wicked fast response times. Basically, it serves your entire site as static files–out of memory.
6. I use CloudFlare, which sits in front of all of this, caches my images and CSS and serves them from local distribution points worldwide, and fields all of my junk traffic such as bots and low-level web attacks.

The results have their own voice.

[ In addition to speed, this configuration allows me to take significant traffic while barely noticing. I've been on the front page of Reddit and Hacker News a number of times using this configuration, and my server giggled slightly. ]

So yeah, this config is rockin’, but I am always looking to improve. Hit me up with questions, comments, and ways you think I can optimize. I love to hear from fellow optimizers.

::