Granted I’m only a couple hundred miles away from my datacenter for this test, but I am loading a fairly typical page of content here in 128ms. That’s pretty sick. We’re talking a tenth of a second for a full page load.

Here’s how I did it.

::

Some people really dislike my thoughts (or enthusiasm?) on the controversial topic of improving website performance.

::

Posted via email from danielmiessler.com | posterous

Posted via email from danielmiessler.com | posterous

Everything should be fine now; let me know if that’s not the case.

Here are the things I need:

1. It needs to be faster than my existing WordPress solution, which means faster than ~300ms-400ms for a typical post’s round trip. If I can’t get a major speed upgrade I probably won’t switch.
2. It’d be nice if it used MySQL as the backend. I have ~5,000 posts I’d like to drop right in. But if I can migrate them without too much effort that’s fine as well.
3. I need my permalink structure to survive.
4. I need to be able to manage the site via git.
5. In general, I need to get my site back to its full running configuration without some sort of major functionality missing.

Any ideas on what platform(s) I should be looking at?

::

[ 2012-01-02 : Well, it looks great on a Mac, anyway. Evidently Optima isn't available on a lot of systems. I looked at some Google Web Font options, as per the commenter below, but I've read bad things about how they compare to Optima, and I'd rather not take the performance hit for something less than optimal. ]

–

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

I run Varnish here on the site, with Nginx as the backend. I’ve written before about my overall setup, and how to improve site performance, so I won’t go into it here.

Here I want to cover a subtlety of putting varnish in front of nginx (or any other web server, really) with respect to redirects. Redirects occur when the client makes a request to the server that cannot (or should not) be served in the manner asked, e.g. asking for /resource instead of /resource/. The trailing slash matters, as those to URLs look the same to humans but are not the same to the web server.

Web servers such as Apache and Nginx handle this naturally by sending clients a redirect for the latter when you ask for the former, i.e. you ask for /resource, you get sent back a redirect (HTTP code 301 or 301) saying that you should be asking for /resource/.

Simple enough. This all happens transparently to the client, as the initial request, the redirect response, and the second request happen so quickly that most people don’t even notice the turnaround.

But things can get a bit weird when you have two web daemons fielding requests, like when you have Varnish in front of Nginx. The problem is that the two daemons have to be listening on different ports. Varnish and Nginx can’t both be on the same exact IP address and port combination (TCP/IP stacks don’t like that), so you have to perform some sort of networking trickery to get them to work.

One common way of doing it is to have Varnish listen out front on port 80, and to have Nginx listen in back on 8080, or 81, or whatever. Then you tell Varnish what port Nginx is on and everything works great.

Until someone asks for /resource.

When that happens, Varnish asks Nginx for the resource and gets a redirect, only the redirect isn’t to a usable URL — it’s to the backend URL. So instead of being sent to site.com/resource/ like you would be if Varnish weren’t there, you get sent to site.com:8080/resource/, which your users shouldn’t be able to connect to (you do have a firewall on your server, right?).

But there’s a simple solution.

TL;DR: Have Varnish listen on 1.2.3.4:80 (your external IP), and have Nginx listen on localhost:80. This way both daemons are convinced they’re authoritative, so when Nginx responds with redirects the won’t be mangled with some alternative port in the URL that external users can’t reach.

Here’s what the localhost config looks like in Nginx:

    server {
        listen  localhost:80;
        server_name  localhost;
    ...

Now all your redirects should look as if Nginx is sitting in front, and you should be good to go. Hope this helps.

[ Note: Don't try to handle this in Varnish itself; this is a backend issue and should be handled by properly situating your daemons, not by rewrite hand-waving on the acceleration server. ]

::

Posted via email from danielmiessler.com | posterous

More than ever, the speed of your website is now affecting its perceived quality, and this is true of both users and search engines. Using the techniques  below you will be able to get the most from limited server resources, typically allowing you to handle millions of requests per day from a relatively small VPS.

1. Use a Faster Web Server
   You may have just set up your site and be using its default web server (most likely Apache). Apache is great, but it’s usually bloated to the point of being rather slow. Consider moving to a web server like Nginx or LightHTTPD that focuses on speed and simplicity rather than being able to do anything and everything. This site uses Nginx.

2. Implement a Cloud Proxy Service
   Services like CloudFlare (used here) sit in front of your site and proxy all sorts of content for you, such as CSS, JavaScript, and images. Not only do they serve these things really quickly, but they also serve them from the closest location to the user, so you’ll see much faster load times from across the globe. In addition, this tier can provide minification, script optimization (e.g. Google Analytics), security, and a number of other services. This site uses CloudFlare.

3. Use an HTTP Accelerator
   One of the most massive improvements you can make to your site is putting an accelerator in front of your main web daemon. Caches serve content by storing responses as static files and serving them directly out of memory. As a result, they can often respond in mere milliseconds. This site uses Varnish for that role, which I highly recommend.

1. Ensure Your Content Caching Settings Are Solid
   You need to make sure you’re telling users’ browsers to cache the content you’re serving, so that subsequent requests to your site won’t require them to pull the CSS, images, etc. again. This will improve your site experience enormously. Give your CSS and other long-term items a significant expiration time. I handle all of my settings through <code>nginx</code> configuration combined with CloudFlare’s optimizations.

2. Put Your Images in CSS
   If you’re serious about getting your site to pop you’ll want to reduce the number of images that get requested when your site is pulled. One way to do that is to simply remove images; another is to place your core images into data URLs served by CSS. This way, when a visitor loads your CSS (only once), the main images are now in cache and will display instantly everywhere else on your page.

3. Use a Content Delivery Network (CDN)
   This is a lot like using a cloud-based caching server in that some of your content is coming from a distributed content provider, but it’s different in that you don’t have to put anything between you and your users. Most people store their images here so that they’re served from very fast, locally-placed servers (e.g. Amazon). Keep in mind that Cloud Proxy services such as CloudFlare give much of the same advantage, so you may want to do one or the other. I switched from S3 to CloudFlare.

4. Get Off of Shared Hosting
   If you’re still on a shared host, get off it. Anyone serious about speed should either be on a VPS or a dedicated server. I use Linode, which I have found to be nothing less than excellent over the years.

5. Reduce the Number of Requests Required
   Fundamentally, the game is simple: reduce the number of requests required to serve your page, and make each response come back quickly. You should consider everything here, including how many DNS lookups will be required, how many external scripts are called, etc. Remove as much as you can from your pages; start from scratch and add only what you absolutely must include.

6. Test, Test, Test
   As you start implementing the techniques above you’ll want to see how they affect your site’s performance. You can use the tools located here to do that.

7. Enjoy
   Once you’ve done these steps you should be enjoying some serious performance improvements. My site went from loading in ~2 seconds to loading in approximately ~250ms using these techniques.

Finally, if you have any other techniques that should be added here, do let me know, and don’t hesitate to ping me with questions or ideas.

::

So you’ve replaced with Apache with Nginx, or put Varnish out in front of your main webserver. Or maybe you just spent time converting all your main site images to CSS Data URLs. Either way, you’re now ready to taste the fruits of your work via speed tests. Here are some of the best resources for doing that.

1. Google Page Speed Online | Google
   This is a brand-new service from Google that replicates it’s Page Speed service in online form. It’s focus is not on displaying a visual of the response, but rather on giving recommendations on how to improve speed.

2. Site-perf.com | Site Performance
   This site is very responsive and lets you test from a few different servers — two in the U.S. and two overseas.

3. Websiteoptimization.com | Website Optimization
   This service (and it’s accompanying documentation and guidance around optimization) is one of the best at giving recommendations on what to fix.

4. Whichloadsfaster.com | Which Site Loads Faster?
   This site allows you to compare your site against another to see which loads faster. It also lets you do multiple tests in a row to get a better baseline. Hint: if you load faster than Google’s front page then you’re doing well.

5. tools.pingdom.com/fpt/ | Pingdom Load Time Test
   Pingdom’s test service is perhaps the cleanest in terms of presentation. I find myself using this one quite often.
   

6. Browser Load Time Stopwatch | LifeHacker
   This is also one of my most-used options for testing speed. It’s different than the others because it’s not a web page, but rather a piece of JavaScript that runs. You add the link to your bookmarks and execute it to test the page you’re currently on.

7. Google Chrome Page Speed | Extension
   The Page Speed chrome extension is phenomenal for getting results that look like a local application in terms of detail and presentation.

8. Google Chrome Developer Tools | Built-in Tool
   Many don’t realize that Google Chrome has a wicked-powerful set of tools built in. By opening the tools from the options menu you not only have the classics that let you manipulate JavaScript, see page layout, etc., but you also have a network display that lets you see what took the longest to load. It’s like a mini Page Speed display without the extension.

9. YSlow | Browser Extension
   YSlow is like Google’s Page Speed, only not quite as advanced and available for more platforms. It’s now available for Chrome as well.

10. webpagetest.org | Web Page Test
    This is another very solid web-based test that gives extremely detailed output with regard to what the request looked like from the browser.

11. Google Analytics | Google
    It’s now possible to track load speed through Google Analytics. Simply get the latest version of the tracking script that has the functionality in it, and you’ll see it as an option within your metrics.

12. Google’s Webmaster Tools
    Google’s Webmaster Tools has a Page Speed menu under Labs that shows you how fast your various pages have been loading. The options is very beta, however, as it gives you no control over what to test and not much information on what has been tested.

13. Pingdom’s Web Monitoring Service
    This offering doesn’t test on demand like the others, but rather is an ongoing service that continuously tests pages that you define to let you know how their responsiveness changes over time. It’s very helpful for noticing how various techniques, like CloudFlare or Varnish, improve site load times. And since their servers are located all over the world, you get to see how fast your site loads from different countries as well.

14. Blitz.io. It is a real-deal traffic simulation platform.

If you know of any I should add to this list, let me know.

[ UPDATE: Yes, I know I'm over ten options now...but it's too late to change the title. :) ]

::

I’ve taken Amazon out of the loop and put CloudFlare back inline, so the content is coming from me directly, but being cached by CloudFlare. This is in contrast to CloudFlare fetching my content from Amazon S3 and then caching it.

So it’s basically just taking one step out of the process. Let me know if you see any change.

::

Ok, so the feeds are back up and running with the proper URL (feeds.danielmiessler.com/danielmiessler)

Even if you didn’t experience any issues, please do me a favor and make sure you’re subscribed to the following address and not any other:

http://feeds.danielmiessler.com/danielmiessler

Thanks.

::

In case you haven’t noticed already, the feeds for the site are borked right now. I’m doing some account Google consolidation and my feedburner properties were a casualty.

I’m working on getting them re-established, but it could be a bit. Sorry for the annoyance.

I’ve not yet updated the RSS feed (I have to do it manually, ugh), but I’ll be doing so soon.

Let me know what you guys think. Is it approachable? Is the CSS inviting enough? Etc.

Thanks.

I’ve been coming repeatedly to a simple conclusion regarding content creation and the use of social services:

If something is worth creating, it’s worth keeping within your own control.

This includes quick little comments on the world, or sharing interesting links (Twitter), to posting and sharing content through something like Google+. I’ve been blogging since 1999, so I have over 5,000 posts saved within mysql, but what about Twitter? Where did those 3,000 posts go? Was any of it worth saving? Why did I put it there and not on my site?

To me it’s quite simple: if it’s worth sharing then it’s good enough for me to feel bad about not having permanently as part of my digital avatar. And if it’s not good enough for that, then maybe I shouldn’t be creating/sharing it in the first place.

The concept of some content being on Twitter, some content being on Facebook, and some content being in Google+ is basically the approach of someone who is not taking what they produce seriously. After five years of doing this, what will you have? And after 20? What history will there be of your experiences and/or contributions?

In 20 years, your Foursquare check-ins, your silly jokes, your poignant observations, your best-find articles, your own deeper thoughts — they will matter to you. Some of it will matter because it will be a history of who you were and what you were doing, but some will matter because the content itself was good.

If the answer is, “I’m not trying to make “contributions”, you uppity fool, I’m just having fun…”, then you now know why I don’t read anything you write, and why nobody else does either. If you don’t take what you produce seriously, why should anyone else? And please don’t confuse seriousness with a lack of silliness or humor — most of the funniest and silliest people you’ll find online take their output very seriously.

Quite simply, if you care about what you put on Twitter, Google+, or other disparate services, consider the possibility that you should be capturing and centralizing that content somewhere that you control, e.g. your own site, or a single trustworthy service that you can reliably export from.

All this content represents who you were and what you did. You’ll want it later. Start now. ::

But when I moved Varnish out of the way, Nginx handled this without issue.

So I set out to figure out a recipe for adding the trailing slash within Varnish. The following solution works for me, but if you know a better way do let me know.

# If the requested URL contains $directory and doesn’t contain index.php, add a trailing slash to the end.

sub vcl_recv {
if ((req.url ~ "/directory" ) && (! (req.url ~ "index.php"))){
   set req.url = req.url "/";
}
}

::

I’ve spend significant time optimizing my site for speed. It’s true that Google favors sites that are fast, but I do it because I hate waste, and I find it enjoyable to optimize things.

The yield from this effort is that I now average ~200ms load times for my WordPress content. In short, my site is fast, and I get a lot of questions and comments on how I get this kind of speed, so here’s my setup:

1. I run highly optimized markup. I wrote my theme myself (it’s the same for my WordPress and custom content), and it’s focused on simplicity and speed.
2. I use inline CSS data to serve my core site images. So, once you pull my CSS you have my logo image and every other image you see in my core template. And that’s cached, so you only get it once. Everything pops after that.
3. I run Nginx instead of Apache. It’s attacked less, extremely lightweight, and very fast.
4. I compress and cache the hell out of everything using nginx.
5. I run Varnish as a front-end to Nginx, which gives wicked fast response times. Basically, it serves your entire site as static files–out of memory.
6. I use CloudFlare, which sits in front of all of this, caches my images and CSS and serves them from local distribution points worldwide, and fields all of my junk traffic such as bots and low-level web attacks.

The results have their own voice.

[ In addition to speed, this configuration allows me to take significant traffic while barely noticing. I've been on the front page of Reddit and Hacker News a number of times using this configuration, and my server giggled slightly. ]

So yeah, this config is rockin’, but I am always looking to improve. Hit me up with questions, comments, and ways you think I can optimize. I love to hear from fellow optimizers.

::

I’ll take that. ::

Posted via email from danielmiessler.com | posterous