Random Musings of an Insane Mind

A while back I wrote a plugin to take care of a particular client traffic problem. As the traffic came in very quickly and unexpectedly, I had only minutes to come up with a solution. As I knew Varnish pretty well, my initial reaction was to put the site behind Varnish. But, there’s a problem with Varnish and WordPress.

WordPress is a cookie monster. It uses and depends on cookies for almost everything – and Varnish doesn’t cache assets that contain cookies. VCL was modified and tweaked, but, the site was still having problems.

So, a plugin was born. Since I was familiar with ESI, I opted to write a quick plugin to cache the sidebar and the content would be handled by Varnish. On each request, Varnish would assemble the Edge Side Include and serve the page – saving the server from a meltdown.

The plugin was never really production ready, though, I have used it for a year or so when particular client needs came up. When Varnish released 3.0, ESI could work with GZipped/Deflated content which significantly increased the utility of the plugin.

If you would like to read a detailed explanation of how the plugin works and why, here’s the original presentation I gave in Florida.

You can find the plugin on WordPress’s plugin hosting at http://wordpress.org/extend/plugins/cd34-varnish-esi/.

I’ve uploaded two plugins that were written for this site.

The first one is cd34-social which uses Async Javascript to load the Google+, Twitter and Facebook social media buttons. Unlike most of the other plugins, this does use Async Javascript so the buttons will appear last, after the rest of the page has loaded. Since it is loaded Async, your page loading speed should be slightly improved.

The second plugin adds the OpenGraph meta tags for Facebook and the link rel/publisher tags for Google+. There is a settings dialog that allows you to set three fields that are user customized.

cd34-social: http://code.google.com/p/cd34-social/downloads/list or, through the WordPress.org plugin directory: cd34-social.

cd34-header: http://code.google.com/p/cd34-header/downloads/list or, through the WordPress.org plugin directory: cd34-header.

Worked with a new client this week and needed to get the VCL working for their installation. They were running W3TC, but, this VCL should work for people running WP-Varnish or any plugin that allows Purging. This VCL is for Varnish 2.x.

There are still some tweaks, but, this appears to be working quite well.

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

acl purge {
    "10.0.1.100";
    "10.0.1.101";
    "10.0.1.102";
    "10.0.1.103";
    "10.0.1.104";
}

sub vcl_recv {
 if (req.request == "PURGE") {
   if (!client.ip ~ purge) {
     error 405 "Not allowed.";
   }
   return(lookup);
 }

  if (req.http.Accept-Encoding) {
#revisit this list
    if (req.url ~ "\.(gif|jpg|jpeg|swf|flv|mp3|mp4|pdf|ico|png|gz|tgz|bz2)(\?.*|)$") {
      remove req.http.Accept-Encoding;
    } elsif (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
    } elsif (req.http.Accept-Encoding ~ "deflate") {
      set req.http.Accept-Encoding = "deflate";
    } else {
      remove req.http.Accept-Encoding;
    }
  }
  if (req.url ~ "\.(gif|jpg|jpeg|swf|css|js|flv|mp3|mp4|pdf|ico|png)(\?.*|)$") {
    unset req.http.cookie;
    set req.url = regsub(req.url, "\?.*$", "");
  }
  if (req.http.cookie) {
    if (req.http.cookie ~ "(wordpress_|wp-settings-)") {
      return(pass);
    } else {
      unset req.http.cookie;
    }
  }
}

sub vcl_fetch {
# this conditional can probably be left out for most installations
# as it can negatively impact sites without purge support. High
# traffic sites might leave it, but, it will remove the WordPress
# 'bar' at the top and you won't have the post 'edit' functions onscreen.
  if ( (!(req.url ~ "(wp-(login|admin)|login)")) || (req.request == "GET") ) {
    unset beresp.http.set-cookie;
# If you're not running purge support with a plugin, remove
# this line.
    set beresp.ttl = 5m;
  }
  if (req.url ~ "\.(gif|jpg|jpeg|swf|css|js|flv|mp3|mp4|pdf|ico|png)(\?.*|)$") {
    set beresp.ttl = 365d;
  }
}

sub vcl_deliver {
# multi-server webfarm? set a variable here so you can check
# the headers to see which frontend served the request
#   set resp.http.X-Server = "server-01";
   if (obj.hits > 0) {
     set resp.http.X-Cache = "HIT";
   } else {
     set resp.http.X-Cache = "MISS";
   }
}

sub vcl_hit {
  if (req.request == "PURGE") {
    set obj.ttl = 0s;
    error 200 "OK";
  }
}

sub vcl_miss {
  if (req.request == "PURGE") {
    error 404 "Not cached";
  }
}

This post is a version of the slideshow presentation I did at Hack and Tell in Fort Lauderdale, Florida at The Whitetable Foundation on Saturday, June 4, 2011.

Briefly, I created a Plugin that enabled Fragment Caching with WordPress and Varnish. The problem we ran into with normal page caching methods was related to the fact that this particular client had people visiting many pages per visit, requiring the sidebar to be regenerated on uncached (cold) pages. By caching the sidebar and the page and assembling the page using Edge Side Includes, we can cache the sidebar which contains the most database intensive queries separately from the page. Thus, a visitor moving from one page to a cold page, only needs to wait for the page to generate and pull the sidebar from the cache.

What problem are we solving?

We had a high traffic site where surfers visited multiple pages, and, a very interactive site. Surfers left a lot of comments which meant we were constantly purging the page cache. This resulted in the sidebar having to be regenerated numerous times – even when it wasn’t truly necesssary.

What are our goals?

First, we want that Time to First Byte to be as quick as possible – surfers hate to wait and if you have a site that takes 12 seconds before they see any visible indication that there is something happening, most will leave.

We needed to keep the site interactive, which meant purging pages from cache when posts were made.

We had to have fast pageloads – accomplished by caching the static version of the page and doing as few calculations as possible to deliver the content.

We needed fast static content loading. Apache does very well, but, isn’t the fastest webserver out there.

How does the WordPress front page work?

The image above is a simple representation of a page that has a header, an article section where three articles are shown and a sidebar. Each of those elements is built from a number of SQL queries, assembled and displayed to the surfer. Each plugin that is used, especially filter plugins that look at content and modify it before output add a little latency – resulting in a slower page display.

How does an Article page work?

An article page works very similar to the frontpage except our content block now only contains the contents from one post. Sometimes additional plugins are called to display the post content dealing with comments, social media sharing icons, greetings based on where you’re visiting from (Google, Digg, Reddit, Facebook, etc) and many more. We also see the same sidebar on our site which contains the site navigation, advertisements and other content.

What Options do we Have?

There are a number of existing caching plugins that I have benchmarked in the past. Notably we have:

* WP-Varnish
* W3 Total Cache
* WP Super Cache
* WordPress-Varnish-ESI
* and many others

Page Caching

With Page Caching, you take the entire generated page and cache it either in ram or on disk. Since the page doesn’t need to be generated from the database, the static version of the page is served much more quickly.

Fragment Caching

With Fragment Caching, we’re able to cache the page and a smaller piece that is often repeated, but, perhaps doesn’t change as often as the page. When a websurfer comments on a post, the sidebar doesn’t need to be regenerated, but, the page does.

WordPress and Varnish

Varnish doesn’t deal well with cookies, and WordPress uses a lot of cookies to maintain information about the current web surfer. Some plugins also add their own cookies to track things so that their plugin works.

Varnish can do domain name normalization which may be desired or not. Many sites redirect the bare domain to the www.domain.com. If you do this, you can modify your Varnish Cache Language (VCL) to make sure it always hands back the proper host header.

There are other issues with Varnish that affect how well it caches. There are a number of situations where Varnish doesn’t work as you would expect, but, this can all be addressed with VCL.

Purging – caching is easy, purging is hard once you graduate beyond a single server setup.

WordPress and Varnish with ESI

In this case, our plugin caches the page and the sidebar separately, and allows Varnish to assemble the page prior to sending it to the server. This is going to be a little slower than page caching, but, in the long run, if you have a lot of page to page traffic, having that sidebar cached will make a significant impact.

Possible Solutions

You could hardcode templates and write modules to cache CPU or Database heavy widgets and in some cases, that is a good solution.

You could create a widget that handles the work to cache existing widgets. There is a plugin called Widget Cache, but, I didn’t find it to have much benefit when testing.

Many of the plugins could be rewritten to use client-side javascript. This way, caching would allow the javascript to be served and the actual computational work would be done on the client’s web browser.

Technical Problems

When the plugin was originally written, Varnish didn’t support compressing ESI assembled pages which resulted in a very difficult to manage infrastructure.

WordPress uses a lot of cookies which need to be dealt with very carefully in Varnish’s configuration.

What sort of Improvement?

Before the ESI Widget	After the ESI Widget
12 seconds time to first byte	.087 seconds time to first byte
.62 requests per second	567 requests per second
Huge number of elements	Moved some elements to a ‘CDN’ url

WordPress Plugin

In the above picture, we can see the ESI widget has been added to the sidebar, and we’ve added our desired widgets to the new ESI Widget Sidebar.

Varnish VCL – vcl_recv

sub vcl_recv {
    if (req.request == "BAN") {
       ban("req.http.host == " + req.http.host +
              "&& req.url == " + req.url);
       error 200 "Ban added";
    }
    if (req.url ~ "\.(gif|jpg|jpeg|swf|css|js|flv|mp3|mp4|pdf|ico|png)(\?.*|)$") {
      unset req.http.cookie;
      set req.url = regsub(req.url, "\?.*$", "");
    }
    If (!(req.url ~ "wp-(login|admin)")) {
      unset req.http.cookie;
    }
}

In vcl_recv, we set up rules to allow the plugin to purge content, we do a little manipulation to cache static assets and ignore some of the cache breaking arguments specified after the ? and we aggressively remove cookies.

Varnish VCL – vcl_fetch

sub vcl_fetch {
  if ( (!(req.url ~ "wp-(login|admin)")) || (req.request == "GET") ) {
                unset beresp.http.set-cookie;
  }
  set beresp.ttl = 12h;

  if (req.url ~ "\.(gif|jpg|jpeg|swf|css|js|flv|mp3|mp4|pdf|ico|png)(\?.*|)$") {
    set beresp.ttl = 365d;
  } else {
    set beresp.do_esi = true;
  }
}

Here, we remove cookies set by the backend. We set our timeout to 12 hours, overriding any expire time. Since the widget purges cached content, we can set this to a longer expiration time – eliminating additional CPU and database work. For static asset, we set a one year expiration time, and, if it isn’t a static asset, we parse it for ESI. The ESI parsing rule needs to be refined considerably as it currently parses objects that wouldn’t contain ESI.

Did Things Break?

Purging broke things and revealed a bug in PHP’s socket handling.

Posting Comments initially broke as a result of cookie handling that was a little too aggressive.

Certain plugins break that rely on being run on each pageload such as WP Greet Box and many of the Post Count and Statistics plugins.

Apache logs are rendered virtually useless since most of the queries are handled by Varnish and never hit the backend. You can log from varnishncsa, but, Google Analytics or some other webbug statistics program is a little easier to use.

End Result

Varnish 3.0, currently in beta, allows compression of ESI assembled pages, and, now can accept compressed content from the backend – allowing the Varnish server to exist at a remote location, possibly opening up avenues for companies to provide Varnish hosting in front of your WordPress site using this plugin.

Varnish ESI powered sites became much easier to deploy with 3.0. Before 2.0, you needed to run Varnish to do the ESI assembly, then, into some other server like Nginx to compress the page before sending it to the surfer, or, you would be stuck handing uncompressed pages to your surfers.

Other Improvements

* Minification/Combining Javascript and CSS
* Proper ordering of included static assets – i.e. include .css files before .js, use Async javascript includes.
* Spriting images – combining smaller images and using CSS to alter the display port resulting in one image being downloaded rather than a dozen tiny social media buttons.
* Inline CSS for images – if your images are small enough, they could be included inline in your CSS – saving an additional fetch for the web browser.
* Multiple sidebars – currently, the ESI widget only handles one sidebar.

How can I get the code?

http://code.google.com/p/wordpress-varnish-esi/

Long ago I wrote the WordPress ESI Widget to help a client’s site stay online during a barrage of traffic. To solve some of the performance problems on high traffic WordPress sites, you have to use caching, but almost all of the caching addons for WordPress do page level caching rather than fragment caching. After the site’s traffic slowed, I stopped development on the widget due to the infrastructure required to support compression.

To compress an ESI assembled page, one needed to run Nginx in front of Varnish and lost some performance as a result. Nginx would take the initial request, pass it to Varnish, Varnish would talk to the backend — which could be the same Nginx server in a somewhat complex configuration — grab the parts, assemble it, hand it back to Nginx which would then compress it and hand it to the surfer.

With Varnish compressing ESI assembled pages, we don’t need the incredibly complex configuration to run ESI. We’re left with a very simple front end cache in front of our backend servers.

Why is Fragment Caching important?

Fragment caching allows the cache to store pieces of the page that may repeat on several pages and assemble those pieces with the rest of the page. The sidebar on your WordPress site only needs to be generated once as someone surfs through your site. This changes the nature of WordPress caching considerably. Compared to the fastest existing WordPress caching plugin, the Varnish ESI widget doubled its performance – bested only by WP Varnish, a plugin that ran Varnish directly and managed cache expiration.

ESI explained simply is probably the best example I have ever found for explaining how ESI works.

But something else is faster

WP Varnish is currently faster, and, for all practical purposes probably always will be on a very busy site. However, on a site that gets a lot of traffic on one page, the second page time to first byte should be faster on an ESI assembled page because the sidebar which contains some of the most computationally expensive parts of the page, doesn’t need to be generated again. While we give up some of the raw speed, we gain an advantage when someone clicks through to read the second page. The perfect use case here is getting publicity for a particular post on your WordPress site, and those surfers decide to read other articles you’ve written.

Varnish’s Original Announcement

From: Poul-Henning Kamp
Date: January 25, 2011 6:04:02 AM EST
To: varnish-misc@varnish-cache.org
Subject: Please help break Varnish GZIP/ESI support before 3.0

One of the major features of Varnish 3.0 is now feature complete, and
I need people to start beating it up and help me find the bugs before
we go into the 3.0 release cycle.

GZIP support
------------

Varnish will ask the backend for gzip'ed objects by default and for
the minority of clients that do not grok that, ungzip during delivery.

If the backend can not or will not gzip the objects, varnish can be
told in VCL to gzip during fetch from the backend.  (It can also
gunzip, but I don't know why would you do that ?)

In addition to bandwidth, this should save varnish storage (one gzip
copy, rather than two copies, one gzip'ed one not).

GZIP support is on by default, but can be disabled with a parameter.

ESI support
-----------

Well, we have ESI support already, the difference is that it also
understands GZIP'ing.  This required a total rewrite of the ESI
parser, much improving the readability of it, I might add.

So now you can use ESI with compression, something that has hitherto
been a faustian bargain, often requiring an afterburner of some kind
to do the compression.

There are a lot of weird cornercases in this code, (such as including
a gzip'ed object in an uncomressed object) so this code really needs
beaten up.

Original message

What else is there?

Another very important fact is that Varnish will use gzip to request assets from the backend. While this doesn’t sound incredibly important, it is. Now, you can run a Varnish server at another data center and not worry as much about latency. Before this version, any ESI assembled page needed to be fetched uncompressed, and, large pages add tiny bits of latency which result in a poorer experience while surfing. Most installations run Varnish on the same machine or on a machine network topologically close, but, this opens the doors for a CDN to run ESI enabled edge servers to supercharge your WordPress site hosted anywhere.

When will it be here?

Varnish moves quickly, and while the changes are substantial in terms of code rewrites, their code is very well written. I don’t expect we’ll see many bugs in the code and it’ll be released in the next few months. This site and a number of other sites we work with will be running it later this week.

In short, caching for WordPress just got an incredible boost. Even before the compression and gzip request from the backend, the ESI Widget was twice as fast as the fastest non-Varnish enabled plugin and over 440 times faster than WordPress out of the box.

Original Info

* WordPress Cache Plugin Benchmarks
* WordPress, Varnish and Edge Side Includes
* ESI Widget Issues in the Varnish, ESI, WordPress experiment
* A WordPress Widget that Enables one to use Varnish and ESI