Analysis of the slashdot effect

Yesterday’s IE7 on Linux post was (still is, for now) featured on the front-page of Slashdot, and subsequently experienced quite a sudden surge in web-traffic. Now, if there’s one thing that’s great about a large data set, it must be the ability to crudely analyse it and make incorrect assumptions!

Most of this is based on about one day’s worth of traffic data from Apache.

To start off with, here are some tips on how to improve your site’s ability to withstand the sudden influx of thousands of visitors (in reverse order of importance):

1. Optimise your CSS
   My CSS file was hit approximately 36 000 times and generated 200 MB of outgoing data, accounting for 3% off the traffic.
2. Optimise your HTML
   HTML accounted for 6.7% of total traffic; I assume CSS was half of this because it is cached for the entire site.
3. OPTIMISE your images!
   This is the one that caught me off-guard, I’ve been using a pretty much out’ve-the-box Wordpress theme, and never even considered image sizes to be an issue - yet looking at the statistics, gif images accounted for a insane 90.2% of traffic! This is roughly 6.27 GB. Looking through the images in the theme, I found one major offender - the top bar - weighing in at 168 K. Using gimp and converting this image to a 90% quality JPG results in a 72% saving on space (47 K), and thus would’ve reduced total bandwidth consumption by 4.5 GB. Sigh, live and learn.

A quick plug for my host, Hostgator, because as far as I know (I wasn’t awake at the beginning) webexpose.org continued to run without problem through the entire day (bandwidth peaked at 3 Mb/second), and I have received no scary emails or anything from them as of yet, despite using 12% of my monthly bandwidth in one day.

Anyway, continuing to look at the day’s data, here’re some figures from AWStats:

Totals
The total hit data from this log file (9 January 2007). Look at the rather sad size/visit rate.

Unique	Visits	Total	Bandwidth
33 930	36 399	267 672	7.02 GB (202.27KB/visit, darn gifs)

Visits by Hour
Hour 16 was the max at 5 387 pages requested, hour 11 is 521. I have no idea what happened between 12 and 15:00, perhaps the site did actually go down? The traffic progression doesn’t seem to indicate this though, you’d expect the site to be down after 16:00 or 17:00.

Referers
Where did all this incoming traffic come from?

Incoming links for other websites	34 232	73.2 %
Direct visits	12 311	26.3 %
Incoming links from search engines	215	0.4 %

Browsers
This is obviously skewed since people running Internet Explorer would probably not want to know how to run it on Linux - and secondly, Slashdot is a pretty tech-aware place, so I’d assume a larger percentage of their readers have already switched to Firefox.

Firefox	74.1 %
Internet Explorer	10.7 %
Mozilla	3.9 %
Opera	3.7 %
Safari	3.2 %
Konqueror	2.7 %
Camino	0.4 %

Operating systems
Not a bad showing from Linux, I don’t really have much to comment on here. Perhaps, except to ask, why would Windows users be reading about IE on Linux (can we assume that a lot of people are at work, under corporate IT policy to use Windows, and are getting their early morning Slashdot fix?).

Windows	54.4 %
Linux	35.3 %
Macintosh	8.2 %
Unknown	1 %
FreeBSD	0.4 %

Countries

USA	29 237
Australia	2 258
Great Britain	2 007
European Union	1 941
Canada	1 769

Other things of interest

GoogleBot hit 79 times, and something called ‘EchO!’ hit a huge 4454 times, using 122 MB of bandwidth. Googling turns up ‘EchO!/2.0′ (http://echo.fr robot) - I’m not quite sure about this crawler, some say it belongs to http://www.voila.com/, a French search engine.

Looking at a sample access log with 15 403 lines, shows 1 161 unique top.gif requests - so that puts it at roughly 15 files per visitor, although AWStats says it’s more like 7.1 files/request. Grepping for ubuntu shows 1 742 results (I just happened to see Ubuntu in a user-agent near the top of the file).