The task is simple, setup a webserver to serve images,videos,audios and other static content as fast as possible and as efficiently as possible. A simplistic view would be to just get some fat pipes and a uber-fast i/o system and pump the bits out with a webserver such as Apache .

A few things to keep in mind. Apache is not the fastest,most efficient webserver when it comes to serving static content. The most common Apache version 1.3.x uses a pre-fork model where different childs serve different clients. newer versions such as Apache 2.0.x use a thread model. As you have more and more clients connecting to a server, the one process per client and one thread per client lead to lot of context switching and memory pressure.

Typically for serving static content, an event-driven webserver is used. This is a single process webserver which handles all the connections and internally multiplexes them and serves the files. It will use high performance API’s such as epoll,kqueue (on BSD systems) and also use zero copy i/o mechanisms such as sendfile.

Our candidates were

1. Cherokee
2. Lighttpd
3. Tux

Whilst dbnet volunteers have worked with Cherokee before and have a good rapport with the author Alvaro Lopez, the current versions of Cherokee have some strange behaviour under load and it was decided to not use it at this time. Using a kernel based webserver wasn’t considered really nice since that could have led to some server instability and lighttpd was relatively easy to learn, the volunteer having downloaded the source only a few hours before and reading the docs

So now we have a high performance event driven webserver, is this the end ?
No, here’s the key insight into delivering content faster to your viewers

Leverage their last mile

Most home users connect via their ISP’s who run large web proxy farms. These web proxy farms cache content (which has been deemed cachable) when their users visit various websites using the connectivity provided by their ISP. Thus the idea is to tell an ISP proxy cache up-front to cache this static content for a while, let’s say upto 4 weeks.

The advantage of filling up an ISP proxy cache is that it not only benefits the user who visits your site again but it also helps other users on that ISP who visit your site.
So if an Etisalat or Singtel or Pacnet user visits mumineen.org and the static content is pushed into the proxy cache, this speeds up access for all subsequent viewers from that region

What’s the trick. Simple. Send an ‘Expires’ header with a date in the future and/or send a Cache-Control: header with an appropiate max-age.

So grab your favourite network tracer out and take a look at the HTTP responses sent out by lighttpd from media.mumineen.org. We want you and particularly your ISP’s proxy cache to take our images and keep them for a while.