I finally figured out why my site has been so slow for some people. After multiple sessions of staring at tcpdump output and grep-ing through error logs, I finally noticed that Apache’s access_log was full of hostnames… Duh! This meant that Apache was doing a reverse lookup on every address that accessed my site.
(For those who don’t undrstand why this is such a serious performance bottleneck, Apache won’t display the page until the hostname is resolved. This will take a couple seconds in a lot of cases, and in the worst case (if there’s no PTR record associated with the client’s IP address), Apache will wait up to 30 seconds before giving up and displaying the page anyway.)
“Ah ha,” I exclaimed, as I fired up vi and scrolled down to HostnameLookups in my httpd.conf, but I became even more confused when I discovered that HostnameLookups was already disabled. Just to be sure, I grep-ed my entire /etc partition for HostnameLookups, but it was definately turned off. So why was Apache continuing to do reverse lookups even though I told it not to? The answer was a faulty “deny” statement in one of my access lists. I was originally commenting my deny statements like this:
Deny from 10.20.30 #some jerk
But apparently, the entire line (including the comment) was being misinterpreted as a hostname, which implicitely enabled HostnameLookups. The solution was to move the comments to their own lines like this:
#some jerk Deny from 10.20.30
I hope this saves someone out there a lot of time and confusion.