For the past few weeks I’ve been dealing with a puzzling problem where this server becomes completely unresponsive. The first few times I caught I had to hit the power button to bring the system down. The third time I managed to catch it in the act with top(1) and discovered the httpd processes going absolutely bonkers. I also happened to be tail -f-ing the webserver access log at the time too. Oddly enough, there were httpd processes going nuts spawning and terminating processes over and over (load average spiked to over 100), but there were very few pages being served up. Took me a while to stop everything and kill off the httpd daemon. A check of the error logs didn’t reveal anything enlightening unfortunately.
An upgrade to Apache 2.0.53 didn’t seem to make things any better. It actually seems to have made things a little worse, since the problem seems to be cropping up more frequently.
At the moment, my prime suspect is mod_security, although I haven’t read or seen anything about mod_security causing this kind of problem. But it started happening a few weeks ago on a previously stable server with no other changes other than adding mod_security to the mix. I haven’t ruled out bad configuration on account of my own stupidity though. Also haven’t ruled out that it could be because of using the 1.9dev1 version of mod_security either. I suppose I ought to test that out first.
Cataloging the library
Found this sweet little tool called OpenDB on Sourceforge.net. It’s an insanely flexible cataloging system, allowing you to define whatever types of objects you want. It comes with several pre-defined types (DVD, VHS, CD, Laser disc, VCD), but oddly enough no book type. You can define users and assign the ownership of each item to a user. If you’re always lending things out to people, you can keep track of that too.
Probably the best feature of the program though, are the plugins that let you do lookups against sites like Amazon.com, Internet Book List, FreeDB and IMDB. For books, all you need to do is enter the ISBN, do the lookup on Amazon and it fills in all the fields you’ve set up for the Book type you define. For other stuff, just enter Amazon’s ASIN and look it up. Entering a title will get you a list of possibilities. Click the one you want and all the fields are filled in for you. It’s awesome. No need to type in anything manually, which is probably the biggest barrier to create and maintain a comprehensive inventory of your stuff.
Runs on PHP and uses MySQL on the back end. The database schema is fairly small and compact for what it stores and tracks.
If you’ve got a collection of stuff you’re looking to keep track of, this is a cool bit of software to check out.
Beating off referrer spam with mod_security
Mod_security so far has been quite effective at beating off the referrer spammers since I installed a few rules. It seems to have gotten even better since using blacklist_to_modsec to generate a bunch of rules based on MT Blacklist.
According to my stats, my server went from serving up at least 50MB of bandwidth/month to spammers (11 000 + requests/month) over the last two months (and that was just 403, 404 and 500 errors) to just 550kB (1500+ requests) so far this month (my default mod_security action is to send a 412 Precondition Failed error). They’re still there and banging away, but instead of having to serve up the entire page requested, all they get is a 350 byte error page. That means a heck of a lot less bandwidth that I have to waste on spammers. There are still a few sneaking through, but it’s not nearly as bad as it was before.
To paraphrase Flo, Kiss my bits spammers.
Dealing with spam
From MT‘s ProNet comes some very good tips for dealing with spam at the webserver level.
I’d just implemented mod_security a little while ago on my server, and while it hasn’t stopped the flood of spammy referrers, it has kept a lot of them sucking up bandwidth. One thing I need to do is start keeping up with adding additional rules to mod_security. Soon I’ll have to split it out into a separate file to include in httpd.conf. And to help with creating mod_security rules is Noel Jackson’s mod_security rule generator.
The other things I’ll be experimenting with are Peter Wood’s blacklist_to_modsec script and the DSBL plugin from Brad Choate.
Wear it like a geek badge
I got one of those USB thumb drives a little while ago. Just a simple little 256 MB PNY Attaché. But it comes with a lanyard that makes it handy to carry around your neck, instead of being relegated to the bottom of your pocket to get mangled by keys or something.
Normally I don’t like to wear things around my neck, but I decided to try the lanyard with the thumb drive. Lots of other fellow computer guys around here carry thumb drives around their neck too. Having a thumb drive around your neck almost seems like some kind of geek identification device or badge. Now other things have made it onto the lanyard: my ID badge and radiation monitoring badge for starters. I’ll have to stop adding things onto it before it gets too heavy.
I’ll probably stop using it eventually because it tends to bounce around when I’m walking along, which is a little bit annoying. In the meantime, I’ll just broadcast my geekiness around my neck.