Thrashing Server
Oct 1st, 2008 by Graham Booker
Well, last Sunday, we released a new version of Perian. It didn’t occur to me at the the time that this would mean a large number of people would be visiting the site. Anyway, Monday morning I noticed that the web server was very slow, which began my fun.
I decided that the best course of action was to increase the number of servers. The system had CPU to spare as well as memory, so this is the natural choice. So, I increased the number, restarted the web server, and it helped, some. So, I increased the number further, reloaded the web server, and watched top for cpu and memory usage. I kept increasing the number, until I realized that reloading the web server didn’t actually reload this part of the configuration and I needed to restart the web server.
So, I restarted the web server, and watched top in horror as the server ran out of memory and started swapping. I quickly issued a “/etc/init.d/apache2 stop” command, but the command never completed. I quickly scrambled to see if there was any other shells I could gain to the server. Everything was running horribly slow because the server was thrashing. It became clear to me about 10 minutes later that the kill process was not keeping up with the new apache processes being created; so I must do something to stop the new apache processes first.
So, I had the sense to issue a “iptables -A INPUT -p tcp –dport 80 -j DROP”. This firewalled off the web server from the entire world. Then, over the course of the next minute, the server starts becoming responsive again. Finally, I managed to actually kill the web server, set it’s child count to a more reasonable value, and start it back up again. Then, a quick flush of the firewall rules, and it was working again.
If I didn’t have the sense to run this, I likely would have had to resort to a reboot of the server into single user mode; a prospect to which I was not very amenable. So, one of these days I’ll reconfigure the thing to use the threaded version of the web server; however, php doesn’t work there, so I guess I’ll have to use some sort of workaround.