• Home
  • Assertions
  • Poetry
  • Programming

Record and Reverie

General things I find interesting

Feed on
Posts
Comments
« Firewall Ban Activated
How to Not Recruit »

Thrashing Server

Oct 1st, 2008 by Graham Booker

Well, last Sunday, we released a new version of Perian. It didn’t occur to me at the the time that this would mean a large number of people would be visiting the site. Anyway, Monday morning I noticed that the web server was very slow, which began my fun.
I decided that the best course of action was to increase the number of servers. The system had CPU to spare as well as memory, so this is the natural choice. So, I increased the number, restarted the web server, and it helped, some. So, I increased the number further, reloaded the web server, and watched top for cpu and memory usage. I kept increasing the number, until I realized that reloading the web server didn’t actually reload this part of the configuration and I needed to restart the web server.
So, I restarted the web server, and watched top in horror as the server ran out of memory and started swapping. I quickly issued a “/etc/init.d/apache2 stop” command, but the command never completed. I quickly scrambled to see if there was any other shells I could gain to the server. Everything was running horribly slow because the server was thrashing. It became clear to me about 10 minutes later that the kill process was not keeping up with the new apache processes being created; so I must do something to stop the new apache processes first.
So, I had the sense to issue a “iptables -A INPUT -p tcp –dport 80 -j DROP”. This firewalled off the web server from the entire world. Then, over the course of the next minute, the server starts becoming responsive again. Finally, I managed to actually kill the web server, set it’s child count to a more reasonable value, and start it back up again. Then, a quick flush of the firewall rules, and it was working again.
If I didn’t have the sense to run this, I likely would have had to resort to a reboot of the server into single user mode; a prospect to which I was not very amenable. So, one of these days I’ll reconfigure the thing to use the threaded version of the web server; however, php doesn’t work there, so I guess I’ll have to use some sort of workaround.

Tags: Servers

Posted in General

Comments are closed.

  • Recent Posts

    • Fighting with Sync by Microsoft
    • How to Not Recruit
    • Thrashing Server
    • Firewall Ban Activated
    • Turn the Firewall Around
    • Decline of Local Bookstores
  • Archives

    2021
    2020
    March 2020 (1)
    2019
    November 2019 (1)
    2018
    June 2018 (1)July 2018 (1)December 2018 (1)
    2017
    January 2017 (2)June 2017 (1)August 2017 (1)
    2016
    June 2016 (1)August 2016 (1)
    2015
    January 2015 (1)February 2015 (1)December 2015 (1)
    2014
    June 2014 (1)July 2014 (1)August 2014 (2)
    2013
    February 2013 (2)March 2013 (1)April 2013 (1)June 2013 (1)November 2013 (1)
    2012
    April 2012 (2)May 2012 (1)June 2012 (1)November 2012 (1)
    2011
    January 2011 (1)October 2011 (1)November 2011 (1)December 2011 (1)
    2010
    February 2010 (2)April 2010 (1)June 2010 (1)July 2010 (1)August 2010 (1)September 2010 (1)October 2010 (2)December 2010 (3)
    2009
    January 2009 (1)February 2009 (1)March 2009 (2)May 2009 (1)July 2009 (3)September 2009 (1)
    2008
    January 2008 (1)February 2008 (4)March 2008 (1)April 2008 (6)May 2008 (1)June 2008 (3)August 2008 (1)September 2008 (2)October 2008 (2)December 2008 (1)
    2007
    January 2007 (1)February 2007 (4)March 2007 (5)April 2007 (4)May 2007 (1)June 2007 (6)August 2007 (3)September 2007 (3)November 2007 (3)December 2007 (4)
    2006
    January 2006 (4)February 2006 (10)March 2006 (4)April 2006 (6)May 2006 (2)June 2006 (4)July 2006 (1)August 2006 (1)September 2006 (4)October 2006 (6)November 2006 (3)December 2006 (3)
    2005
    October 2005 (6)November 2005 (13)December 2005 (1)
    2004
    February 2004 (2)March 2004 (1)April 2004 (1)May 2004 (6)June 2004 (6)July 2004 (3)August 2004 (2)September 2004 (1)November 2004 (5)
    2003
    September 2003 (1)October 2003 (3)November 2003 (1)December 2003 (1)
  • Categories

    • Breakaway (5)
    • Family (4)
    • Friends (2)
    • General (148)
    • Nature Pictures (8)
    • Politics (2)
    • Programming (41)
    • School (11)
    • SysAdmin (8)
    • Teaching (2)
  • Tags

    AC3 Ads Code Frontrow Java Objective-C Open Source Perian Perl permissions plex plugin RSS Sapphire School Servers ZFS

  • Pages

    • Assertions
      • Female Friends Who Won’t Date You
      • Not Dating Friends
    • Poetry
      • Curtis Staying Over
      • Girl Questions
      • Scaring Girls Off
      • Summer’s End
    • Programming
      • Fire Development
      • Kyocera Ringtone Converter for the Mac
      • Perian
      • Text Compression

Record and Reverie © 2021 All Rights Reserved.

WordPress Themes | Web Hosting Bluebook