• Home
  • Assertions
  • Poetry
  • Programming

Record and Reverie

General things I find interesting

Feed on
Posts
Comments
« Google Link Redirection (cont.)
What Objective-C can learn from Java, Part 1 (Generics) »

Trac.fcgi Memory Usage

Oct 16th, 2010 by Graham Booker

I’ve been slowly transitioning to using nginx as the web front-end in an effort to reduce Apache’s memory usage. In keeping with this task, I’m moving more and more off of Apache. One piece I recently moved was trac, transitioning to using it directly by nginx by running it in fast-cgi mode where as previously it was running as cgi though Apache.

While fast-cgi is faster, it has inherent issues, such as any memory leak can result in ever growing memory usage, which is exactly why Apache has a setting for each child to serve a limited number of requests before exiting. Trac.fcgi has no such directive, and has the equivalent of a large memory leak, a non-expiring cache. While it’s not as bad as a memory leak, which will indefinitely grow instead of reaching a limit, if the cache size is larger than the available memory for trac to use, it’s just as serious. The only solution, without fixing trac’s caching mechanisms, is to restart trac periodically, but during the time trac is restarting, all requests are lost, causing bad gateway errors to the user. Additionally, the restart needs to be done manually. Clearly not an ideal solution.

The ideal solution would be for the trac process to be periodically restarted, but all requests be successfully completed. This is what Apache accomplishes with its children, but trac has no such mechanism or even the support for one. So, I had to build it in myself.

My Solution:
First piece is to create a parent process, which holds the fcgi socket, and restarts a child trac process when it dies. This ensures that all waiting requests will be served by either the old or new process. Such a parent process absolutely must have no memory leaks, and so I created one that has only 1 explicit allocation, and it is executed only once.

The second piece, and one that’s considerably harder, is to make the trac process exit gracefully when it’s memory usage gets too big. The first step was to create a subclass of WSGIServer and override its _mainloopPeriodic to run a periodic check. In this check, I do a memory usage check, and if it’s over 90MB, set itself to exit. The problem is there’s no easy way to figure out the memory usage on linux. There is a function, getrusage, which is supposed to give resource usage information, such as memory, but linux gives all zeros (unlike a proper kernel). The only way to get this is to read the information out of /proc, and parse that data. Since this becomes a more expensive operation, I only conduct the test every 100 times.
After doing this, I was still getting periodic bad gateway errors. It turns out that trac spawns a thread to process the request, and that request hadn’t completed when the process exited, dropping the connection and causing the error. In examining the documentation, Python is supposed to wait for the thread to complete before exit. Since it wasn’t, I put in a mechanism to see if any threads are running before exit. Here lies a big problem with Python. I found out that the thread, while created, hasn’t actually started. Since it hasn’t started, it isn’t running, which is why Python exited. Furthermore, Python’s threading is so brain-dead, there seems to be no way at all to differentiate between a thread which hasn’t started and one that is exited but not freed. This means there is no reliable way to detect if all threads have exited. So instead, in order to work around Python, I created a thread-safe counter to count the number of threads. I increment it when the thread is created (not started), and decrement it when the thread completes. I then only allow the main thread to exit when this counter reaches zero (since the main thread does the allocations, this never lets the process die without starting all threads). Given this glaringly bad threading model, I put in another protection mechanism so that the main thread will exit after 30 seconds even if the count isn’t zero, just in case.

With the above two pieces, trac’s memory usage is limited, and no connections are dropped, in the time between one process deciding to go down and when it actually does, nothing is processing requests. So, the last piece is to make trac signal to the parent process that it has decided to exit, and then have the parent process launch a new trac to take over while the previous is exiting. I did this with USR1 signal, where the parent process sends a HUP to the child (in case someone else sent it the USR1 signal), and start a new child. With these modifications in place, trac has been humming along for nearly a month, being restarted about 2-3 times a day with no issues.

Files:
launcher.c – Requires the environment variable WORKER_PATH to be set to know what process to launch. Best run with something like spawn-fcgi
trac.fcgi – Modified trac.fcgi to incorporate the above mentioned changes, complete with commented code for testing/experimentation.
Enjoy

Tags: Code, Servers

Posted in Programming

One Response to “Trac.fcgi Memory Usage”

  1. on 10 Jan 2012 at 6:32 pm1Trac – Time for a diet, you’re eating too much memory | GeekISP Tech Blog

    […] in a FastCGI setup. I did find one person who invested a bit of energy in a custom solution here, but I had some trouble using his ‘launcher’ […]

  • Recent Posts

    • What Objective-C can learn from Java, Part 3 (Single Source File)
    • What Objective-C can learn from Java, Part 2 (Abstract Classes)
    • What Objective-C can learn from Java, Part 1 (Generics)
    • Trac.fcgi Memory Usage
    • Google Link Redirection (cont.)
    • Why you shouldn’t buy A Flip Camera
  • Archives

    2021
    2020
    March 2020 (1)
    2019
    November 2019 (1)
    2018
    June 2018 (1)July 2018 (1)December 2018 (1)
    2017
    January 2017 (2)June 2017 (1)August 2017 (1)
    2016
    June 2016 (1)August 2016 (1)
    2015
    January 2015 (1)February 2015 (1)December 2015 (1)
    2014
    June 2014 (1)July 2014 (1)August 2014 (2)
    2013
    February 2013 (2)March 2013 (1)April 2013 (1)June 2013 (1)November 2013 (1)
    2012
    April 2012 (2)May 2012 (1)June 2012 (1)November 2012 (1)
    2011
    January 2011 (1)October 2011 (1)November 2011 (1)December 2011 (1)
    2010
    February 2010 (2)April 2010 (1)June 2010 (1)July 2010 (1)August 2010 (1)September 2010 (1)October 2010 (2)December 2010 (3)
    2009
    January 2009 (1)February 2009 (1)March 2009 (2)May 2009 (1)July 2009 (3)September 2009 (1)
    2008
    January 2008 (1)February 2008 (4)March 2008 (1)April 2008 (6)May 2008 (1)June 2008 (3)August 2008 (1)September 2008 (2)October 2008 (2)December 2008 (1)
    2007
    January 2007 (1)February 2007 (4)March 2007 (5)April 2007 (4)May 2007 (1)June 2007 (6)August 2007 (3)September 2007 (3)November 2007 (3)December 2007 (4)
    2006
    January 2006 (4)February 2006 (10)March 2006 (4)April 2006 (6)May 2006 (2)June 2006 (4)July 2006 (1)August 2006 (1)September 2006 (4)October 2006 (6)November 2006 (3)December 2006 (3)
    2005
    October 2005 (6)November 2005 (13)December 2005 (1)
    2004
    February 2004 (2)March 2004 (1)April 2004 (1)May 2004 (6)June 2004 (6)July 2004 (3)August 2004 (2)September 2004 (1)November 2004 (5)
    2003
    September 2003 (1)October 2003 (3)November 2003 (1)December 2003 (1)
  • Categories

    • Breakaway (5)
    • Family (4)
    • Friends (2)
    • General (148)
    • Nature Pictures (8)
    • Politics (2)
    • Programming (41)
    • School (11)
    • SysAdmin (8)
    • Teaching (2)
  • Tags

    AC3 Ads Code Frontrow Java Objective-C Open Source Perian Perl permissions plex plugin RSS Sapphire School Servers ZFS

  • Pages

    • Assertions
      • Female Friends Who Won’t Date You
      • Not Dating Friends
    • Poetry
      • Curtis Staying Over
      • Girl Questions
      • Scaring Girls Off
      • Summer’s End
    • Programming
      • Fire Development
      • Kyocera Ringtone Converter for the Mac
      • Perian
      • Text Compression

Record and Reverie © 2021 All Rights Reserved.

WordPress Themes | Web Hosting Bluebook