Python-memory-leak

[1] habrahabr: Устранение утечек памяти в приложении на Питоне

Fixing a memory leak in a running python process with GDB We have a leak…

Today at work we had a really funny challenge: fixing a memory leak in a python process that was already running and couldn’t be stopped.

We discovered in the morning a process in our cluster that was using 3GB of memory in a 32bits sever, which is all the memory a single process can address (even if PAE allows adressing up to 64GB of memory, a single process is still limited to 3GB). This leak was triggered by 2 known bugs and we couldn’t simply recreate the client’s signal because it would affect it with a downtime of one hour. The easiest solution would have been using manhole, but we couldn’t login into manhole because the process couldn’t even allocate enough memory for that. We could have avoid this situation if we had the last release of our code in production, but this was not the case and we needed to choose between 2 options:

Trying to recreate the signal manually in the cluster. Trying to fix the leak in the running process. @xqueralt and @jribas went for the first option and they manage to solve our problem incredibly fast (with lots of black magic included….yeah!!), isolating the affected machine from the cluster and leaving me a patient in coma that needed urgent surgery :)

So even if we managed to recreate the signal without affecting our customer (only with a downtime of a few seconds), we decided to try with the second option too because it was fun, it would help us in the future if a similar problem happens again and because… it was really fun!

Finding a way to fix it: using GDB

The leak was caused by a python Cache object that was storing in a list lots of entries that couldn’t be synced with the Journal because of the first bug. We also have a SIGHUP handler that rotates the Journal and that would have released all this memory, but the second bug was raising a RuntimeError before this happens…. arrgggg. The solution seams pretty clear: releasing the damn list with GDB!

To play with gdb and find a way to clear this list I simulated the bug with the following script.

import signal import glib

class Dummy(object):

    Dummy object to store in the cache 
   s = "open source fuck yeah"

class LeakyCache(object):

    Leaky cache storing entries 

   _bug2 = 'GRRR'

   def __init__(self):
       # Fill the cache with many Dummy objects to simulate the leak
       self._entries = [Dummy()] * 9999999
       print "Cache is now full"

   def rotate(self):
       if self._bug2 is not None:
           raise RuntimeError("Fuuuuu!")

def main():

   cache = LeakyCache()
   # Install a SIGHUP handler
   handler = signal.signal(signal.SIGHUP, lambda signal, frame: cache.rotate())
   glib.MainLoop().run()

if __name__ == "__main__":

   main()

To fix our leak and the 2nd bug we need to:

Get a reference to our LeakyCache instance. Clear the list or replace it with an empty list. Set self._bug2 to None to fix the bug. In C that would have been very easy as you can easily set breakpoints in a line of code or function and access/change a variable, but with Python code it’s not that easy as you need to deal with the C API.

Thus,the first step is to find a breakpoint that will allow us to get a reference to the LeakyCache and here the second bug turned out to be very handy as it’s easy to reach this code sending a SIGHUP signal to the process with kill -1 $PID

Finding a breakpoint and get a pointer to the LeakyCache object

We know the LeakyCache object will raise an exception handling the SIGHUP signal, so that’s our best shot to set the breakpoint. But when you are dealing with Python code in GDB you can’t just set a breakpoint in a function (or I didn’t find yet a way to do it), and you need to find instead a function of Python’s C API.

Exceptions are created calling the function BaseException_new, , and because we now a RuntimeException will be created by the LeakyCache, we can use it as a good breakpoint.

We will start first our script and get the PID of the process

$ python test.py &
[1] 4565

Now we can start gdb on a separte console, attach to the process and set the breakpoint

$ gdb
(gdb) file /usr/bin/python
Reading symbols from /usr/bin/python...Reading symbols from /usr/lib/debug/usr/bin/python2.7...done.
done.
(gdb) attach 2109
Attaching to program: /usr/bin/python, process 2109
[...]
(gdb) break BaseException_new
Breakpoint 1 at 0x80a7207: file ../Objects/exceptions.c, line 31.
(gdb) c
Continuing.

And we can now send the SIGHUP signal:

$ kill -1 2109

In GDB we will see that our program has stopped creating the RuntimeException in the rotate function and we can show a nice traceback to verify it:

(gdb) c
Continuing.
Breakpoint 1, BaseException_new (type=0x823a9c0, args=('Fuuuuu!',), kwds=0x0) at ../Objects/exceptions.c:31
31	../Objects/exceptions.c: No such file or directory.
in ../Objects/exceptions.c
(gdb) bt
#0 BaseException_new (type=0x823a9c0, args=('Fuuuuu!',), kwds=0x0) at ../Objects/exceptions.c:31
#1 0x080cdf23 in type_call (type=0x823a9c0, args=('Fuuuuu!',), kwds=0x0) at ../Objects/typeobject.c:721
#2 0x080a464a in PyObject_Call (func=, arg=('Fuuuuu!',), kw=0x0) at ../Objects/abstract.c:2529
#3 0x080dad43 in do_call (f=
Frame 0x9cf700c, for file test.py, line 18, in rotate (self=, <...>, <...>, <...>, <...>, <...>, <...>, <...>, <...>, <...>, <....(truncated), throwflag=0) at  ../Python/ceval.c:4231
#4 call_function (f=
Frame 0x9cf700c, for file test.py, line 18, in rotate (self=, <...>, <...>, <...>, <...>, <...>, <...>, <...>, <...>, <...>, <....(truncated), throwflag=0) at ../Python/ceval.c:4036
#5 PyEval_EvalFrameEx (f=
Frame 0x9cf700c, for file test.py, line 18, in rotate (self=, <...>, <...>, <...>, <...>, <...>, <...>, <...>, <...>, <...>, <....(truncated), throwflag=0) at ../Python/ceval.c:2666

The frame that we need to pay attention to is the 5th one, which contains a pointer to the Python’s frame, from which we will get the pointer to our LeakyCache object instance and store it a variable named cache.

(gdb) frame 5
#5 PyEval_EvalFrameEx (f=
Frame 0x9cf700c, for file test.py, line 18, in rotate (self=, <...>, <...>, <...>, <...>, <...>, <...>, <...>, <...>, <...>, <....(truncated), throwflag=0) at ../Python/ceval.c:2666 2666	../Python/ceval.c: No such file or directory. in ../Python/ceval.c
(gdb) set cache=f->f_localsplus[0]
(gdb) p cache

$1 =
<LeakyCache(_entries=[<Dummy at remote 0xb783ec8c>, <...>,
<...>, <...>, <...>, <...>, <...>, <...>, <...>,...(truncated)

A Python Frame stores a reference to the frames locals in the array f_localsplus and because the Exception was raised by the LeakyCache, the pointer to the instance of this object can be accessed at f->f_localsplus[0] . For more details you take at look at the implementation of pylocals in Python's gdbinit and to this excellent document.

And... Yeyyy! We now have a pointer to the LeakyCache to modify it through the C API.

Modify the Python object using the C API

Now that we have a pointer to our object we can it modify as if we were in pdb. In this case we needed to free the list and fix the 2nd bug.

For both cases we will use the C equivalent of the builtin function setattr to modify the object's variables: PyObject_SetAttr

First, we will replace the current list with an empty one. setattr will call Py_DECREF in the list, decreasing its refcount and calling its deallocation function.

(gdb) call PyObject_SetAttr(cache, Py_BuildValue("s","_entries"), PyList_New(0))

We will now fix the 2nd bug setting _bug2 to None (Creating a Python string with a null pointer returns None):

(gdb) call PyObject_SetAttr(cache, Py_BuildValue("s","_bug2"), Py_BuildValue("s", 0))

And finally we will force a garbage collection:

(gdb) call PyGC_Collect()

If we continue with the execution of the process we will see that our memory was freed and the SIGHUP handler will not raise an Exception anymore.

Trying the fix in production

Fortunately we had already the problem fixed and now this process was running in a machine completely isolated from the cluster, which means I could break everything without having to worry about anything... and ... it didn't work. Replacing the list didn't free the memory and I tried forcing a garbage collection which killed the process with a segmentation fault :)

It was probably already too late for this kind of madness but like the simulation shows it would have probably worked if the process would have had at least enough memory available for the GC. Anyway, lesson learned!

I'm pretty sure there are other ways to do it or at least to make sure the list is properly deallocated and all its elements and that's why I'm waiting for Feedback and other solutions in comments!

This entry was posted in Hacks, Python. Bookmark the permalink .