How To Deallocate A Typed Numpy Array? Is Setting Callback_free_data A Viable Option?

While using an open source Cython library I found a memory leak. The leak seems to come from a typed numpy array, which is not freed from the memory when it goes out of scope. The

Solution 1:

The problem (solved in the comments) turned out not to be the deallocation of the numpy array. Instead, the numpy array held a bunch of Fiboheap objects, which themselves held pointers to a bunch of Python objects. It's these objects that weren't freed.

When the Python object pointers in the Fiboheap were acquired (in insert) their reference count was incremented to ensure they were kept alive. However, when the Fiboheap was destroyed (in __dealloc__) the reference count of the Python objects it held was not decreased, causing the memory leak. The solution is to ensure that Py_DECREF is called on all the held Python objects during __dealloc__.

---

There's potentially a second, more challenging problem waiting to appear: it might be possible for the objects held by the Fiboheap to themselves contain a reference back to the Fiboheap, maybe indirectly. Python uses the function tp_tranverse to find these loops and tp_clear to break them. Cython will automatically generate a tp_traverse for its cdef classes, however since it has no way of knowing about the Python object pointers hidden within the C Fiboheap structure it won't handle these correctly (maybe generating another memory leak).

This is a probably unlikely to happen in reality, so may not be worth worrying about, but it's something to be aware of. A newsgroup post describes a means of generating custom tp_traverse functions in Cython. For most applications this should not be necessary - it's only the mixture of Cython object and PyObject* that makes it slightly possible here.