21.2. Tools for Detecting Memory Leaks

Tools for analysing memory can be characterised by:

Tool Characteristics

Characteristic	Description
Availability	Does it come with the platform? Is it within Python or the standard library? Does it need third party library installation or requires a special Python build of some sort?
Memory Granularity	How detailed is the memory measurement? Somewhere between every malloc or the overall memory usage as seen by the OS.
Execution Granularity	How detailed is the execution measurement? Per line, per function, for Python or C code?
Memory Cost	What is the extra memory consumption is introduced by using this tool?
Execution Cost	What is the extra runtime introduced by using this tool?
Developer Cost	How hard it is to use the tool?

Each tool makes trade offs between each of these characteristics.

21.2.1. Platform Tools

These tools come ready with the platform. They give a good overall picture of the memory usage.

Characteristic	Description
Availability	Always.
Memory Granularity	Usually the total memory usage by the process.
Execution Granularity	Generally periodic at low frequency, typically of the order of seconds.
Memory Cost	Usually none.
Execution Cost	Usually none.
Developer Cost	Easy.

Here are some tools available on differenct platforms:

21.2.1.1. Windows

A weakness of Windows, especially in corporate environments, is that the OS is usually severely locked down. At best this usually vastly extends the time it takes to find a leak, at worst this hopelessly limits the tools that can be installed or run on the platform. In this case some leaks can never be found and fixed.

21.2.1.1.1. Windows Task Manager

This is the basic tool for reviewing process memory. The columns to monitor are “Working Set (Memory)” which broadly corresponds to the Unix Resident Set Size (RSS) . The Sysinternals procexp (see below) is a more sophisticated version.

21.2.1.1.2. perfmon.exe

This is a Microsoft tool for logging performance and plotting it in real time. It is quite capable but a little fiddly to set up. The third party Python library psutil is a useful alternative and the pymentrace.procces also provides memory plotting of arbitrary processes: pymemtrace Process using gnuplot.

21.2.1.1.3. Sysinternals Suite

The outstanding Windows Sysinternals tools are a wonderful collection of tools and are essential for debugging any Windows application.

21.2.1.2. Linux

TODO: Finish this.

21.2.1.2.1. /proc/<PID>

The /proc/<PID> filesystem is full of good stuff.

21.2.1.2.2. Valgrind

Valgrind is essential on any Linux development platform. There is a tutorial here for building and using Python with Valgrind.

21.2.1.2.3. eBPF

The next big thing after DTrace is eBPF. Truly awesome.

21.2.1.3. Mac OS X

Tools such as vmmap, heap, leaks, malloc_history, vm_stat can all help. See the man pages for further information.

Some useful information for memory tools from Apple

21.2.1.3.1. DTrace

Mac OS X is DTrace aware, this needs a special build of Python, here is an introduction that takes you through building and using a DTrace aware version of Python on Mac OS X.

Some examples of using DTrace with pymemtrace.

21.2.2. Python Tools

21.2.2.1. Modules from the Standard Library

Characteristic	Description
Availability	Always.
Memory Granularity	Various.
Execution Granularity	Various.
Memory Cost	Usually none.
Execution Cost	Usually none.
Developer Cost	Straightforward.

21.2.2.1.1. sys

The sys has a number of useful functions, mostly CPython specific.

sys Tools

Tool	Description	Notes
getallocatedblocks()	Returns the number of allocated blocks, regardless of size.	This has no information about the size of any block. CPython only. Implemented in Objects/obmalloc.c as _Py_GetAllocatedBlocks. As implemented in Python 3.9 this returns the total reference count of every pool in every arena.
getrefcount(object)	Returns the reference count of an object.	This is increased by one for the duration of the call.
getsizeof(object)	Returns the size of an object in bytes.	Builtin objects will return correct results. Others are implementation specific. User defined objects can implement __sizeof__ which will be called if available.
_debugmallocstats(object)	Prints the state of the Python Memory Allocator pymalloc to stderr. pymentrace’s pymemtrace Debug Malloc Stats is a very useful wrapper around sys._debugmallocstats() which can report changes to Python’s small object allocator.	See pymemtrace Debug Malloc Stats for a pymemtrace wrapper that makes this much more useful.

21.2.2.1.2. gc

The gc controls the Python garbage collector. Turning the garbage collector off with gc.disable() is worth trying to see what effect, if any, it has.

21.2.2.1.3. tracemalloc

tracemalloc is a useful module that can trace memory blocks allocate by Python. It is invasive and using it consumes a significant amount of memory itself. See pymemtrace trace_malloc for a pymemtrace wrapper that makes this much more useful by report changes to Python’s memory allocator..

21.2.2.2. Third Party Modules

21.2.2.2.1. psutil

psutil is an excellent, third party, package that can report high level information on a process. psutil on PyPi

Here are some notes on the cost of computing the Resident Set Size (RSS).

21.2.2.2.2. objgraph

objgraph is a wrapper around the Python garbage collector that can take a snapshot of the Python objects in scope. This is quite invasive and expensive but can be very useful in specific cases. It does some excellent visualisations using graphviz, xdot etc.

21.2.2.2.3. Guppy 3

guppy_3 (source: guppy_3_source) is a project that gives a highly detailed picture of the current state of the Python heap.

21.2.2.2.4. memory_profiler

Sadly memory_profiler (source: memory_profiler_src) this excellent project is no longer actively maintained (volunteers welcome, contact f@bianp.net). It gives an annotated version of your source code line by line with the memory usage. It is pure Python and relies just on psutil.

21.2.3. Debugging Tools

Debugging Python (and C/C++ extensions) with GDB:

• GDB support for Python: https://devguide.python.org/gdb/

• Python debugging with GDB: https://wiki.python.org/moin/DebuggingWithGdb and https://pythondev.readthedocs.io/gdb.html

• Python debugging tools: https://pythondev.readthedocs.io/debug_tools.html

21.2.4. Building a Debug Version of Python

This is an essential technique however it is limited, due to speed, to a development environment rather than in production.

Building a debug version of Python in a variety of forms: https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-label

Building a DTrace aware version of Python: https://github.com/paulross/dtrace-py Here are some examples of using DTrace with pymemtrace with some technical notes on using DTrace.