10.1. Debugging Tools¶
First create your toolbox, in this one we have:
Debug version of Python - great for finding out more detail of your Python code as it executes.
Valgrind - the goto tool for memory leaks. It is a little tricky to get working but should be in every developers toolbox.
OS memory monitioring - this is a quick and simple way of identifying whether memory leaks are happening or not. An example is given below: A Simple Memory Monitor
10.1.1. Build a Debug Version of Python¶
There are a large combination of debug builds of Python that you can create and each one will give you extra information when you either:
Invoke Python with a command line option.
Example: a
Py_DEBUG
build invoking Python withpython -X showrefcount
Set an environment variable.
Example: a
Py_DEBUG
build invoking Python withPYTHONMALLOCSTATS=1 python
Additional functions that are added to the
sys
module that can give useful information.Example: a
Py_DEBUG
build an callingsys.getobjects(...)
.
See here Building and Using a Debug Version of Python for instructions on how to do this.
10.1.2. Valgrind¶
See here Building Python for Valgrind for instructions on how to build Valgrind.
See here Using Valgrind for instructions on how to use Valgrind.
Here Finding Where the Leak is With Valgrind is an example of finding a leak with Valgrind.
10.1.3. A Simple Memory Monitor¶
Here is a simple process memory monitor using the psutil
library:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | import sys
import time
import psutil
def memMon(pid, freq=1.0):
proc = psutil.Process(pid)
print(proc.memory_info_ex())
prev_mem = None
while True:
try:
mem = proc.memory_info().rss / 1e6
if prev_mem is None:
print('{:10.3f} [Mb]'.format(mem))
else:
print('{:10.3f} [Mb] {:+10.3f} [Mb]'.format(mem, mem - prev_mem))
prev_mem = mem
time.sleep(freq)
except KeyboardInterrupt:
try:
input(' Pausing memMon, <cr> to continue, ^C to end...')
except KeyboardInterrupt:
print('\n')
return
if __name__ == '__main__':
if len(sys.argv) < 2:
print('Usage: python pidmon.py <PID>')
sys.exit(1)
pid = int(sys.argv[1])
memMon(pid)
sys.exit(0)
|
Lets test it. In one shell fire up Python and find its PID:
>>> import os
>>> os.getpid()
13360
In a second shell fire up pidmon.py with this PID:
$ python3 pidmon.py 13360
pextmem(rss=7364608, vms=2526482432, pfaults=9793536, pageins=24576)
7.365 [Mb]
7.365 [Mb] +0.000 [Mb]
7.365 [Mb] +0.000 [Mb]
...
Pause pidmon.py with Ctrl-C:
^C Pausing memMon, <cr> to continue, ^C to end...
Go back to the first shell and create a large string (1Gb):
>>> s = ' ' * 1024**3
In the second shell continue pidmon.py with <cr> and we see the memory usage:
1077.932 [Mb] +1070.567 [Mb]
1077.932 [Mb] +0.000 [Mb]
1077.932 [Mb] +0.000 [Mb]
...
Go back to the first shell and delete the string:
>>> del s
In the second shell we see the memory usage drop:
1077.953 [Mb] +0.020 [Mb]
1077.953 [Mb] +0.000 [Mb]
272.679 [Mb] -805.274 [Mb]
4.243 [Mb] -268.435 [Mb]
4.243 [Mb] +0.000 [Mb]
...
In the second shell halt pidmon with two Ctrl-C commands:
^C Pausing memMon, <cr> to continue, ^C to end...^C
So we can observe the total memory usage of another process simply and cheaply. This is often the first test to do when examining processes for memory leaks.