Get python application memory usage

Question

My main goal is to know how much memory my python application takes during execution.

I'm using python 2.7.5 on Windows-32 and Windows-64.

I found a way to get some info about my process here: http://code.activestate.com/recipes/578513-get-memory-usage-of-windows-processes-using-getpro/

Putting the code here for convenience:

"""Functions for getting memory usage of Windows processes."""

__all__ = ['get_current_process', 'get_memory_info', 'get_memory_usage']

import ctypes
from ctypes import wintypes

GetCurrentProcess = ctypes.windll.kernel32.GetCurrentProcess
GetCurrentProcess.argtypes = []
GetCurrentProcess.restype = wintypes.HANDLE

SIZE_T = ctypes.c_size_t

class PROCESS_MEMORY_COUNTERS_EX(ctypes.Structure):
    _fields_ = [
        ('cb', wintypes.DWORD),
        ('PageFaultCount', wintypes.DWORD),
        ('PeakWorkingSetSize', SIZE_T),
        ('WorkingSetSize', SIZE_T),
        ('QuotaPeakPagedPoolUsage', SIZE_T),
        ('QuotaPagedPoolUsage', SIZE_T),
        ('QuotaPeakNonPagedPoolUsage', SIZE_T),
        ('QuotaNonPagedPoolUsage', SIZE_T),
        ('PagefileUsage', SIZE_T),
        ('PeakPagefileUsage', SIZE_T),
        ('PrivateUsage', SIZE_T),
    ]

GetProcessMemoryInfo = ctypes.windll.psapi.GetProcessMemoryInfo
GetProcessMemoryInfo.argtypes = [
    wintypes.HANDLE,
    ctypes.POINTER(PROCESS_MEMORY_COUNTERS_EX),
    wintypes.DWORD,
]
GetProcessMemoryInfo.restype = wintypes.BOOL

def get_current_process():
    """Return handle to current process."""
    return GetCurrentProcess()

def get_memory_info(process=None):
    """Return Win32 process memory counters structure as a dict."""
    if process is None:
        process = get_current_process()
    counters = PROCESS_MEMORY_COUNTERS_EX()
    ret = GetProcessMemoryInfo(process, ctypes.byref(counters),
                               ctypes.sizeof(counters))
    if not ret:
        raise ctypes.WinError()
    info = dict((name, getattr(counters, name))
                for name, _ in counters._fields_)
    return info

def get_memory_usage(process=None):
    """Return this process's memory usage in bytes."""
    info = get_memory_info(process=process)
    return info['PrivateUsage']

if __name__ == '__main__':
    import pprint
    pprint.pprint(get_memory_info())

And this is the result:

{'PageFaultCount': 1942L,
 'PagefileUsage': 4624384L,
 'PeakPagefileUsage': 4624384L,
 'PeakWorkingSetSize': 7544832L,
 'PrivateUsage': 4624384L,
 'QuotaNonPagedPoolUsage': 8520L,
 'QuotaPagedPoolUsage': 117848L,
 'QuotaPeakNonPagedPoolUsage': 8776L,
 'QuotaPeakPagedPoolUsage': 117984L,
 'WorkingSetSize': 7544832L,
 'cb': 44L}

But this does not satisfy me. These results give me the whole python process information while what I need is only my specific application that runs on top of the Python framework.

I saw several memory profilers on the internet and also here in Stack Overflow but they are too big for me. The only information that I need is how much memory my app consumes by itself - without taking into account all the Python framework.

How can I achieve this?

Answer 1

Here is a simple easy pythonic way, based on (os, psutil) modules. Thanks to (Dataman) and (RichieHindle) answers.

import os
import psutil


## - Get Process Id of This Running Script -
proc_id = os.getpid()

print '\nProcess ID: ', proc_id


#--------------------------------------------------
## - Get More Info Using the Process Id

ObjInf = psutil.Process(proc_id)

print '\nProcess %s Info:' % proc_id, ObjInf

#--------------------------------------------------
## - Proccess Name of this program 

name = ObjInf.name()

print '\nThis Program Process name:', name

#--------------------------------------------------
## - Print CPU Percentage

CpuPerc = ObjInf.cpu_percent()

print '\nCpu Percentage:', CpuPerc


#---------------------------------------------------
## - Print Memory Usage

memory_inf = ObjInf.memory_full_info()

print '\nMemory Info:', memory_inf, '\n'



## Print available commands you can do with the psutil obj

for c in dir(ObjInf):
    print c 

If your script is made in python, then your script is python itself, so it wouldn't run without it, thus you have to account for python memory usage also, if you want to see how much memory python consumes by itself just run an empty python script and you will deduct from there, that your script will be the main resources consumer, which happens to be made in python thus python.

Now if you want to check memory usage of a thread then this question might help -> Why does python thread consume so much memory?

Answer 2

Here is my script I am using to find the maximum amount of resources used during the execution of another script. I am using psutilto achieve this. You can tweak the script to make it suitable for your purposes.

import psutil, sys
import numpy as np
from time import sleep
pid = int(sys.argv[1])
delay = int(sys.argv[2])
p = psutil.Process(pid)
max_resources_used = -1
while p.is_running():
    ## p.memory_info()[0] returns 'rss' in Unix
    r = int(p.memory_info()[0] / 1048576.0)  ## resources used in Mega Bytes
    max_resources_used = r if r > max_resources_used else max_resources_used
    sleep(delay)
print("Maximum resources used: %s MB." %np.max(max_resources_used))

Usage:

python script.py pid delay_in_seconds

For example:

python script.py  55356  2 

Explanation:

You need to find out the process ID and pass it as an argument to the script plus the time interval for checking the resource usage in seconds (i.e. every how many seconds the script checks the amount of used resources). The script keeps track of the memory usage until the process is running. Finally it returns the maximum amount of memory used in MB.