PyMOTW: resource - System resource management

By Doug Hellmann
September 20, 2009

resource – System resource management

Purpose:Manage the system resource limits for a Unix program.
Python Version:1.5.2

The functions in resource help you probe the current resources consumed by a process, and place limits on them to control how much load your program places on a system.

Current Usage

Use getrusage() to probe the resources used by the current process and/or its children. The return value is a data structure containing several resource metrics based on the current state of the system.

Note

Not all of the resource values gathered are displayed here. Refer to the stdlib docs for a more complete list.

import resource
import time

usage = resource.getrusage(resource.RUSAGE_SELF)

for name, desc in [
('ru_utime', 'User time'),
('ru_stime', 'System time'),
('ru_maxrss', 'Max. Resident Set Size'),
('ru_ixrss', 'Shared Memory Size'),
('ru_idrss', 'Unshared Memory Size'),
('ru_isrss', 'Stack Size'),
('ru_inblock', 'Block inputs'),
('ru_oublock', 'Block outputs'),
]:
print '%-25s (%-10s) = %s' % (desc, name, getattr(usage, name))

Because the test program is extremely simple, the results aren’t that interesting:

$ python resource_getrusage.py
User time (ru_utime ) = 0.015466
System time (ru_stime ) = 0.013189
Max. Resident Set Size (ru_maxrss ) = 0
Shared Memory Size (ru_ixrss ) = 0
Unshared Memory Size (ru_idrss ) = 0
Stack Size (ru_isrss ) = 0
Block inputs (ru_inblock) = 0
Block outputs (ru_oublock) = 1

Resource Limits

Separate from the current actual usage, it is possible to check the limits imposed on the application, and then change them.

import resource

for name, desc in [
('RLIMIT_CORE', 'core file size'),
('RLIMIT_CPU', 'CPU time'),
('RLIMIT_FSIZE', 'file size'),
('RLIMIT_DATA', 'heap size'),
('RLIMIT_STACK', 'stack size'),
('RLIMIT_RSS', 'resident set size'),
('RLIMIT_NPROC', 'number of processes'),
('RLIMIT_NOFILE', 'number of open files'),
('RLIMIT_MEMLOCK', 'lockable memory address'),
]:
limit_num = getattr(resource, name)
soft, hard = resource.getrlimit(limit_num)
print 'Maximum %-25s (%-15s) : %20s %20s' % (desc, name, soft, hard)

The return value for each limit is a tuple containing the soft limit imposed by the current configuration and the hard limit imposed by the operating system.

$ python resource_getrlimit.py
Maximum core file size (RLIMIT_CORE ) : 0 9223372036854775807
Maximum CPU time (RLIMIT_CPU ) : 9223372036854775807 9223372036854775807
Maximum file size (RLIMIT_FSIZE ) : 9223372036854775807 9223372036854775807
Maximum heap size (RLIMIT_DATA ) : 6291456 9223372036854775807
Maximum stack size (RLIMIT_STACK ) : 8388608 67104768
Maximum resident set size (RLIMIT_RSS ) : 9223372036854775807 9223372036854775807
Maximum number of processes (RLIMIT_NPROC ) : 266 532
Maximum number of open files (RLIMIT_NOFILE ) : 256 9223372036854775807
Maximum lockable memory address (RLIMIT_MEMLOCK ) : 9223372036854775807 9223372036854775807

The limits can be changed with setrlimit(). For example, to control the number of files a process can open the RLIMIT_NOFILE value can be set to use a smaller soft limit value.

import resource
import os

soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print 'Soft limit starts as :', soft

resource.setrlimit(resource.RLIMIT_NOFILE, (4, hard))

soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print 'Soft limit changed to :', soft

random = open('/dev/random', 'r')
print 'random has fd =', random.fileno()
try:
null = open('/dev/null', 'w')
except IOError, err:
print err
else:
print 'null has fd =', null.fileno()

$ python resource_setrlimit_nofile.py
Soft limit starts as : 256
Soft limit changed to : 4
random has fd = 3
[Errno 24] Too many open files: '/dev/null'

It can also be useful to limit the amount of CPU time a process should consume, to avoid eating up too much time. When the process runs past the allotted amount of time, it it sent a SIGXCPU signal.

import resource
import sys
import signal
import time

# Set up a signal handler to notify us
# when we run out of time.
def time_expired(n, stack):
print 'EXPIRED :', time.ctime()
raise SystemExit('(time ran out)')

signal.signal(signal.SIGXCPU, time_expired)

# Adjust the CPU time limit
soft, hard = resource.getrlimit(resource.RLIMIT_CPU)
print 'Soft limit starts as :', soft

resource.setrlimit(resource.RLIMIT_CPU, (1, hard))

soft, hard = resource.getrlimit(resource.RLIMIT_CPU)
print 'Soft limit changed to :', soft
print

# Consume some CPU time in a pointless exercise
print 'Starting:', time.ctime()
for i in range(200000):
for i in range(200000):
v = i * i

# We should never make it this far
print 'Exiting :', time.ctime()

Normally the signal handler should flush all open files and close them, but in this case we just print a message and exit.

$ python resource_setrlimit_cpu.py
Soft limit starts as : 9223372036854775807
Soft limit changed to : 1

Starting: Sun Sep 20 12:11:39 2009
EXPIRED : Sun Sep 20 12:11:40 2009
(time ran out)


See also

resource
The standard library documentation for this module.
signal
For details on registering signal handlers.

PyMOTW Home

The canonical version of this article


You might also be interested in:

News Topics

Recommended for You

Got a Question?