Pymalloc, a specialized object allocator written by Vladimir
Marangozov, was a feature added to Python 2.1. Pymalloc is intended
to be faster than the system malloc() and to have less
memory overhead for allocation patterns typical of Python programs.
The allocator uses C's malloc() function to get large
pools of memory and then fulfills smaller memory requests from these
pools.
In 2.1 and 2.2, pymalloc was an experimental feature and wasn't
enabled by default; you had to explicitly enable it when compiling
Python by providing the
--with-pymalloc option to the configure
script. In 2.3, pymalloc has had further enhancements and is now
enabled by default; you'll have to supply
--without-pymalloc to disable it.
This change is transparent to code written in Python; however,
pymalloc may expose bugs in C extensions. Authors of C extension
modules should test their code with pymalloc enabled,
because some incorrect code may cause core dumps at runtime.
There's one particularly common error that causes problems. There are
a number of memory allocation functions in Python's C API that have
previously just been aliases for the C library's malloc()
and free(), meaning that if you accidentally called
mismatched functions the error wouldn't be noticeable. When the
object allocator is enabled, these functions aren't aliases of
malloc() and free() any more, and calling the
wrong function to free memory may get you a core dump. For example,
if memory was allocated using PyObject_Malloc(), it has to
be freed using PyObject_Free(), not free(). A
few modules included with Python fell afoul of this and had to be
fixed; doubtless there are more third-party modules that will have the
same problem.
As part of this change, the confusing multiple interfaces for
allocating memory have been consolidated down into two API families.
Memory allocated with one family must not be manipulated with
functions from the other family. There is one family for allocating
chunks of memory and another family of functions specifically for
allocating Python objects.
To allocate and free an undistinguished chunk of memory use
the ``raw memory'' family: PyMem_Malloc(),
PyMem_Realloc(), and PyMem_Free().
The ``object memory'' family is the interface to the pymalloc
facility described above and is biased towards a large number of
``small'' allocations: PyObject_Malloc,
PyObject_Realloc, and PyObject_Free.
To allocate and free Python objects, use the ``object'' family
PyObject_New(), PyObject_NewVar(), and
PyObject_Del().
Thanks to lots of work by Tim Peters, pymalloc in 2.3 also provides
debugging features to catch memory overwrites and doubled frees in
both extension modules and in the interpreter itself. To enable this
support, compile a debugging version of the Python interpreter by
running configure with --with-pydebug.
To aid extension writers, a header file Misc/pymemcompat.h is
distributed with the source to Python 2.3 that allows Python
extensions to use the 2.3 interfaces to memory allocation while
compiling against any version of Python since 1.5.2. You would copy
the file from Python's source distribution and bundle it with the
source of your extension.
For the full details of the pymalloc implementation, see
the comments at the top of the file Objects/obmalloc.c in the
Python source code. The above link points to the file within the
SourceForge CVS browser.
