Embedding basics

Manual reference counting

Most things in Python are objects. Therefore it is only natural that many of the Python C API functions operate on Python objects. Because C/C++ can't work with Python objects directly, the API defines a PyObject structure and a lot of functions to operate on PyObject pointers.

An important property of Python objects, and therefore of PyObjects, is that they are reference counted. This has major advantages compared to 'dumb' copying: it requiring less memory and it avoids unnecessary copying overhead. However, there is a downside as well. Although the reference counting is transparent from the Python-side, it is quite explicit in the C API. In other words you must increase and decrease the reference counts of PyObjects manually using the Py_XINCREF and Py_XDECREF macros. This is cumbersome, and if you don't do it properly some objects might be released when you still need them, or not be released at all.

I will briefly explain how to update the reference counts correctly, but I'll soon show a better way to do things.

The API functions that return PyObject pointers are listed in the Python C API documentation as either returning a borrowed or a new reference. The difference is in reference ownership.

When a new reference is returned, you own that reference. Therefore you don't need to worry about the object being deallocated while you still need it. You do need to decrease the reference count when you are done with it however, otherwise the object will never be deallocated: you'll have a resource leak.

Here's a simple example:

    // Create a new tuple of 3 elements long
    PyObject* my_tuple = PyTuple_New(3);
    ... // Use my_tuple here
    // We're done with the tuple
    Py_XDECREF(my_tuple);

When a borrowed reference is returned, you do not have ownership of the reference. So if you just want to discard the return value, there is nothing you have to do: you didn't own it anyway. If want to use it however, you'll first have to increase its reference count (to prevent the object's deletion). Then later on when you are done with itm you'll need to decrease the reference count again. Here's another example:

    // Retrieve the first item in the tuple
    PyObject* first = PyTuple_GetItem(my_tuple, 0);
    Py_XINCREF(first);
    ... // Use first here
    // We're done with the first tuple item
    Py_XDECREF(first);

While this certainly works, it's hardly elegant and it's easy to make mistakes, especially when there are multiple execution paths.

Running Python code from C++

To run Python code from C++ there is a family of functions in the API starting with the PyRun_ prefix. You can find the list of these fuctions here. I shall discuss one of them, the others work very similarly.

    PyObject* PyRun_String(char *str, int start, PyObject *globals, PyObject *locals)

PyRun_String takes the code to execute as a null-terminated (C-style) string in its str parameter. The function returns a new reference to the Python object which results from executing the code. If a Python exception occurred during execution, the null pointer is returned.

The start parameter is the start symbol from the Python grammar to use for interpreting the code. The possible values are:

Lastly, the globals and locals parameters are Python dictionaries containing the globals and locals of the context in which to run the code. For most intents and purposes you can use the namespace dictionary of the running module for both globals and locals.

So how do we get a module's namespace dictionary? First we need to get a reference to the module. The function to do this is PyImport_AddModule. Contrary to what you might guess from its name, it returns a borrowed reference to an existing module.

Then once we have a reference to the module, we can use PyModule_GetDict to get a borrowed reference to the module's namespace dictionary.

Since the running module is usually the __main__ module created upon interpreter initialization, the code to execute a Python program from a string becomes:

    // Get the __main__ module namespace
    PyObject* main_module = PyImport_AddModule("__main__");
    Py_XINCREF(main_module);
    PyObject* main_namespace = PyModule_GetDict(main_module);
    Py_XINCREF(main_namespace);
    // Run a single Python expression and retrieve the result
    PyObject* result = PyRun_String("1 + 1", 
        Py_eval_input, main_namespace, main_namespace);
    ... // Use the result
    // Cleanup
    Py_XDECREF(result);
    Py_XDECREF(main_namespace);
    Py_XDECREF(main_module);

Warning Be careful about what Python code you run and where. Running the addition above on your computer can hardly do any harm. But letting users run arbitrary Python code through your program which is running on a webserver can be a security risk.

Next up: How Boost.Python can simplify embedding...

Copyright © 2002 Dirk Gerrits

Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided "as is" without express or implied warranty, and with no claim as to its suitability for any purpose.