Embedding with Boost.Python

Boost.Python to the rescue

Now we get to the good news. If you don't want to do all the error prone reference counting yourself, you can let Boost.Python do all the work. First include <boost/python.hpp> instead of "Python.h" and link to boost_python.lib (or boost_python_debug.lib) instead of pythonXY.lib. Then all we really need to do is replace every PyObject* with handle<> and we can remove all the Py_XINCREFs and Py_XDECREFs! All the reference counting will be done automagically through the power of the Resource Acquisition Is Initialization idiom.

We still need a way to differentiate between new and borrowed references though. Luckily, this is pretty straightforward using the borrowed function. Let's rewrite the example to run Python code from a string to use handles:

    // Get the __main__ module namespace
    handle<> main_module( borrowed(PyImport_AddModule("__main__")) );
    handle<> main_namespace( borrowed(PyModule_GetDict(main_module)) );
    // Run a single Python expression and retrieve the result
    handle<> result( PyRun_String("1 + 1", 
        Py_eval_input, main_namespace.get(), main_namespace.get()) );
    ... // Use the result

The get() member function used here returns the raw PyObject* that is held by the handle.

Boost.Python modules in an embedded program

Now that we know how to call Python code from C++ and C++ code from Python, how about doing both at the same time? Sometimes you might want to call Python code from C++ and have that Python code call C++ code again.

If you built your Boost.Python module and put it in the proper directory, you can just use it in your embedded Python code as you would in a standalone Python program. The embedded interpreter will be able to find your module just as the standalone interpreter would.

However, you can also define the Boost.Python module in the same program that embeds the Python interpreter. Then you won't have to build the module seperately and place it in the proper directory. This also prevents others from using your module in their own Python code. (Unless they start taking your executable apart that is. )

Doing this is relatively straightforward. You just define your Boost.Python module as usual and use the basic embedding steps described earlier. However, before calling Py_Initialize you call PyImport_AppendInittab first.

This function takes the name and initialization function of your module as parameters and adds the module to the interpreter's list of builtin modules. So when the Python interpreter comes across an import statement, it will find the module in its list of builtin modules instead of (unsuccessfully) searching for it in the Python directory.

Your program will look something like this:

    BOOST_PYTHON_MODULE(my_module)
    {
        ...
    }
    ...
    PyImport_AppendInittab("my_module", initmy_module);
    Py_Initialize();
    ...

Warning When creating Boost.Python modules in the same program that embeds interpreter, you must not call Py_Finalize. Boost.Python keeps some PyObject references alive in global data structures, and when those go out of scope after interpreter finalization, Python crashes. This will be fixed in the future. Stay tuned.

Additional reading

A more elaborate example showing these techniques is located at /libs/python/test/embedding.cpp.

Copyright © 2002 Dirk Gerrits

Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided "as is" without express or implied warranty, and with no claim as to its suitability for any purpose.