In a previous post, I described how you could use the ctypes Python library to import a C++ class to make it usable from within Python. There is however, another way to do this; and that’s by using Boost.Python (which as the name suggests is a part of the Boost C++ library suite).

Whilst this method makes our code on the C++ side slightly more complicated; it significantly simplifies the Python code – and it let’s us use some more powerful features.

To begin let’s start with a simple example.

// hello.cpp

#include <boost/python.hpp>
#include <string>

char const* greet()
{
   return "hello, world";
}

std::string multi_bob(int n)
{
   std::string name = "Bob";
   std::string r = "";
   for (int i=0;i<n;i++)
      r += name;
   return r;
}

BOOST_PYTHON_MODULE(hello)
{
    using namespace boost::python;
    def("greet", greet);
    def("multibob", multi_bob);
}

Everything above the BOOST_PYTHON_MODULE line is perfectly ordinary C++ code. The clever bit comes in when we call the BOOST_PYTHON_MODULE macro (which is defined within the Boost.Python header)…

In this case we’re creating a Python module – with two methods: one based on our very simple greet() function (which we’re also going to call greet); and one based on the more complex multi_bob(int) function. Note that for this second function, to show the fact that the linkage between the C++ & Python code can be very flexible, we give it a different name on the Python side. Also note that we don’t need to tell the macro about the signature of the function as this is handled for us by Boost.Python.

To actually build things we have to do a little more work.

Let’s start with a manually created Makefile that will build this on my Mac – with, Python3, Boost, and Boost.Python all installed using homebrew.

# location of the Python header files
 
PYTHON_VERSION = 3.6
PYTHON_INCLUDE = /usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/include/python3.6m/
PYTHON_LIB = /usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin

# location of the Boost Python include files and library
BOOST_INC =/usr/local/include/boost
BOOST_LIB =/usr/local/Cellar/boost-python/1.65.1/lib/ 
 
TARGET = hello
 
$(TARGET).so: $(TARGET).o
    g++ -shared -Wl $(TARGET).o -L$(BOOST_LIB) -lboost_python3 -L$(PYTHON_LIB) -lpython$(PYTHON_VERSION) -o $(TARGET).so
 
$(TARGET).o: $(TARGET).CPP
    g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c $(TARGET).CPP

What’s important to note here is that unlike the ctypes example: here we need to build our library code – and then link that library against both the Boost (well, Boost.Python) & Python libraries… This means that it’s not very portable: as the resulting library is fundamentally dependent on running in the version of Python it was linked against.

Life is generally far too short to manually write your own makefiles however (especially if you’re doing anything more complex than a simple demo): so we’d rather use CMake to build the Makefile for us. Before we get too far into this: I should first point out that there is a known issue with CMake, that effects us when using CMake with Boost.Python and Python3…

The details (if you really want to know, can be found here) but the short version is that CMake will somewhat messily always generate a warning when we run it to build this: but it is a warning that we can safely ignore.

With that in mind, our CMakeLists.txt file needs to look like this:

cmake_minimum_required(VERSION 3.9)
 
project(greeter)

# Find necessary packages
find_package(PythonLibs 3 REQUIRED)
include_directories(${PYTHON_INCLUDE_DIR})
 
find_package(Boost COMPONENTS python3 REQUIRED)

include_directories(${Boost_INCLUDE_DIR})
 
# Build & Link our library
add_library(hello MODULE hello.cpp)
target_link_libraries(hello ${Boost_LIBRARIES} ${PYTHON_LIBRARIES})

# don't prepend wrapper library name with lib
set_target_properties(hello PROPERTIES PREFIX "")

Having run this, and then made the resulting Makefile – we end up with a binary library called hello.so.

To consume this from Python we could use:

# demo.py

import hello

print(hello.greet())
print(hello.multibob(17))

On MacOS (at least at the time of writing) it’s not quite that simple – as for reasons that I don’t (at least as yet) fully understand find_package() fails to correctly find the Python libraries – and therefore doesn’t link the resultant .so with the Python library at all…

On MacOS we can check this using: otool -L hello.so

Which gives us:

$ otool -L hello.sohello.so:

/usr/local/opt/boost-python/lib/libboost_python3-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)

Whereas the equivalent command on Linux (ldd) (run for a file generated using exactly the same scripts) gives us:

 $ ldd hello.so
linux-vdso.so.1 (0x00007ffd1f3fe000)
libboost_python3.so.1.63.0 => /lib64/libboost_python3.so.1.63.0 (0x00007fa9ec603000)
libpython3.6m.so.1.0 => /lib64/libpython3.6m.so.1.0 (0x00007fa9ec0a1000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fa9ebd18000)
libm.so.6 => /lib64/libm.so.6 (0x00007fa9eba02000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fa9eb7eb000)
libc.so.6 => /lib64/libc.so.6 (0x00007fa9eb418000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fa9eb214000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa9eaff5000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007fa9eadf2000)
librt.so.1 => /lib64/librt.so.1 (0x00007fa9eabea000)
/lib64/ld-linux-x86-64.so.2 (0x000055726f8df000)

The key difference being line 3 – where there is a link to libpython3…

We can fix this universally – with a small addition to the CMakeLists.txt file, to override the value for the PYTHON_LIBRARIES variable:

cmake_minimum_required(VERSION 3.9)
 
project(greeter)

# Find necessary packages
find_package(PythonLibs 3 REQUIRED)
include_directories(${PYTHON_INCLUDE_DIR})
 
find_package(Boost COMPONENTS python3 REQUIRED)

include_directories(${Boost_INCLUDE_DIR})
 
# Build & Link our library
add_library(hello MODULE hello.cpp)

if(APPLE)
        set(PYTHON_LIBRARIES "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/libpython3.6.dylib")
endif()

target_link_libraries(hello ${Boost_LIBRARIES} ${PYTHON_LIBRARIES})

# don't prepend wrapper library name with lib
set_target_properties(hello PROPERTIES PREFIX "")

Obviously you’ll need to check where the relevant library file is on your own system, and substitute the correct path for the one shown here – if they’re different.

Next time, we’ll look at how to wrap some more complex C++ classes for use in Python; using Boost.Python.

---

Filed under: C++,Python - @ December 21, 2017 09:55