Advanced C++ / Python integration with Boost.Python (Part 2)

Last time we looked at how to use Boost.Python to wrap a very simple piece of C++ code. This time we’re going to take that one step further along, and do the same thing for a more complex C++ example – which includes a C++ Class.

For the purposes of this example – let us assume that we have a “legacy” C++ class (i.e. one that we’re not going to change): which looks something like this:

class ExampleClass
{
public:

  ExampleClass();
  void print_state();
  unsigned char* exporter();
  void set_data(uint8_t, uint8_t, uint8_t);
  void importer(unsigned char*, int);

private:
  uint8_t a;
  uint8_t b;
  uint8_t c;

};

This (deliberately, slightly contrived) class has three methods (and a constructor). We can set the three 8-bit unsigned integers (uint8_t) directly using the set_data() method; we can print the data to the screen; or we can either import from or export to an unsigned char*.

In order to be able to use the importer & exporter methods with Python (via Boost.Python) we’ll need to further wrap these – as Boost.Python won’t let us use unsigned char* directly.

In order to use the exporter() method; we need a wrapper for the unsigned char* so that in Python it will become a Python bytes object.

If we want to be able to extract the internal state of the object (for example to be able to copy it to a new object to create a copy of the internal state of the original one) we need to define a way for Python to understand the data it’s going to receive. In C++ we’d just create a void* or unsigned char* and use memcpy to copy the arbitrary memory; but we can’t do that for Python. Instead, here we make use of MemoryView a type of PyObject object (a Python built-in type) to contain the data. But in order to get the data into that object we need a little additional work.

However since we’ve already said we don’t want to change our class directly – we could put all of the wrapper code into a separate library.cpp file…

The wrapper code required is here:

PyObject* example_class_export_wrap(ExampleClass& self)
{
    PyObject* pymemview;
    unsigned char* export_data;

    export_data = self.exporter();

    pymemview = PyMemoryView_FromMemory((char*) export_data, 3, PyBUF_READ);
    return pymemview;
}

We create a PyObject* to store the data; and a regular C++ unsigned char* to contain the returned data from the exporter() method. The clever bit comes when we call PyMemoryView_FromMemory the intuition for which is essentially doing the same as our memcpy… It takes three bytes (as specified) from export_data, and makes them available as read-only data to Python.

From Python we can then turn it into a bytes object using .tobytes(). We can then interact with the data as with any other Python data…

import example
example_object = example.ExampleClass()

# do some things...

data = example_object.exporter().tobytes()
for b in data:
  print("0x{:02X}".format(b))

In fact we can go one better – and obviate the need to do the bytes conversion in Python if we use one additional line:

PyObject* example_class_export_wrap(ExampleClass& self)
{
    PyObject* pymemview;
    unsigned char* export_data;

    export_data = self.exporter();

    pymemview = PyMemoryView_FromMemory((char*) export_data, 3, PyBUF_READ);
    return PyBytes_FromObject(pymemview);
}

This way the return value when we call the exporter from Python is just a regular Python bytes object.

What about going back the other way? How can we do this import from Python bytes?

Again we need a wrapper function…

void example_class_import_wrap(ExampleClass& self, boost::python::object py_buffer)
{
  namespace python = boost::python;
  
  python::stl_input_iterator<char> begin(py_buffer), end;

  // Copy the py_buffer into a local buffer with known continguous memory.
  std::vector<char> buffer(begin, end);

  // Cast and delegate to the importer member function.
  self.importer(reinterpret_cast<unsigned char*>(&buffer[0]), buffer.size());
}

This code is based on an answer in Stack Overflow (see here). As described there: “…the auxiliary C++ function would need to populate a continuous block of memory with the elements of from the bytes [object]. The boost::python::stl_input_iterator can provide a convenient way to construct C++ containers, such as std::vector<char>, from a Python object, such as … bytes …”

The original example there (which was designed to be robust with Python 2 code too) – constructs an iterator from the py_buffer object (because, as the answer explains, Python 2 doesn’t have bytes and the str object use instead isn’t iterable). Since it’s nearly 2018, no-one should be using Python 2 any more; so we can skip that additional complexity.

We can then consume this from Python with something like this:

...

pybuf = struct.pack('BBB', 0x49, 0x4A, 0x4B)
e = example.ExampleClass()

e.importer(pybuf.decode())

# If we export this back out – we should get the bytes we started with...
t = e.exporter()
if t == pybuf:
    print("t == pybuff")

The only remaining thing to do is to actually link all of this up – so that Boost.Python knows to call these functions: rather than calling the original class methods directly.

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;
  python::class_<ExampleClass>("ExampleClass")
    .def("print", &ExampleClass::print_state)
    .def("exporter", &example_class_export_wrap)
    .def("set_data", &ExampleClass::set_data)
    .def("importer", &example_class_import_wrap)
    ;
}

Here we see that for the print & set methods we just provide the method directly to the Boost.Python macro; but for the other two (would-be) Python-side methods: we provide the functions discussed above.

The full code to support this example can be found here: https://github.com/Auctoris/boost_python_impex