Boost.Python

May 2002 Progress Report

Introduction
New Features
Documentation
Miscellaneous
What's Next

Introduction

Aside from library development, work on Boost.Python in May was focused on reducing the support burden. In recent weeks, responding to requests for support, espcially surrounding building the library, had begun to impede progress on development. There was a major push to release a stable 1.28.0 of Boost, including documentation of Boost.Build and specific instructions for building Boost.Python v1. The documentation for Boost.Python v2 was also updated as described here.

New Features

Shared Library Support for AIX

The Kull group required the ability to build and test Boost.Python extensions on AIX, a platform with "creatively designed" shared library semantics. Making this work was a multi-pronged effort, involving changes to Boost.Build and some great research by Martin Casado which uncovered the key mechanism required to allow shared libraries to use functions from the Python executable. The current solution used in Boost.Build relies on a Python Script as part of the build process. This is not a problem for Boost.Python, as Python will be available. However, the commands issued by the script are so simple that a 100%-pure-Boost.Jam solution is surely possible. Linking on AIX is sufficiently interesting to have skewed the Boost.Python development schedule a bit.

Class Enhancements

Operators

Support for exposing C++ operators and functions as the corresponding Python special methods was added. Thinking that the Boost.Python v1 interface was a little too esoteric (especially the use of left_operand<...>/right_operand<...> for asymmetric operands), I introduced a simple form of expression templates which allow users to simply write the expressions that should be wrapped, as in this example.

Iterators

Python iterator support as required by the Kull project resulted in a highly flexible interface allowing:

Direct exposure of a class' begin() and end() functions: ... .def("__iter__", iterator<list_int>())
Creation of iterators from member functions... ... .def("__iter__" , range(&my_class::x_begin, &my_class::x_end)) )
...and member data: ... .def("__iter__" , range(&std::pair<char*,char*>::first, &std::pair<char*,char*>::second)) )
The ability to specify CallPolicies, e.g. to prevent copying of heavyweight values: ... .def("__iter__", , range<return_value_policy<copy_non_const_reference> >( &my_sequence<heavy>::begin , &my_sequence<heavy>::end))

Properties

The Kull iteration interfaces also required the ability to iterate over a sequence specified by an instance's attribute:

>>> f = field()
>>> for e in f.elements:
...     print e,

This forced the exposure of the property interface used internally to implement the data member exposure facility described in March. Properties are an incredibly useful idiom, so it's good to be able to provide them at little new development cost.

setattr

class_<> acquired a setattr member function which allows users to easily add new Python objects as class attributes.

module Attribute

Ralf Grosse-Kunstleve has been working on pickling support for v2. To make it work correctly, he had to make sure that a class' __module__ attribute was set correctly.

`back_reference`

The new back_reference<T> template can be used as a function parameter when the user needs access to both a T argument and to the Python object which manages it. The function will only match in the overload resolution process if it would match the same function signature with T substituted for back_reference<T>. This feature is not yet documented.

Documentation

In a major effort to prepare Boost.Python v2 to replace v1, many pages of new reference documentation were added:

CallPolicies.html

Dereferenceable.html

Extractor.html

HolderGenerator.html

ResultConverter.html

call_method.html

callbacks.html

data_members.html

has_back_reference.html

implicit.html

instance_holder.html

operators.html

ptr.html

type_id.html

with_custodian_and_ward.html

Major updates were made to the following pages:

call.html

updated

class.html

reference.html

As usual, careful documentation forces one to consider the interface again, and there were many interface changes associated with this effort, including the elevation of the following components from implementation detail to first-class library citizen:

type_id.hpp

pointee.hpp

lvalue_from_pytype.hpp

Miscellaneous

Converters

It appears that the world of C++ <==> Python conversion rules is an endlessly-rich area of exploration. Completing the conversions for char and char const* types, as described at the end of April's report, uncovered some interesting new shades to the problem. It turns out to be worth distinguishing mutable and immutable lvalue conversions, because despite the fact that Python doesn't understand const, it does understand immutability (c.f. Python strings, which expose an immutable char pointer). It is also worth recognizing types which represent lvalue sequences, to prevent Python "foobar" from being silently truncated to C++ 'f'. More details on this insight can be found in the mailing list archive. I don't plan to do anything about this immediately, but I do think it's the right direction to go in the long run.

Checkins Mailing List

In order to better coordinate changes made by multiple developers, I enabled syncmail for the Boost.Python CVS trees, and established an associated mailing list. Subscribe to this list to receive notices of each new checkin.

Shared Libraries

Beyond the vagaries of dynamic linking on AIX, I have been participating in a more-general discussion of dynamic linking for C++. Needless to say, C++ dynamic linking is of critical importance to Boost.Python: all extension modules are normally built as shared libraries, and Boost.Python extension modules share a common library as well. In fact, there are at least two separate conversations. One in the C++ standard extensions mailing list concerns what can be standardized for C++ and shared libraries; the other, mostly on the gcc mailing list, concerns the behavior of GCC on Posix/ELF platforms. Some of the GCC threads are here:

http://gcc.gnu.org/ml/gcc/2002-05/msg02002.html
http://gcc.gnu.org/ml/gcc/2002-05/msg02945.html
http://gcc.gnu.org/ml/gcc/2002-05/msg01758.html

What's Next

Development is focused on what's needed to be able to retire Boost.Python v1. At the moment, that means deciding the user-friendly interfaces for to_/from_python conversion, and formally exposing the Python object smart pointers and object wrapper classes. Quite a few questions have also been showing up recently about how to embed Python with Boost.Python, and how to link with it statically; the solutions to these issues will probably have to be formalized before long.

Revised 13 November, 2002

May 2002 Progress Report

Contents