|
|
Boost.PythonFrequently Asked Questions (FAQs) |
*= operator
work?
typedef boost::function<void (string s) > funcptr;
void foo(funcptr fp)
{
fp("hello,world!");
}
BOOST_PYTHON_MODULE(test)
{
def("foo",foo) ;
}
And then:
>>> def hello(s): ... print s ... >>> foo(hello) hello, world!The short answer is: "you can't". This is not a Boost.Python limitation so much as a limitation of C++. The problem is that a Python function is actually data, and the only way of associating data with a C++ function pointer is to store it in a static variable of the function. The problem with that is that you can only associate one piece of data with every C++ function, and we have no way of compiling a new C++ function on-the-fly for every Python function you decide to pass to
foo. In other words, this could work if the C++
function is always going to invoke the same Python
function, but you probably don't want that.
If you have the luxury of changing the C++ code you're
wrapping, pass it an object instead and call that;
the overloaded function call operator will invoke the Python
function you pass it behind the object.
For more perspective on the issue, see this posting.
period const& get_floating_frequency() const
{
return boost::python::call_method<period const&>(
m_self,"get_floating_frequency");
}
And you get:
ReferenceError: Attempt to return dangling reference to object of type: class period
In this case, the Python method invoked by call_method
constructs a new Python object. You're trying to return a reference to a
C++ object (an instance of class period) contained within
and owned by that Python object. Because the called method handed back a
brand new object, the only reference to it is held for the duration of
get_floating_frequency() above. When the function returns,
the Python object will be destroyed, destroying the instance of
class period, and leaving the returned reference dangling.
That's already undefined behavior, and if you try to do anything with
that reference you're likely to cause a crash. Boost.Python detects this
situation at runtime and helpfully throws an exception instead of letting
you do that.
Q: I have an object composed of 12 doubles. A const& to this object is returned by a member function of another class. From the viewpoint of using the returned object in Python I do not care if I get a copy or a reference to the returned object. In Boost.Python Version 2 I have the choice of using copy_const_reference or return_internal_reference. Are there considerations that would lead me to prefer one over the other, such as size of generated code or memory overhead?A: copy_const_reference will make an instance with storage for one of your objects, size = base_size + 12 * sizeof(double). return_internal_reference will make an instance with storage for a pointer to one of your objects, size = base_size + sizeof(void*). However, it will also create a weak reference object which goes in the source object's weakreflist and a special callback object to manage the lifetime of the internally-referenced object. My guess? copy_const_reference is your friend here, resulting in less overall memory use and less fragmentation, also probably fewer total cycles.
Ralf W. Grosse-Kunstleve provides these notes:
class_<> wrapper:
class_<std::vector<double> >("std_vector_double")
.def(...)
...
;
This can be moved to a template so that several types (double, int,
long, etc.) can be wrapped with the same code. This technique is used
in the file
scitbx/include/scitbx/array_family/boost_python/flex_wrapper.hin the "scitbx" package. The file could easily be modified for wrapping std::vector<> instantiations.
This type of C++/Python binding is most suitable for containers that may contain a large number of elements (>10000).
void foo(std::vector<double> const& array); // pass by const-reference void foo(std::vector<double> array); // pass by valueSome custom rvalue converters are implemented in the file
scitbx/include/scitbx/boost_python/container_conversions.hThis code can be used to convert from C++ container types such as std::vector<> or std::list<> to Python tuples and vice versa. A few simple examples can be found in the file
scitbx/array_family/boost_python/regression_test_module.cppAutomatic C++ container <-> Python tuple conversions are most suitable for containers of moderate size. These converters generate significantly less object code compared to alternative 1 above.
It would also be useful to also have "custom lvalue converters" such as std::vector<> <-> Python list. These converters would support the modification of the Python list from C++. For example:
C++:
void foo(std::vector<double>& array)
{
for(std::size_t i=0;i<array.size();i++) {
array[i] *= 2;
}
}
Python:
>>> l = [1, 2, 3] >>> foo(l) >>> print l [2, 4, 6]Custom lvalue converters require changes to the Boost.Python core library and are currently not available.
P.S.:
The "scitbx" files referenced above are available via anonymous CVS:
cvs -d:pserver:anonymous@cvs.cctbx.sourceforge.net:/cvsroot/cctbx login cvs -d:pserver:anonymous@cvs.cctbx.sourceforge.net:/cvsroot/cctbx co scitbx
Q: I get this error message when compiling a large source file. What can I do?A: You have two choices:
- Upgrade your compiler (preferred)
- Break your source file up into multiple translation units.
my_module.cpp:... void more_of_my_module(); BOOST_PYTHON_MODULE(my_module) { def("foo", foo); def("bar", bar); ... more_of_my_module(); }more_of_my_module.cpp:void more_of_my_module() { def("baz", baz); ... }If you find that aclass_<...>declaration can't fit in a single source file without triggering the error, you can always pass a reference to theclass_object to a function in another source file, and call some of its member functions (e.g..def(...)) in the auxilliary source file:
more_of_my_class.cpp:void more_of_my_class(class<my_class>& x) { x .def("baz", baz) .add_property("xx", &my_class::get_xx, &my_class::set_xx) ; ... }
Greg Burley gives the following answer for Unix GCC users:
Once you have created a boost python extension for your c++ library or class, you may need to debug the code. Afterall this is one of the reasons for wrapping the library in python. An expected side-effect or benefit of using BPL is that debugging should be isolated to the c++ library that is under test, given that python code is minimal and boost::python either works or it doesn't. (ie. While errors can occur when the wrapping method is invalid, most errors are caught by the compiler ;-).The basic steps required to initiate a gdb session to debug a c++ library via python are shown here. Note, however that you should start the gdb session in the directory that contains your BPL my_ext.so module.
(gdb) target exec python (gdb) run >>> from my_ext import * >>> [C-c] (gdb) break MyClass::MyBuggyFunction (gdb) cont >>> pyobj = MyClass() >>> pyobj.MyBuggyFunction() Breakpoint 1, MyClass::MyBuggyFunction ... Current language: auto; currently c++ (gdb) do debugging stuff
Greg's approach works even better using Emacs' "gdb"
command, since it will show you each line of source as you step through
it.
On Windows, my favorite debugging solution is the debugger that comes with Microsoft Visual C++ 7. This debugger seems to work with code generated by all versions of Microsoft and Metrowerks toolsets; it's rock solid and "just works" without requiring any special tricks from the user.
Raoul Gough has provided the following for gdb on Windows:
gdb support for Windows DLLs has improved lately, so it is now possible to debug Python extensions using a few tricks. Firstly, you will need an up-to-date gdb with support for minimal symbol extraction from a DLL. Any gdb from version 6 onwards, or Cygwin gdb-20030214-1 and onwards should do. A suitable release will have a section in the gdb.info file under Configuration – Native – Cygwin Native – Non-debug DLL symbols. Refer to that info section for more details of the procedures outlined here.
Secondly, it seems necessary to set a breakpoint in the Python interpreter, rather than using ^C to break execution. A good place to set this breakpoint is PyOS_Readline, which will stop execution immediately before reading each interactive Python command. You have to let Python start once under the debugger, so that it loads its own DLL, before you can set the breakpoint:
$ gdb python GNU gdb 2003-09-02-cvs (cygwin-special) [...] (gdb) run Starting program: /cygdrive/c/Python22/python.exe Python 2.2.2 (#37, Oct 14 2002, 17:02:34) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> ^Z Program exited normally. (gdb) break *&PyOS_Readline Breakpoint 1 at 0x1e04eff0 (gdb) run Starting program: /cygdrive/c/Python22/python.exe Python 2.2.2 (#37, Oct 14 2002, 17:02:34) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. Breakpoint 1, 0x1e04eff0 in python22!PyOS_Readline () from /cygdrive/c/WINNT/system32/python22.dll (gdb) cont Continuing. >>> from my_ext import * Breakpoint 1, 0x1e04eff0 in python22!PyOS_Readline () from /cygdrive/c/WINNT/system32/python22.dll (gdb) # my_ext now loaded (with any debugging symbols it contains)
boost-python-runtest rule, you can ask it to launch your
debugger for you by adding "--debugger=debugger" to your bjam
command-line:
bjam -sTOOLS=vc7.1 "--debugger=devenv /debugexe" test bjam -sTOOLS=gcc -sPYTHON_LAUNCH=gdb testIt can also be extremely useful to add the
-d+2 option when
you run your test, because Boost.Build will then show you the exact
commands it uses to invoke it. This will invariably involve setting up
PYTHONPATH and other important environment variables such as
LD_LIBRARY_PATH which may be needed by your debugger in order to get
things to work right.
*= operator work?Q: I have exported my class to python, with many overloaded operators. it works fine for me except the*=operator. It always tells me "can't multiply sequence with non int type". If I usep1.__imul__(p2)instead ofp1 *= p2, it successfully executes my code. What's wrong with me?A: There's nothing wrong with you. This is a bug in Python 2.2. You can see the same effect in Pure Python (you can learn a lot about what's happening in Boost.Python by playing with new-style classes in Pure Python).
>>> class X(object): ... def __imul__(self, x): ... print 'imul' ... >>> x = X() >>> x *= 1To cure this problem, all you need to do is upgrade your Python to version 2.2.1 or later.
gcc (GCC) 3.3 20030304 (Apple Computer, Inc. build 1493)Under 10.2.8 get the August 2003 gcc update (free at http://connect.apple.com/). Under 10.3 get the Xcode Tools v1.0 (also free).
Python 2.3 is required. The Python that ships with 10.3 is fine. Under 10.2.8 use these commands to install Python as a framework:
./configure --enable-framework make make frameworkinstallThe last command requires root privileges because the target directory is /Library/Frameworks/Python.framework/Versions/2.3. However, the installation does not interfere with the Python version that ships with 10.2.8.
It is also crucial to increase the stacksize before starting compilations, e.g.:
limit stacksize 8192kIf the stacksize is too small the build might crash with internal compiler errors.
Sometimes Apple's compiler exhibits a bug by printing an error like the following while compiling a boost::python::class_<your_type> template instantiation:
.../inheritance.hpp:44: error: cannot dynamic_cast `p' (of type `struct cctbx::boost_python::<unnamed>::add_pair* ') to type `void*' (source type is not polymorphic)We do not know a general workaround, but if the definition of your_type can be modified the following was found to work in all cases encountered so far:
struct your_type
{
// before defining any member data
#if defined(__MACH__) && defined(__APPLE_CC__) && __APPLE_CC__ == 1493
bool dummy_;
#endif
// now your member data, e.g.
double x;
int j;
// etc.
};
"I am wrapping a function that always returns a pointer to an already-held C++ object."One way to do that is to hijack the mechanisms used for wrapping a class with virtual functions. If you make a wrapper class with an initial PyObject* constructor argument and store that PyObject* as "self", you can get back to it by casting down to that wrapper type in a thin wrapper function. For example:
class X { X(int); virtual ~X(); ... };
X* f(); // known to return Xs that are managed by Python objects
// wrapping code
struct X_wrap : X
{
X_wrap(PyObject* self, int v) : self(self), X(v) {}
PyObject* self;
};
handle<> f_wrap()
{
X_wrap* xw = dynamic_cast<X_wrap*>(f());
assert(xw != 0);
return handle<>(borrowed(xw->self));
}
...
def("f", f_wrap());
class_<X,X_wrap,boost::noncopyable>("X", init<int>())
...
;
Of course, if X has no virtual functions you'll have to use
static_cast instead of dynamic_cast with no
runtime check that it's valid. This approach also only works if the
X object was constructed from Python, because
Xs constructed from C++ are of course never
X_wrap objects.
Another approach to this requires you to change your C++ code a bit;
if that's an option for you it might be a better way to go. work we've
been meaning to get to anyway. When a shared_ptr<X> is
converted from Python, the shared_ptr actually manages a reference to the
containing Python object. When a shared_ptr<X> is converted back to
Python, the library checks to see if it's one of those "Python object
managers" and if so just returns the original Python object. So you could
just write object(p) to get the Python object back. To
exploit this you'd have to be able to change the C++ code you're wrapping
so that it deals with shared_ptr instead of raw pointers.
There are other approaches too. The functions that receive the Python object that you eventually want to return could be wrapped with a thin wrapper that records the correspondence between the object address and its containing Python object, and you could have your f_wrap function look in that mapping to get the Python object out.
Part of an API that I'm wrapping goes something like this:Yes: Make sure the C++ object is held by auto_ptr:struct A {}; struct B { void add( A* ); } where B::add() takes ownership of the pointer passed to it.However:
a = mod.A() b = mod.B() b.add( a ) del a del b # python interpreter crashes # later due to memory corruption.Even binding the lifetime of a to b via with_custodian_and_ward doesn't prevent the python object a from ultimately trying to delete the object it's pointing to. Is there a way to accomplish a 'transfer-of-ownership' of a wrapped C++ object?
--Bruce Lowery
class_<A, std::auto_ptr<A> >("A")
...
;
Then make a thin wrapper function which takes an auto_ptr parameter:
void b_insert(B& b, std::auto_ptr<A> a)
{
b.insert(a.get());
a.release();
}
Wrap that as B.add. Note that pointers returned via manage_new_object
will also be held by auto_ptr, so this transfer-of-ownership
will also work correctly.
Please refer to the Reducing Compiling Time section in the tutorial.
Please refer to the Creating Packages section in the tutorial.
If you see Microsoft Visual C++ 7.1 (MS Visual Studio .NET 2003) issue an error message like the following it is most likely due to a bug in the compiler:
boost\boost\python\detail\invoke.hpp(76): error C2064: term does not evaluate to a function taking 2 arguments"This message is triggered by code like the following:
#include <boost/python.hpp>
using namespace boost::python;
class FXThread
{
public:
bool setAutoDelete(bool doso) throw();
};
void Export_FXThread()
{
class_< FXThread >("FXThread")
.def("setAutoDelete", &FXThread::setAutoDelete)
;
}
The bug is related to the throw() modifier.
As a workaround cast off the modifier. E.g.:
.def("setAutoDelete", (bool (FXThread::*)(bool)) &FXThread::setAutoDelete)
(The bug has been reported to Microsoft.)
Below is a small, self-contained demo extension module that shows how to do this. Here is the corresponding trivial test:
import custom_string
assert custom_string.hello() == "Hello world."
assert custom_string.size("california") == 10
If you look at the code you will find:
#include <boost/python/module.hpp>
#include <boost/python/def.hpp>
#include <boost/python/to_python_converter.hpp>
namespace sandbox { namespace {
class custom_string
{
public:
custom_string() {}
custom_string(std::string const& value) : value_(value) {}
std::string const& value() const { return value_; }
private:
std::string value_;
};
struct custom_string_to_python_str
{
static PyObject* convert(custom_string const& s)
{
return boost::python::incref(boost::python::object(s.value()).ptr());
}
};
struct custom_string_from_python_str
{
custom_string_from_python_str()
{
boost::python::converter::registry::push_back(
&convertible,
&construct,
boost::python::type_id<custom_string>());
}
static void* convertible(PyObject* obj_ptr)
{
if (!PyString_Check(obj_ptr)) return 0;
return obj_ptr;
}
static void construct(
PyObject* obj_ptr,
boost::python::converter::rvalue_from_python_stage1_data* data)
{
const char* value = PyString_AsString(obj_ptr);
if (value == 0) boost::python::throw_error_already_set();
void* storage = (
(boost::python::converter::rvalue_from_python_storage<custom_string>*)
data)->storage.bytes;
new (storage) custom_string(value);
data->convertible = storage;
}
};
custom_string hello() { return custom_string("Hello world."); }
std::size_t size(custom_string const& s) { return s.value().size(); }
void init_module()
{
using namespace boost::python;
boost::python::to_python_converter<
custom_string,
custom_string_to_python_str>();
custom_string_from_python_str();
def("hello", hello);
def("size", size);
}
}} // namespace sandbox::<anonymous>
BOOST_PYTHON_MODULE(custom_string)
{
sandbox::init_module();
}
If you define custom converters similar to the ones shown above the def_readonly() and def_readwrite() member functions provided by boost::python::class_ for direct access to your member data will not work as expected. This is because def_readonly("bar", &foo::bar) is equivalent to:
.add_property("bar", make_getter(&foo::bar, return_internal_reference()))
Similarly, def_readwrite("bar", &foo::bar) is
equivalent to:
.add_property("bar", make_getter(&foo::bar, return_internal_reference()),
make_setter(&foo::bar, return_internal_reference())
In order to define return value policies compatible with the
custom conversions replace def_readonly() and
def_readwrite() by add_property(). E.g.:
.add_property("bar", make_getter(&foo::bar, return_value_policy<return_by_value>()),
make_setter(&foo::bar, return_value_policy<return_by_value>()))
The quick answer to this is: no.
The longer answer is that it can be patched to be so, but it's complex. You will need to add custom lock/unlock wrapping of every time your code enters Boost.Python (particularly every virtual function override) plus heavily modify boost/python/detail/invoke.hpp with custom unlock/lock wrapping of every time Boost.Python enters your code. You must furthermore take care to not unlock/lock when Boost.Python is invoking iterator changes via invoke.hpp.
There is a patched invoke.hpp posted on the C++-SIG mailing list archives and you can find a real implementation of all the machinery necessary to fully implement this in the TnFOX project at this SourceForge project location.
Revised 12 March, 2006
© Copyright Dave Abrahams 2002-2006.