14.1. C++ RAII Wrappers Around PyObject*
¶
It is sometimes useful to wrap up a PyObject*
in a class that will manage the reference count. Here is a base class that shows the general idea, it takes a PyObject *
and provides:
Construction with a
PyObject *
and access this withoperator PyObject*() const
.PyObject **operator&()
to reset the underlying pointer, for example when using it withPyArg_ParseTupleAndKeywords
.Decrementing the reference count on destruction (potientially freeing the object).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | /** General wrapper around a PyObject*.
* This decrements the reference count on destruction.
*/
class DecRefDtor {
public:
DecRefDtor(PyObject *ref) : m_ref { ref } {}
Py_ssize_t ref_count() const { return m_ref ? Py_REFCNT(m_ref) : 0; }
// Allow setting of the (optional) argument with PyArg_ParseTupleAndKeywords
PyObject **operator&() {
Py_XDECREF(m_ref);
m_ref = NULL;
return &m_ref;
}
// Access the argument
operator PyObject*() const { return m_ref; }
// Test if constructed successfully from the new reference.
explicit operator bool() { return m_ref != NULL; }
~DecRefDtor() { Py_XDECREF(m_ref); }
protected:
PyObject *m_ref;
};
|
14.1.1. C++ RAII Wrapper for a Borrowed PyObject*
¶
There are two useful sub-classes, one for borrowed references, one for new references which are intended to be temporary. Using borrowed references:
1 2 3 4 5 6 7 8 9 10 | /** Wrapper around a PyObject* that is a borrowed reference.
* This increments the reference count on construction and
* decrements the reference count on destruction.
*/
class BorrowedRef : public DecRefDtor {
public:
BorrowedRef(PyObject *borrowed_ref) : DecRefDtor(borrowed_ref) {
Py_XINCREF(m_ref);
}
};
|
This can be used with borrowed references as follows:
void function(PyObject *obj) {
BorrowedRef(obj); // Increment reference here.
// ...
} // Decrement reference here.
14.1.2. C++ RAII Wrapper for a New PyObject*
¶
Here is a sub-class that wraps a new reference to a PyObject *
and ensures it is free’d when the wrapper goes out of scope:
/** Wrapper around a PyObject* that is a new reference.
* This owns the reference so does not increment it on construction but
* does decrement it on destruction.
*/
class NewRef : public DecRefDtor {
public:
NewRef(PyObject *new_ref) : DecRefDtor(new_ref) {}
};
This new reference wrapper can be used as follows:
void function() {
NewRef(PyLongFromLong(9)); // New reference here.
// Use static_cast<PyObject*>(NewRef) ...
} // Decrement the new reference here.
14.2. Handling Default Arguments¶
Handling default, possibly mutable, arguments in a pythonic way is described here: Being Pythonic with Default Arguments. It is quite complicated to get it right but C++ can ease the pain with a generic class to simplify handling default arguments in CPython functions:
1 2 3 4 5 6 7 8 9 10 11 12 13 | class DefaultArg {
public:
DefaultArg(PyObject *new_ref) : m_arg { NULL }, m_default { new_ref } {}
// Allow setting of the (optional) argument with PyArg_ParseTupleAndKeywords
PyObject **operator&() { m_arg = NULL; return &m_arg; }
// Access the argument or the default if default.
operator PyObject*() const { return m_arg ? m_arg : m_default; }
// Test if constructed successfully from the new reference.
explicit operator bool() { return m_default != NULL; }
protected:
PyObject *m_arg;
PyObject *m_default;
};
|
Suppose we have the Python function signature of def function(encoding='utf8', cache={}):
then in C/C++ we can do this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | PyObject *
function(PyObject * /* module */, PyObject *args, PyObject *kwargs) {
/* ... */
static DefaultArg encoding(PyUnicode_FromString("utf8"));
static DefaultArg cache(PyDict_New());
/* Check constructed OK. */
if (! encoding || ! cache) {
return NULL;
}
static const char *kwlist[] = { "encoding", "cache", NULL };
if (! PyArg_ParseTupleAndKeywords(args, kwargs, "|OO", const_cast<char**>(kwlist), &encoding, &cache)) {
return NULL;
}
/* Then just use encoding, cache as if they were a PyObject* (possibly
* might need to be cast to some specific PyObject*). */
/* ... */
}
|
14.3. Homogeneous Python Containers and C++¶
Here are some useful generic functions that can convert homogeneous Python containers to and from their C++ STL equivalents. They use templates to identify the C++ type and function pointers to convert from Python to C++ objects and back. These functions must have a these characteristics on error:
Converting from C++ to Python, on error set a Python error (e.g with
PyErr_SetString
orPyErr_Format
) and return NULL.Converting from Python to C++ set a Python error and return a default C++ object (for example an empty
std::string
).
For illustration here are a couple of such functions that convert PyBytesObject*
to and from std::string
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | std::string py_bytes_to_std_string(PyObject *py_str) {
std::string r;
if (PyBytes_Check(py_str)) {
r = std::string(PyBytes_AS_STRING(py_str));
} else {
PyErr_Format(PyExc_TypeError,
"Argument %s must be bytes not \"%s\"",
__FUNCTION__, Py_TYPE(py_str)->tp_name);
}
return r;
}
PyObject *std_string_to_py_bytes(const std::string &str) {
return PyBytes_FromStringAndSize(str.c_str(), str.size());
}
|
We can use this for a variety of containers, first Python lists of bytes
.
14.3.1. Python Lists and C++ std::vector<T>
¶
14.3.1.1. Python list
to C++ std::vector<T>
¶
This converts a python list to a std::vector<T>
. ConvertToT
is a function pointer to a function that takes a PyObject*
and returns an instance of a T
type. On failure to to convert a PyObject*
this function should set a Python error (making PyErr_Occurred()
non-NULL) and return a default T
. On failure this function sets PyErr_Occurred()
and the return value will be an empty vector.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | template <typename T>
std::vector<T>
py_list_to_std_vector(PyObject *py_list, T (*ConvertToT)(PyObject *)) {
assert(cpython_asserts(py_list));
std::vector<T> cpp_vector;
if (PyList_Check(py_list)) {
cpp_vector.reserve(PyList_GET_SIZE(py_list));
for (Py_ssize_t i = 0; i < PyList_GET_SIZE(py_list); ++i) {
cpp_vector.emplace(cpp_vector.end(),
(*ConvertToT)(PyList_GetItem(py_list, i)));
if (PyErr_Occurred()) {
cpp_vector.clear();
break;
}
}
} else {
PyErr_Format(PyExc_TypeError,
"Argument \"py_list\" to %s must be list not \"%s\"",
__FUNCTION__, Py_TYPE(py_list)->tp_name);
}
return cpp_vector;
}
|
If we have a function std::string py_bytes_to_std_string(PyObject *py_str);
(above) we can use this thus, we have to specify the C++ template specialisation:
std::vector<std::string> result = py_list_to_std_vector<std::string>(py_list, &py_bytes_to_std_string);
if (PyErr_Occurred()) {
// Handle error condition.
} else {
// All good.
}
14.3.1.2. C++ std::vector<T>
to Python list
¶
And the inverse that takes a C++ std::vector<T>
and makes a Python list
. ConvertToPy
is a pointer to a function that takes an instance of a T
type and returns a PyObject*
, this should return NULL on failure and set PyErr_Occurred()
. On failure this function sets PyErr_Occurred()
and returns NULL.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | template <typename T>
PyObject*
std_vector_to_py_list(const std::vector<T> &cpp_vec,
PyObject *(*ConvertToPy)(const T&)
) {
assert(cpython_asserts());
PyObject *r = PyList_New(cpp_vec.size());
if (! r) {
goto except;
}
for (Py_ssize_t i = 0; i < cpp_vec.size(); ++i) {
PyObject *item = (*ConvertToPy)(cpp_vec[i]);
if (! item || PyErr_Occurred() || PyList_SetItem(r, i, item)) {
goto except;
}
}
assert(! PyErr_Occurred());
assert(r);
goto finally;
except:
assert(PyErr_Occurred());
// Clean up list
if (r) {
// No PyList_Clear().
for (Py_ssize_t i = 0; i < PyList_GET_SIZE(r); ++i) {
Py_XDECREF(PyList_GET_ITEM(r, i));
}
Py_DECREF(r);
r = NULL;
}
finally:
return r;
}
|
If we have a function PyObject *std_string_to_py_bytes(const std::string &str);
(above) we can use this thus:
std::vector<std::string> cpp_vector;
// Initialise cpp_vector...
PyObject *py_list = std_vector_to_py_list(cpp_vector, &std_string_to_py_bytes);
if (! py_list) {
// Handle error condition.
} else {
// All good.
}
14.3.2. Python Sets, Frozensets and C++ std::unordered_set<T>
¶
14.3.2.1. Python set
to C++ std::unordered_set<T>
¶
Convert a Python set
or frozenset
to a std::unordered_set<T>
. ConvertToT
is a function pointer to a function that takes a PyObject*
and returns an instance of a T
type. This function should make PyErr_Occurred()
true on failure to convert a PyObject*
and return a default T
. On failure this sets PyErr_Occurred()
and the return value will be an empty container.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | template <typename T>
std::unordered_set<T>
py_set_to_std_unordered_set(PyObject *py_set, T (*ConvertToT)(PyObject *)) {
assert(cpython_asserts(py_set));
std::unordered_set<T> cpp_set;
if (PySet_Check(py_set) || PyFrozenSet_Check(py_set)) {
// The C API does not allow direct access to an item in a set so we
// make a copy and pop from that.
PyObject *set_copy = PySet_New(py_set);
if (set_copy) {
while (PySet_GET_SIZE(set_copy)) {
PyObject *item = PySet_Pop(set_copy);
if (! item || PyErr_Occurred()) {
PySet_Clear(set_copy);
cpp_set.clear();
break;
}
cpp_set.emplace((*ConvertToT)(item));
Py_DECREF(item);
}
Py_DECREF(set_copy);
} else {
assert(PyErr_Occurred());
}
} else {
PyErr_Format(PyExc_TypeError,
"Argument \"py_set\" to %s must be set or frozenset not \"%s\"",
__FUNCTION__, Py_TYPE(py_set)->tp_name);
}
return cpp_set;
}
|
14.3.2.2. C++ std::unordered_set<T>
to Python set
or frozenset
¶
Convert a std::unordered_set<T>
to a new Python set
or frozenset
. ConvertToPy
is a pointer to a function that takes an instance of a T
type and returns a PyObject*
, this function should return NULL on failure. On failure this function sets PyErr_Occurred()
and returns NULL.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | template <typename T>
PyObject*
std_unordered_set_to_py_set(const std::unordered_set<T> &cpp_set,
PyObject *(*ConvertToPy)(const T&),
bool is_frozen=false) {
assert(cpython_asserts());
PyObject *r = NULL;
if (is_frozen) {
r = PyFrozenSet_New(NULL);
} else {
r = PySet_New(NULL);
}
if (! r) {
goto except;
}
for (auto &iter: cpp_set) {
PyObject *item = (*ConvertToPy)(iter);
if (! item || PyErr_Occurred() || PySet_Add(r, item)) {
goto except;
}
}
assert(! PyErr_Occurred());
assert(r);
goto finally;
except:
assert(PyErr_Occurred());
// Clean up set
if (r) {
PySet_Clear(r);
Py_DECREF(r);
r = NULL;
}
finally:
return r;
}
|
14.3.3. Python Dicts and C++ std::unordered_map<K, V>
¶
14.3.3.1. Python dict
to C++ std::unordered_map<K, V>
¶
Convert a Python dict
to a std::unordered_map<K, V>
. PyKeyConvertToK
and PyKeyConvertToK
are function pointers to functions that takes a PyObject*
and returns an instance of a K
or V
type. On failure to convert a PyObject*
this function should make PyErr_Occurred()
true and return a default value.
On failure this function will make PyErr_Occurred()
non-NULL and return an empty map.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | template <typename K, typename V>
std::unordered_map<K, V>
py_dict_to_std_unordered_map(PyObject *dict,
K (*PyKeyConvertToK)(PyObject *),
V (*PyValConvertToV)(PyObject *)
) {
Py_ssize_t pos = 0;
PyObject *key = NULL;
PyObject *val = NULL;
std::unordered_map<K, V> cpp_map;
if (! PyDict_Check(dict)) {
PyErr_Format(PyExc_TypeError,
"Argument \"dict\" to %s must be dict not \"%s\"",
__FUNCTION__, Py_TYPE(dict)->tp_name);
return cpp_map;
}
while (PyDict_Next(dict, &pos, &key, &val)) {
K cpp_key = (*PyKeyConvertToK)(key);
if (PyErr_Occurred()) {
cpp_map.clear();
break;
}
V cpp_val = (*PyValConvertToV)(val);
if (PyErr_Occurred()) {
cpp_map.clear();
break;
}
cpp_map.emplace(cpp_key, cpp_val);
}
return cpp_map;
}
|
The following expects a Python dict of {bytes : bytes}
and will convert it to a std::unordered_map<std::string, std::string>
:
std::unordered_map<std::string, std::string> result;
result = py_dict_to_std_unordered_map(py_dict,
&py_bytes_to_std_string,
&py_bytes_to_std_string);
if (PyErr_Occurred()) {
// Handle failure...
} else {
// Success...
}
14.3.3.2. C++ std::unordered_map<K, V>
to Python dict
¶
This generic function converts a std::unordered_map<K, V>
to a new Python dict
. KeyConvertToPy
, ValConvertToPy
are pointers to functions that takes an instance of a K
or V
type and returns a PyObject*
. These should return a new reference on success, NULL on failure.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | template <typename K, typename V>
PyObject*
std_unordered_map_to_py_dict(const std::unordered_map<K, V> &cpp_map,
PyObject *(*KeyConvertToPy)(const K&),
PyObject *(*ValConvertToPy)(const V&)
) {
PyObject *key = NULL;
PyObject *val = NULL;
PyObject *r = PyDict_New();
if (!r) {
goto except;
}
for (auto &iter: cpp_map) {
key = (*KeyConvertToPy)(iter.first);
if (! key || PyErr_Occurred()) {
goto except;
}
val = (*ValConvertToPy)(iter.second);
if (! val || PyErr_Occurred()) {
goto except;
}
if (PyDict_SetItem(r, key, val)) {
goto except;
}
}
assert(! PyErr_Occurred());
assert(r);
goto finally;
except:
assert(PyErr_Occurred());
// Clean up dict
if (r) {
PyDict_Clear(r);
Py_DECREF(r);
}
r = NULL;
finally:
return r;
}
|
The following will convert a std::unordered_map<std::string, std::string>
to a Python dict {bytes : bytes}
:
1 2 3 4 5 6 7 8 9 10 11 12 13 | std::unordered_map<std::string, std::string> cpp_map {
{"Foo", "Bar"}
};
PyObject *py_dict = std_unordered_map_to_py_dict<std::string, std::string>(
cpp_map,
&std_string_to_py_bytes,
&std_string_to_py_bytes
);
if (! py_dict) {
// Handle failure...
} else {
// All good...
}
|