How to achieve polymorphism in Python C API?

Question

I'm writing functools.partial object alternative, that accumulates arguments until their number become sufficient to make a call.

I use C API and I have tp_call implementation which when its called, returns modified version of self or PyObject*.

At first I followed Defining New Types guide and then realized, that I just can't return different types (PyObject * and MyObject*) from tp_call implementation. Then I tried to not use struct with MyObject* definition and use PyObject_SetAttrString in tp_init instead, just like we do that in Python. But in that case I got AttributeError, because you can't set arbitrary attributes on object instances in Python.

What I need here is to make my tp_call implementation polymorphic, and make it able to return either MyObject which is subclass of PyObject, or PyObject type itself.

What is the sane way to do that?

UPDATE #0

That's the code:

static PyObject *Curry_call(Curry *self, PyObject *args, 
                            PyObject *kwargs) {
    PyObject * old_args = self->args;
    self->args = PySequence_Concat(self->args, args);
    Py_DECREF(old_args);
    if (self->kwargs == NULL && kwargs != NULL) {
        self->kwargs = kwargs;
        Py_INCREF(self->kwargs);
    } else if (self->kwargs != NULL && kwargs != NULL) {
        PyDict_Merge(self->kwargs, kwargs, 1);
    }

    if ((PyObject_Size(self->args) +
         (self->kwargs != NULL ? PyObject_Size(self->kwargs) : 0)) >=
        self->num_args) {
        return PyObject_Call(self->fn, self->args, self->kwargs);
    } else {
        return (PyObject *)self;
    }
}

UPDATE #1

Why I initially abandoned this implementation - because I get segfault with it on subsequent calls of partial object. I thought that It happens because of casting Curry * to PyObject* issues. But now I have fixed the segfault by adding Py_INCREF(self); before return (PyObject *)self;. Very strange to me. Should I really INCREF self if I return it by C API ownership rules?

How can `MyObject` be a subclass of `PyObject`? There are no classes in C — Valentin Lorentz, Apr 11 '15 at 08:43
@ValentinLorentz: PyObject and its environment mimick polymorphism by including the "mother" struct into each "daughter" struct. Quite clever, and quite common in C. Read a bit of linux kernel, and you'll find that pattern here and there. — Marcus Müller, Apr 11 '15 at 08:44
@ValentinLorentz In C API there is [the pattern of defining custom types](https://docs.python.org/2/extending/newtypes.html#the-basics), so a type defined in such way becomes a subclass of `PyObject` semantically, but not in C level of course. So I'm searching for the way to operate `MyObject` in a polymorphic way in C. — Gill Bates, Apr 11 '15 at 08:48
To see an example of what @MarcusMüller suggested, take a look at this, focusing on how a struct instance is embedded in another struct and the usage of `container_of` macro: http://lxr.free-electrons.com/source/Documentation/rbtree.txt?v=3.18 — holgac, Apr 11 '15 at 08:50
@holgac: would you mind proofreading my [answer](http://stackoverflow.com/a/29575910/4433386)? — Marcus Müller, Apr 11 '15 at 09:00

score 2 · Answer 1 · answered Apr 11 '15 at 08:48

2

If you've defined your MyObject type correctly, you should be able to simply cast your MyObject * to a PyObject * and return that. The first member of a MyObject is a PyObject, and C lets you cast a pointer to a struct to a pointer to the struct's first member and vice versa. I believe the feature exists specifically to allow things like this.

answered Apr 11 '15 at 08:48

user2357112

260,549
28
431
505

I like your answer even more than I like mine :) so [have an upvote](http://www.quickmeme.com/img/ca/ca211d0239f0cfe1e5a4cf3194a73db792e1eafe821967f4f73eeb4a1da4ee08.jpg) – Marcus Müller Apr 11 '15 at 08:49
People say, that [such casting can cause undefined behavior](http://stackoverflow.com/a/3995987/1818608) – Gill Bates Apr 11 '15 at 08:52
@GillBates: They say that in situations where this specific exception does not apply. In the question you link, neither CMAcceleration nor Vector3d has an object of the other type as its first member. – user2357112 Apr 11 '15 at 08:54
@GillBates: But that's how scripting language interpreters in C work :D – Marcus Müller Apr 11 '15 at 08:54
Note, that `tp_call` should be able to receive that casted object as a first argument. – Gill Bates Apr 11 '15 at 08:54
@GillBates: And it can. `tp_call` takes a `PyObject *` as its first argument, and if it was called correctly, you can cast that pointer to a `MyObject *` and proceed to interact with it as you would with any other `MyObject *`. – user2357112 Apr 11 '15 at 09:00
@user2357112 The problem is that I get segfault on subsequent call of my partial object. During first call, execution goes to my `tp_call`, then it accumulates arguments and returns `PyObject*`, but in second call, execution goes to `PyObject`'s `tp_call` and that causes segfault. – Gill Bates Apr 11 '15 at 09:12
@GillBates: You most likely have an entirely unrelated bug. – user2357112 Apr 11 '15 at 09:13
@user2357112 Yeah, you right, and my previous statement is completely wrong. Segfault disappears when I add `Py_INCREF(self);` before `return (PyObject *)self;`. Very strange to me. Should I really INCREF `self` if I return it by C API ownership rules? – Gill Bates Apr 11 '15 at 10:32
@GillBates: You should. – user2357112 Apr 11 '15 at 15:41

Marcus Müller · Answer 2 · 2015-04-11T08:56:33.720

1

I don't really know your whole code, but as long as MyObject is a PyObject (compatible, i.e. has the same "header" fields, make sure you have a length field), CPython is designed to just take your MyObject as a PyObject; simply cast the pointer to PyObject before returning it.

As you can see here, that is one of the things that is convenient when using C++: You can actually have subclasses with type safety, and you don't have to worry about someone just copying over half of your subclass' instance, for example.

EDIT: because it was asked "isn't this unsafe": yes. It is. But its only as unsafe as type handling in user code gets; CPython lets you do this, because it stores and checks the PyTypeObject *ob_type member of the PyObject struct contained. That's about as safe as for example C++'s runtime type checking is -- but it's implemented by python developers as opposed to GCC/clang/MSVC/icc/... developers.

edited Apr 11 '15 at 08:56

answered Apr 11 '15 at 08:47

Marcus Müller

34,677
4
53
94

Wouldn't cast from `MyObject*` to `PyObject*` and than again to `MyObject*` cause undefined behavior? – Gill Bates Apr 11 '15 at 08:49
not really, since a struct in C is really just the concatenation of its fields in memory, so it's safe to cast a pointer to a struct to a pointer to its first element; since that needs to be a unique thing, the opposite is true, too. The "type" safety gets emulated by using `PyTypeObject *ob_type`. – Marcus Müller Apr 11 '15 at 08:53
*since a struct in C is really just the concatenation of its fields in memory* but if you cast bigger struct to smaller, it lets compiler think that, freed memory from bigger struct is available and it can start using it for something else, doesn't it? – Gill Bates Apr 11 '15 at 08:57
no. the compiler doesn't take part in the freeing at all. When you free something, your runtime/OS looks into a table of allocated memory and says "ah right, there's an entry that someone allocated X bytes at that address, let's remove that entry". There is no runtime type safety in C. – Marcus Müller Apr 11 '15 at 08:59
I don't know about python-c api, but as you said, if both structs are compatible, casting should work fine (if python api wants to work with PyObject). But embedding `PyObject` in `MyObject` might be a choice too. Have a PyObject field in MyObject, and send it to python api. then when you receive a PyObject from python api, if you're sure it's MyObject, convert it to MyObject using a `container_of` macro. This way, the PyObject field in MyObject can be anywhere, not necessarily at the top. – holgac Apr 11 '15 at 09:09

How to achieve polymorphism in Python C API?

UPDATE #0

UPDATE #1

2 Answers2