Python dictionary "plus-equal" behavior

Question

I'm trying to understand the exact mechanism behind updating a python dictionary using d[key] += diff. I have some helper classes to trace magic method invocations:

class sdict(dict):
    def __setitem__(self, *args, **kargs):
        print "sdict.__setitem__"
        return super(sdict, self).__setitem__(*args, **kargs)
    def __delitem__(self, *args, **kargs):
        print "sdict.__delitem__"
        return super(sdict, self).__delitem__(*args, **kargs)
    def __getitem__(self, *args, **kargs):
        print "sdict.__getitem__"
        return super(sdict, self).__getitem__(*args, **kargs)
    def __iadd__(self, *args, **kargs):
        print "sdict.__iadd__"
        return super(sdict, self).__iadd__(*args, **kargs)
    def __add__(self, *args, **kargs):
        print "sdict.__add__"
        return super(sdict, self).__add__(*args, **kargs)

class mutable(object):
    def __init__(self, val=0):
        self.value = val
    def __iadd__(self, val):
        print "mutable.__iadd__"
        self.value = self.value + val
        return self
    def __add__(self, val):
        print "mutable.__add__"
        return mutable(self.value + val)

With these tools, let's go diving:

>>> d = sdict()
>>> d["a"] = 0
sdict.__setitem__
>>> d["a"] += 1
sdict.__getitem__
sdict.__setitem__
>>> d["a"]
sdict.__getitem__
1

We don't see any __iadd__ operation invoked here, which makes sense because the left-hand side expression d["a"] returns an integer that does not implement the __iadd__ method. We do see python magically converting the += operator into __getitem__ and __setitem__ calls.

Continuing:

>>> d["m"] = mutable()
sdict.__setitem__
>>> d["m"] += 1
sdict.__getitem__
mutable.__iadd__
sdict.__setitem__
>>> d["m"]
sdict.__getitem__
<__main__.mutable object at 0x106c4b710>

Here the += operator successfully invokes an __iadd__ method. It looks like the += operator is actually being used twice:

• Once for the magic translation to __getitem__ and __setitem__ calls
• A second time for the __iadd__ call.

Where I need help is the following:

• What is the exact, technical mechanism for translating the += operator into __getitem__ and __setitem__ calls?
• In the second example, why is the += operator used twice? Doesn't python translate the statement to d["m"] = d["m"] + 1 (In which case wouldn't we see __add__ be invoked instead of __iadd__?)

Answer 1

In the first example, you didn't apply the += operator to the dictionary. You applied it to the value stored in the d['a'] key, and that's a different object altogether.

In other words, Python will retrieve d['m'] (a __getitem__ call), apply the += operator to that, then set the result of that expression back to d['m'] (the __setitem__ call).

The __iadd__ method either returns self mutated in-place, or a new object, but Python cannot know for sure what the method returned. So it has to call d.__setitem__('m', <return_value_from_d['m'].__iadd__(1)>), always.

The exact same thing happens if you did:

m = d['m']
m += 1
d['m'] = m

but without the extra name m in the global namespace.

If the mutable() instance was not stored in a dictionary but in the global namespace instead, the exact same sequence of events takes place, but directly on the globals() dictionary, and you wouldn't get to see the __getitem__ and __setitem__ calls.

This is documented under the augmented assignment reference documentation:

An augmented assignment evaluates the target (which, unlike normal assignment statements, cannot be an unpacking) and the expression list, performs the binary operation specific to the type of assignment on the two operands, and assigns the result to the original target.

where d['m'] is the target; evaluating the target here involves __getitem__, assigning the result back to the original target invokes __setitem__.

Answer 2

Because, as specified in docs, __iadd__ might perform the operation in-place but the result will be either self or a new object, thus __setitem__ is called.