Python weakmethods

Python is my favorite language. It is so simple to code and readable that doing just about anything is quicker and easier than any other language I’ve used. At the same time it is so advanced that I am always discovering new features/techniques/oddities.

One of the oddities I recently ran into is the inability of python’s weakref module to create weak references to bound methods.

Problem
Any application where you pass methods as callbacks could run into this situation. Two recent occurences involve GUI programming and network programming (with twisted). In both cases there is an event system which you need to register callbacks that are called to notify your class. They usually look something like:

    eo = EventObject()

    class MyClass(object):
        def __init__(self):
           eo.register(self.callback)
           
        def callback(self):
            ...

This works great, and it’s usually what you’ll see in just about every tutorial on pygtk or twisted (using their own api of course). What if your program is long running and object that generates your events is expected to outlive your class instance? Your program will ‘leak memory’ because, unless you can explicitly unregister your callback, the event object will hold a reference to your bound method which keeps your instance alive.

A real-world application where this is a problem is Exaile, which uses a global event manager that lives as long as the program runs.

WeakRef
One (elegant) solution is provided by Python’s weakref module:

This module lets you pass weak references to other classes, so when you dispose of your object, it is not kept alive (good examples on that site). The problem with weakref, though, is that it doesn’t act as you might expect on bound methods.

Since bound methods are ‘first class objects’, unless you store a separate reference to the method (which requires extra bookkeeping), the weakref created from a bound method is dead on arrival. The following example illistrates this:

    import weakref
    class A(object):
        def fun(self):
            return 'fun!'
            
    def notify_dead(ref):
        '''called by weakref when the referent dies'''
        print '{0} now dead'.format(repr(ref))

    print 'start'
    a = A()
    r = weakref.ref(a.fun, notify_dead) # dead on arrival
    print a.fun()   # fun!
    print r()       # None
    print r()()     # Exception

WeakMethod
I’ve seen a few ways to work around this, but the cleanest I’ve seen is what’s done in Exaile. The general idea is to create a class that holds a weakref to the instance object and generates the bound method when called if the referent is still alive. The following is a simplified version (no exception handling):

import types
import weakref

class WeakMethod(object):
    def __init__(self, meth, notify=None):
        if meth.im_self is not None:
            raise ValueError ('unbound method')
        self.obj = weakref.ref(meth.im_self, notify)
        self.func = meth.im_func
        self.cls = meth.im_class
        
    def __call__(self):
        obj = self.obj()
        if obj is None:
            return None
        else:
            return types.MethodType(self.func, self.obj, self.cls)

The previous example then becomes:

    class A(object):
        def fun(self):
            return 'fun!'
            
    def notify_dead(ref):
        '''called by weakref when the referent dies'''
        print '{0} now dead'.format(repr(ref))

    print 'start'
    a = A()
    r = WeakMethod(a.fun, notify_dead)
    print a.fun()   # fun!
    print r()       # 
    print r()()     # fun!
    del a           # dies
    print r()       # None

A similar WeakMethodProxy class can be made to behave like a proxy object, if you need to pass something that acts like a method.
# notes on lambdas,closures

Note:
These classes are only valid for methods. Unbound methods, functions, lambdas, and closures/nested functions will not work. You creat weakrefs to these objects as you would any object with weakref.ref, but since these objects are usually created and used in-place (not stored), the weakrefs will be invalid beyond the scope that they are defined in.
Because of this, it is not very useful to create weakrefs of lambdas and closures, more on this in the next post.

One final note:
atexit
This is a convenient function for making sure resources are cleaned up on program termination. The problem is that there is no way to ‘unregister’ a function, and if you pass an instance method the instance is kept alive for the length of the program. You can pass a weakref proxy, but this will cause an exception when atexit tries to execute an invalid weakref proxy. Most solutions I’ve seen create a global ‘cleanup’ function or class that they then wrap their cleanup functions in, supplementing atexit with their own register/unregister or exception handling.

Advertisements

Tags:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: