pickling instances of metaclass generated classes

Post by lars van gemerden
Hello,
I am using metaclasses to make classes and these classes to make
instances. Now I want to use multiprocessing, which needs to pickle
these instances.
Pickle cannot find the class definitions of the instances. I am trying
to add a line to the __new__ of the metaclass to add the new class
under the right name in the right module/place, so pickle can find
it.
Is this the right approach? Can anyone explain to me where/how to add
these classes for pickle to find and maybe why?

Can you post some code (preferably pared down to a minimal example that fails)?
I'm not really clear on what you are doing. I would expect that a class defined
by a class statement would usually work fine unless if the metaclass is doing
something particularly weird to it.

In any case, you can probably just explicitly register a reduction function for
each type using copy_reg.pickle():

http://docs.python.org/library/copy_reg

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Ian Kelly

2011-12-29 19:55:04 UTC

It sounds like you're trying to do something like this?
... pass
...

instance = MetaClass('<Anonymous>', (object,), {})()
instance

<__main__.<Anonymous> object at 0x00CC00F0>

import pickle
pickle.dumps(instance)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "c:\python27\lib\pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "c:\python27\lib\pickle.py", line 224, in dump
self.save(obj)
File "c:\python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "c:\python27\lib\pickle.py", line 401, in save_reduce
save(args)
File "c:\python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "c:\python27\lib\pickle.py", line 562, in save_tuple
save(element)
File "c:\python27\lib\pickle.py", line 295, in save
self.save_global(obj)
File "c:\python27\lib\pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
it's not found as __main__.<Anonymous>

Yeah, pickle's not going to work with anonymous classes. As you
suggest, you could dynamically add the classes to the module namespace
so that pickle.dumps will find them, but bear in mind that they will
also have to exist when calling pickle.loads, so you will need to be
able to reconstruct the same anonymous classes before unpickling later
on.

Cheers,
Ian

lars van gemerden

2011-12-30 09:50:26 UTC

It sounds like you're trying to do something like this?
... ? ? pass
...>>> instance = MetaClass('<Anonymous>', (object,), {})()

Post by lars van gemerden
instance

<__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle

Post by lars van gemerden
pickle.dumps(instance)

? File "<stdin>", line 1, in <module>
? File "c:\python27\lib\pickle.py", line 1374, in dumps
? ? Pickler(file, protocol).dump(obj)
? File "c:\python27\lib\pickle.py", line 224, in dump
? ? self.save(obj)
? File "c:\python27\lib\pickle.py", line 331, in save
? ? self.save_reduce(obj=obj, *rv)
? File "c:\python27\lib\pickle.py", line 401, in save_reduce
? ? save(args)
? File "c:\python27\lib\pickle.py", line 286, in save
? ? f(self, obj) # Call unbound method with explicit self
? File "c:\python27\lib\pickle.py", line 562, in save_tuple
? ? save(element)
? File "c:\python27\lib\pickle.py", line 295, in save
? ? self.save_global(obj)
? File "c:\python27\lib\pickle.py", line 748, in save_global
? ? (obj, module, name))
it's not found as __main__.<Anonymous>
Yeah, pickle's not going to work with anonymous classes. ?As you
suggest, you could dynamically add the classes to the module namespace
so that pickle.dumps will find them, but bear in mind that they will
also have to exist when calling pickle.loads, so you will need to be
able to reconstruct the same anonymous classes before unpickling later
on.
Cheers,
Ian

Thank you Ian for the minimal example. This is almost the case i was
trying to discribe, however the classes will be named at runtime and
the class definitions will be persisted in a database.

Can you help me with how to add the classes to the correct namespace?
The application spans multiple modules (or compared to the example,
the metaclass definition will be in another module then one where the
class and instance will be generated).

Cheers, Lars

Peter Otten

2011-12-30 11:41:10 UTC

It sounds like you're trying to do something like this?
... pass
...>>> instance = MetaClass('<Anonymous>', (object,), {})()

Post by lars van gemerden
instance

<__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle

Post by lars van gemerden
pickle.dumps(instance)

File "<stdin>", line 1, in <module>
File "c:\python27\lib\pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "c:\python27\lib\pickle.py", line 224, in dump
self.save(obj)
File "c:\python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "c:\python27\lib\pickle.py", line 401, in save_reduce
save(args)
File "c:\python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "c:\python27\lib\pickle.py", line 562, in save_tuple
save(element)
File "c:\python27\lib\pickle.py", line 295, in save
self.save_global(obj)
File "c:\python27\lib\pickle.py", line 748, in save_global
(obj, module, name))
it's not found as __main__.<Anonymous>
Yeah, pickle's not going to work with anonymous classes. As you
suggest, you could dynamically add the classes to the module namespace
so that pickle.dumps will find them, but bear in mind that they will
also have to exist when calling pickle.loads, so you will need to be
able to reconstruct the same anonymous classes before unpickling later
on.
Cheers,
Ian

Thank you Ian for the minimal example. This is almost the case i was
trying to discribe, however the classes will be named at runtime and
the class definitions will be persisted in a database.
Can you help me with how to add the classes to the correct namespace?
The application spans multiple modules (or compared to the example,
the metaclass definition will be in another module then one where the
class and instance will be generated).

If the metaclass is global in whatever module you don't need to bother about
that. The problem is that you cannot access the classes (metaclass
instances) under a dotted name. One workaround is a pseudo-module that does
whatever is needed to recreate the class:

import pickle
import sys

class MetaClass(type):
pass

class M(object):
def __init__(self, module):
self.__module = module
def __getattr__(self, name):
print "creating class", name
class_ = MetaClass(name, (), {"__module__": self.__module})
setattr(self, name, class_)
return class_

sys.modules["m"] = M("m")
import m
c = m.x
s = pickle.dumps(c)
print repr(s)
d = pickle.loads(s)

assert c is d

sys.modules["m"] = M("m")
e = pickle.loads(s)

assert c is not e

The official way is probably what Robert mentioned, via the copy_reg module,
but I didn't get it to work.

Peter Otten

2012-01-01 12:51:05 UTC

import pickle
import sys
pass
self.__module = module
print "creating class", name
class_ = MetaClass(name, (), {"__module__": self.__module})
setattr(self, name, class_)
return class_
sys.modules["m"] = M("m")
import m
c = m.x
s = pickle.dumps(c)
print repr(s)
d = pickle.loads(s)
assert c is d
sys.modules["m"] = M("m")
e = pickle.loads(s)
assert c is not e
The official way is probably what Robert mentioned, via the copy_reg
module, but I didn't get it to work.

I will look further into this. does "sys.modules["m"] = M("m")" create
a new module?

Assigning to sys.modules[modulename] can put arbitrary objects into the

Post by Peter Otten
import sys
sys.modules["x"] = 42
import x
x

Post by Peter Otten
sys.modules["x"] = "spam"
import x
x

'spam'

Cheers, Lars
PS: I get an error when posting this to the usenet group

Sorry, that seems to happen when I post via gmane and don't manually clear
the follow-up that my newsreader helpfully (knode) inserts. I've not yet
found a permanent fix, but if that was the problem you should be able to
answer this post.

lars van gemerden

2011-12-30 11:16:10 UTC

It sounds like you're trying to do something like this?
... ? ? pass
...>>> instance = MetaClass('<Anonymous>', (object,), {})()

Post by lars van gemerden
instance

<__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle

Post by lars van gemerden
pickle.dumps(instance)

Ian also wrote:

'''
Actually, I was wrong, you probably don't need to do that. I suggest
going with Robert Kern's suggestion to either register the class with
the copy_reg module, or (perhaps better since it won't leak
registrations) implement a __reduce__ method on the class. For
... cls = MetaClass.build_class(*metaclass_args)
... self = cls.__new__(cls)
... return self
...
... @classmethod
... def build_class(mcs, arg1, arg2, arg3):
... # Do something useful with the args...
... class _AnonymousClass(object):
... __metaclass__ = mcs
... def __reduce__(self):
... return (reconstructor, ('foo', 'bar', 'baz'),
self.__dict__)
... return _AnonymousClass
...

instance = MetaClass.build_class('foo', 'bar', 'baz')()
instance

<__main__._AnonymousClass object at 0x011DB410>

instance.banana = 42
import pickle
s = pickle.dumps(instance)
s

"c__main__\nreconstructor
\np0\n(S'foo'\np1\nS'bar'\np2\nS'baz'\np3\ntp4\nRp5\n(dp6\nS'banana'\np7\nI42\nsb."

inst2 = pickle.loads(s)
inst2

<__main__._AnonymousClass object at 0x011DBE90>

inst2.banana

inst2.__class__ is instance.__class__

False

Cheers,
Ian

'''

lars van gemerden

2011-12-30 15:56:11 UTC

It sounds like you're trying to do something like this?
... ? ? pass
...>>> instance = MetaClass('<Anonymous>', (object,), {})()

Post by lars van gemerden
instance

<__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle

Post by lars van gemerden
pickle.dumps(instance)

'''
Actually, I was wrong, you probably don't need to do that. ?I suggest
going with Robert Kern's suggestion to either register the class with
the copy_reg module, or (perhaps better since it won't leak
registrations) implement a __reduce__ method on the class. ?For
... ? ? cls = MetaClass.build_class(*metaclass_args)
... ? ? self = cls.__new__(cls)
... ? ? return self
... ? ? ? ? # Do something useful with the args...
... ? ? ? ? ? ? __metaclass__ = mcs
... ? ? ? ? ? ? ? ? return (reconstructor, ('foo', 'bar', 'baz'),
self.__dict__)
... ? ? ? ? return _AnonymousClass
...>>> instance = MetaClass.build_class('foo', 'bar', 'baz')()

Post by lars van gemerden
instance

<__main__._AnonymousClass object at 0x011DB410>>>> instance.banana = 42

Post by lars van gemerden
import pickle
s = pickle.dumps(instance)
s

"c__main__\nreconstructor
\np0\n(S'foo'\np1\nS'bar'\np2\nS'baz'\np3\ntp4\nRp5\n(dp6\nS'banana'\np7\nI 42\nsb.">>> inst2 = pickle.loads(s)

Post by lars van gemerden
inst2

<__main__._AnonymousClass object at 0x011DBE90>>>> inst2.banana
42

Post by lars van gemerden
inst2.__class__ is instance.__class__

False
Cheers,
Ian
'''

Interesting, though I cannot say I completely understand this solution
(looked up __reduce__, but still). I am trying to adapt this example
to a situation where the metaclass generated classes are named at
runtime (not anonymous), but cannot figure it out.

Cheers, Lars

lars van gemerden

2011-12-30 16:51:38 UTC

It sounds like you're trying to do something like this?
... ? ? pass
...>>> instance = MetaClass('<Anonymous>', (object,), {})()

Post by lars van gemerden
instance

<__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle

Post by lars van gemerden
pickle.dumps(instance)

Post by lars van gemerden
instance

<__main__._AnonymousClass object at 0x011DB410>>>> instance.banana = 42

Post by lars van gemerden
import pickle
s = pickle.dumps(instance)
s

"c__main__\nreconstructor
\np0\n(S'foo'\np1\nS'bar'\np2\nS'baz'\np3\ntp4\nRp5\n(dp6\nS'banana'\np7\nI 42\nsb.">>> inst2 = pickle.loads(s)

Post by lars van gemerden
inst2

<__main__._AnonymousClass object at 0x011DBE90>>>> inst2.banana
42

Post by lars van gemerden
inst2.__class__ is instance.__class__

False
Cheers,
Ian
'''

Found a way to name the classes:

def reconstructor(*metaclass_args):
cls = MetaClass2.build_class(*metaclass_args)
self = cls.__new__(cls)
return self

class MetaClass(type):
@classmethod
def build_class(mcs, name, arg1, arg2, arg3):
return mcs(name, (object,), {"__reduce__": lambda e:
(reconstructor2, (name, arg1, arg2, arg3), e.__dict__)})

I still wonder whether it might be easier to add the class to the
namespace. Can anyone help me with that?

Regards, Lars

Ian Kelly

2011-12-30 17:09:35 UTC

Post by lars van gemerden
I still wonder whether it might be easier to add the class to the
namespace. Can anyone help me with that?

from mypackage import mymodule

setattr(mymodule, myclass.__name__, myclass)

lars van gemerden

2012-01-03 11:43:17 UTC

On Dec 29 2011, 10:55?am, lars van gemerden <l... at rational-it.com>

Ok,

After reading all posts (thanks a lot), I am considering to use the
following base metaclass for all metaclasses that must lead to
pickleable instances (not pickleable classes):

import sys

class Meta(type):
def __new__(mcls, name, bases, attrs):
cls = type.__new__(mcls, name, bases, attrs)
setattr(sys.modules[__name__], name, cls)
return cls

if __name__ == '__main__':
instance = Meta("Klass", (str,),{})("apple")
s = pickle.dumps(instance)
delattr(sys.modules[__name__], "Klass")
Meta("Klass", (str,),{})
inst = pickle.loads(s)
print instance
print inst
print type(instance) is type(inst)

Can anyone see any drawbacks to this approach? I've also tested the
case where the Meta metaclass is defined in another module.

Cheers, Lars

Ian Kelly

2012-01-03 18:04:52 UTC