multiprocessing Pool workers cannot create subprocesses

[Jason Grout]

In a recent application, a student of mine tried to create child
processes inside of a multiprocessing Pool worker (for security and
convenience reasons, we wanted to run some code inside of a child
process). Here is some test code for python 2.7:

=============================
import multiprocessing as mp

def run_computation(x):
print 2*x

def f(x):
curr_proc=mp.current_process()
# uncomment following line to get this to work
#curr_proc.daemon=False

p = mp.Process(target=run_computation, args=(x,))
p.start()
p.join()


pool = mp.Pool(processes=4)
pool.map(f, range(10))

===============================

The result is:

Traceback (most recent call last):
File "daemon.py", line 17, in <module>
pool.map(f, range(10))
File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py",
line 199, in map
return self.map_async(func, iterable, chunksize).get()
File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py",
line 491, in get
raise self._value
AssertionError: daemonic processes are not allowed to have children

The problem appears to be that multiprocessing sets its workers to have
the daemon flag set to True, which prevents workers from creating child
processes. If I uncomment the line indicated in the code, I can create
child processes and the program works (it prints out the even numbers
from 0 to 18).

It makes me nervous to just change the daemon status of the process like
that, especially when I don't know the reason the workers have
daemon=True to begin with. What is the reasoning behind that decision?
What issues do we need to worry about if we just set the daemon mode
flag like in the above code?

Thanks,

Jason

[Ned Deily]

In article <4D838D28.5090705 at creativetrax.com>,

Post by Jason Grout
The problem appears to be that multiprocessing sets its workers to have
the daemon flag set to True, which prevents workers from creating child
processes. If I uncomment the line indicated in the code, I can create
child processes and the program works (it prints out the even numbers
from 0 to 18).
It makes me nervous to just change the daemon status of the process like
that, especially when I don't know the reason the workers have
daemon=True to begin with. What is the reasoning behind that decision?
What issues do we need to worry about if we just set the daemon mode
flag like in the above code?

http://docs.python.org/library/multiprocessing.html#multiprocessing.Proce
ss.daemon

"When a process exits, it attempts to terminate all of its daemonic
child processes.

Note that a daemonic process is not allowed to create child processes.
Otherwise a daemonic process would leave its children orphaned if it
gets terminated when its parent process exits. Additionally, these are
not Unix daemons or services, they are normal processes that will be
terminated (and not joined) if non-daemonic processes have exited."

[Jason Grout]

In article<4D838D28.5090705 at creativetrax.com>,

Post by Jason Grout
The problem appears to be that multiprocessing sets its workers to have
the daemon flag set to True, which prevents workers from creating child
processes. If I uncomment the line indicated in the code, I can create
child processes and the program works (it prints out the even numbers
from 0 to 18).
It makes me nervous to just change the daemon status of the process like
that, especially when I don't know the reason the workers have
daemon=True to begin with. What is the reasoning behind that decision?
What issues do we need to worry about if we just set the daemon mode
flag like in the above code?

http://docs.python.org/library/multiprocessing.html#multiprocessing.Proce
ss.daemon
"When a process exits, it attempts to terminate all of its daemonic
child processes.
Note that a daemonic process is not allowed to create child processes.
Otherwise a daemonic process would leave its children orphaned if it
gets terminated when its parent process exits. Additionally, these are
not Unix daemons or services, they are normal processes that will be
terminated (and not joined) if non-daemonic processes have exited."

Right; thanks. Let me rephrase my questions:

1. Why is important that the multiprocessing Pool worker processors have
daemon=True (I think this is the same as asking: why is it important
that they be terminated with terminate() rather than join() )?

2. Is it safe for us to reset a Pool worker process to have daemon=False
before starting a subprocess from that worker, like in the code from the
original message?

Thanks,

Jason

[John L. Stephens]

This post might be inappropriate. Click to display it.

[Jason Grout]

Post by John L. Stephens

Post by Jason Grout
1. Why is important that the multiprocessing Pool worker processors
have daemon=True (I think this is the same as asking: why is it
important that they be terminated with terminate() rather than join() )?
2. Is it safe for us to reset a Pool worker process to have
daemon=False before starting a subprocess from that worker, like in
the code from the original message?
Thanks,
Jason

Jason,
I just happen to be dealing with a project that uses multiprocessing.
What I have learned is this...
If a child thread (pool worker) is not set to daemon (daemon=False), if
for some reason the parent thread terminates either normally or
abnormally and the worker thread has not completed its task, the child
thread will terminate by throwing all sorts of nasty errors. However, in
daemon mode, multiprocessing will terminate the child thread 'cleanly'.
That's not to say that the worker has a chance to complete its work or
shut itself down. Multiprocessing will absorb the exceptions and not
pass them along.
You may terminate a child thread using join(). That is probably the
safest way to do it. terminate() will just kill the worker thread
immediately without any regard to whether or not it has completed its
tasks. I believe multiprocessing uses terminate() as well to kill a
daemon thread if the parent thread disappears.
join() will, however, block until the task has competed and returned. If
you want to continue doing work in the parent thread while the child
thread is off doing its thing, then another means of syncing up the
parent and children threads is needed.
As for allowing children threads to spawn off children of its own using
subprocess runs the risk of creating a little army of zombie
'grandchildren' if either the parent or child threads terminate before
the subprocess completes and returns.
Hope that helps....

Thanks. That helps tremendously!

Jason