python script hangs when run from subprocess

[Larry.Martell]

I have a python script and when I run it directly from the command line it runs to completion. But I need to run it from another script. I do that like this:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
rv = p.wait()
out_buf = p.stdout.read()

When I do this, wait never returns. If I trace the underlying script it's always in the same write to stderr that never seems to complete:

write(2, "/KA22/05Feb12/Images/12063LBO003"..., 24457

I run many other scripts and commands in the same manner, and they all complete, it's just this one. Anyone have any ideas why this is happening, and how I can further debug or fix this?

TIA!
-larry

[Peter Otten]

Post by Larry.Martell
I have a python script and when I run it directly from the command line it
runs to completion. But I need to run it from another script. I do that
p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT) rv = p.wait()
out_buf = p.stdout.read()
When I do this, wait never returns. If I trace the underlying script it's
write(2, "/KA22/05Feb12/Images/12063LBO003"..., 24457
I run many other scripts and commands in the same manner, and they all
complete, it's just this one. Anyone have any ideas why this is happening,
and how I can further debug or fix this?

The script writes to an OS buffer, and when that buffer is full it blocks
forever. p.wait() in turn then waits forever for the script to terminate...

As a fix try

out_buf = subprocess.Popen(...).communicate()[0]


This uses threads or select (depending on the OS) to avoid the problem --
and is prominently mentionend in the documentation:

http://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait

[Nobody]

Post by Larry.Martell
I have a python script and when I run it directly from the command line
it runs to completion. But I need to run it from another script. I do
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
rv = p.wait()
out_buf = p.stdout.read()
When I do this, wait never returns.

The last two statements are the wrong way around. If you're reading a
process' output via a pipe, you shouldn't wait() for it until it has
closed its end of the pipe.

As it stands, you have a potential deadlock. If the subprocess tries to
write more data than will fit into the pipe, it will block until the
parent reads from the pipe. But the parent won't read from the pipe until
after the subprocess has terminated, which won't happen because the
subprocess is blocked waiting for the parent to read from the pipe ...

[Larry.Martell]

Post by Nobody

Post by Larry.Martell
I have a python script and when I run it directly from the command line
it runs to completion. But I need to run it from another script. I do
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
rv = p.wait()
out_buf = p.stdout.read()
When I do this, wait never returns.

The last two statements are the wrong way around. If you're reading a
process' output via a pipe, you shouldn't wait() for it until it has
closed its end of the pipe.
As it stands, you have a potential deadlock. If the subprocess tries to
write more data than will fit into the pipe, it will block until the
parent reads from the pipe. But the parent won't read from the pipe until
after the subprocess has terminated, which won't happen because the
subprocess is blocked waiting for the parent to read from the pipe ...

Thanks. I reversed the order of the wait and read calls, and it no longer hangs.

[Larry.Martell]

Post by Peter Otten

Post by Larry.Martell
I have a python script and when I run it directly from the command line it
runs to completion. But I need to run it from another script. I do that
p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT) rv = p.wait()
out_buf = p.stdout.read()
When I do this, wait never returns. If I trace the underlying script it's
write(2, "/KA22/05Feb12/Images/12063LBO003"..., 24457
I run many other scripts and commands in the same manner, and they all
complete, it's just this one. Anyone have any ideas why this is happening,
and how I can further debug or fix this?

The script writes to an OS buffer, and when that buffer is full it blocks
forever. p.wait() in turn then waits forever for the script to terminate...
As a fix try
out_buf = subprocess.Popen(...).communicate()[0]
This uses threads or select (depending on the OS) to avoid the problem --
http://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait

Thanks. I hadn't seen this. I'll check it out.