subprocess (spawned by os.system) inherits open TCP/UDP/IP port

[alf]

You can avoid this, if you like. Set FD_CLOEXEC on the socket after you
old = fcntl.fcntl(s.fileno(), fcntl.F_GETFD)
fcntl.fcntl(s.fileno(), fcntl.F_SETFD, old | fcntl.FD_CLOEXEC)

thx for responding (I was about to send the question to twisted mailing
list too ;-).


still would like to find out why it is happening (now FD_CLOEXEC
narrowed may yahooing/googling searches). While realize that file
descriptors are shared by forked processes it is still weird why the
port moves to the child process once parent gets killed. what it the
parent got multiple subprocesses.

Plus it is kind of unintuitive os.system does not protect from such
behavoir which is for me more an equivalent of like issuing a ne
wcommand/ starting a process from the shell.

Thx,

[alf]

Hi,

I need a help with explaining following behavior. Although it is not
python issue per say, python helped me to write sample programs and
originally I encountered the issue using python software. So let's
assume we have two following programs:



[myhost] ~> cat ss.py
import socket
UDPSock=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
UDPSock.bind(("",12345))
import os
os.system('python cc.py')

[myhost] ~> cat cc.py
import time
time.sleep(2036)



then I start master one, do ctrl-Z and bg.

[myhost] ~> python ss.py

Suspended
[myhost] ~> bg
[1] python ss.py &
[myhost] ~> ps
UID PID PPID C STIME TTY TIME CMD
myuser00 3192 3189 0 14:57 pts/0 00:00:00 -tcsh
myuser00 3247 3192 0 14:57 pts/0 00:00:00 python ss.py
myuser00 3248 3247 0 14:57 pts/0 00:00:00 python cc.py



[myhost] ~> netstat -uanp |grep 12345
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
udp 0 0 0.0.0.0:12345 0.0.0.0:*
3247/python



As expected netstat indicates process 3247 having port allocated. Then I
kill the master process and interestingly the child inherits the port open.


[myhost] ~> kill 3247
[myhost] ~> netstat -uanp | grep 12345
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
udp 0 0 0.0.0.0:12345 0.0.0.0:*
3248/python
[1] + Terminated python ss.py



Why it is happening? I know that os.system uses fork, but still this is
something unexpected.


Thx, Alf

[Jeff McNeil]

What's unexpected about it? Child processes inherit all of the open file
descriptors of their parent. A socket is simply another open file
descriptor. When your parent process exits, your child still holds a valid,
open file descriptor.

import sys
import socket
import os
import time

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, 0)
s.bind(('', 8888))

pid = os.fork()
if pid > 0:
time.sleep(100)
elif pid == 0:
time.sleep(100)

$ lsof | grep python | grep jeff | grep 8888
python 13048 jeff 3u IPv4 212689 UDP
*:8888
python 13049 jeff 3u IPv4 212689 UDP
*:8888

-Jeff

Post by alf
Hi,
I need a help with explaining following behavior. Although it is not
python issue per say, python helped me to write sample programs and
originally I encountered the issue using python software. So let's
[myhost] ~> cat ss.py
import socket
UDPSock=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
UDPSock.bind(("",12345))
import os
os.system('python cc.py')
[myhost] ~> cat cc.py
import time
time.sleep(2036)
then I start master one, do ctrl-Z and bg.
[myhost] ~> python ss.py
Suspended
[myhost] ~> bg
[1] python ss.py &
[myhost] ~> ps
UID PID PPID C STIME TTY TIME CMD
myuser00 3192 3189 0 14:57 pts/0 00:00:00 -tcsh
myuser00 3247 3192 0 14:57 pts/0 00:00:00 python ss.py
myuser00 3248 3247 0 14:57 pts/0 00:00:00 python cc.py
[myhost] ~> netstat -uanp |grep 12345
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
udp 0 0 0.0.0.0:12345 0.0.0.0:*
3247/python
As expected netstat indicates process 3247 having port allocated. Then I
kill the master process and interestingly the child inherits the port open.
[myhost] ~> kill 3247
[myhost] ~> netstat -uanp | grep 12345
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
udp 0 0 0.0.0.0:12345 0.0.0.0:*
3248/python
[1] + Terminated python ss.py
Why it is happening? I know that os.system uses fork, but still this is
something unexpected.
Thx, Alf
--
http://mail.python.org/mailman/listinfo/python-list

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20070719/2c8aecfb/attachment.html>

[Jeff McNeil]

The open file descriptor/socket shouldn't "move" between processes when you
kill the parent. When os.system forks, anything you've got open in the
parent will transfer to the child. Both descriptors reference the same open
port. Running the same 'ss' and 'cc' code you've supplied behaves like that
on my workstation:

[jeff at marvin ~]$ cat ss.py
import socket
UDPSock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
UDPSock.bind(('', 12345))

import os
os.system("python cc.py")

[jeff at marvin ~]$ cat cc.py
import time
time.sleep(2036)

Corresponding lsof output when the above code runs shows the following
before killing the parent.

[jeff at marvin ~]$ /usr/sbin/lsof | grep jeff | grep UDP
python 18713 jeff 3u IPv4 34004 UDP *:italk

python 18714 jeff 3u IPv4 34004 UDP *:italk



I don't believe FD_CLOEXEC will not stop the child from inheriting your open
file descriptors. Rather, it will ensure they're closed when execl(3) and
friends are called.

-Jeff

Post by alf

You can avoid this, if you like. Set FD_CLOEXEC on the socket after you
old = fcntl.fcntl(s.fileno(), fcntl.F_GETFD)
fcntl.fcntl(s.fileno(), fcntl.F_SETFD, old | fcntl.FD_CLOEXEC)

thx for responding (I was about to send the question to twisted mailing
list too ;-).
still would like to find out why it is happening (now FD_CLOEXEC
narrowed may yahooing/googling searches). While realize that file
descriptors are shared by forked processes it is still weird why the
port moves to the child process once parent gets killed. what it the
parent got multiple subprocesses.
Plus it is kind of unintuitive os.system does not protect from such
behavoir which is for me more an equivalent of like issuing a ne
wcommand/ starting a process from the shell.
Thx,
--
alf
--
http://mail.python.org/mailman/listinfo/python-list

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20070720/c728d040/attachment.html>

[Jean-Paul Calderone]

Post by alf
Hi,
I need a help with explaining following behavior. Although it is not
python issue per say, python helped me to write sample programs and
originally I encountered the issue using python software. So let's
[snip - file descriptors inherited by child process]

You can avoid this, if you like. Set FD_CLOEXEC on the socket after you
open it, before you call os.system:

old = fcntl.fcntl(s.fileno(), fcntl.F_GETFD)
fcntl.fcntl(s.fileno(), fcntl.F_SETFD, old | fcntl.FD_CLOEXEC)

Jean-Paul

[Hrvoje Niksic]

Post by alf
still would like to find out why it is happening (now FD_CLOEXEC
narrowed may yahooing/googling searches). While realize that file
descriptors are shared by forked processes it is still weird why the
port moves to the child process once parent gets killed. what it the
parent got multiple subprocesses.

Netstat probably shows only one of the processes that hold to the
port, possibly the one with the lowest PID (the parent).

Post by alf
Plus it is kind of unintuitive os.system does not protect from such
behavoir which is for me more an equivalent of like issuing a ne
wcommand/ starting a process from the shell.

It is considered a feature that fork/exec'ed programs inherit file
descriptors -- that's how stdin and stdout get inherited all the time.
It doesn't occur often with network connections because shells rarely
have reason to open them.

[Steve Holden]

Post by Hrvoje Niksic

Post by alf
still would like to find out why it is happening (now FD_CLOEXEC
narrowed may yahooing/googling searches). While realize that file
descriptors are shared by forked processes it is still weird why the
port moves to the child process once parent gets killed. what it the
parent got multiple subprocesses.

Netstat probably shows only one of the processes that hold to the
port, possibly the one with the lowest PID (the parent).

Post by alf
Plus it is kind of unintuitive os.system does not protect from such
behavoir which is for me more an equivalent of like issuing a ne
wcommand/ starting a process from the shell.

It is considered a feature that fork/exec'ed programs inherit file
descriptors -- that's how stdin and stdout get inherited all the time.
It doesn't occur often with network connections because shells rarely
have reason to open them.

It would be hard to figure out how (x)inetd or TCP wrappers could work
without the open connection being passed to the spawned server process.

regards
Steve

[Jean-Paul Calderone]

Post by alf

You can avoid this, if you like. Set FD_CLOEXEC on the socket after you
old = fcntl.fcntl(s.fileno(), fcntl.F_GETFD)
fcntl.fcntl(s.fileno(), fcntl.F_SETFD, old | fcntl.FD_CLOEXEC)

thx for responding (I was about to send the question to twisted mailing
list too ;-).
still would like to find out why it is happening (now FD_CLOEXEC
narrowed may yahooing/googling searches). While realize that file
descriptors are shared by forked processes it is still weird why the
port moves to the child process once parent gets killed. what it the
parent got multiple subprocesses.

It doesn't actually move. The file descriptor is a handle onto the port.
A port can have multiple handles which refer to it. When you fork, you
effectively copy the handle into another process. Now there are two
handles onto the same port. If one of the processes exits, the other one
still has a handle, and so the port still exists. If a process forks
multiple times, then multiple copies are made. Each process can accept
connections on the port through its own handle. Exiting just drops the
handle, it doesn't close the port.

Post by alf
Plus it is kind of unintuitive os.system does not protect from such
behavoir which is for me more an equivalent of like issuing a ne
wcommand/ starting a process from the shell.

Possibly, but intuitive and software often don't go hand in hand. :)

Jean-Paul

[Gabriel Genellina]

Is it possible to cause this sort of thing to happen on Windows.
Specifically, I'm looking for a way to cause multiple processes to
accept new connections on a bound socket. on UNIX, I can just fork()
after binding the server socket to a port and the children can
accept() on that same socket, but on Windows, I don't know how to make
that work. Any ideas? Thanks!

Sockets are inherited by default, at least on any more-or-less-recent
Windows version (that is, not on Win9x). There is a specific Winsock
function (WSADuplicateSocket) and you can use the generic
DuplicateHandle too with bInheritHandle=TRUE.
Read about this on Microsoft site <http://msdn2.microsoft.com/en-us/
library/ms683463.aspx>

Note: Better to post a NEW message instead of replying to this old
thread.

--
Gabriel Genellina

[Seun Osewa]

Is it possible to cause this sort of thing to happen on Windows.
Specifically, I'm looking for a way to cause multiple processes to
accept new connections on a bound socket. on UNIX, I can just fork()
after binding the server socket to a port and the children can
accept() on that same socket, but on Windows, I don't know how to make
that work. Any ideas? Thanks!

It doesn't actually move. Thefiledescriptor is a handle onto the port.
A port can have multiple handles which refer to it. When you fork, you
effectively copy the handle into another process. Now there are two
handles onto the same port. If one of the processes exits, the other one
still has a handle, and so the port still exists. If a process forks
multiple times, then multiple copies are made. Each process can accept
connections on the port through its own handle. Exiting just drops the
handle, it doesn't close the port.