Multiple processes (forking) in Python

Archive - Originally posted on "The Horse's Mouth" - 2010-03-25 19:16:18 - Graham Ellis
What is forking

If you want a process to head off in two different directions, you can fork it. That results in identical copies of the process being created - the original (parent) and the copy (child), each with their own complete set of variables, and program counter, etc.

The clever thing is that the parent can tell it's the parent because the fork function return the process id of the child to it, and the child can tell its the child because it gets a false return from the fork. Thus you can test the code and have parent and child go off in different logic:

gerald = fork()

if not gerald:

  # child - doesn't know parent's PID

else:

  # parent - gets process id of child

Full Sample source

Why would you fork a process?

If you're writing something like a print driver, then it can be a good idea to fork a process off to print, while you continue with the main parent code.

We've got an example of this technique in use to "ping" to a whole series of remote systems ... forking off a new child each time that an IP address is entered, with potentially lots of children waiting for slow / inaccessible hosts in parallel

Full Sample Source

What is you want to correlate responses from the children?

The "trick" here is to open a pipe before you fork; forking connects the read handle in the child to the write handle in the parent, and vice versa, so that the child can remain in touch with the child.

Full Sample source

Note that that parent process will now have to wait for the child - so you're not getting the same parallel benefits. You can use signals, and methods that check whether data's available in the buffers, to add greater flexibility if you need to do so.

Alternatives

Python threads are a lighter weight way of parallel processing - where execution remains within a single process. I've got an article about that [here] in the solution centre.

You can also start off two completely independent processes - on the same computer or on different computers - and have them talk through a network port - TCP or UDP. That's a subject for a further post! Now made [here]

Main Content

Multiple processes (forking) in Python