multiprocessing.Queue

The multiprocessing.Queues module offers a Queue implementation to be used as a message passing mechanism between multiple related processes. The Python example demonstrates the Queue with one parent process, two writer-child processes and one reader-child process.

Queue Updated 7 Feb 2019 1 min read

Overview

  • The Queue class in Multiprocessing module of Python Standard Library provides a mechanism to pass data between a parent process and the descendent processes of it.
  • Multiprocessing.Queues.Queue uses pipes to send data between related * processes.
python
A Pipe is a message passing mechanism between processes in Unix-like operating systems.
  • For any read/write (get/put method calls) operation the Multiprocessing.Queues.Queue class deserialises/serializes data using Python's pickle module and sends through the underlying pipe.
  • Python's Multiprocessing.Queues.Queue also uses semaphores and locks to take care of the scenarios involving multiple reader/writer processes.

Example

python
import multiprocessing

import time

# Producer/Writer

def procFunction0(messageQueue):

    for i in range(1,10):

        messageQueue.put("Child1:Message%d"%i)

        time.sleep(1)

# Consumer/Reader

def procFunction1(messageQueue):

    while messageQueue.empty() is False:

        print("From reader:%s"%messageQueue.get())

        time.sleep(1)

# Producer/Writer

def procFunction2(messageQueue):

    for i in range(1,10):

        messageQueue.put("Child3:Message%d"%i)

        time.sleep(1)

if __name__ == "__main__":

    multiprocessing.set_start_method("fork")

    messageQueue  = multiprocessing.Queue()

    # Create child processes

    childProcess0 = multiprocessing.Process(target=procFunction0, args=(messageQueue,))

    childProcess1 = multiprocessing.Process(target=procFunction1, args=(messageQueue,))

    childProcess2 = multiprocessing.Process(target=procFunction2, args=(messageQueue,))

    # Start all the child processes - Writer, Reader, Writer

    childProcess0.start()

    childProcess1.start()

    childProcess2.start()

    # Wait for child processes to finish

    childProcess0.join()

    childProcess1.join()

    childProcess2.join()

Output

Output
From reader:Child1:Message1

From reader:Child3:Message1

From reader:Child1:Message2

From reader:Child3:Message2

From reader:Child1:Message3

From reader:Child3:Message3

From reader:Child1:Message4

From reader:Child3:Message4

From reader:Child3:Message5

From reader:Child1:Message5

From reader:Child1:Message6

From reader:Child3:Message6

From reader:Child1:Message7

From reader:Child3:Message7

From reader:Child1:Message8

From reader:Child3:Message8

From reader:Child3:Message9

From reader:Child1:Message9