Multiprocessing in Python - Running Multiple Processes in Parallel

Why Use Multiprocessing In Python?

Performing multiple operations for a single processor becomes challenging. As the number of processes keeps increasing, the processor will have to halt the current process and move to the next, to keep them going. Thus, it will have to interrupt each task, thereby hampering the performance.

You can think of it as an employee in an organization tasked to perform jobs in multiple departments. If the employee has to manage the sales, accounts, and even the backend, he will have to stop sales when he is into accounts and vice versa.

Suppose there are different employees, each to perform a specific task. It becomes simpler, right? That’s why multiprocessing in Python becomes essential. The smaller task threads act like different employees, making it easier to handle and manage various processes. A multiprocessing system can be represented as:

• A system with more than a single central processor
• A multi-core processor, i.e., a single computing unit with multiple independent core processing units

In multiprocessing, the system can divide and assign tasks to different processors.

Want a Top Software Development Job? Start Here!

Full Stack Developer - MERN StackExplore Program

What Is the Multiprocessing Module?

The Python multiprocessing module provides multiple classes that allow us to build parallel programs to implement multiprocessing in Python. It offers an easy-to-use API for dividing processes between many processors, thereby fully leveraging multiprocessing. It overcomes the limitations of Global Interpreter Lock (GIL) by using sub-processes instead of threads. The primary classes of the Python multiprocessing module are:

• Process
• Queue
• Lock

Let’s use an example to better understand the use of the multiprocessing module in Python.

Example - Using the Process Class to Implement Multiprocessing in Python

# importing Python multiprocessing module

import multiprocessing

def prnt_cu(n):

print("Cube: {}".format(n * n * n))

def prnt_squ(n):

print("Square: {}".format(n * n))

if __name__ == "__main__":

# creating multiple processes

proc1 = multiprocessing.Process(target=prnt_squ, args=(5, ))

proc2 = multiprocessing.Process(target=prnt_cu, args=(5, ))

# Initiating process 1

proc1.start()

# Initiating process 2

proc2.start()

# Waiting until proc1 finishes

proc1.join()

# Waiting until proc2 finishes

proc2.join()

# Processes finished

print("Both Processes Completed!")

Output

Now, it’s time to understand the above code and see how the multiprocessing module and process class help build parallel programs.

• You first used the “import multiprocessing” command to import the module.
• Next, you created the Process class objects: proc1 and proc2. The arguments passed in these objects were:
• target: The functions to be used
• args: Arguments to be given in the functions
• After the object construction, you must use the start() method to start the processes.
• Lastly, you used the join() method to stop the current program’s execution until it executes the processes. Thus, the program will first run proc1 and proc2. It will then move back to the following statements of the running program.

What Are Pipes Used For In Multiprocessing In Python?

While using multiprocessing in Python, Pipes acts as the communication channel. Pipes are helpful when you want to initiate communication between multiple processes. They return two connection objects, one for each end of the Pipe, and use the send() & recv() methods to communicate. Let’s look at an example for a clear understanding. In the below code, you will use a Pipe to send some info from the child to the parent connection.

import multiprocessing

from multiprocessing import Process, Pipe

def exm_function(c):

    c.send(['Hi! This is child info'])

    c.close()

if __name__ == '__main__':

    par_c, chi_c = Pipe()

    mp1 = multiprocessing.Process(target=exm_function, args=(chi_c,))

    mp1.start()

print (par_c.recv() )

mp1.join()

Output

Want a Top Software Development Job? Start Here!

Full Stack Developer - MERN StackExplore Program

What Are the Queues Used For In Multiprocessing In Python?

The Queue in Python is a data structure based on the FIFO (First-In-First-Out) concept. Like the Pipe, even a queue helps in communication between different processes in multiprocessing in Python. It provides the put() and get() methods to add and receive data from the queue. Here’s an example to show the use of queue for multiprocessing in Python. This code will create a function to check if a number is even or odd and insert it in the queue. You will then initiate the process and print the numbers.

import multiprocessing

def even_no(num, n):

    for i in num:

        if i % 2 == 0:

            n.put(i)

if __name__ == "__main__":

    n = multiprocessing.Queue()

    p = multiprocessing.Process(target=even_no, args=(range(10), n))

    p.start()

    p.join()

while n:

    print(n.get())

Output

What Are The Locks Used For In Multiprocessing In Python?

The lock is used for locking the processes while using multiprocessing in Python. With its acquire() and release() methods, you can lock and resume processes. Thus, it allows you to execute specific tasks based on priority while stopping the other processes. The below code uses the lock mechanism on an ATM-like system.

import multiprocessing

# Withdrawal function

def wthdrw(bal, lock):

for _ in range(10000):

lock.acquire()

bal.value = bal.value - 1

lock.release()

# Deposit function

def dpst(bal, lock):

for _ in range(10000):

lock.acquire()

bal.value = bal.value + 1

lock.release()

def transact():

# initial balance

bal = multiprocessing.Value('i', 100)

# creating lock object

lock = multiprocessing.Lock()

# creating processes

proc1 = multiprocessing.Process(target=wthdrw, args=(bal,lock))

proc2 = multiprocessing.Process(target=dpst, args=(bal,lock))

# starting processes

proc1.start()

proc2.start()

# waiting for processes to finish

proc1.join()

proc2.join()

# printing final balance

print("Final balance = {}".format(bal.value))

if __name__ == "__main__":

for _ in range(10):

# performing transaction process

transact()

Output

Conclusion

In this article, you learned about what is multiprocessing in Python and how to use it. The most practical use of multiprocessing is sharing CPU resources and ATM operations, as you have seen in the last example. Because of the ease it provides to manage multiple processes, the concept of multiprocessing in Python will undoubtedly get a lot of traction. It will be a wise move to get a firm understanding and hands-on practice. 

Do you have any questions for us? Leave them in the comments section of this article. Our experts will get back to you on the same, ASAP.

Happy learning!