Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

How to use a Python multiprocessing module

July 27, 2023
Rodolfo Olivieri
Related topics:
Python
Related products:
Red Hat Enterprise Linux

Share:

    In this article, we will learn how to work with a specific Python class from the multiprocessing module, the process class. I will give you a quick overview with examples.

    What is a Python multiprocessing module?

    What better way of describing what the module than to pull from the official documentation? Multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads.

    The threading module is not the focus of this article, but in summary, the threading module will handle a small segment of code execution (lightweight and with shared memory), while the multiprocessing one will handle a program execution (heavier, and totally isolated).

    If you want to learn more about the difference between a process and a thread, read this amazing article by Jong Hyuck Won, Process vs Thread: What’s the difference?

    In general, the multiprocessing module offers a variety of other classes, functions and utilities that one could use to handle multiple processes executing during your program execution. That module is specially designed to be the main point of interaction if a program needs to apply parallelism in its workflow. We won't go over all classes and utilities from the multiprocessing module, but rather, we will focus on a very specific class, the process class.

    What is the process class?

    In this section, we will try to give a better scope of what a process is, and how you can identify, use and manage processes within Python. As explained in the GNU C Library: "Processes are the primitive units for allocation of system resources. Each process has its own address space and (usually) one thread of control. A process executes a program; you can have multiple processes executing the same program, but each process has its own copy of the program within its own address space and executes it independently of the other copies."

    But what does that look like in Python? So far, we have managed to give some descriptions and references to what a process is, the difference between a process and a thread, but we haven't touched any code so far. Well, let's change that and do a very simple example of a process in Python:

    #!/usr/bin/env python
    import os
    
    # A very, very simple process.
    if __name__ == "__main__":
        print(f"Hi! I'm process {os.getpid()}")
    

    Which will produce the following output:

    [r0x0d@fedora ~]$ python /tmp/tmp.iuW2VAurGG/scratch.py
    Hi! I'm process 144112
    

    As you can see, any running Python script or program is a process of its own.

    Creating a child process from your parent

    And what about spawning different child processes inside your parent process? Well, to do that, we have the aid of the Process class from multiprocessing module, and it looks like this:

    #!/usr/bin/env python
    import os
    import multiprocessing
    
    def child_process():
        print(f"Hi! I'm a child process {os.getpid()}")
    
    if __name__ == "__main__":
        print(f"Hi! I'm process {os.getpid()}")
    
        # Here we create a new instance of the Process class and assign our
        # `child_process` function to be executed.
        process = multiprocessing.Process(target=child_process)
    
        # We then start the process
        process.start()
    
        # And finally, we join the process. This will make our script to hang and
        # wait until the child process is done.
        process.join()
    

    Which will produce the following output:

    [r0x0d@fedora ~]$ python /tmp/tmp.iuW2VAurGG/scratch.py
    Hi! I'm process 144078
    Hi! I'm a child process 144079
    

    A very important note about the previous script: if you don't use the process.join() to wait for your child process to execute and finish, then any other subsequent code that point will actually execute and may become a bit harder to synchronize your workflow.

    Consider the following example:

    #!/usr/bin/env python
    import os
    import multiprocessing
    
    def child_process():
        print(f"Hi! I'm a child process {os.getpid()}")
    
    if __name__ == "__main__":
        print(f"Hi! I'm process {os.getpid()}")
    
        # Here we create a new instance of the Process class and assign our
        # `child_process` function to be executed.
        process = multiprocessing.Process(target=child_process)
    
        # We then start the process
        process.start()
    
        # And finally, we join the process. This will make our script to hang and
        # wait until the child process is done.
        #process.join()
    
        print("AFTER CHILD EXECUTION! RIGHT?!")
    

    This snippet will produce the following output:

    [r0x0d@fedora ~]$ python /tmp/tmp.iuW2VAurGG/scratch.py
    Hi! I'm process 145489
    AFTER CHILD EXECUTION! RIGHT?!
    Hi! I'm a child process 145490

    Of course, it is not correct to affirm that the above snippet is wrong. It will all depend on how you want to use the module and how your child processes will execute. So use it wisely.

    Creating various child processes from a parent process

    If you want to spawn multiple processes, you can take advantage of for-loops (or any other type of loops). They will let you create as many references to the processes you need, and at a later stage, start/join them.

    #!/usr/bin/env python
    import os
    import multiprocessing
    
    def child_process(id):
        print(f"Hi! I'm a child process {os.getpid()} with id#{id}")
    
    if __name__ == "__main__":
        print(f"Hi! I'm process {os.getpid()}")
        list_of_processes = []
    
        # Loop through the number 0 to 10 and create processes for each one of
        # them.
        for i in range(0, 10):
            # Here we create a new instance of the Process class and assign our
            # `child_process` function to be executed. Note the difference now that
            # we are using the `args` parameter now, this means that we can pass
            # down parameters to the function being executed as a child process.
            process = multiprocessing.Process(target=child_process, args=(i,))
            list_of_processes.append(process)
    
        for process in list_of_processes:
            # We then start the process
            process.start()
    
            # And finally, we join the process. This will make our script to hang
            # and wait until the child process is done.
            process.join()
    

    That will produce the following output:

    [r0x0d@fedora ~]$ python /tmp/tmp.iuW2VAurGG/scratch.py
    Hi! I'm process 146056
    Hi! I'm a child process 146057 with id#0
    Hi! I'm a child process 146058 with id#1
    Hi! I'm a child process 146059 with id#2
    Hi! I'm a child process 146060 with id#3
    Hi! I'm a child process 146061 with id#4
    Hi! I'm a child process 146062 with id#5
    Hi! I'm a child process 146063 with id#6
    Hi! I'm a child process 146064 with id#7
    Hi! I'm a child process 146065 with id#8
    Hi! I'm a child process 146066 with id#9

    Communicating data between child process and parent process

    In the previous section, I described the addition of a new parameter to the multiprocessing.Process class constructor, the args. This parameter allows you to pass down values to your child process to be used inside of the function. But do you know how to return data from the child process?

    You may be thinking that to return data from the child, one must use the return statement inside of it to actually be able to retrieve the data. A process is wonderful to execute functions in an isolated way, without interfering with shared resources meaning that the normal and usual way that we know about returning data from functions. Here, is not allowed because of its isolation.

    Instead, we can use the queue class, which will provide us an interface to communicate data between the parent process and its child processes. A queue, in this context, is a normal FIFO (First In First Out) that has a built-in mechanism for working with multiprocessing.

    Consider the following example:

    #!/usr/bin/env python
    import os
    import multiprocessing
    
    def child_process(queue, number1, number2):
        print(f"Hi! I'm a child process {os.getpid()}. I do calculations.")
        sum = number1 + number2
    
        # Putting data into the queue
        queue.put(sum)
    
    if __name__ == "__main__":
        print(f"Hi! I'm process {os.getpid()}")
    
        # Defining a new Queue()
        queue = multiprocessing.Queue()
    
        # Here we create a new instance of the Process class and assign our
        # `child_process` function to be executed. Note the difference now that
        # we are using the `args` parameter now, this means that we can pass
        # down parameters to the function being executed as a child process.
        process = multiprocessing.Process(target=child_process, args=(queue,1, 2))
    
        # We then start the process
        process.start()
    
        # And finally, we join the process. This will make our script to hang and
        # wait until the child process is done.
        process.join()
    
        # Accessing the result from the queue.
        print(f"Got the result from child process as {queue.get()}")
    

    It will give the following output:

    [r0x0d@fedora ~]$ python /tmp/tmp.iuW2VAurGG/scratch.py
    Hi! I'm process 149002
    Hi! I'm a child process 149003. I do calculations.
    Got the result from child process as 3
    

    Exception handling for the process class

    Handling exceptions is a special and somewhat difficult task that we have to go through from time to time while working with the process module. The reason for that is, by default, any exception that occurs inside a child process will always be handled by the Process class that spawned it.

    The code below is raising an Exception with text:

    #!/usr/bin/env python
    import os
    import multiprocessing
    
    def child_process():
        print(f"Hi! I'm a child process {os.getpid()}.")
        raise Exception("Oh no! :(")
    
    if __name__ == "__main__":
        print(f"Hi! I'm process {os.getpid()}")
    
        # Here we create a new instance of the Process class and assign our
        # `child_process` function to be executed. Note the difference now that
        # we are using the `args` parameter now, this means that we can pass
        # down parameters to the function being executed as a child process.
        process = multiprocessing.Process(target=child_process)
    
        try:
            # We then start the process
            process.start()
    
            # And finally, we join the process. This will make our script to hang and
            # wait until the child process is done.
            process.join()
    
            print("AFTER CHILD EXECUTION! RIGHT?!")
        except Exception:
            print("Uhhh... It failed?")
    

    This results in:

    [r0x0d@fedora ~]$ python /tmp/tmp.iuW2VAurGG/scratch.py
    Hi! I'm process 149505
    Hi! I'm a child process 149506.
    Process Process-1:
    Traceback (most recent call last):
      File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
        self.run()
      File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
        self._target(*self._args, **self._kwargs)
      File "/tmp/tmp.iuW2VAurGG/scratch.py", line 7, in child_process
        raise Exception("Oh no! :(")
    Exception: Oh no! :(
    AFTER CHILD EXECUTION! RIGHT?!

    If you follow up the code, you will be able to notice that there is a print statement carefully placed after the process.join() call to simulate that the parent process is still running, even after an unhandled exception raised in its child.

    One way of overcoming this situation is to actually handle the exception inside your child process as follows:

    #!/usr/bin/env python
    import os
    import multiprocessing
    
    def child_process():
        try:
            print(f"Hi! I'm a child process {os.getpid()}.")
            raise Exception("Oh no! :(")
        except Exception:
            print("Uh, I think it's fine now...")
    
    if __name__ == "__main__":
        print(f"Hi! I'm process {os.getpid()}")
    
        # Here we create a new instance of the Process class and assign our
        # `child_process` function to be executed. Note the difference now that
        # we are using the `args` parameter now, this means that we can pass
        # down parameters to the function being executed as a child process.
        process = multiprocessing.Process(target=child_process)
    
        # We then start the process
        process.start()
    
        # And finally, we join the process. This will make our script to hang and
        # wait until the child process is done.
        process.join()
    
        print("AFTER CHILD EXECUTION! RIGHT?!")
    

    Now your exceptions will be handled inside your child process, meaning you can control what will happen to it and what should be done in such cases.

    Final thoughts

    The multiprocessing module is very powerful when working and implementing solutions that will depend on executing in a parallel way, especially if used with the Process class. That adds this amazing possibility to execute any function in its own isolated process.

    Last updated: August 15, 2023

    Related Posts

    • Build your first application using Python with Red Hat Container Development Kit (CDK)

    • How to install multiple versions of Python on Red Hat Enterprise Linux

    • Profiling Python Programs

    • OpenShift Online adds Python 3.3, PHP 5.4 and more

    • Build your first application using Python 3.5 on RHEL 7 with containers and Red Hat Software Collections

    Recent Posts

    • How to run AI models in cloud development environments

    • How Trilio secures OpenShift virtual machines and containers

    • How to implement observability with Node.js and Llama Stack

    • How to encrypt RHEL images for Azure confidential VMs

    • How to manage RHEL virtual machines with Podman Desktop

    What’s up next?

    Ready to level up your Linux knowledge? Our Intermediate Linux Cheat Sheet presents a collection of Linux commands and executables for developers and system administrators who want to move beyond the basics.

    Get the cheat sheet
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue