Python Subprocess: Master External Command Execution [2024]

Introduction to the Python Subprocess

Subprocess in Python is used to run new programs and scripts by spawning new processes. And it enables the user to launch new programs right from the current Python program thanks to the subprocess module. 

• So, you may use a subprocess in Python to run external applications from a git repository or code from C or C++ programs. 
• Using subprocesses in Python, you can also obtain exit codes and input, output, or error streams.
• The Subprocess in Python can be useful if you've ever intended to streamline your command-line scripting or utilize Python alongside command-line apps—or any applications, for that matter. The Python subprocess module may help with everything from starting GUI interfaces to executing shell commands and command-line programs.

And don't forget that all the subprocess tasks are complete within a parent process. The numerous background operations and activities that Python has been referred to as processes. This is called the parent process. 

Processes frequently have tasks that must be performed before the process can be finished. And this parent process needs someone to take care of these tasks. That's where this parent process gives birth to the subprocess. When a process needs to finish a quick task, it can create a subprocess to handle it while the main process is still running. This makes managing data and memory easier and more effective. You can run more processes concurrently by dividing larger tasks into smaller subprocesses in Python that can handle simpler tasks within a process. And this is called multiprocessing.

How to Start a Process in Python?

To start a new process, or in other words, a new subprocess in Python, you need to use the Popen function call. It is possible to pass two parameters in the function call. The first parameter is the program you want to start, and the second is the file argument. In the example below, you will use Unix’s cat command and example.py as the two parameters. The cat command is short for ‘concatenate’ and is widely used in Linux and Unix programming. It is like “cat example.py.” You can start any program unless you haven’t created it.

from subprocess import Popen, PIPE

process = Popen(['cat', 'example.py'], stdout=PIPE, stderr=PIPE)

stdout, stderr = process.communicate()

print(stdout)

Output:

In the above code, the process.communicate() is the primary call that reads all the process’s inputs and outputs. The “stdout” handles the process output, and the “stderr” is for handling any sort of error and is written only if any such error is thrown.

What Is the Subprocess Call()?

Subprocess in Python has a call() method that is used to initiate a program. The syntax of this subprocess call() method is:

subprocess.check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False)

Parameters of Subprocess Call()

The call() method from the subprocess in Python accepts the following parameters:

• args: It is the command that you want to execute. You can pass multiple commands by separating them with a semicolon (;)
• stdin: This refers to the standard input stream’s value passed as (os.pipe())
• stdout: It is the standard output stream’s obtained value
• stderr: This handles any errors that occurred from the standard error stream
• shell: It is the boolean parameter that executes the program in a new shell if kept true

Learn 15+ In-Demand Tools and Skills!

Automation Testing Masters ProgramExplore Program

Return Value of the Call() Method from Subprocess in Python

The Python subprocess call() function returns the executed code of the program. If there is no program output, the function will return the code that it executed successfully. It may also raise a CalledProcessError exception.

Now, use a simple example to call a subprocess for the built-in Unix command “ls -l.” The ls command lists all the files in a directory, and the -l command lists those directories in an extended format.

import subprocess

subprocess.call(["ls", "-l"])

Output:

As simple as that, the output displays the total number of files along with the current date and time.

What is Save Process Output (stdout) in Subprocess in Python?

The save process output or stdout allows you to store the output of a code directly in a string with the help of the check_output function. The syntax of this method is:

subprocess.check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False)

As you can see, the function accepts the same parameters as the call() method except for one additional parameter, which is:

• universal_newlines: It is a boolean parameter that opens the files with stdout and stderr in a universal newline when kept true

The method also returns the same value as the call() function. Now, look at a simple example again. This time you will use Linux’s echo command used to print the argument that is passed along with it. You will store the echo command’s output in a string variable and print it using Python’s print function.

import subprocess

s = subprocess.check_output(["echo", "Hello World!"])

print(s)

Output:

Example of Using Subprocess in Python

Let’s combine both the call() and check_output() functions from the subprocess in Python to execute the “Hello World” programs from different programming languages: C, C++, and Java. You will first have to create three separate files named Hello.c, Hello.cpp, and Hello.java to begin. After making these files, you will write the respective programs in these files to execute “Hello World!” Let’s begin with our three files.

Frequently Used Arguments

• args: This refers to the command you want to run a subprocess in Python. By using a semicolon to separate them, you can pass numerous commands (;)
• stdin: This is referring to the value sent as (os.pipe()) for the standard input stream.
• stdout: It represents the value that was retrieved from the standard output stream.
• stderr: stderr handles any kind of error that occurs in a shell. 
• shell: shell is the boolean parameter that executes the program in a new shell if only kept true.

Our Free Courses with Certificate

Python for Beginners	Enroll Now!
Python Libraries for Data Science	Enroll Now!
Python Pandas Basics Course	Enroll Now!
Getting Python Interview Ready	Enroll Now!
Machine Learning using Python	Enroll Now!

Popen Function

The Popen function is the name of an upgrade function for the call function. The popen() function will execute the command supplied by the string command. The function should return a pointer to a stream that may be used to read from or write to the pipe while also creating a pipe between the calling application and the executed command. Immediately after starting, the Popen function returns data, and it does not wait for the subprocess to finish.

For example,

Furthermore, using the same media player example, we can see how to launch theSubprocess in python using the popen function. You can start the Windows Media Player as a subprocess for a parent process.

program = "mediaplayer.exe"

subprocess.Popen(program)

/*response*/

<subprocess.Popen object at 0x01EE0430>

The above code is successfully starting the subprocess, but you won't get any update for the rest of the process if there were any errors. To know more about the complete process and if there are any errors occurring, then it is advisable to use the popen wait method.

By Using the Wait Method

The wait method holds out on returning a value until the subprocess in Python is complete. In this case, if it is returning zero, then let's assume that there were no errors.

program = "mediaplayer.exe"

process = subprocess.Popen(program)

code = process.wait()

print(code)

/*result*/

0

In some scenarios, such as when using stdout/stderr=PIPE commands, you cannot use the wait() function because it may result in a deadlock that will cause your application to halt until it is resolved. In certain circumstances, you can utilize the communicate() function instead of the wait() method. Next, let's examine how that is carried out at Subprocess in Python by using the communication method.

args = ["ping", "mediaplayer"]

process = subprocess.Popen(args, stdout=subprocess.PIPE)

data = process.communicate()

print(data)

Popen Constructor

The Popen class manages the Popen constructor, which is the module's fundamental mechanism for construction. It provides a lot of flexibility so that programmers can manage the less typical circumstances that aren't handled by the convenience functions.

Want a Top Software Development Job? Start Here!

Full Stack Developer - MERN StackExplore Program

Exceptions

Exceptions are referred to as raised in the child process at Subprocess in Python before the new program's execution and will be raised again in the parent. OSError is the most commonly encountered exception. Keep in mind that the child will only report an OSError if the chosen shell itself cannot be found when shell=True. Examining the return code or output from the subprocess is required to ascertain whether the shell was unable to locate the requested application.

Security Considerations

This approach will never automatically invoke a system shell, in contrast to some others. All characters, including shell metacharacters, can now be safely passed to child processes. If the shell is explicitly invoked with the shell=True flag, the application must ensure that all white space and meta characters are accurately quoted. This prevents shell injection vulnerabilities in Subprocess in Python. Shlex.quote() can be used for this escape on some platforms.

Popen Objects

• Popen.poll()

 Verify whether the child's procedure has ended at Subprocess in Python

• Popen.wait(timeout=None)

Watch for the child's process to finish..

• Popen.send signal(signal).

Send the signal ‌ to the child by using Popen.send signal(signal).

• Popen.kill()

Murder the child. The function on POSIX OSs sends SIGKILL to the child. 

• Popen.returncode

The poll() and wait() functions, as well as communicate() indirectly, set the child return code. A value of None signifies that the process has not yet come to an end.

Windows Popen Helpers

The Windows popen program is created using a subset of the Windows STARTUPINFO structure. The following parameters can be passed as keyword-only arguments to set the corresponding characteristics.

• dwFlags
• hStdInput.
• hStdOutput.

Window Constants

The Subprocess in the Python module exposes the following constants.

• subprocess.STD_INPUT_HANDLE

The widely utilized input method. 

• subprocess.STD_OUTPUT_HANDLE

The common output mechanism. 

• subprocess.STD_ERROR_HANDLE

The device for standard error. 

• subprocess.SW_HIDE

It helps to hide the window.

Older, High-level API

The high-level APIs are meant for straightforward operations where performance is not a top priority at Subprocess in Python. The high-level APIs, in contrast to the full APIs, only call for a single object handler, similar to a C++ fstream or a Python file I/O idiom.

Replacing Older Functions With the Subprocess Module

In arithmetic, if a becomes b means that b is replacing a to produce the desired results. Likewise, in the subprocess in Python, the subprocess is also altering certain modules to optimize efficacy.

Examples

• Replacing /bin/sh shell ommand substitution

Replacing /bin/sh shell command substitution means,

output=$(mycmd myarg) 

And it becomes,

output = check_output(["myarg", "myarg"]) 

• Replacing shell pipeline:

Generally, the pipeline is a mechanism for trans interaction that employs data routing. Pipelines involve the shell connecting several subprocesses together via pipes and running external commands inside each subprocess. As a result, the data transmitted across the pipeline is not directly processed by the shell itself.

As a fallback, the shell's own pipeline support can still be utilized directly for trustworthy input. For example, 

output=$(dmesg | grep hda) becomes 

output = check output("dmesg | grep hda", shell=True)

• Replacing os.system()

OS falls under the umbrella of Subprocess in Python’s common utility modules and it is generally known as miscellaneous operating system interfaces. This module provides a portable way to use operating system-dependent functionality. The command (a string) is executed by the os. system() method in a subshell.

sts = os.system("mycmd" + " myarg") 

And it becomes 

retcode = call("mycmd" + " myarg", shell=True) 

From the above program,

• The call() return value is encoded compared to the os.system().
• When utilizing the subprocess module, the caller must execute this individually because the os.system() function ignores SIGINT and SIGQUIT signals while the command is under execution.

• Replacing the os.spawn family:

The Os.spawn family gives programmers extra control over how their code is run. We can understand how the subprocess is using the spawn family by the below examples.

P_NOWAIT example:

           pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg") 

==> 

pid = Popen(["/bin/mycmd", "myarg"]).pid 

P_WAIT example:

            retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg") 

==> 

retcode = call(["/bin/mycmd", "myarg"]) 

Vector example:

os.spawnvp(os.P_NOWAIT, path, args) 

==> 

Popen([path] + args[1:]) 

Environment example:

            os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env) 

==> 

Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"}) 

• Replacing os.popen(), os.popen2(), os.popen3()

In Subprocess in python, every popen using different arguments like,

• os.popen() -> stdout

• os.popen2() -> (stdin, stdout)

• os.popen3() -> (stdin, stdout, stderr)

Legacy Shell Invocation Functions

The Subprocess in the Python module also delivers the legacy 2.x commands module functionalities. These operations implicitly invoke the system shell, and these functions are commensurate with exception handling.

In this article, we discussed subprocesses in Python. Now, it's simple to spawn external programs from the Python code by using the subprocess module. The call() and pippen() functions are the two main functions of this module. You won't have any difficulty generating and utilizing subprocesses in Python after you practice and understand how to utilize these two functions properly.

Want a Top Software Development Job? Start Here!

Full Stack Developer - MERN StackExplore Program

Code to Include in Hello.c File

Create a Hello.c file and write the following code in it.

#include<stdio.h>

int main()

{

    printf("C says Hello World!");

    // returning with 0 is essential to call it from Python

    return 0;

}

If you are not familiar with the terms, you can learn the basics of C programming from here.

Code to Write in Hello.cpp File

After the Hello.c, create Hello.cpp file and write the following code in it.

#include <iostream>

using namespace std;

int main()

{

    int a, b;

    cin >> a >> b;

    cout << "C++ says Hello World! Values are:" << a << " " << b;

return 0;

}

Here, this demo has also used integer variables to store some values and pass them through the stdin parameter. If you are not familiar with the terms, you can learn the basics of C++ programming from here.

Code to Write in Hello.java File

Use the following code in your Hello.java file.

class Hello{

public static void main(String args[]){

System.out.print("Java says Hello World!");

}

}

If you are not familiar with the terms, you can learn the basics of Java programming from here.

Code to Write in Main.py File to Execute Subprocess in Python

Once you have created these three separate files, you can start using the call() and output() functions from the subprocess in Python. Here’s the code that you need to put in your main.py file.

import subprocess

import os

def C_Execution():

# storing the output

s = subprocess.check_call("gcc Hello.c -o out1;./out1", shell = True)

print(", return code", s)

def Cpp_Execution():

    # creating a pipe to child process

    data, temp = os.pipe()

    # writing inputs to stdin and using utf-8 to convert it to byte string

    os.write(temp, bytes("7 12\n", "utf-8"));

    os.close(temp)

    # storing output as a byte string

    s = subprocess.check_output("g++ Hello.cpp -o out2;./out2", stdin = data, shell = True)

    # decoding to print a normal output

    print(s.decode("utf-8"))

def Java_Execution():

# storing the output

s = subprocess.check_output("javac Hello.java;java Hello", shell = True)

print(s.decode("utf-8"))

# Driver functions

if __name__=="__main__":

C_Execution()

Cpp_Execution()

Java_Execution()

Output:

FAQs

1. What does subprocess run do?

Subprocess runs the command, waits for the procedure to complete, and then returns a "Completed process" artifact.

2. Is subprocess a standard Python library?

With the help of the subprocess library, we can run and control subprocesses right from Python.

3. How do you run a subprocess in Python?

If you want to run a Subprocess in python, then you have to create the external command to run it.

4. What is PIPE in Python?

Summing It Up

In this article, you have learned about subprocess in Python. You can now easily use the subprocess module to run external programs from your Python code. The two primary functions from this module are the call() and output() functions. Once you practice and learn to use these two functions appropriately, you won’t face much trouble creating and using subprocess in Python. 

If you are a newbie, it is better to start from the basics first. You can refer to Simplilearn’s Python Tutorial for Beginners. The tutorial will help clear the basics and get a firm understanding of some Python programming concepts. After the basics, you can also opt for our Online Python Certification Course. The certification course comes with hours of applied and self-paced learning materials to help you excel in Python development.

If you are on the other hand looking for a free course that allows you to explore the fundamentals of Python in a systematic manner - allowing you the freedom to decide whether learning the language is indeed right for you, you could check out our Full Stack Web Developer - MERN. This course, in collaboration with Caltech CTME, can help you hone the right skills and make you job-ready in no time.

Have any questions for us? Leave them in the comments section of this article. Our experts will get back to you on the same, ASAP!