How to use Python subprocess to execute external commands and programs

Python subprocess is a module with which you can control, execute and evaluate external programs within the code. The two most important functions of this tool are run() and Popen().

What is Python subprocess?

The Python module subprocess has been part of the programming language inventory since version 2.4. It is a very comprehensive and powerful tool that you can use to execute other programs or commands within your code. It is not only possible to open programs, but also to control and adapt the data flow. Python subprocess offers numerous methods and functions, the most important of which we will look at in this article and explain using practical examples.

How subprocess functions with run()

Before that, however, it makes sense to take a look at the structure and basic functionality of Python subprocess. The module is used to execute subprocesses. Python functions in the parent-child hierarchy as a parent process that creates a subordinate process. The function most commonly used within the module is run(). This allows you to start a process via Python and only begins further steps once it has been completed.

Example of how Python subprocess works with run()

We will now use this function for our first small example to illustrate how Python subprocess works. To do this, we first import the subprocess and sys modules and then execute a simple request. The corresponding code looks like this:

import subprocess
import sys
result = subprocess.run([sys.executable, "-c", "print('hello')"])

The output would be:

hello

• subprocess.run: This is the core function. It takes a list of strings representing the command to be executed. run() then starts a new Python process.
• sys.executable: sys.executable refers to the absolute path of the Python interpreter that was used to run your program. An example path might be /local/user/bin/example.
• -c: -c is a command-line option that passes the provided string for execution. In our case, it’s a script that prints the word ‘hello’.

How to execute Python subprocess with a script

To test how to use the module for a custom script, you can try out the following example. To do this, first create a simple script in .py format and save it as ‘example-script.py’:

print("The weather is nice today")

Use the following code to execute this file with Python subprocess:

import subprocess
result = subprocess.run(["python", "examplescript.py"], capture_output=True, text=True)
print(result.stdout)

The corresponding output will then look like this:

The weather is nice today

How to open external programs

In principle, it is possible to open any program with Python’s subprocess and the run() function. The only requirement is that you know the exact name or path where the program is located on your system. In the following code, for example, we open Notepad:

import subprocess
subprocess.run(["notepad"])

CompletedProcess capturing external output

After these straightforward examples, we will now turn our attention to capturing external output. You will execute an external program using Python subprocess as demonstrated earlier, but this time, a CompletedProcess object will be returned. We’ve previously made the necessary adjustments in a brief example, but now we’ll explore them in more detail. We will start again with our initial code, but this time we’ll modify it.

import subprocess
import sys
result = subprocess.run([sys.executable, "-c", "print('hello')"], capture_output=True, text=True)
print("The standard output is:", result.stdout)
print("This is the standard error:", result.stderr)

Once again, we instruct the system to output the string ‘hello’. This takes place in a subprocess. What’s new, however, are the two keyword arguments capture_output=True and text=True, which we also pass to run(). If the command runs successfully and no errors occur, a CompletedProcess object is returned and assigned to result. This object contains information about the program’s exit code and provides the output through result.stdout and any errors through result.stderr. stdout refers to standard output, while stderr handles any potential errors. We use text=True to display the output as a string. Since no errors are expected, our result looks as follows:

The standard output is: hello
This is the standard error:

To better illustrate how this works, let’s create the next example so that stderr is not left empty this time. The corresponding code is this:

import subprocess
import sys
result = subprocess.run([sys.executable, "-c", "raise ValueError('error')"], capture_output=True, text=True)
print("The standard output is:", result.stdout)
print("This is the standard error:", result.stderr)

While the standard output remains empty this time, there is now an output for stderr:

The standard output is:
This is the standard error: Traceback (most recent call last):
	File "<string>", line 1, in <module>
ValueError: error

How to execute from a function

Even if you want to include a command directly in the code, Python subprocess offers you this option. In this case, the code could look like this example:

import subprocess
result = subprocess.run(["C:/Users/name/anaconda3/python", "-c", "print('This output was taken directly from a function')"], capture_output=True, text=True, shell=True)
print("The standard output is:", result.stdout)

Our output will look like this:

The standard output is: This output was taken directly from a function

How to stop or end processes

Another very useful use of Python subprocess is achieved through the interaction of run() with the timeout argument. It allows you to stop an external program if it takes too long to run. Use the function time.sleep for this. The appropriate code is this:

import subprocess
import sys
result = subprocess.run([sys.executable, "-c", "import time; time.sleep(3)"], timeout=1)

The subordinate process uses time.sleep to suspend for three seconds. However, since you have instructed the system via timeout=1 to trigger a timeout after one second, the result is a TimeoutExpired exception.

Python subprocess with Popen()

Even though run() is the Python subprocess function that is used most frequently, there are other important classes that can be very useful. One of these is Popen(). This class is more or less the substructure of Python subprocess and is somewhat more complex to use than run(). However, Popen() gives you more control over the execution and allows you to interact with the input and output. The class owes its name to a UNIX command and stands for ‘pipe open’.

Almost all arguments that you can use with run() are also permitted for Popen(). In contrast to run(), however, this function does not wait until a process is completed, but starts a second one in parallel. We can illustrate how this works with a simple example:

import subprocess
from time import sleep
def poll_and_read(process):
    print(f"This is the output after poll(): {process.poll()}")
    print(f"This is the standard output: {process.stdout.read().decode('utf-8')}")
process = subprocess.Popen(["python", "timer.py", "3"], stdout=subprocess.PIPE)
poll_and_read(process)
sleep(2)
poll_and_read(process)
sleep(2)
poll_and_read(process)
process.wait()
print(f"exit code of the process: {process.returncode}")

Here we use the method .poll() to check whether the process is still running or has already been completed. As long as it is still running, the value ‘none’ is output. The method then outputs the exit code. With .read() all bytes that are currently under .stdout are to be read out. If you execute the code, you will first receive the value ‘None’ and then the value previously contained in .stdout. This continues until the process has run through. Then poll() receives the value ‘0’.