Python Subprocess - Execute Shell Commands

Python subprocess is a module in python that helps us to execute shell commands, get the input/output/error pipes, and the error codes for the executed command from the python language.

Why Python Subprocess Module?

Do you know any other module in python which can help us to execute the shell commands? Your answer will be the ‘os’ module in python.

The os.system() can help us to execute the shell commands from the python program but the problem is that we cannot get the output of the executed command in our python program.

>>> import os

>>> os.system('ls')

main.py  requirements.txt  venv

The output is generated on the terminal. We can only retrieve the error code for the executed shell command.

Example:

import os

out = os.system('ls')

>>> print(out)

0

One way to get the output of the executed shell command is by using the pipeline operator to store the output in a text file and then read the file in the python program.

Example:

import os

os.system('ls > output.txt')

file = open('output.txt','r')

>>> print(file.read())

output.txt
main.py
requirements.txt
venv

Python subprocess module is a high-level interface for executing shell commands from the python program and is intended to replace the following python modules and functions:

1. os.system()
2. os.spawn*()
3. os.popen*()
4. popen2.*()
5. commands.*()

According to the python official documentation, the subprocess module will rule these previously stated functions as they are going to be deprecated very soon.

Now when you know why to choose subprocess in python and what functions it is going to replace. So let’s learn about Python Subprocess Module.

Python Subprocess Run Function

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, capture_output=False, shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None, text=None, env=None, universal_newlines=None, **other_popen_kwargs)

The subprocess.run() function was added in Python 3.5 and it is recommended to use the run() function to execute the shell commands in the python program.

The args argument in the subprocess.run() function takes the shell command and returns an object of CompletedProcess in Python.

Example:

out = subprocess.run('ls', shell=True)

>>> print(out)

CompletedProcess(args='ls', returncode=0)

Executing Commands

Executing shell commands using the subprocess.run() function can be done by an iterator. If using the string literal space will act as the delimiter.

Learn what are iterators and how you can generate iterators from the python generator.

Example:

subprocess.run(
'ls -la', shell=True, check=True)
subprocess.run(
['ls', '-la'], shell=True, check=True)
subprocess.run(
{'ls', '-la'}, shell=True, check=True)
subprocess.run(
('ls', '-la'), shell=True, check=True)

Shell Parameter

Windows have a different command for listing the directories i.e dir command. But when we try to execute this command from the subprocess.run() function, it might throw an error.

This is because the dir command is specifically built for the Windows shell, and thus we have to set the shell parameter to True.

Setting this parameter to True also has some security hazards when you are working with user inputs. So this is only recommended when you are passing the arguments yourself.

capture_output parameter

By default the capture_output argument is False.

Example:

out = subprocess.run('ls', shell=True)

>>> print(out.stdout)

None

>>> print(out.stderr)

None

When capture_output is set to True the stdout and stderr will be captured and represented in the CompletedProcess Object.

out = subprocess.run('ls', shell=True, capture_output=True)

>>> print(out)
CompletedProcess(args='ls', returncode=0, stdout=b'output\nmain.py\ntest.csv\nvenv\n', stderr=b'')

>>> print(out.stdout)
b'output\nsr.py\ntest.csv\nvenv\n'

>>> print(out.stderr)
b''

This can also be achieved if we set the stdout and stderr parameters to subprocess.PIPE, But this method captures both the stdout and stderr separately.

Example:

out = subprocess.run('ls', shell=True,
 stdout=subprocess.PIPE, stderr=subprocess.PIPE)

>>> print(out)

CompletedProcess(args='ls', returncode=0, stdout=b'output\nsr.py\ntest.csv\nvenv\n', stderr=b'')

In order to capture both the parameters together, we can set the stdout to subprocess.PIPE and the stderr to subprocess.STDOUT.

If you want no output and error from the command, use the subprocess.DEVNULL for both stdout and stderr subprocess.run() parameters.

Example:

subprocess.run('ls', shell=True, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL)

Decode the Output

Setting the capture_output parameter to True captures the output and stores it into the stdout class field in form of bytes. Let’s decode the output.

out = subprocess.run('ping 127.0.0.1', shell=True, capture_output=True)



>>> print(out.stdout)

b'\r\nPinging 127.0.0.1 with 32 bytes of data:\r\nReply from 127.0.0.1: bytes=32 time<1ms TTL=128\r\nReply from 127.0.0.1: bytes=32 time<1ms TTL=128\r\nReply from 127.0.0.1: bytes=32 time<1ms TTL=128\r\nReply from 127.0.0.1: bytes=32 time<1ms TTL=128\r\n\r\nPing statistics for 127.0.0.1:\r\n    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),\r\nApproximate round trip times in milli-seconds:\r\n    Minimum = 0ms, Maximum = 0ms, Average = 0ms\r\n'



>>> print(out.stdout.decode())

Pinging 127.0.0.1 with 32 bytes of data:
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128

Ping statistics for 127.0.0.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

You can clearly see the difference from the output. When not using the decode function the new line and tab are shown as escape sequence characters but the decode function prettifies the output.

The decode function accepts several decoding parameters like ‘ASCII’, ‘UTF-8’, etc.

One of the easier ways to do this is to set the text parameter in the subprocess.run() function to True.

Example:

out = subprocess.run('ping 127.0.0.1', shell=True,
capture_output=True, text=True)

When the text parameter is set to True, we don’t have to decode the stdout, and the output captured is already decoded or in readable text format.

Timeout Parameter in Python Subprocess

The timeout parameter in the Python subprocess.run() function is used to set the expiry time(in seconds) for the command execution.

Example:

>>> subprocess.run('ping 127.0.0.1', shell=True, timeout=2)

subprocess.TimeoutExpired: Command 'ping 127.0.0.1' timed out after 2.0 seconds

When the timeout is expired the subprocess is killed and the TimeoutExpired exception is raised after the child process has terminated.

Check Parameter

If the check parameter is set to True, and the process ends with a non-zero error code, then a CalledProcessError exception is raised by the python subprocess module.

Example:

subprocess.run(
'cat output.txt', shell=True, check=True)

>>> print(out)

subprocess.CalledProcessError: Command 'cat output.txt' returned non-zero exit status 1.

In the above command, we want to see the content of the output.txt file, but this file doesn’t exist and hence we get a non-zero error code.

If we don’t set the check parameter to True, we will just get the error code and no information about the error.

out = subprocess.run(
'cat output.txt', shell=True, check=True)

>>> print(out)

CompletedProcess(args='cat output.txt', returncode=1)

Save Output in a File

We can also save the output of the executed command to a file.

Example:

with open('output.txt', 'w') as f:
    out = subprocess.run('ping 127.0.0.1', shell=True, stdout=f, text=True)

In the above code, we have opened a text file in write mode and passing the opened file to the stdout parameter.

A new file ‘output.txt’ will be created and the output of the executed command will be saved in it.

Input Parameter in Python Subprocess Run Function

In Python Subprocess Module, we can take the output from an executed command and make it as input for another command, using the input parameter.

Example:

You are
learning
about Python
Subprocess
Module.

out1 = subprocess.run('cat output.txt', shell=True,
                      text=True, capture_output=True)

out2 = subprocess.run('grep -n Subprocess', shell=True, text=True,
                      input=out1.stdout, capture_output=True)

>>> print(out2.stdout)

4:Subprocess

In this example, the first subprocess i.e out1 is just capturing the output of the output.txt file, and the second subprocess executes the grep command to find the Subprocess word from the out1.stdout.

Executing C Program from python

We can also build and execute the c program from the python program using the Python subprocess module.

Example:

#include 

int main()
{
    printf("Hello World from C");
    return 0;
}

import subprocess
import os

subprocess.run(
    ['gcc', '-o', 'test.exe', 'test.c'], shell=True)

out = subprocess.run(['.\\test.exe'], shell=True, capture_output=True)

>>> print(out)

CompletedProcess(args=['.\\test.exe'], returncode=0, stdout=b'Hello World from C', stderr=b'')

The stdout class field captures the output of the command.

Hope you like it!

Learn more about python from here.