Python Subprocess - Execute Shell Commands
Advertisement
Python subprocess is a module in python that helps us to execute shell commands, get the input/output/error pipes, and the error codes for the executed command from the python language.
Why Python Subprocess Module?
Do you know any other module in python which can help us to execute the shell commands? Your answer will be the ‘os’ module in python.
The os.system()
can help us to execute the shell commands from the python program but the problem is that we cannot get the output of the executed command in our python program.
>>> import os
>>> os.system('ls')
main.py requirements.txt venv
The output is generated on the terminal. We can only retrieve the error code for the executed shell command.
Example:
import os
out = os.system('ls')
>>> print(out)
0
One way to get the output of the executed shell command is by using the pipeline operator to store the output in a text file and then read the file in the python program.
Example:
import os
os.system('ls > output.txt')
file = open('output.txt','r')
>>> print(file.read())
output.txt
main.py
requirements.txt
venv
Python subprocess module is a high-level interface for executing shell commands from the python program and is intended to replace the following python modules and functions:
- os.system()
- os.spawn*()
- os.popen*()
- popen2.*()
- commands.*()
According to the python official documentation, the subprocess module will rule these previously stated functions as they are going to be deprecated very soon.
Now when you know why to choose subprocess in python and what functions it is going to replace. So let’s learn about Python Subprocess Module.
Python Subprocess Run Function
subprocess.run
(args, *, stdin=None, input=None, stdout=None, stderr=None, capture_output=False, shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None, text=None, env=None, universal_newlines=None, **other_popen_kwargs)
The subprocess.run()
function was added in Python 3.5 and it is recommended to use the run()
function to execute the shell commands in the python program.
The args argument in the subprocess.run()
function takes the shell command and returns an object of CompletedProcess in Python.
Example:
out = subprocess.run('ls', shell=True)
>>> print(out)
CompletedProcess(args='ls', returncode=0)
Executing Commands
Executing shell commands using the subprocess.run()
function can be done by an iterator. If using the string literal space will act as the delimiter.
Learn what are iterators and how you can generate iterators from the python generator.
Example:
subprocess.run(
'ls -la', shell=True, check=True)
subprocess.run(
['ls', '-la'], shell=True, check=True)
subprocess.run(
{'ls', '-la'}, shell=True, check=True)
subprocess.run(
('ls', '-la'), shell=True, check=True)
Shell Parameter
Windows have a different command for listing the directories i.e dir command. But when we try to execute this command from the subprocess.run()
function, it might throw an error.
This is because the dir command is specifically built for the Windows shell, and thus we have to set the shell parameter to True.
Setting this parameter to True also has some security hazards when you are working with user inputs. So this is only recommended when you are passing the arguments yourself.
capture_output parameter
By default the capture_output argument is False.
Example:
out = subprocess.run('ls', shell=True)
>>> print(out.stdout)
None
>>> print(out.stderr)
None
When capture_output is set to True the stdout and stderr will be captured and represented in the CompletedProcess Object.
out = subprocess.run('ls', shell=True, capture_output=True)
>>> print(out)
CompletedProcess(args='ls', returncode=0, stdout=b'output\nmain.py\ntest.csv\nvenv\n', stderr=b'')
>>> print(out.stdout)
b'output\nsr.py\ntest.csv\nvenv\n'
>>> print(out.stderr)
b''
This can also be achieved if we set the stdout and stderr parameters to subprocess.PIPE, But this method captures both the stdout and stderr separately.
Example:
out = subprocess.run('ls', shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> print(out)
CompletedProcess(args='ls', returncode=0, stdout=b'output\nsr.py\ntest.csv\nvenv\n', stderr=b'')
In order to capture both the parameters together, we can set the stdout to subprocess.PIPE and the stderr to subprocess.STDOUT.
If you want no output and error from the command, use the subprocess.DEVNULL for both stdout and stderr subprocess.run()
parameters.
Example:
subprocess.run('ls', shell=True, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL)
Decode the Output
Setting the capture_output parameter to True captures the output and stores it into the stdout class field in form of bytes. Let’s decode the output.
out = subprocess.run('ping 127.0.0.1', shell=True, capture_output=True)
>>> print(out.stdout)
b'\r\nPinging 127.0.0.1 with 32 bytes of data:\r\nReply from 127.0.0.1: bytes=32 time<1ms TTL=128\r\nReply from 127.0.0.1: bytes=32 time<1ms TTL=128\r\nReply from 127.0.0.1: bytes=32 time<1ms TTL=128\r\nReply from 127.0.0.1: bytes=32 time<1ms TTL=128\r\n\r\nPing statistics for 127.0.0.1:\r\n Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),\r\nApproximate round trip times in milli-seconds:\r\n Minimum = 0ms, Maximum = 0ms, Average = 0ms\r\n'
>>> print(out.stdout.decode())
Pinging 127.0.0.1 with 32 bytes of data:
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Ping statistics for 127.0.0.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
You can clearly see the difference from the output. When not using the decode function the new line and tab are shown as escape sequence characters but the decode function prettifies the output.
The decode function accepts several decoding parameters like ‘ASCII’, ‘UTF-8’, etc.
One of the easier ways to do this is to set the text parameter in the subprocess.run() function to True.
Example:
out = subprocess.run('ping 127.0.0.1', shell=True,
capture_output=True, text=True)
When the text parameter is set to True, we don’t have to decode the stdout, and the output captured is already decoded or in readable text format.
Timeout Parameter in Python Subprocess
The timeout parameter in the Python subprocess.run()
function is used to set the expiry time(in seconds) for the command execution.
Example:
>>> subprocess.run('ping 127.0.0.1', shell=True, timeout=2)
subprocess.TimeoutExpired: Command 'ping 127.0.0.1' timed out after 2.0 seconds
When the timeout is expired the subprocess is killed and the TimeoutExpired exception is raised after the child process has terminated.
Check Parameter
If the check parameter is set to True, and the process ends with a non-zero error code, then a CalledProcessError exception is raised by the python subprocess module.
Example:
subprocess.run(
'cat output.txt', shell=True, check=True)
>>> print(out)
subprocess.CalledProcessError: Command 'cat output.txt' returned non-zero exit status 1.
In the above command, we want to see the content of the output.txt file, but this file doesn’t exist and hence we get a non-zero error code.
If we don’t set the check parameter to True, we will just get the error code and no information about the error.
out = subprocess.run(
'cat output.txt', shell=True, check=True)
>>> print(out)
CompletedProcess(args='cat output.txt', returncode=1)
Save Output in a File
We can also save the output of the executed command to a file.
Example:
with open('output.txt', 'w') as f:
out = subprocess.run('ping 127.0.0.1', shell=True, stdout=f, text=True)
In the above code, we have opened a text file in write mode and passing the opened file to the stdout parameter.
A new file ‘output.txt’ will be created and the output of the executed command will be saved in it.
Input Parameter in Python Subprocess Run Function
In Python Subprocess Module, we can take the output from an executed command and make it as input for another command, using the input parameter.
Example:
You are
learning
about Python
Subprocess
Module.
out1 = subprocess.run('cat output.txt', shell=True,
text=True, capture_output=True)
out2 = subprocess.run('grep -n Subprocess', shell=True, text=True,
input=out1.stdout, capture_output=True)
>>> print(out2.stdout)
4:Subprocess
In this example, the first subprocess i.e out1 is just capturing the output of the output.txt file, and the second subprocess executes the grep command to find the Subprocess word from the out1.stdout.
Executing C Program from python
We can also build and execute the c program from the python program using the Python subprocess module.
Example:
#include
int main()
{
printf("Hello World from C");
return 0;
}
import subprocess
import os
subprocess.run(
['gcc', '-o', 'test.exe', 'test.c'], shell=True)
out = subprocess.run(['.\\test.exe'], shell=True, capture_output=True)
>>> print(out)
CompletedProcess(args=['.\\test.exe'], returncode=0, stdout=b'Hello World from C', stderr=b'')
The stdout class field captures the output of the command.
Hope you like it!
Learn more about python from here.
Python subprocess is a module in python that helps us to execute shell commands, get the input/output/error pipes, and the error codes for the executed command from the python language.
If you want no output and error from the command, use the subprocess.DEVNULL for both stdout and stderr subprocess.run()
parameters.