Python Subprocess: Run Shell Commands with Ease

In the world of programming, executing shell commands through code can be a necessary task for many developers. Python’s subprocess module provides an interface for executing shell commands and handling input/output streams. In this blog, we will explore Python subprocess module and how it can be used to execute shell commands in Python.

Check out our other blogs on Python development, including Python decorators, generators, enumerate function and list comprehension to enhance your Python skills.

What is a process?

Before we dive into the details of the Python subprocess module, let’s understand what a process in Operating System is.

In the world of computers, a process refers to an instance of a program running on an operating system. It represents an active entity engaged in executing specific instructions and performing computations.

Processes have their own allocated resources, such as memory and file handles, and operate independently with their own state and data. They can interact and communicate with other processes, enabling collaboration and coordination.

Operating systems handle the scheduling of processes, ensuring fair execution and resource allocation. Additionally, processes can spawn new processes, enabling complex interactions within a program or between multiple programs.

What is Python Subprocess?

Subprocess in Python is a powerful module that enables you to launch and manage new processes directly from your Python scripts. It allows seamless execution of external programs, scripts, or system commands.

With subprocess, you can integrate Python with command-line apps, utilize code from C or C++ programs, and even run applications from git repositories. It extends the capabilities of your Python code by enabling interactions with command-line programs and the execution of GUI interfaces.

The subprocess module operates within a parent process, which handles background operations. By dividing tasks into subprocesses, you can achieve multiprocessing and improve efficiency.

Subprocess not only launches processes but also provides access to exit codes, input/output streams, and error handling. You have fine-grained control over subprocess behavior, including timeouts, environment variables, and signal handling.

Subprocess module was included in the standard Python library since version 2.4. Therefore, it is available on all platforms supported by Python, including Linux, macOS, and Windows.

When to use Python Subprocess

Python subprocess is a powerful tool for executing shell commands from within Python code. However, it is important to understand when and why to use subprocess in your code.

We should use Python Subprocess while:

1. Executing External Commands: Easily run external commands and complex shell commands from your Python code.
2. Run Background processes: Run background processes such as web servers, allowing your Python script to handle tasks simultaneously while the server runs independently.
3. Interacting with Other Programs: Seamlessly communicate with and exchange data with other programs or scripts.
4. Automating System Tasks: Automate system-related tasks like file manipulation, program launching, and administrative actions.
5. Parallel Processing: Harness the power of parallel processing by spawning multiple processes concurrently.
6. Cross-platform Compatibility: Enjoy the convenience of consistent operation across different operating systems.
7. External Data Processing: Process external data sources or interact with external APIs to build data-driven solutions.

Why Python Subprocess Module?

Do you have any additional Python modules that could assist us in executing the shell commands? Isn’t it true that your answer will be Python’s OS module?

The OS module in Python can execute shell commands from within the python program.

However, the problem with the os.system() function is that it only returns the exit code and does not provide the output of the executed shell command.

>>> import os
>>> os.system('ls')
main.py  requirements.txt  venv

The output of the ‘ls’ command is printed on the terminal, and the only output that the python program receives is the exit code of the command.

import os
out = os.system('ls')
>>> print(out)
0

To overcome this limitation of the ‘os’ module, we can use the pipeline operator to store the output of the command in a text file and then read the file in the python program.

However, this approach is cumbersome, and the ‘subprocess’ module provides an easier and more efficient way to execute shell commands from within the python program.

Example:

import os
os.system('ls > output.txt')
file = open('output.txt','r')
>>> print(file.read())
output.txt
main.py
requirements.txt
venv

The ‘subprocess’ module is a high-level interface for executing shell commands and is intended to replace the following python modules and functions:

1. os.system()
2. os.spawn*()
3. os.popen*()
4. popen2.*()
5. commands.*()

The official Python documentation states that the ‘subprocess’ module will replace these functions as they are going to be deprecated very soon.

Subprocess vs OS module in Python

The Subprocess and OS modules differ from each other in several ways, including:

Now when you know the difference between Python Subprocess and OS Module. So let’s learn more about Subprocess library.

Python subprocess run

The subprocess.run() function is the simplest way to execute a shell command. This is a higher-level function that combines the functionality of Popen() and communicate() functions into a single convenient call.

Syntax:

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, capture_output=False, shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None, text=None, env=None, universal_newlines=None)

Example:

import subprocess
subprocess.run(['ls', '-l'])

Output:

-rw-r--r-- 1 DS 197121 1074 May 22 09:23 LICENSE
-rw-r--r-- 1 DS 197121   91 May 22 09:23 README.md
-rw-r--r-- 1 DS 197121  117 May 22 09:24 subprocess_run.py

The run() function waits for the command to complete and returns a CompletedProcess object that contains the output, return code, and other information about the process.

result = subprocess.run(['ls', '-l'])
print(result) # CompletedProcess(args=['ls', '-l'], returncode=0)
print(type(result)) # <class 'subprocess.CompletedProcess'>

Handle Subprocess Return Codes

When a process finishes its execution, it returns a status code that indicates whether it completed successfully or not.

In Python subprocess, we can obtain the return code of a process using the returncode attribute of the CompletedProcess object.

import subprocess
result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE)
if result.returncode == 0:
    print("Command executed successfully")
else:
    print("Command execution failed")

Python Subprocess run parameters

Let’s take a closer look at some of the important parameters of the run() function:

1. args

import subprocess
result = subprocess.run(['ls', '-l'])
print(result)

The stdout parameter is used to capture the standard output of the command. We can pass a file object or a subprocess.PIPE object to this parameter.

import subprocess
result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE)
print(result.stdout.decode('utf-8'))

3. stderr

The stderr parameter is used to capture the standard error of the command. We can pass a file object or a subprocess.PIPE object to this parameter.

import subprocess
result = subprocess.run(['ls', '-l', 'nonexistent_file'], stderr=subprocess.PIPE)
print(result.stderr.decode('utf-8'))

In the above code, we are running the ls -l nonexistent_file command, which will generate an error because the file does not exist.

4. check

The check parameter is used to specify whether to raise an exception if the command fails to execute.

If check is set to True and the command fails, a CalledProcessError exception will be raised.

import subprocess
try:
    result = subprocess.run(['ls', '-l', 'nonexistent_file'], check=True)
except subprocess.CalledProcessError as e:
    print(type(e)) # <class 'subprocess.CalledProcessError'>
    print(e) # Command '['ls', '-l', 'nonexistent_file']' returned non-zero exit status 2.

5. stdin

The stdin parameter is used to specify the input to the command. We can pass a file object or a string to this parameter.

import subprocess
result = subprocess.run(
    ['grep', 'Hack The Developer'], input=b'Hello, Hack The Developer\n', capture_output=True)
print(result.stdout.decode('utf-8'))

In this example, the command being executed is grep with the argument 'Hack The Developer'. The input parameter is set to b'Hello, Hack The Developer\n', which is the input passed to the grep command.

The capture_output=True parameter instructs the subprocess.run() function to capture the output of the command execution. The result of the command is stored in the result variable.

Finally, result.stdout.decode('utf-8') is used to decode and print the captured output of the command execution. It will display the line 'Hello, Hack The Developer\n' if it matches the specified search pattern.

6. capture_output

The capture_output parameter is a boolean value that specifies whether to capture both standard output and standard error.

If it is set to True, stdout and stderr parameters will be set to subprocess.PIPE.

import subprocess
result = subprocess.run(['ls', '-l', 'nonexistent_file'], capture_output=True)
print(result.stdout.decode('utf-8'))
print(result.stderr.decode('utf-8'))

7. timeout

The timeout parameter is used to specify a timeout in seconds for the command to complete.

If the command does not complete within the timeout period, a TimeoutExpired exception will be raised.

import subprocess
try:
    result = subprocess.run(['sleep', '10'], timeout=5)
except subprocess.TimeoutExpired as e:
    print(e)

8. env

The env parameter is used to specify the environment variables to be used when executing the command.

We can pass a dictionary of environment variables to this parameter.

import subprocess
env = {'PATH': '/usr/local/bin:/usr/bin:/bin'}
result = subprocess.run(['echo', '$PATH'], env=env, shell=True, capture_output=True)
print(result.stdout.decode('utf-8'))

Python Subprocess Popen

The subprocess.Popen function is a more powerful and flexible way to execute shell commands in Python.

Unlike the subprocess.run() function, which runs a command and waits for it to complete, subprocess.

Popen starts a new process and returns a Popen object that can be used to communicate with the process while it is running.

import subprocess
process = subprocess.Popen(['python', '-c', 'print("Hello, world!")'])
print(type(process)) # <class 'subprocess.Popen'>

Popen Methods and Attributes

The <class 'subprocess.Popen'> object returned by subprocess.Popen has several methods and attributes that can be used to communicate with the process.

Some of the most commonly used methods and attributes in subprocess.Popen are:

1. communicate

The communicate method is used to send input to the command and retrieve its output and error streams.

We can pass a string or a bytes object to this method to send input to the command, and it returns a tuple containing the output and error streams.

import subprocess
process = subprocess.Popen(['python', '-c', 'print(input())'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, error = process.communicate(input=b'Test input')
print(output.decode('utf-8'))

2. poll

The poll method is used to check whether the process has completed or not.

It returns None if the process is still running, and its return code if the process has completed.

import subprocess
process = subprocess.Popen(['python', '-c', 'print("Hello, world!")'])
while process.poll() is None:
    print("Process is still running")
print("Process has completed")

In this example, The code inside the while loop is executed until the poll method returns a non-None value, indicating that the process has completed.

3. wait

The wait method is used to wait for the process to complete and retrieve its return code.

It blocks until the process has completed and returns the process’s return code.

import subprocess
process = subprocess.Popen(['python', '-c', 'print("Hello, world!")'])
return_code = process.wait()
print(f"Process has completed with return code {return_code}")

4. send_signal

The send_signal method is used to send a signal to the process. This method takes a signal number as its argument, which can be one of the standard signal numbers defined in the signal module, such as signal.SIGTERM or signal.SIGKILL.

import subprocess
import signal

process = subprocess.Popen(['python', '-c', 'import time; time.sleep(10)'])
process.send_signal(signal.SIGTERM)

5. terminate

The terminate method is used to send the signal.SIGTERM signal to the process, which will cause it to terminate gracefully.

import subprocess

process = subprocess.Popen(['python', '-c', 'import time; time.sleep(10)'])
process.terminate()

6. kill

The kill method is used to send the signal.SIGKILL signal to the process, which will cause it to terminate abruptly.

import subprocess

process = subprocess.Popen(['python', '-c', 'import time; time.sleep(10)'])
process.kill()

Python Subprocess call

The subprocess.call() function is used to run a command in a similar way to the run() function.

However, the call() function does not capture the output of the command. Instead, it returns the return code.

Subprocess Call Syntax

subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False)

The parameters of the call() function are the same as the run() function.

• args: This parameter is a list that specifies the command to run and its arguments.
• stdin, stdout, and stderr: These parameters are used to specify the input, output, and error streams of the command.
• shell: This parameter is used to specify whether to run the command in a shell or not.

Example:

import subprocess
return_code = subprocess.call(['ls', '-l'])
if return_code == 0:
    print("Command executed successfully")
else:
    print("Command execution failed")

Using the shell=True Parameter

By default, the subprocess module does not run commands in a shell. However, we can run commands in a shell by passing shell=True as a parameter to the run() or call() function.

Example:

import subprocess
subprocess.call('echo "Hello World"', shell=True)

Note: Running commands in a shell can be a security risk if the command is constructed from untrusted input. Therefore, it is recommended to avoid using the shell=True parameter whenever possible.

Python Subprocess Check Output

The subprocess.check_output() function is used to execute a command and capture its output. It also takes a list of arguments that represent the command to be run.

This function is similar to the run function, but it only captures the standard output of the command, not its standard error. If the command fails, it raises a CalledProcessError exception.

Subprocess check_output syntax

subprocess.check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None, encoding=None, errors=None)

Parameters:

• args: A string or a sequence of arguments to be executed as a command
• stdin: Specifies the input for the command. It can be a string, a byte string, or a file object
• stderr: Specifies where to redirect the standard error of the command. It can be a file object or a subprocess.PIPE object
• shell: If shell is True, the command will be executed through the shell
• universal_newlines: If universal_newlines is True, the function will return a string instead of bytes
• timeout: Timeout for the command execution in seconds
• encoding: Specifies the character encoding for the input and output streams
• errors: Specifies the error handling for the input and output streams

Example:

import subprocess
output = subprocess.check_output(['ls', '-l'])
print(output.decode('utf-8'))

In the above code, we are running the ls -l command and capturing its output using the check_output function. We are decoding the output as a string using the decode() method and printing it.

Handling Exceptions in Subprocess check_output

If the command fails, the check_output function raises a CalledProcessError exception.

This exception contains the return code of the command and its standard error output.

Example:

import subprocess
try:
    output = subprocess.check_output(['ls', 'nonexistent'])
except subprocess.CalledProcessError as e:
    print("Command execution failed with return code", e.returncode)
    print("Error output:", e.output.decode('utf-8'))

Input Output (I/O) Streams in Subprocess

In Python subprocess, we can interact with the standard input, output, and error streams of the process using the stdin, stdout, and stderr parameters respectively.

These parameters can be passed to the Popen function to specify the input/output streams of the process.

Standard Input

To handle standard input in a subprocess, we need to use the stdin parameter of the Popen function.

The stdin parameter can take a file object or a subprocess.PIPE object as its value.

If we want to pass input to the subprocess interactively, we can use the subprocess.PIPE object.

Example:

import subprocess

process = subprocess.Popen(['python', '-c', 'print(input("Enter your name: "))'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
output, _ = process.communicate(input=b'John Doe\n')
print(output.decode('utf-8'))

Standard Output

To handle standard output in a subprocess, we need to use the stdout parameter of the Popen function.

The stdout parameter can take a file object or a subprocess.PIPE object as its value. If we want to capture the output of the subprocess, we can use the subprocess.PIPE object.

Example of how to handle standard output in Python subprocess using subprocess.PIPE:

import subprocess

process = subprocess.Popen(['python', '-c', 'print("Hello, World!")'], stdout=subprocess.PIPE)
output, _ = process.communicate()
print(output.decode('utf-8'))

Standard Error

To handle standard error in a subprocess, we need to use the stderr parameter of the Popen function.

The stderr parameter can take a file object or a subprocess.PIPE object as its value. If we want to capture the error output of the subprocess, we can use the subprocess.PIPE object.

Example:

import subprocess

process = subprocess.Popen(['python', '-c', 'print(1/0)'], stderr=subprocess.PIPE)
_, error = process.communicate()
print(error.decode('utf-8'))

Decode Standard Streams in Subprocess

When we execute shell commands using Python subprocess, we often need to handle the output and error streams of the command.

By default, these streams are returned as bytes. However, it is often more convenient to work with them as strings. To do this, we need to decode the streams.

Python provides several encoding options for decoding standard streams. Some of the commonly used encoding options are:

1. utf-8: This is the most commonly used encoding option. It is a variable-length encoding that can represent any Unicode character. It is widely supported and is the default encoding for most platforms.
2. ascii: This is a 7-bit encoding that can represent only a limited subset of characters. It is commonly used in the United States and other English-speaking countries.
3. latin-1: This is a 8-bit encoding that can represent any character in the ISO-8859-1 character set. It is commonly used in Western Europe.

To decode the standard streams, we can use the decode() method of the bytes object.

import subprocess
process = subprocess.Popen(['ls'], stdout=subprocess.PIPE)
output = process.stdout.read()
output_str = output.decode('utf-8')
print(output_str)

Encode Input in Subprocess

When we pass an input to a subprocess, we need to ensure that it is encoded in a format that the subprocess can handle. This is important because different operating systems and subprocesses may have different encoding requirements.

In Python 3, strings are Unicode by default, which means that they can contain characters from any language. However, subprocesses may not be able to handle Unicode strings, and may expect input in a different encoding, such as ASCII or UTF-8.

To ensure that the input is encoded correctly, we can use the encode() method of the string object to convert it to a byte string. We can then pass this byte string to the subprocess.

import subprocess

input_str = "Hello, World!"
input_bytes = input_str.encode('utf-8')

process = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
output, error = process.communicate(input=input_bytes)

print(output.decode('utf-8'))

Execute External Program using Subprocess

Python subprocess module can be used to execute C and JAVA code in addition to shell commands.

Now, we are going to explore how to run C and JAVA code using Python subprocess module.

Execute C Code in Python

To execute C code using Python subprocess module, we need to compile the code first using the GCC compiler.

We can do this by running the gcc command with the source file name and the output file name.

Once the code is compiled, we can run the output file using the subprocess module.

import subprocess

# Compile the C code
subprocess.run(["gcc", "hello.c", "-o", "hello"])

# Run the compiled code
result = subprocess.run("./hello", stdout=subprocess.PIPE)

# Print the output
print(result.stdout.decode("utf-8"))

Execute JAVA Code in Python

To execute JAVA code using Python subprocess module, we need to compile the code first using the javac compiler.

We can do this by running the javac command with the source file name.

Once the code is compiled, we can run the class file using the subprocess module.

import subprocess

# Compile the JAVA code
subprocess.run(["javac", "Hello.java"])

# Run the compiled code
result = subprocess.run(["java", "Hello"], stdout=subprocess.PIPE)

# Print the output
print(result.stdout.decode("utf-8"))

Python Subprocess Best Practices

Python subprocess module provides a powerful and flexible way of executing shell commands in Python.

However, as with any programming tool, there are some best practices that should be followed to ensure that the code is secure, efficient, and maintainable.

Use a list of arguments instead of a string

When passing arguments to the Popen constructor or run function, it is recommended to use a list of arguments instead of a string.

This is because a list of arguments is more secure than a string. If you use a string, it may be vulnerable to shell injection attacks.

import subprocess
args = ['ls', '-l']
process = subprocess.Popen(args, stdout=subprocess.PIPE)
output, error = process.communicate()
print(output.decode('utf-8'))

Use full paths for executables

When running an executable file with subprocess, it is recommended to use full path of the executable instead of just name.

This is because system may have multiple executables with same name, and if you just use name, wrong executable may be executed.

import subprocess
args = ['/usr/bin/ls', '-l']
process = subprocess.Popen(args, stdout=subprocess.PIPE)
output, error = process.communicate()
print(output.decode('utf-8'))

Use absolute paths for files

When passing file paths to Python subprocess, it is recommended to use absolute paths instead of relative paths.

This is because the current working directory may change, and if you use a relative path, the wrong file may be accessed.

import subprocess
args = ['cat', '/home/user/file.txt']
process = subprocess.Popen(args, stdout=subprocess.PIPE)
output, error = process.communicate()
print(output.decode('utf-8'))

Use communicate() function to capture output

When capturing the output of a command with Python subprocess, it is recommended to use the communicate() function instead of reading directly from the stdout or stderr file objects.

This is because the communicate() function waits for the command to finish before returning the output, which ensures that all the output is captured.

import subprocess
args = ['ls', '-l']
process = subprocess.Popen(args, stdout=subprocess.PIPE)
output, error = process.communicate()
print(output.decode('utf-8'))

Use shell=False whenever possible

When executing a command with Python subprocess, it is recommended to set the shell parameter to False whenever possible.

This is because when shell is set to True, it opens up the possibility of shell injection attacks.

import subprocess
args = ['ls', '-l']
process = subprocess.Popen(args, stdout=subprocess.PIPE, shell=False)
output, error = process.communicate()
print(output.decode('utf-8'))

Wrapping Up

In conclusion, Python subprocess is a powerful module that makes it easy to execute shell commands from within a Python script. It allows you to interact with system processes, run commands with arguments, and capture their output.

We have explored various subprocess functions, including Popen, call, check_call, and check_output, and learned how to use them to perform a wide range of tasks, from executing simple commands to more complex operations.

Subprocess is an essential tool for any Python developer who needs to work with shell commands and system processes. By leveraging the power of subprocess, you can easily integrate system-level functionality into your Python scripts and automate your workflow. With its rich set of features and versatility, subprocess is a must-have module for any serious Python programmer.

References

1. Python Subprocess documentation