I’ve had numerous cases where I’ve needed to execute bash commands and return the results in Python for some additional manipulation. My most recent example involved me working with BigQuery schemas. Getting schemas from the CLI was easy peasy but I’m not the best at BASH programming so I naturally turned to something more familiar for the manipulation. I’ll also use the BigQuery CLI for this example.
My Setup
This isn’t too important. Python behaves fairly similar across platforms. We’re not working with files, but regardless, here’s my setup
- MacBook M1 Chip
- Python 3.9
Package
We’ll use the subprocess package. This is built into Python so you shouldn’t need to PIP install anything
Solution
Execute Bash Commands and Return Results in Python
import subprocess
import json
response = subprocess.check_output("bq show --schema --format=prettyjson yourProject:yourDataset.yourTable", shell=True)
goodData = json.loads(response)
Trinket Example – Echo JSON and Load into Python Variable
Gotchas
CLI Returns String Data
There are a couple of gothcas when executing bash commands and returning the results in Python. My biggest one was that my cli would return data as a string instead of JSON for easy manipulation. I used json.loads() on the data my cli returned to get the data into something workable.
Check for Errors!
In some cases my CLI would error out so make sure to wrap your cli call in a try: catch: and handle for potential failures. This a great rec for any api calls, cli calls, and just typical good form. You might be iterating through something huge and might need keep track of where you are so a random failure mid iteration might be a costly mistake.
Other Usecases
There are zillions of usecases. I’d probably look into if there’s a Python client before resorting to executing bash commands. Sometimes I’m lazy and don’t want to find the Python client equivalent and only care about time.
And that’s pretty much it! Feel free to reach out if you have any questions. Happy coding folks! Need more practice? Check out this blog post that covers dropping dups from arrays.