Defensive Scripting with bash

DEFENSIVE PROGRAMMING IN BASH Bash is a very powerful tool, but it is important to employ strategies that catch errors early, provide clear logging, and handle unexpected situations cleanly. The scripts you write may be subjected to unexpected behaviours and events such as a required directory not present, a required shell command not present, or, commonly, permission errors. I have been writing Bash scripts for a while, and I have gathered the best techniques from structuring scripts to seamless error handling, which we will explore: some key practices to enhance reliability of your scripts. This guide assumes prior exposure to Bash. Structuring a Bash script Bash being a flexible language as it is follows no strict structure, but here is the one I recommend. #!/bin/bash # global variables immutable and referenced by whole script readonly PROGNAME=$(basename "$0") # sourced script source my_script.sh # program logic grouped into functions do_something() { # logic return $? } main() { # main procedure return $? } # calling main main || { printf "program failed" >&2 exit 1 } export vs local vs global - Try to keep global variables to a minimum - SHOUTING_SNAKE_CASE VAR NAMING - readonly declaration - use variable names to replace cryptic $1 Globals I recommend you always use in your programs: readonly PROG_NAME=$(basename "$0") readonly PROG_DIR=$(readlink -m $(dirname "$0")) # or even better export SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" Sourcing and splitting a script Splitting the logic of your script into multiple scripts will show easier to manage as the script gets larger. All other variables should be local. #!/usr/bin/env bash change_owner_of_file() { local filename=$1 local user=$2 local group=$3 chown $user:$group $filename } Modular coding, functions and arguments Grouping procedures into functions and specifying its return code in case of failure or success is the way to go. Some distinctive functions I use are: info() -- for logging successful events - track all changes made by script, especially cron jobs #!/bin/bash info() { echo "[INFO] [$TIME] $" >> "$info_logs" } # or you can use separate info files for each event, but that might take up space when doing scripting on small-space devices like Raspberry Pi boards error() -- for logging failed events and warnings - error logging very important for building daemons - easily get reason why script may have failed mid-run #!/bin/bash error() { echo "[ERROR] [$TIME] $" >> "$err_logs" } main() -- single point of entry for your program - keeps code clean - procedures become descriptive main() { # check req files # check req dirs # install missing deps # confirm installation # run data collections # run inference engine and generate results # run decision engine and generate results # run tui if -q flag isn't given, which just pretty-print results and decision # exit check_req_files || { printf "Required files missing" >&2 exit "$ERR_FAILURE" } check_req_dirs || { printf "Required directories missing" >&2 exit "$ERR_FAILURE" } parse_args "$@" # installing missing dependencies install_missing_deps # running data collections write_cpu_json & write_memory_json write_thermal_json write_disk_json write_sound_json # run engine python3 "$ENGINE_DIR"/inference.py && python3 "$ENGINE_DIR"/decision.py display_tui || { exit "$ERR_FAILURE" } exit "$ERR_SUCCESS" } main "$@" # code taken from sysk Commenting From my experience, Bash syntax isn't really presentable. It gets messy fast as the program expands. All the cryptic ?&&^^ symbols become confusing, because a single symbol can be used in 2 or more cases representing different things. A simple comment # this does this will go a long way. giving strict instructions: # DONT TOUCH THIS # USE PRINTF NOT ECHO Wrapping lines When using a code editor like VS Code, they give you automatic wrap-line options. Yes, it works, but when migrating the code to another IDE it breaks. So if code is too long, wrap with the \ . Also makes script look clean. jq -n \ --arg name "$names" \ -f "$file" /dev/null and /dev/zero Think of /dev/null as a black hole. It is essentially the equivalent of a write-only file. Everything written to it disappears. Attempts to read or output from it result in nothing. Suppressing stdout. cat $filename >/dev/null # Contents of the file will not list to stdout. It can be very useful when you need a command to run without any output. Debugging Bash has no built-in debugger, but there are many workarounds. - using shellcheck - ShellCheck is one of the best ways to write a safe script. Sometimes it might be annoying to use because of the many options, but a script which can pass through ShellCheck successfully has probably no syntax error. But it can't catch runtime errors or logic errors. - using set -x andset +x - this echoes each line before producing output - use to debug a particular piece of code where you feel the error might be from set -x echo "use case" >> $file set +x - using sh -n scriptname - catches syntax errors without executing script - some syntax errors may escape Handling failures and testing using if statements Setting error-handling options Bash is set by default to run to completion even when it encounters an error. #!/bin/bash echo "program starts" ech "program continues" # command doesn't exist echo "program end" Running this gives: program starts ./test.sh: line 3: ech: command not found program end This and other common unexpected behaviours can be avoided using shell options. - set -o errexit orset -e : this option ensures that the script exits when any command returns a non-zero status unless handled with|| - set -o nounset orset -u : ensures the script treats unset variables as errors - set -o pipefail : this option causes the pipeline to fail if any command within it fails These commands can be combined as set -euo pipefail . #!/bin/bash set -euo pipefail echo "program starts" ech "program continues" # command doesn't exist echo "program end" Running this we get: program starts ./test.sh: line 4: ech: command not found Some problems with the set -euo pipefail While it might seem very convenient to use the guard line, there are some unexpected behaviours which might be unwanted, especially with the set -e flag, which is very problematic. - Conditions disable set -e - the test of if, while and until - on the left of && and|| - inverted with the ! flag false && echo never # false fails but script continues false && echo this runs - Command substitution and subshells don't automatically inherit the set -e flag: set -e result=$(false) echo "gets here" # still successfully runs This can be fixed with shopt -s inherit_errexit shell option (Bash 4.4+). - local ,export andreadonly : they mask return values so they always succeed even if substituted command fails export answer=$(find /bin/bash) # bad # better declare word=$() export word This can be fixed by declaring and assigning on separate lines. This demonstrates set -e can give a fake sense of safety. Explicitly catching and handling errors As shown above, the shell options are a good way to perform error handling in a script, but they are not perfect. That said, the best way to handle errors is to use conditionals. In Bash, an exit status of 0 represents success; any other represents a failure, but there are special exit codes that specify a particular event. The exit code of the last command to run is stored in the $? special variable. echo "hello world" echo "$?" # outputs 0 # some special return codes readonly ERR_SUCCESS=0 # operation successful readonly ERR_FAILURE=1 # operation failed general readonly ERR_PERMISSION_DENIED=126 # permission denied readonly ERR_COMMAND_FAILED=126 # command runs but fails readonly ERR_NOT_FOUND=127 # command or file not found readonly ERR_BAD_USAGE=2 # bad use of script or function - Catching these codes with || and&& is the way to go. find "$file" || { echo "no file with name $file" exit 127 } Handling unexpected exit events using traps Trap exit events to perform cleanup tasks or handle unexpected trap [command] [signals] Let's take a script designed to run non-stop for 60 seconds, maybe performing data movement or log parsing, and we don't want any interruption, maybe from Ctrl + C . #!/bin/bash trap "echo Booh!" SIGINT SIGTERM echo "pid is $$" while true; do sleep 60 # script last 60 seconds done You can do more research on traps. Conclusion Following these practices will ensure to a certain level that your Bash script will be robust. Reference Note: this article summarises my discoveries with Bash. There are many pitfalls and gotchas which have not been mentioned here. Remember there is no perfect guide to Bash. The best way to learn is to practice and encounter the errors yourself. Thanks for reading to this level. If you have any doubts, please don't hesitate to share in the comments. Happy scripting. Top comments (0)