# Writing better Bash scripts A series of 10 article(s), in reading order. # Bash - ASCII art > Learn how to add eye-catching ASCII art banners to your Bash scripts for a bigger visual impact. Includes a full sample code implementation. --- ![Bash - ASCII art](/img/v2/ascii_art.webp) **TL;DR** This short article shows how to add an ASCII-art banner to Bash scripts for a bit of visual flair, generated with the free patorjk.com/software/taag tool and echoed at the top of the script. I write a lot of Bash scripts, and I like to have a similar approach for each one: a banner, [a logging function](/blog/bash-logging), [doc blocks above each function](/blog/linux-generate-documentation-from-bash-scripts). One of the things I always do is to include a good old-fashioned *ASCII Art* banner, perhaps for the geek factor, but mostly to make a bigger visual impact. I use to create my banners, so let's take a closer look. *The same idea, applied to a website instead of a console: [Inject ASCII Art in any HTML pages rendered by Docusaurus](/blog/docusaurus-ascii-art).* ![Sample ASCII art](./images/sample.webp) For my part, I'm implementing the banner like this: **script.sh** ```bash #!/usr/bin/env bash set -euo pipefail function __main() { clear # http://patorjk.com/software/taag/#p=display&f=Big&t=My%20Nice%20Script cat <<\EOF __ __ _ _ _ _____ _ _ | \/ | | \ | (_) / ____| (_) | | | \ / |_ _ | \| |_ ___ ___ | (___ ___ _ __ _ _ __ | |_ | |\/| | | | | | . ` | |/ __/ _ \ \___ \ / __| '__| | '_ \| __| | | | | |_| | | |\ | | (_| __/ ____) | (__| | | | |_) | |_ |_| |_|\__, | |_| \_|_|\___\___| |_____/ \___|_| |_| .__/ \__| __/ | | | |___/ |_| EOF printf "%s\n\n" "(c) Copyright ..." # Program the functions of your script here, # such as calling your functions return 0 } __main "$@" ``` And this is how it'll look in my bash terminal: ![Terminal](./images/terminal.webp) --- # Bash - Loading environment variables from a file > Learn how to load environment variables from a .env file into a Bash script, so you can externalize configuration and reuse variables across other applications like Laravel. --- ![Bash - Loading environment variables from a file](/img/v2/bash.webp) **TL;DR** This article shows the correct way to load a `.env` file's variables into a Bash script's environment: `set -o allexport; source .env; set +o allexport`, which handles values containing spaces correctly — unlike the common but unreliable `export $(... | xargs)` pattern. Imagine you have a `.env` file like this one and you wish to process that file in a Bash script. **.env** ``` DOCKER_GIT_USEREMAIL="christophe@me.com" DOCKER_GIT_FULLNAME="Christophe Avonture" DOCKER_GIT_USERNAME="Me and myself" ``` Using a configuration file will enable you to externalize the management of your constants, as well as reuse variables from another application, such as a site developed in Laravel. Let's take a look at how to do this as correctly as possible. The snippet comes from You can load that file easily in your environment using the following instructions: ```bash set -o allexport source .env set +o allexport ``` This done, variables will be accessible like any environment variables in your bash script. When the script exits, the added variables are removed (just like in a sub-shell). Using source is the best solution to avoid problems with f.i. spaces like in *Me and myself* i.e. using other solutions like `export $(... | xargs)` will always give unpredictable results. To illustrate this, simply create a file called f.i. `test.sh` with this content: **test.sh** ```bash #!/usr/bin/env bash set -o allexport source .env set +o allexport echo "${DOCKER_GIT_FULLNAME} (${DOCKER_GIT_USEREMAIL})" ``` Using the `.env` file provided here above, we'll obtain this output: ```bash $ ./test.sh Christophe Avonture (christophe@me.com) ``` This is, I think, the best and, too, the easiest way to use an external file to store configuration items in Bash. Two more `.env` companions from this blog: [Compare environment files in the Linux console](/blog/compare-env-files-cli), to spot the key you forgot to report from `.env.example`, and [Batch edit of environment file](/blog/update-env-files-cli), to change the same variable in a dozen projects at once. --- # Bash - Script to add logging features to your script > Add robust logging features to your Bash scripts with this simple, reusable library. Get automatic log files, timestamps, and full function call traces for easy debugging. --- ![Bash - Script to add logging features to your script](/img/v2/bash.webp) **TL;DR** This article shares a reusable `log.sh` Bash library that adds logging to any script: source it, call `log::write "message"` wherever needed, and get timestamped log entries automatically written to a logfile — each entry includes a full call trace (which function, on which line, called from where) to make debugging non-interactive/cron scripts easier. > Also read [Bash - Echo on the console and in a logfile in the same time](/blog/bash-console-log-together) When you write Bash scripts and certainly when you foresee running them in a cron, you should implement a logfile. Every action fired by your script should be logged somewhere so you can start the script in a non-interactive mode and in case of need, consult the last logfile. *Logging becomes indispensable as soon as your script runs [several jobs in parallel](/blog/bash-parallel-task): the console output of ten concurrent tasks is unreadable, a timestamped logfile is not.* Below is a script I've developed in the form of a library, which means you can easily include it in your existing code without having to change anything. You just need to include the file in your script (i.e. add a `source log.sh` line) then here and there foresee a `log::write "Something to log"`. Easy no? To illustrate how the log feature works, we will create two files. Your script (here, I'll call it `run.sh`) and the one containing the log implementation (`log.sh`). ## The result By running the script, we can see our different `echo` statements, but also the content of the generated log. ![Using log](./images/logging.webp) You can, of course, choose not to display the log automatically — just comment out the line `log::__displayLog`. ### Trace Each statement logged will have a trace as you can see. For instance, the block `[Function __main line 43]` means that the caller was function called `__main` and it was on line `43`. ```log [2024-03-25T12:29:08+0100] Start ./run.sh [2024-03-25T12:29:08+0100] Running main with these parameters: --help [Function __main line 43] [2024-03-25T12:29:08+0100] Doing some stuff in test2 [Function test2 line 22;Function test1 line 30;Function __main line 48] [2024-03-25T12:29:08+0100] I'm doing something in the test5 function... [Function test5 line 4;Function test4 line 10;Function test3 line 16;Function test2 line 24;Function test1 line 30;Function __main line 48] [2024-03-25T12:29:08+0100] Logfile /tmp/run.sh.uplzaefiWq.log ended for ./run.sh. [2024-03-25T12:29:08+0100] Duration: 0 second(s) ``` When there are multiple parents, you'll get the entire trace, f.i. `[Function test5 line 4;Function test4 line 10;Function test3 line 16;Function test2 line 24;Function test1 line 30;Function __main line 48]`: the function `__main` on line 48 called `test1`, which then called `test2`, and so on. Each statement in the log will then contain the entire trace, making debugging easier. ## Your script If you already have a script, please open it. Otherwise, just create a new one and copy/paste the code below. Name the file just as you want (I will name it `run.sh` for example) and make sure the script can be executed by running `chmod +x run.sh` in your console. **run.sh** ```bash #!/usr/bin/env bash set -o nounset -o errexit function test5() { log::write "I'm doing something in the test5 function..." return 0 } function test4() { test5 return 0 } function test3() { test4 return 0 } function test2() { log::write "Doing some stuff in test2" test3 return 0 } function test1() { test2 return 0 } function __init() { source ./log.sh return 0 } function __main() { __init log::write "Running main with these parameters: $*" echo "Welcome in this script" echo "Please do ..." test1 return 0 } __main "$@" ``` ## The log helper Please create a file called `log.sh`, in the exact same folder of your script, and copy/paste this content — that's what produces the trace and log file shown above. **log.sh** ````bash #!/usr/bin/env bash # region - Intro block ----------------------------------------------- # # Working with log files # # ## How to use this script? # # 1. Include the script in your own Bash like this `source log.sh` (adjust path if needed). # 2. Here and there, where you want a log, just type `log::write "This is a log info"` # # ## Dependencies # # None # # endregion - Intro block -------------------------------------------- # let script exit if an unset variable is used or if a command fails set -o nounset set -o errexit # region - private function log::__main # # ## Description # # This function will initialize important global variables # This function is called as soon as this file is "sourced" (like "source log.sh") # to make sure variables are initialized as soon as possible. # # ## Examples # # ```bash # log::__main # ``` # # ## Exit code # # * 0 If successful. # # endregion function log::__main() { # "trap ... EXIT" allow to define a function to call when the script exits, # so, here, call our __logDestruct function trap log::__logDestruct EXIT # Example of a filename: /tmp/myScript.sh.1J6Sqb62.log # Set LOG_FILE="" if you don't want to write to a log file (but simpler to not source this file then) [[ ! -v LOG_FILE ]] && LOG_FILE="$(mktemp /tmp/"$(basename "$0")".XXXXXXXXXX.log)" # Quick function to export the system date in the chosen format. log::__logNow() { date +"%Y-%m-%dT%H:%M:%S%z"; } # Generate a trace dump like [test5,line 728;test4,line 731;test3,line 734;test2,line 737;test1,line 740;main,line 755] # i.e. the latest called function, then the one who called that function, ... until the very first function called # (the entry point function) log::__trace() { local traceInfo traceInfo="" LEN=${#BASH_LINENO[@]} # INDEX=0 is this __trace function, INDEX=1 is the log::write function so start at 2. for (( INDEX=2; INDEX < LEN-1; INDEX++ )); do traceInfo="${traceInfo}Function ${FUNCNAME[(( INDEX ))]} line ${BASH_LINENO[ (( INDEX - 1)) ]};" done traceInfo="$(echo "${traceInfo}" | sed 's/;$//g')" echo "[${traceInfo}]" return 0 } # From now, writing to 3 will write to the log file [[ -z "${LOG_FILE}" ]] || exec 3>"${LOG_FILE}" return 0 } # region - private function log::__logDestruct # # ## Description # # Before exiting the script, display or not the logfile content (and remove it then) # # ## Examples # # ```bash # log::__logDestruct # ``` # # ## Exit code # # * 0 If successful. # # endregion function log::__logDestruct() { # Only if the file isn't empty if [[ -s ${LOG_FILE} ]]; then logEntry="$(printf '[%s] Logfile %s ended for %s.' "$(log::__logNow)" "$LOG_FILE" "$0")" echo "${logEntry}" >&3 logEntry="$(printf '[%s] Duration: %d second(s)' "$(log::__logNow)" "$SECONDS")" echo "$logEntry" >&3 log::__displayLog fi return 0 } # region - public function log::write() # # ## Description # # Write a sentence to a logfile # # ## Examples # # ```bash # log::write 'This is a log entry' # ``` # # ## Parameters # # * @arg string The string to write to the logfile # # ## Exit code # # * 0 If successful. # * 1 An error has occurred. # # endregion function log::write() { # When LOG_FILE was not defined, it means no log so just exit [[ -z ${LOG_FILE} ]] && return 0 local -r message="${1:-}" [[ -z ${message} ]] && return 0 # No message so nothing to write in the log # Create the log file if not yet there [[ ! -f ${LOG_FILE} && ! "${LOG_FILE}" = "/dev/stdout" ]] && touch "${LOG_FILE}" # If the file isn't writable, error and return if [[ ! -w ${LOG_FILE} ]]; then errorMessage="ERROR - The ${LOG_FILE} file isn't writable. Exiting..." printf "\033[1;${__RED:-31}m%s\033[0m\n" "ERROR - ${message} Exiting..." >&2 # Write on stderr fi # If the file size is 0, echo a start info date/time; do it once if [[ ! -s ${LOG_FILE} ]]; then printf '[%s] %s\n' "$(date +"%Y-%m-%dT%H:%M:%S%z")" "Start $0" >&3 fi printf '[%s] %s %s\n' "$(date +"%Y-%m-%dT%H:%M:%S%z")" "${message}" "$(log::__trace)" >&3 return 0 } # region - private function log::__displayLog # # ## Description # # Display the content of the logfile # # ## Examples # # ```bash # log::__displayLog # ``` # # ## Exit code # # * 0 If successful. # # endregion function log::__displayLog() { # If the file is empty, nothing to do [[ ! -s ${LOG_FILE} ]] && return 0 printf "%s\n" "--------------------- LOG FILE CONTENT (${LOG_FILE}) ---------------------" # Show the log content but make sure to remove, if any, ANSI colors. # (@see https://stackoverflow.com/a/51141872/1065340) printf "\033[1;${__GRAY:-30}m%s\n" "" sed "s/\x1B\[[0-9;]\{1,\}[A-Za-z]//g" "${LOG_FILE}" printf "\033[0m\n\n" echo "" return 0 } # This script didn't contains executable code; only helpers. sourced=0 (return 0 2> /dev/null) && sourced=1 || sourced=0 if [ "${sourced}" -eq "0" ]; then printf "\033[1;31m%s\033[0m\n" "ERROR, the $0 script is meant to be sourced. Try 'source $0' and use public functions." >&2 # Write on stderr exit 1 fi log::__main ```` --- # Bash - Echo on the console and in a logfile in the same time > Tired of silent Bash scripts? Learn the simple, effective method to simultaneously echo command output to the console and write it to a log file line-by-line. --- ![Bash - Echo on the console and in a logfile in the same time](/img/v2/bash.webp) **TL;DR** This article shows how to make a long-running Bash command output to both the console and a log file simultaneously (rather than only one or the other), using `eval` to run the command and echoing each line to the screen while also appending it to the logfile in real time — instead of buffering everything and dumping it at the end. In my previous article [Bash - Script to add logging features to your script](/blog/bash-logging), I've shared a way to write information in a logfile. By running `ls -alh /tmp` you will get the list of all files in the `/tmp` folder and display the list on your console. By running `ls -alh /tmp >> application.log` you won't see the list in your console since everything will be written in the `application.log` file. How can we display the output of a command like `ls` f.i. both on the console and in a logfile? ## Result ```bash $ ./script.sh total 8.0K drwxr-xr-x 2 node node 4.0K Aug 9 18:04 . drwxr-xr-x 3 node node 4.0K Aug 9 18:04 .. -rw-r--r-- 1 node node 0 Aug 9 18:04 archive.zip -rw-r--r-- 1 node node 0 Aug 9 18:04 notes.txt -rw-r--r-- 1 node node 0 Aug 9 18:04 report.csv $ cat /tmp/test.log [2026-08-09T18:04:09+0000] total 8.0K [2026-08-09T18:04:09+0000] drwxr-xr-x 2 node node 4.0K Aug 9 18:04 . [2026-08-09T18:04:09+0000] drwxr-xr-x 3 node node 4.0K Aug 9 18:04 .. [2026-08-09T18:04:09+0000] -rw-r--r-- 1 node node 0 Aug 9 18:04 archive.zip [2026-08-09T18:04:09+0000] -rw-r--r-- 1 node node 0 Aug 9 18:04 notes.txt [2026-08-09T18:04:09+0000] -rw-r--r-- 1 node node 0 Aug 9 18:04 report.csv ``` Same lines, twice: once printed live on the console, once appended to `/tmp/test.log` — proof the two outputs really happen at the same time, not one after the other. For the illustration above, the script listed a small demo folder instead of `/tmp` for readability; in practice you'd point it at whatever long-running command you need to trace. ## Why it works The command runs through `eval`, and its output is piped into a `while read` loop. For every single line that comes out, the loop does two things immediately, before reading the next line: `echo` it to the console, then append it to the logfile. Nothing is buffered and dumped at the end — that's what keeps the two outputs in lockstep. For example, let's get the list of all files below `/tmp` and this recursively. For this, our command will be `ls -alhR /tmp`. This command will probably take time to run and this is the objective: run a long command and make sure our user will see the output on the screen so they know that the script is doing something, and also redirect the output to a log file for further analysis in case of f.i. problems (or just because the script is fired in a cron). *If "the script is doing something" is all your user needs to know, a [progression bar](/blog/bash-progression-bar) is often nicer than a wall of scrolling output.* **script.sh** ```bash #!/usr/bin/env bash set -euo pipefail declare -r command="ls -alhR /tmp" declare -r LOGFILE="/tmp/test.log" touch "${LOGFILE}" function log::write() { printf '[%s] %s%s\n' "$(date +"%Y-%m-%dT%H:%M:%S%z")" "${line}" >>"${LOGFILE}" return 0 } eval "${command}" | while IFS= read -r line; do echo "${line}" # Output on the console (stdout) log::write "${line}" # And append in the log file done ``` The main part is the `eval` function. The command (`ls -alhR /tmp`) will be fired and, for every single line, we'll `echo` the line on the screen and, also, redirect it in a text file. The example below does almost the same thing i.e., it executes a command and displays it both in a file and on the console but ... as you can see, the command will be run and everything will be redirected to the file so, for several seconds, the user will have no screen output and might think the script is stuck. Then, once the command has successfully finished, and only then, the log file will be displayed, in a block, on the console. **script.sh** ```bash #!/usr/bin/env bash set -euo pipefail declare -r command="ls -alhR /tmp" declare -r LOGFILE="/tmp/test.log" eval "${command}" > "${LOGFILE}" cat "${LOGFILE}" ``` Don't do things like that! --- # Linux - Compare two versions of the same script > Use a simple Linux command-line trick with diff, grep, and awk to compare two versions of the same Bash script. Quickly identify missing or new functions across file versions or entire folders. --- ![Linux - Compare two versions of the same script](/img/v2/bash.webp) **TL;DR** This article shows how to compare two versions of the same Bash script by extracting each file's function names with `grep`/`awk`, sorting them, and diffing the two sorted lists with `diff --side-by-side` — revealing which functions were added, removed, or renamed regardless of their order in the file. It's then extended into a `compare.sh` script that runs the same comparison across every script in two folders. Still faced with the problem of having to compare two versions of the same script (see the article [Linux - Comparing two folders/files in the console](/blog/linux-diff-file-folder)), this time we're going to consider that the file we need to compare is a Bash script and that we have two versions of it. And that we may have made the two files evolve in different ways, i.e. that one or other, or even both, of the files may have been modified. The aim is therefore to compare the versions and highlight the differences. ## Result ```bash {` ( FILE1=/tmp/bash/console.sh FILE2=/tmp/bash/console_v2.sh printf "\e[33;1m%-39s %s\e[0;1m\n" "Left side: \${FILE1}" "Right side: \${FILE2}" diff --side-by-side --width 83 \ <(grep -P "^(function\s+.*)\(\)" "\${FILE1}" | awk '{print \$2}' | sort) \ <(grep -P "^(function\s+.*)\(\)" "\${FILE2}" | awk '{print \$2}' | sort) ) `} ``` ![Compare the two versions of the same Bash script and shows which functions are in one and not the other](./images/compare_functions.webp) Two sorted function lists, side by side: whatever's only on one side is a function the other version doesn't have — regardless of where in the file it was declared. ## Create some files for this blog post To be able to reproduce examples used in this blog post, if you need it, please create two files. Let's say `/tmp/bash/console.sh` for the first file with this content: **/tmp/bash/console.sh** ```bash #!/usr/bin/env bash function console::printCyan() { } function console::askYesNo() { } function console::printRed() { } function console::printError() { } function console::banner() { } ``` And, for `/tmp/bash/console_v2.sh`, we'll add two new functions, `printGreen` and `printBlue`, in that order: **/tmp/bash/console\_v2.sh** ```bash #!/usr/bin/env bash function console::printCyan() { } function console::askYesNo() { } function console::printRed() { } function console::printError() { } function console::banner() { } function console::printGreen() { } function console::printBlue() { } ``` ## Get the list of functions in a Bash script To get the list of functions declared in a script (let's say `/tmp/bash/console.sh`), just run the command below: ```bash $ `grep -P "^(function\s+.*)\(\)" "/tmp/bash/console.sh" | awk '{print \$2}' | sort` ``` This function will retrieve any functions in the file i.e. lines starting with the word `function` followed by any whitespace character then the name of the function then followed by parenthesis. And, finally, sort the list: ![Get the list of functions in a Bash script](./images/functions_list.webp) ## Compare the two versions of the same Bash script and shows which functions are in one and not the other Regardless of the order in which the functions are declared, the script below, which you can copy and paste into the console, will find the list of functions in the two files, sort the list and compare which functions are in one version and which are not in the other — that's the command and result shown at the top of this article, where we immediately see that the second file has two additions. Of course, if we add a new function in `console.sh` and f.i. remove an existing one, we can see it too: ![Compare the two versions of the same Bash script and shows which functions are not the same in both files](./images/compare_functions_both_side.webp) On the image here above, we see three indicators in the middle of the screen: ```diff > console::printBlue() console::printPurple() | console::printGreen() console::verbose() | console::printRed() ``` The line with the `>` indicator means that `console::printBlue()` was found only in the second file so we know our first file didn't declare `printBlue`. And lines with the `|` indicator highlight that we've a mismatch. On both sides we had sorted our list and so, if the same function had been on both sides, we would have had a match. But that's not the case here. On one side we have, for example, `printPurple` while on the other we have `printGreen`. And we've `verbose` and on the other side `printRed`. We can conclude that, in the first file, we don't have `printGreen` and `printRed` and, we don't have `printPurple` and `verbose` in the second file. By adding `printBlue`, `printGreen` and `printRed` in our first file, now we got: ![Compare the two versions of the same Bash script and shows which functions are not the same in both files](./images/compare_functions_both_side_bis.webp) So now, we can see a new indicator `<`: the function was retrieved only in the first file and not in the second one. Once again, we've the confirmation that we've two new functions (`printPurple` and `verbose`) in our first file and not in the second one. Right now, our `/tmp/bash/console.sh` contains this: **/tmp/bash/console.sh** ```bash #!/usr/bin/env bash function console::printCyan() { } function console::askYesNo() { } function console::printPurple() { } function console::verbose() { } function console::printError() { } function console::banner() { } function console::printBlue() { } function console::printGreen() { } function console::printRed() { } ``` and, for `/tmp/bash/console_v2.sh`: **/tmp/bash/console\_v2.sh** ```bash #!/usr/bin/env bash function console::printCyan() { } function console::askYesNo() { } function console::printRed() { } function console::printError() { } function console::banner() { } function console::printGreen() { } function console::printBlue() { } ``` > **The order does not matter:** > > Unlike a comparison with `diff`, the order in which the functions appear in the scripts is irrelevant, as we are sorting them. *If you'd rather fix the cause than work around it, [Linux - Sort functions in a Bash script](/blog/linux-sort-functions-in-script) reorders the functions in the files themselves.* ## Compare Bash scripts in two folders In the previous chapter, we've seen how to compare two versions of the same script. Let's go one step further and compare two folders: for each script in the two folders, let's run a comparison. To do this, create the `compare.sh` script on your hard drive with this content: **compare.sh** ```bash #!/usr/bin/env bash clear if [ ! $# -eq 2 ]; then echo "Usage: $0 " exit 1 fi sourceFolder="$1" compareWithFolder="$2" [[ ! -d "${sourceFolder}" ]] && echo "Error: Source folder ${sourceFolder} not found." && exit 1 [[ ! -d "${compareWithFolder}" ]] && echo "Error: CompareWith folder ${compareWithFolder} not found." && exit 1 pushd "${sourceFolder}" >/dev/null || exit 1 printf "\e[37;1m%s\e[0;1m\n\n" "Compare .sh scripts and detects functions that are in one of the files but not in the other between these two folders:" printf "\e[37;1m%s\e[0;1m\n" "LEFT SIDE = ${sourceFolder}" printf "\e[37;1m%s\e[0;1m\n\n" "RIGHT SIDE = ${compareWithFolder}" printf "\e[37;1m%s\e[0;1m\n\n" "Legend:" printf "\e[37;1m1/ \e[34;1m%s\e[0;1m\n" "name1 <" printf "\e[37;1m\t%s\e[0;1m\n\n" "That function only exist in the first file; not in the second one" printf "\e[37;1m2/ \e[34;1m%s\e[0;1m\n" "name1 | name2" printf "\e[37;1m\t%s\e[0;1m\n\n" "The function name1 only exists in the first file and name2 only exists in the second file" printf "\e[37;1m3/ \e[34;1m%s\e[0;1m\n" " > name2" printf "\e[37;1m\t%s\e[0;1m\n\n" "The function name2 only exists in the second file" printf "\e[41;1m%s\e[0;1m\n" "Warning: this script checks whether a function is present in one of the files and not in the other," printf "\e[41;1m%s\e[0;1m\n" "but does not check whether the function code is identical. It just checks whether it is present or not." for bashScript in *.sh; do if [[ -f "${compareWithFolder}/${bashScript}" ]]; then ( FILE1="${sourceFolder}/${bashScript}" FILE2="${compareWithFolder}/${bashScript}" printf "\e[32;1m%s\e[0;1m\n" "File ${bashScript}" diff --suppress-common-lines --side-by-side --width 83 \ <(grep -P "^(function\s+.*)\(\)" "${FILE1}" | awk '{print $2}' | sort) \ <(grep -P "^(function\s+.*)\(\)" "${FILE2}" | awk '{print $2}' | sort) && \ printf "\e[32;1m%s\e[0;1m\n" "The two files are identical" ) fi done popd >/dev/null || exit 1 ``` Now, to run it, just start `./compare.sh foldername1 foldername2`. You'll get something like below i.e. for each script in both folders (scripts present in just one folder are ignored), you'll get the name of the script (like `array.sh`) followed by the text `The two files are identical` if both files are identical or, if not, a list of function names and the indicator already seen i.e. `<`, `>` or `|`. ![Compare Bash scripts in two folders](./images/compare_folders.webp) Using this script, it is easy to keep one of the two folders as the *master* (the left-hand folder), i.e. to reproduce in it all the functions that would have been added in a script in the second folder (the one corresponding to the right-hand column). Once a function has been identified, simply open the *right* file and copy/paste the function into the corresponding file in the *left* folder. By adding `| grep -E -v "<$"` to our `grep` statement, we can improve the script by ignoring cases when functions are only added in the left-hand folder. Imagine the requirement *In the right-hand folder, if I've added functions to my scripts; what are these functions so that I can copy/paste them into the scripts in the left-hand folder?* **compare.sh** ```bash #!/usr/bin/env bash clear if [ ! $# -eq 2 ]; then echo "Usage: $0 " exit 1 fi sourceFolder="$1" compareWithFolder="$2" [[ ! -d "${sourceFolder}" ]] && echo "Error: Source folder ${sourceFolder} not found." && exit 1 [[ ! -d "${compareWithFolder}" ]] && echo "Error: CompareWith folder ${compareWithFolder} not found." && exit 1 pushd "${sourceFolder}" >/dev/null || exit 1 printf "\e[37;1m%s\e[0;1m\n\n" "Compare .sh scripts and detects functions that are in one of the files but not in the other between these two folders:" printf "\e[37;1m%s\e[0;1m\n" "LEFT SIDE = ${sourceFolder}" printf "\e[37;1m%s\e[0;1m\n\n" "RIGHT SIDE = ${compareWithFolder}" printf "\e[37;1m%s\e[0;1m\n\n" "Legend:" printf "\e[37;1m1/ \e[34;1m%s\e[0;1m\n" "name1 <" printf "\e[37;1m\t%s\e[0;1m\n\n" "That function only exist in the first file; not in the second one" printf "\e[37;1m2/ \e[34;1m%s\e[0;1m\n" "name1 | name2" printf "\e[37;1m\t%s\e[0;1m\n\n" "The function name1 only exists in the first file and name2 only exists in the second file" printf "\e[37;1m3/ \e[34;1m%s\e[0;1m\n" " > name2" printf "\e[37;1m\t%s\e[0;1m\n\n" "The function name2 only exists in the second file" printf "\e[41;1m%s\e[0;1m\n" "Warning: this script checks whether a function is present in one of the files and not in the other," printf "\e[41;1m%s\e[0;1m\n" "but does not check whether the function code is identical. It just checks whether it is present or not." for bashScript in *.sh; do if [[ -f "${compareWithFolder}/${bashScript}" ]]; then ( FILE1="${sourceFolder}/${bashScript}" FILE2="${compareWithFolder}/${bashScript}" printf "\e[32;1m%s\e[0;1m\n" "File ${bashScript}" diff --suppress-common-lines --side-by-side --width 83 \ <(grep -P "^(function\s+.*)\(\)" "${FILE1}" | awk '{print $2}' | sort) \ <(grep -P "^(function\s+.*)\(\)" "${FILE2}" | awk '{print $2}' | sort) | grep -E -v "<$" ) fi done popd >/dev/null || exit 1 ``` If we run this newer script on the exact same files, now, lines ending with `<` are hidden (only in the left-hand folder) and we just obtain cases when a function is present in the right-hand folder: ![Hide when functions are just in the left-hand folder](./images/no_more_left.webp) This makes it even easier to identify these functions and copy/paste them to the left, for example. --- # Linux - Sort functions in a Bash script > Learn how to use diff and sort in Bash to check if functions in your shell scripts are defined in alphabetical order. Includes a script to scan an entire directory. --- ![Linux - Sort functions in a Bash script](/img/v2/bash.webp) **TL;DR** This article checks whether functions in a Bash script are declared in alphabetical order, by extracting function names with `grep`/`awk` and diffing the as-is list against a sorted copy using `diff --side-by-side`. It's then turned into an `order.sh` script that scans every `.sh` file in a folder and silently reports nothing for correctly sorted files, making misordered scripts easy to spot. In a previous [article](/blog/linux-compare-two-versions-of-the-same-script), we've seen a simple CLI command to display the list of functions present in a script but what about a Bash script that will scan an entire folder; retrieve any `.sh` scripts and check if functions are sorted in the files? ## Result `diff --side-by-side` between the as-declared function order and the sorted order tells you instantly whether a script's functions are alphabetized — mismatched rows point straight at what to move: ![Bad sorter](./images/bad_sorter.webp) Fix the order and rerun until every function lines up and you get a friendly confirmation: ![Congratulations](./images/congratulations.webp) ## Create a simple Bash script to play with this blog post Please create the `/tmp/bash/console.sh` file on your disk with this content: **/tmp/bash/console.sh** ```bash #!/usr/bin/env bash function console::printCyan() { } function console::askYesNo() { } function console::printRed() { } function console::printError() { } function console::banner() { } function console::printGreen() { } function console::printBlue() { } ``` As we can see, we'll just create empty functions in no particular order. ## Get the list of functions in a Bash script To get the list of functions declared in a script, just run the command below: ```bash $ `grep -P "^(function\s+.*)\(\)" "/tmp/bash/console.sh" | awk '{print \$2}' | sort` ``` ![Get the list of functions](./images/display_list_of_functions.webp) Ok, now, we know how to sort the list of functions in the console. Let's use `diff` to compare our existing file (left-hand side) and the sorted list of functions (right-hand side) — this is what produced the comparison shown at the top of this article: ```bash ( FILE=/tmp/bash/console.sh printf "\e[33;1m%-39s %s\e[0;1m\n" "Left side: AS IS" "Right side: Using correct sorter" diff --side-by-side --width 83 \ <(grep -P "^(function\s+.*)\(\)" "${FILE}" | awk '{print $2}') \ <(grep -P "^(function\s+.*)\(\)" "${FILE}" | awk '{print $2}' | sort) ) ``` As we can see on the image shown above, the column at the left show that the first-defined function in the `/tmp/bash/console.sh` script is `console::printCyan` followed by `console::askYesNo` and `console::printRed` for the third function declared (in that order) in the Bash script. In the right column, we can see that the first, sorted, function is `console::askYesNo`, the second is `console::banner` and so on. Back to the left column: the name displayed in white are already in the correct order! Let's update partially the `/tmp/bash/console.sh` file and reorder some functions: **/tmp/bash/console.sh** ```bash #!/usr/bin/env bash function console::askYesNo() { } function console::banner() { } function console::printBlue() { } function console::printCyan() { } function console::printRed() { } function console::printError() { } function console::printGreen() { } ``` Now, rerunning the same command: ![Almost correct](./images/almost_correct.webp) We just need to put `console::printRed()` at the end and we'll be fine and, to give some positive feedback, use this enhanced version: ```bash ( FILE=/tmp/bash/console.sh printf "\e[33;1m%-39s %s\e[0;1m\n" "Left side: AS IS" "Right side: Using correct sorter" diff --side-by-side --width 83 \ <(grep -P "^(function\s+.*)\(\)" "${FILE}" | awk '{print $2}') \ <(grep -P "^(function\s+.*)\(\)" "${FILE}" | awk '{print $2}' | sort) && printf "\n%s\n" "🎉 🎊 🕺 💃 👏 CONGRATULATIONS" ) ``` If you see the confirmation message shown at the top of this article, perfect, functions are correctly ordered in your script. ## Process all scripts from a specific folder We have just seen, on the command line, how to check whether a Bash script declaring functions does so in alphabetical order. Let's move on to the industrialization of this concept: a script that will scan each .sh file in a specific folder and check whether the functions are defined in the file in alphabetical order. *A sorted script is also a script that documents itself better: [Linux - Generate documentation from Bash scripts](/blog/linux-generate-documentation-from-bash-scripts) generates one Markdown page per script from the doc block of each function, in declaration order.* If this is the case, we won't have any display so as not to pollute our console. If this is not the case, the left-hand side of the screen shows the current order of function declarations and the right-hand side shows the expected order, sorted alphabetically. To do this, create the `order.sh` script on your hard drive with this content: **order.sh** ```bash #!/usr/bin/env bash clear if [ ! $# -eq 1 ]; then echo "Usage: $0 " exit 1 fi sourceFolder="$1" [[ ! -d "${sourceFolder}" ]] && echo "Error: Source folder ${sourceFolder} not found." && exit 1 printf "\e[37;1m%s\e[0;1m\n\n" "Check if functions declared in Bash .sh script in folder ${sourceFolder} are correctly ordered in the file." printf "\e[33;1m%-39s %s\e[0;1m\n\n" "Left side: AS IS" "Right side: Using correct sorter" pushd "${sourceFolder}" >/dev/null set +e for bashScript in *.sh; do if ! result="$(diff --side-by-side --width 83 \ <(grep -P "^(function\s+.*)\(\)" "${bashScript}" | awk '{print $2}') \ <(grep -P "^(function\s+.*)\(\)" "${bashScript}" | awk '{print $2}' | sort))"; then printf "\e[33;1m%s\e[0;1m\n" "The file ${bashScript} isn't correctly ordered" printf "\e[37;1m%s\e[0;1m\n" "${result}" fi done set -e popd >/dev/null ``` And now, start the script like this: `./order.sh ~/helpers`. The expected parameter is the name of a folder containing `.sh` files. ![Running the batch script](./images/batch_script.webp) What does that mean? My `~/helpers/api.sh` script is actually (left side) really poorly sorted since there are a lot of differences with the right column (perfect ordering). The first function in my file is `api::__injectLogsToApplicationLog` while there is an `api::__assertHttpMethod` function later in the code. By editing my file and moving `api::__assertHttpMethod` at the top of my script, now, by rerunning the script: ![Move the assert method first](./images/api_move_assert_first.webp) Ok, so now, still at the right side, we can see `api::__debugCurlStatement` is expected in the second position, `api::__doSomeCleaning` as the third one then `api::__executeCall()` and so on (as we can see in the right side). As soon as the `api.sh` file is correctly sorted, rerunning the script won't mention anymore the script but other ones. ![The console.sh script isn't ordered](./images/console.webp) The final objective is thus: we expect no output with the script. If all Bash scripts are correctly sorted then the `order.sh` script is no longer going to find any differences, which is in fact what we want. --- # Linux - Generate documentation from Bash scripts > Learn to auto-generate documentation for your Bash scripts on Linux. This guide provides a custom script to parse function doc blocks and create structured Markdown files automatically. --- ![Linux - Generate documentation from Bash scripts](/img/v2/bash.webp) **TL;DR** This article shares a custom `generate_doc.sh` Bash script that scans a folder of `.sh` files, extracts PHP-Docblock-style comment blocks above each function, and generates one Markdown file per script (plus a summary `readme.md`) — filling the gap left by the lack of a Bash equivalent to tools like phpDocumentor. When writing Bash scripts, I'm always putting some description block in front of any functions like I do in any language (think to [PHP Docblock](https://docs.phpdoc.org/guide/getting-started/what-is-a-docblock.html)). With PHP, there are a few tools like phpDocumentor for extracting these blocks and generate documentation but do such tools exist for Bash? I don't know, I haven't found any. So I've written a small Bash script to accomplish this i.e. parse any `.sh` file present in a folder, extract doc blocks and create one markdown document for any retrieved script. Each function's documentation will then be copied in Markdown, then a table of contents will be appended and, finally, a generic readme.md file will display the list of markdown files retrieved. ## Result Running the script against a folder of `.sh` files: ![Generate the documentation](./images/generate_doc.webp) For one file found in the `helpers` folder, a matching file is created in the `documentation` folder — here, `string.md`, with any function that has a doc-block described: **documentation/string.md** ```markdown # Documentation string.sh ## List of functions * [public function string::upper](#public-function-stringupper) * [Description](#description) * [Examples](#examples) * [Return](#return) * [Exit code](#exit-code) ### public function string::upper #### Description Make a string uppercase. #### Examples echo "HEllow WoRLD!" | string::upper #Output HELLO WORLD! title="$(echo "Project title" | string::upper)" #### Return The string in uppercase #### Exit code * 0 If successful. * 1 An error has occurred. https://github.com/jmcantrell/bashful/blob/master/bin/bashful-utils#L33 ``` And once every `.sh` file is processed, a final `documentation/readme.md` file is created: **documentation/readme.md** ```markdown # List of helpers * [string.md](string.md) ``` ## How to reproduce it Let's consider this `/tmp/bash/helpers/string.sh` file: **/tmp/bash/helpers/string.sh** ````bash # region - public function string::upper # # ## Description # # Make a string uppercase. # # ## Examples # # ```bash # echo "HEllo WoRLD!" | string::upper # #Output # HELLO WORLD! # # title="$(echo "Project title" | string::upper)" # ``` # # ## Return # # The string in uppercase # # ## Exit code # # * 0 If successful. # * 1 An error has occurred. # # https://github.com/jmcantrell/bashful/blob/master/bin/bashful-utils#L33 # # endregion function string::upper() { tr '[:lower:]' '[:upper:]' } ```` We now need to create the `generate_doc.sh` script. Since it's a big script, please click on the next line to get its content. Click again to hide the code. Please create the file `/tmp/bash/generate_doc.sh` with this content: **/tmp/bash/generate\_doc.sh** ````bash #!/usr/bin/env bash # Get the location (foldername) of this script; not the running one (don't use $0) #! Should be defined here and in below during the initialisation of the params array SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd)" # region - public function array::sort # # ## Description # # Sort an array (not associative one) # # ## Examples # # ```bash # arr=("f" "a" "e" "c" "b") # # #! As you can see, don't use the dollar sign before the array variable name # array::sort arr # sort the array "a", "b", "c", ... # ``` # # ## Parameters # # * @arg array Non-associative array to sort # # ## Return # # The sorted array # # ## Exit code # # * 0 If successful. # * 1 An error has occurred. # * 2 Function missing arguments. # # endregion function array::sort() { local -n arr=$1 # shellcheck disable=SC2207 IFS=$'\n' arrSorted=($(sort <<< "${arr[*]}")) unset IFS echo "${arrSorted[@]}" } # region - private function bash::__getIntroBlock # # ## Description # # Extract the introduction block present in the bash file. # # The block will look like below # # # region - Intro block ------------------------------------------- # # # # A long description # # # # endregion - Intro block ---------------------------------------- # # Just like in this script, see the top of this file # # ## Examples # # ```bash # bash::__getIntroBlock "any_script.sh" # ``` # # ## Parameters # # * @arg string The name of the script like "any_script.sh" # # ## Return # # The introduction block as a string # # ## Exit code # # * 0 If successful i.e. no syntax error retrieved in the script # * 1 An error has occurred i.e. there are syntax errors in the script # * 2 Function missing arguments. # # endregion function bash::__getIntroBlock() { local -r filename="$1" # The filename we need to scan (f.i. "any_script.sh") startTag="^# region - Intro block" endTag="^# endregion - Intro block" # Extract the region; including the tags regionIntroBlock="$(sed -n "/${startTag}/,/${endTag}/p" "${filename}")" # Remove the tags line regionIntroBlock="$(echo "$regionIntroBlock" | sed -E 's/^# region - Intro block.*//')" regionIntroBlock="$(echo "$regionIntroBlock" | sed -E 's/^# endregion - Intro block.*//')" # Remove the starting "#" followed or not by a space on each line # shellcheck disable=SC2001 regionIntroBlock="$(echo "$regionIntroBlock" | sed -E 's/^#\s?//')" echo "$regionIntroBlock" } # region - public function bash::addBanner # # ## Description # # Add a banner "Don't modify this file" at the top of the generated file # # ## Examples # # ```bash # bash::addBanner "documentation.md" # ``` # # ## Parameters # # * @arg string The name of the file where the banner has to be added # * @arg string Optional, the name of the script used to generate the documentation. # # ## Exit code # # * 0 If successful. # * 1 An error has occurred. # * 2 Function missing arguments. # # endregion function bash::addBanner() { local filename="$1" # The filename where the banner should be added local generateScript="${2:-}" # The name of the script used to generate the documentation # Comment that will be added at the top of the file. header="" file::addBanner "$filename" "$header\n$header2\n$header3\n$header4\n$header5\n$header6" return 0 } # region - public function bash::generateBashDocumentation # # ## Description # # Scan the script and generate a documentation (markdown one). # # ## Examples # # ```bash # bash::generateBashDocumentation "any_script.sh" # ``` # # ## Parameters # # * @arg string The name of the script like "my_script.sh" # * @arg boolean Optional, Did we need to make a table of contents? # * @arg string Optional, The start tag to retrieve like "# region - public function". # # ## Return # # When successful, the return will be a long string that is the Markdown documentation # # ## Exit code # # * 0 If successful. # * 1 An error has occurred. # * 2 Function missing arguments. # # endregion function bash::generateBashDocumentation() { local filename="$1" # The filename we need to scan (f.i. "my_script.sh") local createTOC="${2:-true}" # Should we need to create a table of contents local startTag="^${3:-# region - public function}" # The start tag if any (f.i. "# region - public function") local endTag="^# endregion" local generateScript="${2:-}" # The name of the script used to generate the documentation # Extract the region; including the tags regionBlock="$(sed -n "/${startTag}/,/${endTag}/p" "${filename}")" # Remove the starting "#" followed or not by a space on each line # shellcheck disable=SC2001 regionBlock="$(echo "$regionBlock" | sed -E 's/^#\s?//')" # Replace the starting "region -" by "# " so we've a heading one in markdown regionBlock="$(echo "$regionBlock" | sed -E 's/^region\s?-\s/# /')" # Replace any "#" in the comment by "###" i.e. increase the level regionBlock="$(echo "$regionBlock" | sed -E 's/# /### /')" # And the final "endregion" line regionBlock="$(echo "$regionBlock" | sed -E 's/^endregion//')" tmpFile="$(mktemp)" # The syntax $'\n' will concatenate substrings with a linefeed echo "# Documentation $(file::getBasename "${filename}")" > "${tmpFile}" # shellcheck disable=SC2129 bash::__getIntroBlock "${filename}" >> "${tmpFile}" if [[ ${createTOC} == "true" ]]; then # Add a table of contents # shellcheck disable=SC2129 echo "" >> "${tmpFile}" echo "## List of functions" >> "${tmpFile}" echo "" >> "${tmpFile}" markdown::generateTableOfContents "${regionBlock}" >> "${tmpFile}" echo "${regionBlock}" >> "${tmpFile}" fi bash::addBanner "${tmpFile}" "${generateScript}" cat "${tmpFile}" rm -f "${tmpFile}" return 0 } # region - public function file::addBanner # # ## Description # # Add a text at the very top of a file # # ## Examples # # ```bash # file::addBanner "test.md" "" # ``` # # ## Parameters # # * @arg string The filename where the banner should be added # * @arg string The banner text to add # # ## Exit code # # * 0 If successful i.e. the banner was successfully added # * 1 An error has occurred i.e. the banner wasn't added # * 2 Function missing arguments. # # endregion function file::addBanner() { local -r filename="$1" sed -i "1 i \\$2\n" "${filename}" return 0 } # region - public function file::getBasename # # ## Description # # Get the filename without parent folder's name. # # ## Examples # # ```bash # file::getBasename "/app/.docker/entrypoint.sh" # will return "entrypoint.sh" # ``` # # ## Parameters # # * @arg string A filename # # ## Return # # Will return only the filename, without parent folder's name # # ## Exit code # # * 0 If successful. # * 1 An error has occurred. # * 2 Function missing arguments. # # endregion function file::getBasename() { basename -- "$1" } # region - private function generate_doc::__main # # ## Description # # Run the script # # ## Examples # # ```bash # generate_doc::__main $* # ``` # # ## Parameters # # * @arg CLI arguments # # ## Exit code # # * 0 If successful. # * 1 An error has occurred. # * 2 Function missing arguments. # # endregion function generate_doc::__main() { # shellcheck disable=SC2178 arrFiles=$(find "${SCRIPT_DIR}/helpers" -name "*.sh") # And sort the array based on filenames thus arrFiles=$(array::sort arrFiles) local readme readme="# List of helpers"$'\n' [[ ! -d "${SCRIPT_DIR}"/documentation ]] && mkdir -p "${SCRIPT_DIR}"/documentation for filename in "${arrFiles[@]}"; do echo "Process $filename" # Replace the .sh extension by .md outputFilename="${filename/%.sh/.md}" # and replace "helpers" (the folder) by "documentation" (where docs will be created) outputFilename="${outputFilename/\/helpers/\/documentation}" echo "Create $outputFilename" bash::generateBashDocumentation "$filename" "true" "# region - public function" > "${outputFilename}" # Keep only the basename like "markdown.md" relativeOutputFilename="$(file::getBasename "${outputFilename}")" # $'\n' is to add a linefeed readme="${readme}"$'\n'"* [${relativeOutputFilename}](${relativeOutputFilename})" done # Generate the readme.md file so we can have a look on each doc through GitLab echo "$readme" > "${SCRIPT_DIR}/documentation/readme.md" # Add the last modification date/time in the readme.md file bash::addBanner "${SCRIPT_DIR}/documentation/readme.md" "$(basename "${BASH_SOURCE[0]}")" echo "" echo "Success! The documentation has been created/updated." return 0 } # region - public function markdown::generateTableOfContents # # ## Description # # This function will retrieve any headings (from `H2` to `H6`) from the markdown content # and will generate a table of contents # # ## Examples # # ```bash # markdown::generateTableOfContents "any markdown long, multi-lines, text" # markdown::generateTableOfContents "$(cat test.md)" # ``` # # ## Parameters # # * @arg string A long and multi-lines text # # ## Return # # A bullet list that will be used as a table of content # # For instance (partial output): # # ```markdown # * [public function console::printGray()](#public-function-console-printgray) # * [Description](#description) # * [Examples](#examples) # * [Parameters](#parameters) # * [Exit code](#exit-code) # * [public function console::printGrayNoNewLine()](#public-function-console-printgraynonewline) # * [Description](#description) # * [Examples](#examples) # ``` # # ## Exit code # # * 0 If successful. # * 1 An error has occurred. # * 2 Function missing arguments. # # endregion function markdown::generateTableOfContents() { local -r markdownContent="$1" # Extract H2 and more local headings headings="$(printf "%s" "$markdownContent" | grep '^##')" # Make sure we've Unix LF style headings="${headings//\r$/}" local toc="" # Process each entries in the list of titles (we can't yet call it a TOC) while IFS= read -r heading; do slug="" # region-slug # Try to find a slug in the title f.i. `## How to use the program?{#how-to}` # In this case, we've a slug and it's `how-to` pattern='\{\#([^}]*)\}' # Generate a slug slug="$(echo "$heading" | string::slugify)" if [[ $heading =~ $pattern ]]; then # The title already contains a slug, use it match="${BASH_REMATCH[0]}" # f.i. `{#run-hooks}` slug="${BASH_REMATCH[1]}" # f.i. `{#how-to}` # Remove the slug in the title heading=${heading//$match/} fi # endregion # H2 should become H1 in the TOC, H3 to H2, ... heading=${heading//## /# } # `# My title` should become `* My title` # `## My subtitle` should become ` * My title` # The two replace below achieve this purpose heading=${heading//# /* } heading=${heading//#/ } # heading is something like: # `* Installation` # ` * Settings` # ` * JSON file` # i.e. # 1. an indentation or not followed by a bullet followed by a space # 2. a title # Parse the title # ([^*]* ?\* ) Anything before a wildcard with or without a following space # Then a wildcard character and followed by a space ==> we've our indentation # (.*) Anything ==> we've our title pattern='([^*]* ?\* )(.*)' if [[ $heading =~ $pattern ]]; then # shellcheck disable=SC2034 indent="${BASH_REMATCH[1]}" # f.i. ` * ` title=${BASH_REMATCH[2]} # f.i. `JSON file` # Remove leading and trailing whitespace title="$(echo "$title" | string::trim)" # Add the heading in the table of content and add a LF character to go to the next line toc+=${heading//$title/[$title](#$slug)}$'\n' fi done < <(printf '%s\n' "$headings") # Process $headings, line by line echo "$toc" } # region - public function string::ltrim # # ## Description # # Removes all leading whitespace (from the left) # Specify the character to remove (default is a space) as the first argument # to remove that character; f.i. a starting slash # # ## Examples # # ```bash # $(echo " Hello world " | string::ltrim)" # "Hello world " # $(echo "/tmp/folder/" | string::ltrim "/")" # "tmp/folder/" # ``` # # ## Parameters # # * @arg String The character to remove, default is a space # # ## Return # # The left-trimmed string # # ## Exit code # # * 0 If successful. # * 1 An error has occurred. # # @link https://github.com/jmcantrell/bashful/blob/master/bin/bashful-utils#L233 # # endregion function string::ltrim() { local -r char=${1:-[:space:]} sed "s%^[${char//%/\\%}]*%%" } # region - public function string::rtrim # # ## Description # # Removes all trailing whitespace (from the right). # Specify the character to remove (default is a space) as the first argument # to remove that character; f.i. an ending slash # # ## Examples # # ```bash # $(echo " Hello world " | string::rtrim)" # " Hello world" # $(echo "/tmp/folder/" | string::rtrim "/")" # "/tmp/folder" # ``` # # ## Parameters # # * @arg String The character to remove, default is a space # # ## Return # # The right-trimmed string # # ## Exit code # # * 0 If successful. # * 1 An error has occurred. # # @https://github.com/jmcantrell/bashful/blob/master/bin/bashful-utils#L245 # # endregion function string::rtrim() { local -r char=${1:-[:space:]} sed "s%[${char//%/\\%}]*$%%" } # region - public function string::slugify # # ## Description # # Create a slug from any text # Return "how-to" f.i. the text is "How to?" # https://gist.github.com/oneohthree/f528c7ae1e701ad990e6 # # ## Examples # # ```bash # echo "HEllo WoRLD!" | string::slugify # will return `hello-world` # ``` # # ## Parameters # # * @arg string A string to "slugify" # # ## Return # # The slug i.e. a string with spaces, special characters, ... # # ## Exit code # # * 0 If successful. # * 1 An error has occurred. # endregion function string::slugify() { # Replace the "– dash" sign and use the minus instead "-" # Use sed to remove some characters then convert uppercase to lower sed -E 's/[~\^]+//g' | # Remove purely and simply some characters sed -E 's/://g' | sed -E 's/[^a-zA-Z0-9]+/-/g' | sed -E 's/^-+\|-+$//g' | sed -E 's/^-+://g' | sed -E 's/-+$//g' | tr "[:upper:]" "[:lower:]" | sed -e "s/^-//" } # region - public function string::trim # # ## Description # # Removes all leading/trailing whitespace. # Specify the character to remove (default is a space) as the first argument # to remove that character; f.i. a slash # # ## Examples # # ```bash # $(echo " Hello world " | string::trim)" # "Hello world" # $(echo "/tmp/folder/" | string::trim "/")" # "tmp/folder" # ``` # # ## Parameters # # * @arg String to trim # # ## Return # # The trimmed string (left and right) # # ## Exit code # # * 0 If successful. # #https://github.com/jmcantrell/bashful/blob/master/bin/bashful-utils#L219 # # endregion function string::trim() { local -r char=${1:-[:space:]} string::ltrim "$@" | string::rtrim "$@" } generate_doc::__main "$@" ```` Once you've created the `/tmp/bash/generate_doc.sh` file and make it executable (`chmod +x ./generate_doc.sh`), just run `./generate_doc.sh` in the console — you'll get the output and generated files shown above. ## Going further Two companion ideas, once your doc blocks are the single source of truth: * keep your scripts readable so the generated documentation stays readable too; [Linux - Sort functions in a Bash script](/blog/linux-sort-functions-in-script) shows a small script that reorders functions alphabetically and * turn that folder of Markdown files into a real, published documentation site, as described in [Quarto - How I Built a Self-Documenting Ecosystem for 50+ Projects](/blog/quarto-industrialisation). --- # Linux - Take advantage of the number of CPUs you have; start concurrent jobs > Optimize your Bash scripts! Learn how to run concurrent jobs and leverage all your CPU cores using process IDs (pids), wait, and nproc to dramatically reduce execution time. --- ![Linux - Take advantage of the number of CPUs you have; start concurrent jobs](/img/v2/linux_parallel.webp) **TL;DR** This article shows how to run Bash function calls concurrently instead of sequentially, using `&` to background each call, `nproc` to size a thread pool matching available CPU cores, and tracking process IDs with `jobs | wc -l` to throttle how many run at once — turning a 10-call, 30-second sequential loop into a \~3-second parallel one (with a note to also respect any rate limits imposed by a target API). In my professional activity, I've been faced with the following requirement: process each line of a CSV file and make a POST API call to upload a document. One line of the CSV contained information that needed to be communicated to an API service, and each line corresponded to a PDF file. So if there are 1000 lines in the CSV file, I have to make 1000 API calls to upload 1000 PDFs. I wrote my script in Linux Bash and then it was time to optimize: not just one API call at a time, but as many as possible. Let's see how we can start more than one task at a time using Linux Bash. Consider this small demo: **demo.sh** ```bash #!/bin/bash set -euo pipefail function demo { echo "Sleeping for 3 seconds..." sleep 3 return 0 } function main { for i in {1..10}; do demo done } start_time=$(date +%s) main end_time=$(date +%s) total_time=$((end_time - start_time)) echo "Total running time: $total_time seconds" ``` Before calling the `main` function, I remember the actual start time, call `main` then calculate the elapsed time in seconds. The `main` function is quite basic here, I'll do a loop from 1 till 10 and, each time, call the `demo` function. That function will sleep for 3 seconds. So, by running the script, since we're calling our demo function ten times and the function is waiting for three seconds, we won't be surprised by the total duration: ```bash Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Total running time: 30 seconds ``` > **Hey dude! I've more than one CPU:** > > But, that's code is so old fashion now? How many CPU did I have? Just one? Oh thank you computer gods, I've got more than that! So, why just using one? ## Now, the optimized version Ok, now, we'll not start our function in a sequential order but we'll call it once and **don't wait until the function is finished**, we'll continue our loop and call the same function a second time, a third time, ... We just need to make sure we'll not kill our performances and for this, we'll retrieve the number of CPU cores we've on the machine. In Linux, this is very easy: the `nproc` command will do this. On my computer, I've 32 logical processors and since I can start 2 threads by CPU, I can calculate the maximum number of threads like this: `NUMBER_OF_THREADS=$(( $(nproc) * 2 ))`. > **Also pay attention to some limitation imposed by the third party:** > > In the Bash script below, no problem, it's just my computer but in my introduction, I've mentioned, "I need to call an API POST service". Here, I just make sure I will not be blacklisted by the web server. For instance, perhaps, there is a limitation like "Not more than 32 calls in a second for the same IP". In that case, I should take this info into account and don't start more than 32 process at a time (I'll then set `NUMBER_OF_THREADS=32`). We'll adapt our sample like this: **demo.sh** ```bash #!/bin/bash set -euo pipefail function demo { echo "Sleeping for 3 seconds..." sleep 3 return 0 } function main { # highlight-next-line # Array to store the PIDs of every jobs started in background # highlight-next-line declare -a pids # highlight-next-line pids=() # highlight-next-line # Get the number of logical cores on the machine and multiply by 2 # highlight-next-line # So if the machine has 4 cores, we'll start 8 threads in the same time # highlight-next-line NUMBER_OF_THREADS=$(( $(nproc) * 2 )) # highlight-next-line # highlight-next-line echo "Number of threads: ${NUMBER_OF_THREADS}" # highlight-next-line for i in {1..10}; do # highlight-next-line # Run our function in the background; don't wait that the function finish # highlight-next-line demo & # highlight-next-line # Remember the pid i.e. the ID of the function we've just started # highlight-next-line pids+=($!) # highlight-next-line # As soon as we've started the max of concurrent threads; wait and allow all of them to finish. # highlight-next-line if [[ $( jobs | wc -l ) -ge ${NUMBER_OF_THREADS} ]]; then # highlight-next-line for pid in "${pids[@]}"; do # highlight-next-line wait "${pid}" # highlight-next-line done # highlight-next-line # highlight-next-line # And reset the array since all jobs are done # highlight-next-line pids=() # highlight-next-line fi done # highlight-next-line # And now, make sure all parallel jobs are finished # highlight-next-line for pid in "${pids[@]}"; do # highlight-next-line wait "${pid}" # highlight-next-line done } start_time=$(date +%s) main end_time=$(date +%s) total_time=$((end_time - start_time)) echo "Total running time: $total_time seconds" ``` ### In depth By adding the `&` character, I'm asking Bash to not wait for the function to finish its job before returning. So, the code below will run `demo` 10 times even before the first `demo` call is finished. ```bash for i in {1..10}; do demo & done ``` If I need to do 100, 500 or 1,000 calls, my computer will start to comply and, perhaps, didn't respond anymore. My computer can't handle 1,000 calls in the same time. What should I do? I know I've a specific number of cores (see the `nproc` command) and I know I can start two threads by cores *(so, in my situation, I can start 64 jobs in the same time)* So before running the next call of `demo`, I need to do two things: 1. I'll keep the process id (`pid`) of the just fired `demo` function (each call has its own `pid`) and store that value in an array, 2. I'll need to check how many jobs are running and not yet finished (retrieved using `$( jobs | wc -l )`) and compare that number with the number of threads we can start (*64* for me). As soon as the number of running jobs is equal to my max, ok, I should wait. I shouldn't start a 65th job but wait until all the previous 64 are finished. And, when it's done, I can continue for the next wave and so on. Finally, after the loop, I do the same i.e. make sure that all jobs defined in our `pids` array are done. ### How many seconds would I have waited? Did you know how many times I need to wait? Remember, in the first version of the script, I've waited 30 seconds. With the optimized version here above and to do **exactly the same thing**, I waited ... just three seconds: ```bash Number of threads: 64 Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Sleeping for 3 seconds... Total running time: 3 seconds ``` > **Running 50 times the function:** > > In the first version of the script, by changing the line `for i in {1..10}; do` to `for i in {1..50}; do`, I'll wait 150 seconds; right? With the optimized version, just 4 seconds. Why 4 and not 3? Probably some delay introduced by the processor (who should handle 50 concurrent threads). > > Since jobs now finish out of order, two companion articles become handy: [Linux - Using a progression bar in your script](/blog/bash-progression-bar) to keep an eye on the overall advancement and [Bash - Script to add logging features to your script](/blog/bash-logging) to know which call failed and when. --- # Linux - Using a progression bar in your script > Add a visual progression bar to your Bash scripts on Linux. This guide shows you how to easily implement a progress bar to track concurrent jobs, visualize progress, and improve your terminal user interface. --- ![Linux - Using a progression bar in your script](/img/v2/linux_progress_bar.webp) **TL;DR** This follow-up article adds a visual progress bar to Bash scripts (building on the earlier parallel-jobs post) using a reusable `progress_bar.sh` helper: source it, pipe your loop's output through `progress_bar::process "Doing some stuff" 50`, and echo `Progress=N` from inside the loop to drive the bar — useful for long-running batch jobs like CSV-driven API uploads. In my previous article; [Linux - Take advantage of the number of CPUs you have; start concurrent jobs](/blog/bash-parallel-task), we've seen how to start jobs in parallel. The next cool thing is to show a progression bar in your console. This has a number of advantages, including a clear view of what's been done and what's still to be done, as well as an attractive interface. *When the number of items isn't known in advance, a live counter is the better fit; see [Keep running and count the number of files in a folder using inotifywait](/blog/linux-inotifywait). And to keep a trace of what happened once the bar is gone, [Bash - Script to add logging features to your script](/blog/bash-logging).* Months ago, I've found this French blog post in my RSS feeds: and, just, **wow!!!** Let's play with it. ## Result ![Progression bar](./images/progression_bar.gif) That's a plain Bash loop, no external tool, driving a live progress bar in the console. ## Why it works There are three things a script needs to do to get this bar: 1. Source the `progress_bar.sh` script ([credits to ephase](https://xieme-art.org/post/bash-avance-barre-de-progression/)); 2. Run the function that does the actual work in a specific way; not just `main` but `main > >(progress_bar::process "Doing some stuff" 50)`. The figure `50` should be set to the number of occurrences the loop will run; 3. From inside that function, `echo` one thing to the console: `Progress=` followed by a number (from 1 till 50) — that's the signal the bar reads to advance. Not so difficult, right? ## Create demo.sh First, please create a new folder on your disk and jump into that folder: `mkdir /tmp/progress && cd $_`. Then, create a `demo.sh` file in the folder and copy/paste the bash script below: **demo.sh** ```bash #!/bin/bash set -euo pipefail function demo { echo "Progress=$1" sleep 0.1 return 0 } function main { for i in {1..50}; do demo "$i" done } source "progress_bar.sh" main > >(progress_bar::process "Doing some stuff" 50) ``` Make the script executable by running `chmod +x ./demo.sh`. ## Create progress\_bar.sh Here is the content you'll need to copy/paste into a file called `progress_bar.sh`, created in the same folder as the `demo.sh` one. Please create a new file on your hard disk; name it `progress_bar.sh` and copy/paste the content below into it. **progress\_bar.sh** ````bash #!/usr/bin/env bash # region - Intro block ----------------------------------------------- # # Progress bar # # ## How to use this script? # # Just include this script in your own Bash like this `source progress_bar.sh` # # Now, you can use any public functions defined here like `progress_bar::draw``. # # ## Example # # The example below will just loop for 1 till 50 and the most important part is the "Progress=${i}" # echo statement. The progression of the bar depends on a regex defined in the progress_bar::process() function. # # ```bash # function main() { # for i in {1..50} # do # echo "Progress=${i}" # sleep 0.1 # done # } # # main > >(progress_bar::process "Doing some stuff" 50) # ``` # # ## Namespace # # `progress_bar::` # # cspell:ignore endregion # # shellcheck disable=SC1090,SC1091 # # ## Source # # The source comes from a French blog post: # https://xieme-art.org/post/bash-avance-barre-de-progression/ # # shellcheck disable=SC2059 # # endregion - Intro block -------------------------------------------- # let script exit if an unset variable is used or if a command fails set -o nounset set -o errexit # The different bar to display (done) PROGRESS_BAR_CHAR=(█ ▒) # Show or not the introduction text (f.i. "Doing some stuff") PROGRESS_BAR_DISPLAY_INFO=1 PROGRESS_BAR_DISPLAY_COMP=1 # Display template PROGRESS_BAR_INFO_TEMPLATE=' %s [ %3d/%3d ] ' PROGRESS_BAR_COMP_TEMPLATE=' %3d%% ' PROGRESS_BAR_TEMPLATE='\033[1m%s%s\033[0m%s%s\r' # region - private function progress_bar::__change_column_size # # ## Description # # This function will capture de WINCH signal. That one is fired by Linux when # the size of the window (i.e. the console) is updated so allow us to retrieve the # max. number of columns that can fit in the window. # # endregion function progress_bar::__change_column_size() { printf >&2 "%${COLUMNS}s" "" printf >&2 "\033[0K\r" COLUMNS=$(tput cols) } # region - private function progress_bar::__draw # # ## Description # # Draw the progress bar / progression # # endregion function progress_bar::__draw() { # function parameters local -r progress=${1?"progress is mandatory"} local -r total=${2?"total elements is mandatory"} local -r info=${3:-"In progress"} # function local variables local progress_segment local todo_segment local info_segment="" local comp_segment="" if [[ ${PROGRESS_BAR_DISPLAY_INFO:-1} -eq 1 ]]; then printf -v info_segment "${PROGRESS_BAR_INFO_TEMPLATE}" \ "$info" "$progress" "$total" fi local -r progress_ratio=$((progress * 100 / total)) if [[ ${PROGRESS_BAR_DISPLAY_COMP:-0} -eq 1 ]]; then printf -v comp_segment "${PROGRESS_BAR_COMP_TEMPLATE}" \ "$progress_ratio" fi # progress bar construction # calculate each element sizes, bar must fit in ou screen local -r bar_size=$((COLUMNS - ${#info_segment} - ${#comp_segment})) local -r progress_segment_size=$((bar_size * progress_ratio / 100)) local -r todo_segment_size=$((bar_size - progress_segment_size)) printf -v progress_segment "%${progress_segment_size}s" "" printf -v todo_segment "%${todo_segment_size}s" "" printf >&2 "$PROGRESS_BAR_TEMPLATE" \ "$info_segment" \ "${progress_segment// /${PROGRESS_BAR_CHAR[0]}}" \ "${todo_segment// /${PROGRESS_BAR_CHAR[1]}}" \ "$comp_segment" } # region - public function progress_bar::process # # ## Description # # Display a progress bar # # ## Example # # ```bash # function main() { # for i in {1..50} # do # echo "Progress=${i}" # sleep 0.1 # done # } # # main > >(progress_bar::process "Doing some stuff" 50) # ``` # # ## Parameters # # * @arg $1 The text to display like "Upload in progress" # * @arg $2 The number of positions in the bar (f.i. if there are 4 rows to proceed, it has to be initialize to 4) # # endregion function progress_bar::process() { local -r action="${1:-Upload in progress}" local -r max=${2:-100} # Capture the resize of the window so the progress bar becomes responsive trap progress_bar::__change_column_size WINCH while read -r line; do # Each line with "Progress=(digit)" will increase the progress bar if [[ $line =~ Progress=([[:digit:]]{1,}) ]]; then progress_bar::__draw "${BASH_REMATCH[1]}" "${max}" "${action}" fi done } # This script didn't contains executable code; only helpers. sourced=0 (return 0 2> /dev/null) && sourced=1 || sourced=0 if [ "${sourced}" -eq "0" ]; then printf "\033[1;31m%s\033[0m\n" "ERROR, the $0 script is meant to be sourced. Try 'source $0' and use public functions." >&2 # Write on stderr exit 1 fi return 0 ```` ## Time to run the code Simply return to the console and run `./demo.sh` and... wow; that's pretty cool — you'll get the bar shown above. In a real-life example, I've implemented this feature in a Bash script where I need to read a CSV file and for each record retrieved in the file, I need to run a `curl` statement for calling a RESTful POST API and upload a PDF. And, when started in an interactive way, the screen output is really practical and terribly effective. Of course, I'm also using a log file where I redirect a maximum amount of information. --- # Linux - Bash scripts - Running unit tests with bats/bats > Learn how to effectively unit test your Bash scripts on Linux using the powerful bats (Bash Automated Testing System) framework. See practical examples for assertions, output checks, and failures. --- ![Linux - Bash scripts - Running unit tests with bats/bats](/img/v2/bash.webp) **TL;DR** This article provides a comprehensive introduction to unit testing Bash scripts using the `bats-core` framework. It shows how to get started quickly by running tests within a Docker container, eliminating complex setup. The guide covers writing basic tests, using various assertion types like `assert_output`, `assert_failure`, and `assert_equal`, and demonstrates how to structure tests for real-world shell functions. More advanced topics such as mocking commands, testing multi-line and colored output, checking file contents, and using `setup`/`teardown` functions for test fixtures are also explained with practical examples. Like all command-line developers, I write Linux Bash scripts. Like any programmer, I'm supposed to write unit tests. Well, I have to admit that I rarely write them. Some time ago, I wrote a set of Bash scripts that make up a library of functions (just like a framework) and here, the interest in having unit tests is even greater since these functions are supposed to be stable and can be used as foundations for more advanced scripts. *Two habits that make such a library testable in the first place: [keeping the number of function parameters small](/blog/linux-bash-too-many-function-parameters) and [documenting each function with a doc block](/blog/linux-generate-documentation-from-bash-scripts).* In this article, we look at how to write unit tests for Bash scripts. If you're after the PHP equivalent, see [Write PHP unit tests using Pest](/blog/pest_tips); for functional/end-to-end testing rather than unit tests, I've also covered Behat, [Cypress](/blog/cypress) and [Pest v4's browser testing](/blog/pest-functional-testing); and if you'd rather have a first draft of the suite written for you, [a local LLM can generate the missing cases](/blog/ollama-test-generator). ## Let's start with a little hors d'oeuvre Like always, first, we'll create a new folder so we can play with some examples. Please run `mkdir -p /tmp/bats && cd $_` to create a temporary folder and jump in it; then run `code .` to start VSCode and open the folder. We'll create a simple illustration file, let's call it `tests/simple.bats` and copy/paste the following content: **tests/simple.bats** ```bats #!/usr/bin/env bats setup() { bats_load_library bats-support bats_load_library bats-assert } @test "Asserting it's hello" { run echo "hello" assert_output "hello" assert_success } @test "Simple.bats exists" { run ls simple.bats assert_output --regexp "[Ss]imple\.bats" assert_success } ``` And now, the very difficult part is, ouch no, really easy in fact, to run [Bats-core](https://bats-core.readthedocs.io/en/stable/): ```bash $ docker run --rm -it -w /code/tests -v .:/code bats/bats:latest simple.bats ``` And ... it works. ![Simple test](./images/simple_test.webp) ### What did we do? We've created a very basic example with two checks. The first one is called *Asserting it's hello* and we've fired `echo "hello"` just like we can make in a Shell script. Then running `assert_output "hello"` we are checking the output of the fired command is `hello` and it's a success (so it's `hello` and nothing else). This is done using this script: **tests/simple.bats** ```bats @test "Asserting it's hello" { run echo "hello" assert_output "hello" assert_success } ``` And the second check is called *Simple.bats exists* and we're just doing a `ls simple.bats` and check if, for the illustration, `Simple.bats` or `simple.bats` is returned in the list of files (this, to illustrate the use of a regular expression). **tests/simple.bats** ```bats @test "Simple.bats exists" { run ls simple.bats assert_output --regexp "[Ss]imple\.bats" assert_success } ``` ### Asserting a failure Of course we can also assert a failure. Edit your `simple.bats` file like this: **tests/simple.bats** ```bats #!/usr/bin/env bats setup() { bats_load_library bats-support bats_load_library bats-assert } @test "Asserting it's hello" { run echo "hello" assert_output "hello" assert_success } @test "Simple.bats exists" { run ls simple.bats assert_output --regexp "[Ss]imple\.bats" assert_success } # highlight-start @test "Removing an inexistent file" { run rm INEXISTENT_FILE assert_failure } # highlight-end ``` So, in the new test function, we'll just try to remove a not-existing file and we expect, for sure, a failure: ![Asserting for a failure](./images/assert_failure.webp) ## Some real-world use cases Imagine the following, simplified, tree structure: ```tree expanded=true showJSX=false debug=false title="" . ├── src │ └── assert.sh └── tests ├── assert.bats ``` The file `src/assert.sh` contains your Linux shell code you want to test. Your test scenario should be stored in the `tests` folder. Since we'll write tests for the `src/assert.sh` file, let's create the `tests/assert.bats`. Below the content of the `src/assert.sh`. Let's start with a simple function: checking if a given binary is installed on the system or not. The name of the binary has to be passed as a parameter to the function. **src/assert.sh** ```bash #!/usr/bin/env bash function assert::notEmpty() { local -r variable="${1:-}" local -r errorMessage="${2:-"The variable can't be empty."}" [[ -z ${variable} ]] && echo "${errorMessage}" >&2 return 0 } function assert::binaryExists() { local -r binary="${1:-}" assert::notEmpty "${binary}" \ "You should specify the name of the program whose existence needs to be checked." if [[ -z "$(command -v "$binary" || true)" ]]; then echo "The ${binary} binary wasn't found on your system." >&2 exit 1 fi return 0 } ``` Below the content of the `tests/assert.bats`. **tests/assert.bats** ```bats #!/usr/bin/env bats setup() { bats_load_library bats-assert bats_load_library bats-support source ./../src/assert.sh # Create a dummy file every time a test function is called tmpFile="/tmp/bats/assert.tmp" [[ ! -d "/tmp/bats" ]] && mkdir "/tmp/bats" touch "${tmpFile}" } # Once the test function has been called, remove the temporary teardown() { [[ -f "${tmpFile}" ]] && rm -f "${tmpFile}" return 0 } @test "assert::binaryExists - Assert some binaries exists" { # clear is a native Linux command run assert::binaryExists "clear" # We expect thus binaryExists return a zero exit code (=success) assert_success # and no output message since it's successful assert_output "" # Another test with the native ls command run assert::binaryExists "ls" assert_output "" assert_success } @test "assert::binaryExists - Assert some binaries didn't exist" { run assert::binaryExists "fakeProgram" # Since fakeProgram didn't exist, we expect binaryExists exit with code 1 assert_failure 1 # We also expect this, exact, error message assert_output "The fakeProgram binary wasn't found on your system." } ``` The command line: ```bash $ docker run --rm -it -w /code/tests -v .:/code bats/bats:latest assert.bats ``` By running it, we expect a success for the binaryExists for `clear` and `ls` commands (since well installed on our system) and we expect a failure for `fakeProgram` but, since we're using `assert_failure`, our tests scenario should work. Let's add a check to see if a specific Docker image already exists or not on the system: **src/assert.sh** ```bash #!/usr/bin/env bash function assert::notEmpty() { local -r variable="${1:-}" local -r errorMessage="${2:-"The variable can't be empty."}" [[ -z ${variable} ]] && echo "${errorMessage}" >&2 return 0 } function assert::binaryExists() { local -r binary="${1:-}" assert::notEmpty "${binary}" \ "You should specify the name of the program whose existence needs to be checked." if [[ -z "$(command -v "$binary" || true)" ]]; then echo "The ${binary} binary wasn't found on your system." >&2 exit 1 fi return 0 } # highlight-start function assert::dockerImageExists() { local -r imageName="${1:-}" local -r errorMessage="${2:-The Docker image \"$imageName\" did not exist}" assert::notEmpty "${imageName}" "You should specify the name of the Docker image whose existence needs to be checked." # When the image exists, "docker images -q" will return his ID (f.i. `5eed474112e9`), an empty string otherwise if [[ "$(docker images -q "$imageName" 2> /dev/null)" == "" ]]; then echo "${errorMessage}" >&2 exit 1 fi return 0 } # highlight-end ``` and the updated `tests/assert.bats` file: **tests/assert.bats** ```bats #!/usr/bin/env bats setup() { bats_load_library bats-assert bats_load_library bats-support source ./../src/assert.sh # Create a dummy file every time a test function is called tmpFile="/tmp/bats/assert.tmp" [[ ! -d "/tmp/bats" ]] && mkdir "/tmp/bats" touch "${tmpFile}" } # Once the test function has been called, remove the temporary teardown() { [[ -f "${tmpFile}" ]] && rm -f "${tmpFile}" return 0 } @test "assert::binaryExists - Assert some binaries exists" { # clear is a native Linux command run assert::binaryExists "clear" # We expect thus binaryExists return a zero exit code (=success) assert_success # and no output message since it's successful assert_output "" # Another test with the native ls command run assert::binaryExists "ls" assert_output "" assert_success } @test "assert::binaryExists - Assert some binaries didn't exist" { run assert::binaryExists "FakeProgram" # Since FakeProgram didn't exist, we expect binaryExists exit with code 1 assert_failure 1 # We also expect this, exact, error message assert_output "The FakeProgram binary wasn't found on your system." } # highlight-start @test "assert::dockerImageExists - Assert docker image exists - using mockup" { # Mock - we'll create a very simple docker override and return a fake ID # This will simulate the `docker images -q "AN_IMAGE_NAME"` which return # the ID of the image when found docker() { echo "feb5d9fea6a5" } run assert::dockerImageExists "A-great-Docker-Image" assert_output "" assert_success } @test "assert::dockerImageExists - Assert docker image didn't exist - using mockup" { # Mock - we'll create a very simple docker override and return a fake ID # This will simulate the `docker images -q "AN_IMAGE_NAME"` which return # the ID of the image when found; here return an empty string to simulate # a non-existent image docker() { echo "" } run assert::dockerImageExists "Fake/image" "ERROR - Bad choice, that image didn't exist" assert_failure 1 assert_output --partial "Bad choice, that image didn't exist" } # highlight-end ``` ## More examples ### assert\_equal Simply verify that both values are equals. Here, we'll call a function that will return the length of an array and verify it's the expected value. **tests/simple.bats** ```bats @test "array::length - Calculate the length of an array" { arr=("one" "two" "three" "four" "five") assert_equal $(array::length arr) 5 } function array::length() { local -n array_length=$1 echo ${#array_length[@]} } ``` ### assert\_failure `assert::binaryExists` will exit 1 if the binary can't be retrieved. An error message like *The binary can't be found* will be echoed on the console. **tests/simple.bats** ```bats @test "Assert binary didn't exist" { # Simulate which and return an error meaning "No, that binary didn't exist on the host" which() { exit 1 } run assert::binaryExists "Inexistent_Binary" "Ouch, no, no, that binary didn't exist on the system" assert_failure assert_output --partial "Ouch, no, no, that binary didn't exist on the system" } function assert::binaryExists() { local binary="${1}" local msg="${2:-${FUNCNAME[0]} - File \"$binary\" did not exist}" [[ -z "$(which "$binary" || true)" ]] && echo "$msg" && exit 1 return 0 } ``` ### assert\_output `assert_output` has two nice options: `--partial` and `--regexp`. Using `--partial` will allow f.i. the check if the output contains some raw text like, for a help screen, a given sentence. **tests/simple.bats** ```bats @test "Show the help screen" { source 'src/helpers.sh' arrArguments=("--help") run helpers::showHelp ${arrArguments[@]} assert_output --partial "Convert a revealJs slideshow to a PDF document" } ``` Using `--regexp` will allow to use a regular expression: **tests/simple.bats** ```bats @test "Show the help screen" { source 'src/helpers.sh' arrArguments=("--input InvalidFile") run helpers::__process ${arrArguments[@]} assert_output --regexp "ERROR - The input file .* doesn't exist." } ``` ### assert\_success `assert::binaryExists` will return 0 when the binary can be retrieved. The function will run in silent (no output). **tests/simple.bats** ```bats @test "Assert binary exists" { run assert::binaryExists "clear" # clear is a native Linux command assert_output "" # No output when success assert_success } function assert::binaryExists() { local binary="${1}" local msg="${2:-${FUNCNAME[0]} - File \"$binary\" did not exist}" [[ -z "$(which "$binary" || true)" ]] && echo "$msg" && exit 1 return 0 } ``` ### Check for ANSI colors Imagine the following code: **script.sh** ```bash __RED=31 function console::printRed() { for line in "$@"; do printf "\e[1;${__RED}m%s\e[0m\n" "$line" done } ``` We wish to check that the line will be echoed in red. **tests/simple.bats** ```bats @test "console::printRed - The sentence should be displayed" { run console::printRed "This line should be echoed in Red" assert_output "�[1;31mThis line should be echoed in Red�[0m" assert_success } ``` ### Check for multi-line output Imagine the following code: **script.sh** ```bash function console::banner() { printf "%s\n" "============================================" printf "= %-40s =\n" "$@" printf "%s\n" "============================================" } ``` This will write three lines on the console, like f.i. ```bash # ============================================ # = Step 1 - Initialisation = # ============================================ ``` To check for multi-lines, use the `$lines` array like this: **tests/simple.bats** ```bats @test "console::banner - The sentence should be displayed" { run console::banner "Step 1 - Initialisation" assert_equal "${lines[0]}" "============================================" assert_equal "${lines[1]}" "= Step 1 - Initialisation =" assert_equal "${lines[2]}" "============================================" assert_success } ``` ### Check against a file Imagine a function that will parse a file and f.i. remove some paragraphs. We need to check if the content is correct, once the function has been fired. For this, imagine a `removeTopOfFileComments` function. The function will parse the file and remove the HTML comments (``) present at the top of the file. For the test, we'll create a file with three empty lines, then a HTML comment block, then two empty lines, then the HTML code. So, by removing the HTML comment, we'll have five empty lines followed by the HTML block so, we need to check our file contains six lines. The tip used is: * `cat --show-ends --show-tabs "$tempfile"` i.e. get the content of the file but with `$` where we've a line feed and, here, also `^I` for tabs. * then we'll pipe the result with `tr "\n" "#"` so, instead of getting six lines, we'll get only one by replacing line feed by `#`. Now, bingo, since we've a variable with only one line (in our example: `$#$#$#$#$#$#`), we can compare with our expectation: **tests/simple.bats** ```bats @test "html::removeTopOfFileComments - remove HTML comments - with empty lines" { tempfile="$(mktemp)" # Here, we'll have extra, empty, lines. They should be removed too echo '' >$tempfile echo '' >>$tempfile echo '' >>$tempfile echo ' ' >>$tempfile # We also add extra spaces after the end tag echo '' >>$tempfile echo '' >>$tempfile echo '' >>$tempfile run html::removeTopOfFileComments "$tempfile" # Get now the content of the file # We expect three empty lines (the three first) # The HTML comment has been removed # Then there are two more empty line (so we'll five empty lines) # And we'll have our "" block. # # cat --show-ends --show-tabs will show the dollar sign (end-of-line) and f.i. ^I for tabulations # tr "\n" "#" will then convert the line feed character to a hash so, in fact, fileContent will # be a string like `$#$#$#$#$#$#` fileContent="$(cat --show-ends --show-tabs "$tempfile" | tr "\n" "#")" # Once we've our string, compare the fileContent with our expectation assert_equal "$fileContent" "\$#\$#\$#\$#\$#\$#" } ``` ### Check against a file using a regex A second scenario can be: you have a write function (think to a logfile) and you want to check the presence of a given line in the file. The example below relies on `bats-file` and its `assert_file_contains` method. That method asks for a filename and a regex pattern. **tests/simple.bats** ```bats setup() { bats_load_library bats-support bats_load_library bats-assert bats_load_library bats-file #! "grep" without the "-P" argument seems to not support repetition like "\d{4}" # # a date like `2022-04-07` regexDate="[0-9][0-9][0-9][0-9]\-[0-9][0-9]\-[0-9][0-9]" # a time like `17:41:22` regexTime="[0-9][0-9]\:[0-9][0-9]\:[0-9][0-9]" # a time zone difference like "0200" regexUTC="[0-9]*" #! Should be [0-9][0-9][0-9][0-9] but didn't work??? # The final pattern so we can match f.i. `[2022-04-07T18:00:20+0200] ` __DATE_PATTERN="\[${regexDate}\T${regexTime}\+${regexUTC}.*\]\s" return 0 } @test "log::write - Write a line in the log" { local sentence="" sentence="This is my important message" run write "${sentence}" assert_file_exist "/tmp/bats_log.tmp" echo "${__DATE_PATTERN}${sentence}" >/tmp/regex.tmp assert_file_contains "/tmp/bats_log.tmp" "${__DATE_PATTERN}${sentence}" assert_success } ``` ### Check that a value is NOT in a file Another use of the assert\_failure can be to start a command like a grep and expect to get an error: ```bash run grep "REGEX_SOMETHING_THAT_SHOULD_BE_MISSING" "/tmp/test.log" assert_failure 1 ``` ### Some special functions #### setup The `setup` function is called before running a test. For each `@test` function present in the scenario, the `setup()` function will be called. In the following example, since there are two test functions, `setup()` will be called twice. **tests/simple.bats** ```bats setup() { bats_load_library bats-support bats_load_library bats-assert source 'src/env.sh' } @test "env::assertFileExists - Assert .env file exists - The path isn't initialised" { run env::assertFileExists assert_failure } @test "env::assertFileExists - Assert .env file exists - The file exists" { ENV_ROOT_DIR="/tmp" ENV_FILENAME=".env.bats.testing" touch ${ENV_ROOT_DIR}/${ENV_FILENAME} run env::assertFileExists assert_success } ``` #### teardown Just like `setup`, the `teardown` function will be called for each test but once the test has been fired. This is the good place for, f.i., removing some files created during the execution of a test. ```bash teardown() { rm -f /tmp/bats } ``` ### Running the tests The command `docker run --rm -it -w /code/tests -v .:/code bats/bats:latest simple.bats` will run the `simple.bats` file while `docker run --rm -it -w /code/tests -v .:/code bats/bats:latest` will run all `.bats` file in the working directory. #### Mocking We can override a function during a test. Consider the following use case: we've a function that will return 0 when a give Docker image is present on the host. The function will return 1 and echo an error on the console if the image isn't retrieved. **script.sh** ```bash function assert::dockerImageExists() { local image="${1}" local msg="${2:-The Docker image \"$image\" did not exist}" # When the image exists, "docker images -q" will return his ID (f.i. `5eed474112e9`), an empty string otherwise [[ "$(docker images -q "$image" 2>/dev/null)" == "" ]] && echo "$msg" && exit 1 return 0 } ``` So, we need to override the docker answer. When the image is supposed to be there, we just need to return a non-empty string, anything but not an empty string. Let's return a fake ID to really simulate the answer of `docker images -q`. **tests/simple.bats** ```bats @test "Assert docker image exists" { # Mock - we'll create a very simple docker override and return a fake ID # This will simulate the `docker images -q "AN_IMAGE_NAME"` which return # the ID of the image when found docker() { echo "feb5d9fea6a5" } source assert.sh run assert::dockerImageExists "A-great-Docker-Image" assert_output "" # No output when success assert_success } ``` And return an empty string to simulate an inexistent image. **tests/simple.bats** ```bats @test "Assert docker image didn't exist" { # Mock - we'll create a very simple docker override and return a fake ID # This will simulate the `docker images -q "AN_IMAGE_NAME"` which return # the ID of the image when found; here return an empty string to simulate # an inexistent image docker() { echo "" } source assert.sh run assert::dockerImageExists "Fake/image" "Bad choice, that image didn't exist" assert_output --partial "Bad choice, that image didn't exist" assert_failure } ``` ## Conclusion Bats gives Bash the thing it always lacked: a way to prove a script still does what it did last month. The setup cost is close to zero — a `.bats` file, the `bats/bats` image, and no runner to install — and the payoff shows up the first time a refactor breaks a function you had forgotten about. Start small. Take the script you're most afraid to touch, write three tests for its happy path, and add one every time you fix a bug in it. That's usually enough to turn a Bash file you no longer dare modify into one you can. ---