Once a upon a time I moved to brand new project. And without much thought, I decided to take DevOps responsibilities (after a long period of Frontend). Huge mono-repository (Angular and Node.js) gives rise to many specific problems. And this project was no exception. At the very beginning CI/CD duration was about 1.5h. And that was the biggest problem to take care of.

But at first, I want to talk about "Bourne again shell", cause CI/CD automation almost entirely was implemented by means of shell scripts (Bash). In context of huge mono-repo even regular build becomes a piece of odd stuff. That`s why a huge amount of scripts with complicated logic inside (build, test, deploy, generation of release notes, collection of logs and metrics, ...) was another significant problem.

Regardless of lang, quality of complex logic should be under control. It is no secret that key aspect of code quality control - is tests. For example, to refactor safely major logic should be covered with tests. We decided to keep shell scripts untouched and cover the major ones with tests before any refactoring. Of course, it is possible to use Jest (or Mocha) with a bunch of awful utils to test shell scripts. This approach is a bit wordy and has no value if scripts under test are written in Bash. Also I have no idea how to mock external shell commands (such as curl, ls, touch, npm, ...) with Jest or similar framework.

In our particular case it was great to have the following to test CI/CD scripts:

Tiny testing framework with no assertions API. Double square brackets statement covers majority of cases; check this out for more detail.
Tests isolation (for parallelization).
Mock API (to mock external shell commands).
JUnit reporting format (to integrate with CI).

It was not hard to implement appropriate testing framework from scratch https://github.com/redneckz/red-shell-unit

How to run tests

Lets start from the very basic example:

add.sh

#!/usr/bin/env bash
echo $(( $1 + $2 ))

This script just adds the first argument to the second one and outputs the result to stdout.

One of the approaches is to place tests along with modules. This approach simplifies things, cause technically there is no need to explicitly point out path to shell script under test. Path can be computed relative to spec.

add.spec.sh

#!/usr/bin/env red-shu.sh

it 'should add two numbers'
  # Act
  result=$(run 2 2)
  # Assert
  [[ $result -eq 4 ]]
ti

In order to make this test up and running shebang should reference red-shu.sh. It is a wrapper which provides core functionality (it/ti/run/mock) and tests isolation.

it/ti commands declare isolated block to test the only requirement. For each block subshell is initiated. If block exits with non-zero code, appropriate requirement turns red. Framework provides run command to execute script/command under test. Such decoupling makes refactoring less expensive. For example, you can move or rename scripts along with corresponding tests without difficulty.

The best technical solution is a solution that does not require any new components or modules. That`s why exit codes are used to interpret test results. In this case, no dedicated assertion API is needed. Bash`s double square brackets statement is powerful enough to cover most needs and assert anything you ever wanted. Please check this out.

Also commands like grep or diff are suitable for more complicated kinds of assertions.

How to mock external commands

Mocks are just necessary to write unit tests. Most dependencies should be mocked to test module functionality in isolation and highlight external parts of contract.

The implementation of mocks turned out to be surprisingly straightforward. To mock something just write regular function named appropriately:

function node() {
  mock::log node "$@"
  if [[ "$1" == --version ]]; then
    echo v16.4.0;
  fi
}

mock::log command makes it possible to assert invocation of mocked commands as follows:

node --version
mock::called node --version

Under the hood, mock::log writes every and each invocation to temporary file; mock::called - just greps this file. Also regular expressions could be used to assert args:

mock::called node '.*'
[[ $(mock::called_times node '.*') -eq 999 ]]

Optionally, the following utility allows to declare mocks without implementation:

mock node
node --version

Lots of shell commands require stdin as well as args. That was one of the reasons this shell-based framework to take its place. For example, to pipe something to script/command under test just use basic shell structures:

echo "Lorem ipsum..." | command under test

Or:

diff ./file.txt <(printf '%s\n' 'first line' 'second line' 'third line')

To mock command which require stdin and assert such input:

# Arrange
mock::stdin wc
# Act
run
# Assert
mock::consumed wc "some line"

How to report test results

Ofcourse, test result should be binary - either passed or failed. There should be no need to interpret test result manually. This framework offers a simple approach. As mentioned above, test is considered red if exit code is non-zero, otherwise it is green.

On the other hand, parsable output is a mandatory feature for each testing framework to make it possible to integrate with IDE, CI/CD and whatever else.

The framework produces TSV . For example:

$ add.spec.sh
REDSHU CASE 2020-02-24T12:39:01+0300 add.sh 1 should add two numbers
REDSHU PASS 2020-02-24T12:39:02+0300 add.sh 1

Columns:

REDSHU - constant prefix
CASE/PASS/FAIL - test case status
Timestamp
Script/command under test
Test case ID
Requirement description

Jenkins, Bamboo, GitLab CI/CD and all of them support JUnit reporting format. This framework comes with special utility to reformat test output to JUint format:

$ add.spec.sh | red-shu-2-junit.sh
<testsuites name="CI Tests">
  <testsuite name="add.sh" tests="1" timestamp="2020-02-24T12:39:43+0300" time="0">
    <testcase name="add.sh should add two numbers"></testcase>
  </testsuite>
</testsuites>

...

I would be grateful if you share your experience of testing of CI/CD scripts. Please try Red Shell Unit. I`m open to your proposals and new ideas.

Even shell scripts require unit tests