6. Testing System Components¶
The DIMS project has adopted use of the Bats: Bash Automated Testing System
(known as bats
) to perform simple tests in a manner that produces
parsable output following the Test Anything Protocol (TAP).
Bats is a TAP Producer, whose output can be processed by one of many TAP Consumers, including the Python program tap.py.
6.1. Organizing Bats Tests¶
This section covers the basic functionality of bats
and
how it can be used to produce test results.
We should start by looking at the --help
output for bats
to understand
how it works in general.
$ bats -h
Bats 0.4.0
Usage: bats [-c] [-p | -t] <test> [<test> ...]
<test> is the path to a Bats test file, or the path to a directory
containing Bats test files.
-c, --count Count the number of test cases without running any tests
-h, --help Display this help message
-p, --pretty Show results in pretty format (default for terminals)
-t, --tap Show results in TAP format
-v, --version Display the version number
For more information, see https://github.com/sstephenson/bats
As is seen, multiple tests – files that end in .bats
– can be passed
as a series of arguments on the command line. This can be either individual
arguments, or a wildcard shell expression like *.bats
.
If the argument evaluates to being a directory, bats
will look through that
directory and run all files in it that end in .bats
.
Caution
As we will see, bats
has some limitations that do not allow mixing file
arguments and directory arguments. You can either give bats
one or more
files, or you can give it one or more directories, but you cannot mix
files and directories.
To see how this works, let us start with a simple example that has tests that
do nothing other than report success with their name. In this case, test
a.bats
looks like this:
#!/usr/bin/env bats
@test "a" {
[[ true ]]
}
We produce three such tests, each in their own directory, following this organizational structure:
$ tree tests
tests
├── a
│ └── a.bats
└── b
├── b.bats
└── c
└── c.bats
3 directories, 3 files
Since the hierarchy shown here does not contain tests itself, but rather holds directories that in turn hold tests, how does we run the tests?
Running bats
with an argument that includes the highest level of the
directory hierarchy does not work to run any of the tests in subordinate
directories:
$ bats tests
0 tests, 0 failures
Running bats
and passing a directory that contains files with
names that end in .bats
runs all of the tests in that
directory.
$ bats tests/a
✓ a
1 test, 0 failures
If we specify the next directory tests/b
, then bats
will run the tests in that directory that end in .bats
,
but will not traverse down into the tests/b/c/
directory.
$ bats tests/b
✓ b
1 test, 0 failures
To run the tests in the lowest directory, that specific directory must be given on the command line:
$ bats tests/b/c
✓ c
1 test, 0 failures
Attempting to pass all of the directories along as arguments does not work, as seen here:
$ bats tests/a /tests/b tests/b/c
bats: /tmp/b does not exist
/usr/local/Cellar/bats/0.4.0/libexec/bats-exec-suite: line 20: let: count+=: syntax error: operand expected (error token is "+=")
This means that we can separate tests into subdirectories, to any depth or directory organizational structure, as needed, but tests must be run on a per-directory basis, or identified and run as a group of tests passed as file arguments using wildcards:
$ bats tests/a/*.bats tests/b/*.bats tests/b/c/*.bats
✓ a
✓ b
✓ c
3 tests, 0 failures
Because specifying wildcards in this way, with arbitrary
depths in the hierarchy of directories below tests/
is too hard to predict, use a program like find
to identify tests by name (possibly using wildcards or
grep
filters for names), passing the results on to
a program like xargs
to invoke bats
on each
identified test:
$ find tests -name '*.bats' | xargs bats
1..3
ok 1 a
ok 2 b
ok 3 c
Note
Note that the output changed from the examples above, which include the
arrow (“✓”) character, to now include the word ok
instead in TAP
format. This is because the default for terminals (i.e., a program that is
using a TTY device, not a simple file handle to something like a pipe). To
get the pretty-print output, add the -p
flag, like this:
$ find tests -name '*.bats' | xargs bats -p ✓ a ✓ b ✓ c 3 tests, 0 failures
A more realistic test is seen here. This file, pycharm.bats
, is the product
of a Jinja template that is installed by Ansible along with the PyCharm Community
Edition Python IDE.
#!/usr/bin/env bats
#
# Ansible managed: /home/dittrich/dims/git/ansible-playbooks/v2/roles/pycharm/templates/../templates/tests/./system/pycharm.bats.j2 modified on 2016-09-15 20:14:38 by dittrich on dimsdemo1 [ansible-playbooks v1.3.33]
#
# vim: set ts=4 sw=4 tw=0 et :
load helpers
@test "[S][EV] Pycharm is not an installed apt package." {
! is_installed_package pycharm
}
@test "[S][EV] Pycharm Community edition is installed in /opt" {
results=$(ls -d /opt/pycharm-community-* | wc -l)
echo $results >&2
[ $results -ne 0 ]
}
@test "[S][EV] \"pycharm\" is /opt/dims/bin/pycharm" {
assert "pycharm is /opt/dims/bin/pycharm" type pycharm
}
@test "[S][EV] /opt/dims/bin/pycharm is a symbolic link to installed pycharm" {
[ -L /opt/dims/bin/pycharm ]
}
@test "[S][EV] Pycharm Community installed version number is 2016.2.3" {
assert "2016.2.3" bash -c "file $(which pycharm) | sed 's|\(.*/pycharm-community-\)\([^/]*\)\(/.*$\)|\2|'"
}
$ test.runner --level system --match pycharm
[+] Running test system/pycharm
✓ [S][EV] Pycharm is not an installed apt package.
✓ [S][EV] Pycharm Community edition is installed in /opt
✓ [S][EV] "pycharm" is /opt/dims/bin/pycharm
✓ [S][EV] /opt/dims/bin/pycharm is a symbolic link to installed pycharm
✓ [S][EV] Pycharm Community installed version number is 2016.2.3
5 tests, 0 failures
6.2. Organizing tests in DIMS Ansible Playbooks Roles¶
The DIMS project uses a more elaborate version of the above example, which
uses a drop-in model that allows any Ansible role to drop its own
tests into a structured hierarchy that supports fine-grained test
execution control. This drop-in model is implemented by the
tasks/bats-tests.yml
task playbook.
To illustrate how this works, we start with an empty test directory:
$ tree /opt/dims/tests.d
/opt/dims/tests.d
0 directories, 0 files
The base
role has the largest number of tests, since it does
the most complex foundational setup work for DIMS computer systems.
The template/tests
directory is filled with Jinja template
Bash scripts and/or bats
tests, in a hierarchy that includes
subdirectories for each of the defined test levels from Section
Test levels of dimstp.
$ tree roles/base/templates/tests
roles/base/templates/tests
├── component
├── helpers.bash.j2
├── integration
│ ├── dims-coreos.bats.j2
│ └── proxy.bats.j2
├── README.txt
├── rsyslog.bats
├── system
│ ├── deprecated.bats.j2
│ ├── dims-accounts.bats.j2
│ ├── dims-accounts-sudo.bats.j2
│ ├── dims-base.bats.j2
│ ├── dims-coreos.bats.j2
│ ├── dns.bats.j2
│ ├── iptables-sudo.bats.j2
│ ├── reboot.bats.j2
│ └── updates.bats.j2
├── unit
│ ├── ansible-yaml.bats.j2
│ ├── bats-helpers.bats.j2
│ ├── dims-filters.bats.j2
│ └── dims_functions.bats.j2
└── user
├── user-account.bats.j2
└── user-deprecated.bats.j2
5 directories, 20 files
After running just the base
role, the highlighted subdirectories that
correspond to each of the test levels are now present in the
/opt/dims/tests.d/
directory:
$ tree /opt/dims/tests.d/
/opt/dims/tests.d
├── component
│ └── helpers.bash -> /opt/dims/tests.d/helpers.bash
├── helpers.bash
├── integration
│ ├── dims-coreos.bats
│ ├── helpers.bash -> /opt/dims/tests.d/helpers.bash
│ └── proxy.bats
├── system
│ ├── deprecated.bats
│ ├── dims-accounts.bats
│ ├── dims-accounts-sudo.bats
│ ├── dims-base.bats
│ ├── dims-ci-utils.bats
│ ├── dims-coreos.bats
│ ├── dns.bats
│ ├── helpers.bash -> /opt/dims/tests.d/helpers.bash
│ ├── iptables-sudo.bats
│ ├── reboot.bats
│ └── updates.bats
├── unit
│ ├── ansible-yaml.bats
│ ├── bats-helpers.bats
│ ├── dims-filters.bats
│ ├── dims_functions.bats
│ └── helpers.bash -> /opt/dims/tests.d/helpers.bash
└── user
├── helpers.bash -> /opt/dims/tests.d/helpers.bash
├── user-account.bats
└── user-deprecated.bats
5 directories, 24 files
Here is the directory structure for tests in the docker
role:
/docker/templates/tests
└── system
├── docker-core.bats.j2
└── docker-network.bats.j2
1 directories, 2 files
If we now run the docker
role, it will drop these files into
the system
subdirectory. There are now 3 additional files (see emphasized
lines for the new additions):
$ tree /opt/dims/tests.d
/opt/dims/tests.d
├── component
│ └── helpers.bash -> /opt/dims/tests.d/helpers.bash
├── helpers.bash
├── integration
│ ├── dims-coreos.bats
│ ├── helpers.bash -> /opt/dims/tests.d/helpers.bash
│ └── proxy.bats
├── system
│ ├── deprecated.bats
│ ├── dims-accounts.bats
│ ├── dims-accounts-sudo.bats
│ ├── dims-base.bats
│ ├── dims-ci-utils.bats
│ ├── dims-coreos.bats
│ ├── dns.bats
│ ├── docker-core.bats
│ ├── docker-network.bats
│ ├── helpers.bash -> /opt/dims/tests.d/helpers.bash
│ ├── iptables-sudo.bats
│ ├── reboot.bats
│ └── updates.bats
├── unit
│ ├── ansible-yaml.bats
│ ├── bats-helpers.bats
│ ├── dims-filters.bats
│ ├── dims_functions.bats
│ └── helpers.bash -> /opt/dims/tests.d/helpers.bash
└── user
├── helpers.bash -> /opt/dims/tests.d/helpers.bash
├── user-account.bats
└── user-deprecated.bats
5 directories, 26 files
You will see the tests being installed during ansible-playbook
runs, for
example (from the base
role):
. . .
TASK [base : Identify bats test templates] ************************************
Sunday 03 September 2017 13:05:45 -0700 (0:00:05.496) 0:03:48.846 ******
ok: [dimsdemo1.devops.develop -> 127.0.0.1]
TASK [base : Initialize bats_test_templates list] *****************************
Sunday 03 September 2017 13:05:46 -0700 (0:00:01.152) 0:03:49.998 ******
ok: [dimsdemo1.devops.develop]
TASK [base : Set fact with list of test templates] ****************************
Sunday 03 September 2017 13:05:47 -0700 (0:00:01.047) 0:03:51.046 ******
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/system/dims-coreos.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/system/dims-base.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/system/dims-accounts-sudo.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/system/updates.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/system/reboot.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/system/dns.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/system/deprecated.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/system/dims-accounts.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/system/iptables-sudo.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/integration/dims-coreos.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/integration/proxy.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/user/user-account.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/user/user-deprecated.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/unit/bats-helpers.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/unit/dims-filters.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/unit/ansible-yaml.bats.j2)
ok: [dimsdemo1.devops.develop] => (item=/home/dittrich/dims/git/ansible-dims-playbooks/roles/base/templates/tests/unit/dims_functions.bats.j2)
TASK [base : debug] ***********************************************************
Sunday 03 September 2017 13:06:04 -0700 (0:00:17.532) 0:04:08.578 ******
ok: [dimsdemo1.devops.develop] => {
"bats_test_templates": [
"system/dims-coreos.bats.j2",
"system/dims-base.bats.j2",
"system/dims-accounts-sudo.bats.j2",
"system/updates.bats.j2",
"system/reboot.bats.j2",
"system/dns.bats.j2",
"system/deprecated.bats.j2",
"system/dims-accounts.bats.j2",
"system/iptables-sudo.bats.j2",
"integration/dims-coreos.bats.j2",
"integration/proxy.bats.j2",
"user/user-account.bats.j2",
"user/user-deprecated.bats.j2",
"unit/bats-helpers.bats.j2",
"unit/dims-filters.bats.j2",
"unit/ansible-yaml.bats.j2",
"unit/dims_functions.bats.j2"
]
}
TASK [base : Make defined bats tests present] *********************************
Sunday 03 September 2017 13:06:05 -0700 (0:00:01.053) 0:04:09.631 ******
changed: [dimsdemo1.devops.develop] => (item=system/dims-coreos.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=system/dims-base.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=system/dims-accounts-sudo.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=system/updates.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=system/reboot.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=system/dns.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=system/deprecated.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=system/dims-accounts.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=system/iptables-sudo.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=integration/dims-coreos.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=integration/proxy.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=user/user-account.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=user/user-deprecated.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=unit/bats-helpers.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=unit/dims-filters.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=unit/ansible-yaml.bats.j2)
changed: [dimsdemo1.devops.develop] => (item=unit/dims_functions.bats.j2)
. . .
Tests can now be run by level, multiple levels at the same time,
or more fine-grained filtering can be performed using find
and grep
filtering.
6.3. Running Bats Tests Using the DIMS test.runner
¶
A test runner script (creatively named test.runner
) is available to This
script builds on and extends the capabilities of scipts like test_runner.sh
from the GitHub docker/swarm/test/integration repository.
$ base/templates/tests/test.runner --help
usage: test.runner [options] args
flags:
-d,--[no]debug: enable debug mode (default: false)
-E,--exclude: tests to exclude (default: '')
-L,--level: test level (default: 'system')
-M,--match: regex to match tests (default: '.*')
-l,--[no]list-tests: list available tests (default: false)
-t,--[no]tap: output tap format (default: false)
-S,--[no]sudo-tests: perform sudo tests (default: false)
-T,--[no]terse: print only failed tests (default: false)
-D,--testdir: test directory (default: '/opt/dims/tests.d/')
-u,--[no]usage: print usage information (default: false)
-v,--[no]verbose: be verbose (default: false)
-h,--help: show this help (default: false)
To see a list of all tests under a given test level, specify the level using
the --level
option. (The default is system
). The following example
shows a list of all the available system
level tests:
$ test.runner --list-tests
system/dims-base.bats
system/pycharm.bats
system/dns.bats
system/docker.bats
system/dims-accounts.bats
system/dims-ci-utils.bats
system/deprecated.bats
system/coreos-prereqs.bats
system/user/vpn.bats
system/proxy.bats
To see all tests under any level, use *
or a space-separated list
of levels:
$ test.runner --level "*" --list-tests
system/dims-base.bats
system/pycharm.bats
system/dns.bats
system/docker.bats
system/dims-accounts.bats
system/dims-ci-utils.bats
system/deprecated.bats
system/coreos-prereqs.bats
system/user/vpn.bats
system/proxy.bats
unit/dims-filters.bats
unit/bats-helpers.bats
Certain tests that require elevated privileges (i.e., use of sudo
)
are handled separately. To list or run these tests, use the --sudo-tests
option:
$ test.runner --list-tests --sudo-tests
system/dims-accounts-sudo.bats
system/iptables-sudo.bats
A subset of the tests can be selected using the --match
option.
To see all tests that include the word dims
, do:
$ test.runner --level system --match dims --list-tests
system/dims-base.bats
system/dims-accounts.bats
system/dims-ci-utils.bats
The --match
option takes a an egrep
expression to filter
the selected tests, so multiple substrings (or regular expressions)
can be passed with pipe separation:
$ test.runner --level system --match "dims|coreos" --list-tests
system/dims-base.bats
system/dims-accounts.bats
system/dims-ci-utils.bats
system/coreos-prereqs.bats
There is a similar option --exclude
that filters out tests by
egrep
regular expression. Two of the four selected tests are
then excluded like this:
$ test.runner --level system --match "dims|coreos" --exclude "base|utils" --list-tests
system/dims-accounts.bats
system/coreos-prereqs.bats
6.4. Controlling the Amount and Type of Output¶
The default for the bats
program is to use --pretty
formatting when
standard output is being sent to a terminal. This allows the use of colors and
characters like ✓ and ✗ to be used for passed and failed tests (respectively).
$ bats --help
[No write since last change]
Bats 0.4.0
Usage: bats [-c] [-p | -t] <test> [<test> ...]
<test> is the path to a Bats test file, or the path to a directory
containing Bats test files.
-c, --count Count the number of test cases without running any tests
-h, --help Display this help message
-p, --pretty Show results in pretty format (default for terminals)
-t, --tap Show results in TAP format
-v, --version Display the version number
For more information, see https://github.com/sstephenson/bats
Press ENTER or type command to continue
We will limit the tests in this example to just those for pycharm
and coreos
in their names. These are relatively small tests, so it is
easier to see the effects of the options we will be examining.
$ test.runner --match "pycharm|coreos" --list-tests
system/pycharm.bats
system/coreos-prereqs.bats
The DIMS test.runner
script follows this same default output
style of bats
, so just running the two tests above gives
the following output:
$ test.runner --match "pycharm|coreos"
[+] Running test system/pycharm.bats
✓ [S][EV] Pycharm is not an installed apt package.
✓ [S][EV] Pycharm Community edition is installed in /opt
✓ [S][EV] "pycharm" is /opt/dims/bin/pycharm
✓ [S][EV] /opt/dims/bin/pycharm is a symbolic link to installed pycharm
✓ [S][EV] Pycharm Community installed version number is 2016.2.2
5 tests, 0 failures
[+] Running test system/coreos-prereqs.bats
✓ [S][EV] consul service is running
✓ [S][EV] consul is /opt/dims/bin/consul
✓ [S][EV] 10.142.29.116 is member of consul cluster
✓ [S][EV] 10.142.29.117 is member of consul cluster
✓ [S][EV] 10.142.29.120 is member of consul cluster
✓ [S][EV] docker overlay network "ingress" exists
✗ [S][EV] docker overlay network "app.develop" exists
(from function `assert' in file system/helpers.bash, line 18,
in test file system/coreos-prereqs.bats, line 41)
`assert 'app.develop' bash -c "docker network ls --filter driver=overlay | awk '/app.develop/ { print \$2; }'"' failed
expected: "app.develop"
actual: ""
✗ [S][EV] docker overlay network "data.develop" exists
(from function `assert' in file system/helpers.bash, line 18,
in test file system/coreos-prereqs.bats, line 45)
`assert 'data.develop' bash -c "docker network ls --filter driver=overlay | awk '/data.develop/ { print \$2; }'"' failed
expected: "data.develop"
actual: ""
8 tests, 2 failures
To get TAP compliant output, add the --tap
(or
-t
) option:
$ test.runner --match "pycharm|coreos" --tap
[+] Running test system/pycharm.bats
1..5
ok 1 [S][EV] Pycharm is not an installed apt package.
ok 2 [S][EV] Pycharm Community edition is installed in /opt
ok 3 [S][EV] "pycharm" is /opt/dims/bin/pycharm
ok 4 [S][EV] /opt/dims/bin/pycharm is a symbolic link to installed pycharm
ok 5 [S][EV] Pycharm Community installed version number is 2016.2.2
[+] Running test system/coreos-prereqs.bats
1..8
ok 1 [S][EV] consul service is running
ok 2 [S][EV] consul is /opt/dims/bin/consul
ok 3 [S][EV] 10.142.29.116 is member of consul cluster
ok 4 [S][EV] 10.142.29.117 is member of consul cluster
ok 5 [S][EV] 10.142.29.120 is member of consul cluster
ok 6 [S][EV] docker overlay network "ingress" exists
not ok 7 [S][EV] docker overlay network "app.develop" exists
# (from function `assert' in file system/helpers.bash, line 18,
# in test file system/coreos-prereqs.bats, line 41)
# `assert 'app.develop' bash -c "docker network ls --filter driver=overlay | awk '/app.develop/ { print \$2; }'"' failed
# expected: "app.develop"
# actual: ""
not ok 8 [S][EV] docker overlay network "data.develop" exists
# (from function `assert' in file system/helpers.bash, line 18,
# in test file system/coreos-prereqs.bats, line 45)
# `assert 'data.develop' bash -c "docker network ls --filter driver=overlay | awk '/data.develop/ { print \$2; }'"' failed
# expected: "data.develop"
# actual: ""
When running a large suite of tests, the total number of individual tests
can get very large (along with the resulting output). To increase the signal
to noise ratio, you can use the --terse
option to filter out all of
the successful tests, just focusing on the remaining failed tests. This is
handy for things like validation of code changes and regression testing
of newly provisioned Vagrant virtual machines.
$ test.runner --match "pycharm|coreos" --terse
[+] Running test system/pycharm.bats
5 tests, 0 failures
[+] Running test system/coreos-prereqs.bats
✗ [S][EV] docker overlay network "app.develop" exists
(from function `assert' in file system/helpers.bash, line 18,
in test file system/coreos-prereqs.bats, line 41)
`assert 'app.develop' bash -c "docker network ls --filter driver=overlay | awk '/app.develop/ { print \$2; }'"' failed
expected: "app.develop"
actual: ""
✗ [S][EV] docker overlay network "data.develop" exists
(from function `assert' in file system/helpers.bash, line 18,
in test file system/coreos-prereqs.bats, line 45)
`assert 'data.develop' bash -c "docker network ls --filter driver=overlay | awk '/data.develop/ { print \$2; }'"' failed
expected: "data.develop"
actual: ""
8 tests, 2 failures
Here is the same examples as above, but this time using the TAP compliant output:
$ test.runner --match "pycharm|coreos" --tap
[+] Running test system/pycharm.bats
1..5
ok 1 [S][EV] Pycharm is not an installed apt package.
ok 2 [S][EV] Pycharm Community edition is installed in /opt
ok 3 [S][EV] "pycharm" is /opt/dims/bin/pycharm
ok 4 [S][EV] /opt/dims/bin/pycharm is a symbolic link to installed pycharm
ok 5 [S][EV] Pycharm Community installed version number is 2016.2.2
[+] Running test system/coreos-prereqs.bats
1..8
ok 1 [S][EV] consul service is running
ok 2 [S][EV] consul is /opt/dims/bin/consul
ok 3 [S][EV] 10.142.29.116 is member of consul cluster
ok 4 [S][EV] 10.142.29.117 is member of consul cluster
ok 5 [S][EV] 10.142.29.120 is member of consul cluster
ok 6 [S][EV] docker overlay network "ingress" exists
not ok 7 [S][EV] docker overlay network "app.develop" exists
# (from function `assert' in file system/helpers.bash, line 18,
# in test file system/coreos-prereqs.bats, line 41)
# `assert 'app.develop' bash -c "docker network ls --filter driver=overlay | awk '/app.develop/ { print \$2; }'"' failed
# expected: "app.develop"
# actual: ""
not ok 8 [S][EV] docker overlay network "data.develop" exists
# (from function `assert' in file system/helpers.bash, line 18,
# in test file system/coreos-prereqs.bats, line 45)
# `assert 'data.develop' bash -c "docker network ls --filter driver=overlay | awk '/data.develop/ { print \$2; }'"' failed
# expected: "data.develop"
# actual: ""
$ test.runner --match "pycharm|coreos" --tap --terse
[+] Running test system/pycharm.bats
1..5
[+] Running test system/coreos-prereqs.bats
1..8
not ok 7 [S][EV] docker overlay network "app.develop" exists
# (from function `assert' in file system/helpers.bash, line 18,
# in test file system/coreos-prereqs.bats, line 41)
# `assert 'app.develop' bash -c "docker network ls --filter driver=overlay | awk '/app.develop/ { print \$2; }'"' failed
# expected: "app.develop"
# actual: ""
not ok 8 [S][EV] docker overlay network "data.develop" exists
# (from function `assert' in file system/helpers.bash, line 18,
# in test file system/coreos-prereqs.bats, line 45)
# `assert 'data.develop' bash -c "docker network ls --filter driver=overlay | awk '/data.develop/ { print \$2; }'"' failed
# expected: "data.develop"
# actual: ""
Figure Using test.runner in Vagrant Provisioning shows the output of
test.runner --level system --terse
at the completion of provisioning
of two Vagrants. The one on the left has passed all tests, while the Vagrant
on the right has failed two tests. Note that the error result has been
passed on to make
, which reports the failure and passes it along
to the shell (as seen by the red $
prompt on the right, indicating
a non-zero return value).
6.4.1. Using DIMS Bash functions in Bats tests¶
The DIMS project Bash shells take advantage of a library of functions
that are installed by the base
role into $DIMS/bin/dims_functions.sh
.
Bats has a pre- and post-test hooking feature that is very tersely documented (see setup and teardown: Pre- and post-test hooks):
You can define special setup and teardown functions, which run before and after each test case, respectively. Use these to load fixtures, set up your environment, and clean up when you’re done.
What this means is that if you define a setup()
function, it will be run
before every @test
, and if you define a teardown()
function, it will
be run after every @test
.
We can take advantage of this to source the common DIMS dims_functions.sh
library, making any defined functions in that file available to be called
directly in a @TEST
the same way it would be called in a Bash script.
An example of how this works can be seen in the unit tests for the
dims_functions.sh
library itself. (We are only showing a sub-set of the
tests.)
- Lines 9-16 perform the
setup()
actions (e.g., creating directories used in a later test.) - Lines 18-21 perform the
teardown()
actions. - All of the remaining highlighted lines use functions defined in
dims_functions.sh
just as if sourced in a normal Bash script.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | #!/usr/bin/env bats
#
# {{ ansible_managed }} [ansible-playbooks v{{ ansibleplaybooks_version }}]
#
# vim: set ts=4 sw=4 tw=0 et :
load helpers
function setup() {
source $DIMS/bin/dims_functions.sh
touch --reference=/bin/ls /tmp/bats.ls-marker
for name in A B; do
mkdir -p /tmp/bats.tmp/${name}.dir
touch /tmp/bats.tmp/${name}.txt
done
}
function teardown() {
rm -f /tmp/bats.ls-marker
rm -rf /tmp/bats.tmp
}
@test "[U][EV] say() strips whitespace properly" {
assert '[+] unce, tice, fee times a madie...' say 'unce, tice, fee times a madie... '
}
@test "[U][EV] say_raw() does not strip whitespace" {
assert '[+] unce, tice, fee times a madie... ' say_raw 'unce, tice, fee times a madie... '
}
# This test needs to directly source dims_functions in bash command string because of multi-command structure.
@test "[U][EV] add_on_exit() saves and get_on_exit() returns content properly" {
assert "'([0]=\"cat /dev/null\")'" bash -c ". $DIMS/bin/dims_functions.sh; touch /tmp/foo; add_on_exit cat /dev/null; get_on_exit"
}
@test "[U][EV] get_hostname() returns hostname" {
assert "$(hostname)" get_hostname
}
@test "[U][EV] is_fqdn host.category.deployment returns success" {
is_fqdn host.category.deployment
}
@test "[U][EV] is_fqdn host.subdomain.category.deployment returns success" {
is_fqdn host.subdomain.category.deployment
}
@test "[U][EV] is_fqdn 12345 returns failure" {
! is_fqdn 12345
}
@test "[U][EV] parse_fqdn host.category.deployment returns 'host category deployment'" {
assert "host category deployment" parse_fqdn host.category.deployment
}
@test "[U][EV] get_deployment_from_fqdn host.category.deployment returns 'deployment'" {
assert "deployment" get_deployment_from_fqdn host.category.deployment
}
@test "[U][EV] get_category_from_fqdn host.category.deployment returns 'category'" {
assert "category" get_category_from_fqdn host.category.deployment
}
@test "[U][EV] get_hostname_from_fqdn host.category.deployment returns 'host'" {
assert "host" get_hostname_from_fqdn host.category.deployment
}
|
Attention
Note that there is one test, shown on lines 31 through 34, that has multiple
commands separated by semicolons. That compound command sequence needs to be
run as a single command string using bash -c
, which means it is going
to be run as a new sub-process to the assert
command line. Sourcing
the functions in the outer shell does not make them available in the sub-process,
so that command string must itself also source the dims_functions.sh
library
in order to have the functions defined at that level.
Another place that a bats
unit test is employed is the python-virtualenv
role, which loads a number of pip
packages and utilities used for DIMS
development. This build process is quite extensive and produces thousands of
lines of output that may be necessary to debug a problem in the build process,
but create a huge amount of noise if no needed. To avoid spewing out so
much noisy text, it is only shown if -v
(or higher verbosity level)
is selected.
Here is the output when a failure occurs without verbosity:
$ run.playbook --tags python-virtualenv
. . .
TASK [python-virtualenv : Run dimsenv.build script] ***************************
Tuesday 01 August 2017 19:00:10 -0700 (0:00:02.416) 0:01:13.310 ********
changed: [dimsdemo1.devops.develop]
TASK [python-virtualenv : Run unit test for Python virtualenv] ****************
Tuesday 01 August 2017 19:02:16 -0700 (0:02:06.294) 0:03:19.605 ********
fatal: [dimsdemo1.devops.develop]: FAILED! => {
"changed": true,
"cmd": [
"/opt/dims/bin/test.runner",
"--tap",
"--level",
"unit",
"--match",
"python-virtualenv"
],
"delta": "0:00:00.562965",
"end": "2017-08-01 19:02:18.579603",
"failed": true,
"rc": 1,
"start": "2017-08-01 19:02:18.016638"
}
STDOUT:
# [+] Running test unit/python-virtualenv
1..17
ok 1 [S][EV] Directory /opt/dims/envs/dimsenv exists
ok 2 [U][EV] Directory /opt/dims/envs/dimsenv is not empty
ok 3 [U][EV] Directories /opt/dims/envs/dimsenv/{bin,lib,share} exist
ok 4 [U][EV] Program /opt/dims/envs/dimsenv/bin/python exists
ok 5 [U][EV] Program /opt/dims/envs/dimsenv/bin/pip exists
ok 6 [U][EV] Program /opt/dims/envs/dimsenv/bin/easy_install exists
ok 7 [U][EV] Program /opt/dims/envs/dimsenv/bin/wheel exists
ok 8 [U][EV] Program /opt/dims/envs/dimsenv/bin/python-config exists
ok 9 [U][EV] Program /opt/dims/bin/virtualenvwrapper.sh exists
ok 10 [U][EV] Program /opt/dims/envs/dimsenv/bin/activate exists
ok 11 [U][EV] Program /opt/dims/envs/dimsenv/bin/logmon exists
not ok 12 [U][EV] Program /opt/dims/envs/dimsenv/bin/blueprint exists
# (in test file unit/python-virtualenv.bats, line 54)
# `[[ -x /opt/dims/envs/dimsenv/bin/blueprint ]]' failed
not ok 13 [U][EV] Program /opt/dims/envs/dimsenv/bin/dimscli exists
# (in test file unit/python-virtualenv.bats, line 58)
# `[[ -x /opt/dims/envs/dimsenv/bin/dimscli ]]' failed
not ok 14 [U][EV] Program /opt/dims/envs/dimsenv/bin/sphinx-autobuild exists
# (in test file unit/python-virtualenv.bats, line 62)
# `[[ -x /opt/dims/envs/dimsenv/bin/sphinx-autobuild ]]' failed
not ok 15 [U][EV] Program /opt/dims/envs/dimsenv/bin/ansible exists
# (in test file unit/python-virtualenv.bats, line 66)
# `[[ -x /opt/dims/envs/dimsenv/bin/ansible ]]' failed
not ok 16 [U][EV] /opt/dims/envs/dimsenv/bin/dimscli version is 0.26.0
# (from function `assert' in file unit/helpers.bash, line 13,
# in test file unit/python-virtualenv.bats, line 71)
# `assert "dimscli 0.26.0" bash -c "/opt/dims/envs/dimsenv/bin/dimscli --version 2>&1"' failed with status 127
not ok 17 [U][EV] /opt/dims/envs/dimsenv/bin/ansible version is 2.3.1.0
# (from function `assert' in file unit/helpers.bash, line 18,
# in test file unit/python-virtualenv.bats, line 76)
# `assert "ansible 2.3.1.0" bash -c "/opt/dims/envs/dimsenv/bin/ansible --version 2>&1 | head -n1"' failed
# expected: "ansible 2.3.1.0"
# actual: "bash: /opt/dims/envs/dimsenv/bin/ansible: No such file or directory"
#
PLAY RECAP ********************************************************************
dimsdemo1.devops.develop : ok=49 changed=7 unreachable=0 failed=1
. . .
To find out what the problem is, run the build again and add at least one -v
:
$ run.playbook -v --tags python-virtualenv
. . .
TASK [python-virtualenv : Run dimsenv.build script] ***************************
Tuesday 01 August 2017 18:54:22 -0700 (0:00:02.437) 0:01:32.394 ********
changed: [dimsdemo1.devops.develop] => {
"changed": true,
"cmd": [
"bash",
"/opt/dims/bin/dimsenv.build",
"--verbose",
"2>&1"
],
"delta": "0:02:08.917329",
"end": "2017-08-01 18:56:32.631252",
"rc": 0,
"start": "2017-08-01 18:54:23.713923"
}
STDOUT:
[+] Starting /opt/dims/bin/dimsenv.build
[+] Unpacking /opt/dims/src/Python-2.7.13.tgz archive
[+] Configuring/compiling Python-2.7.13
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
. . .
[ 10129 lines deleted! ]
. . .
virtualenvwrapper.user_scripts creating /opt/dims/envs/dimsenv/bin/get_env_details
Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',))': /source/python_dimscli-0.26.0-py2.py3-none-any.whl
Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',))': /source/python_dimscli-0.26.0-py2.py3-none-any.whl
Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',))': /source/python_dimscli-0.26.0-py2.py3-none-any.whl
Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',))': /source/python_dimscli-0.26.0-py2.py3-none-any.whl
Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',))': /source/python_dimscli-0.26.0-py2.py3-none-any.whl
Exception:
Traceback (most recent call last):
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/commands/install.py", line 335, in run
wb.build(autobuilding=True)
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/req/req_set.py", line 620, in _prepare_file
session=self.session, hashes=hashes)
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/download.py", line 821, in unpack_url
hashes=hashes
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/download.py", line 659, in unpack_http_url
hashes)
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/download.py", line 853, in _download_http_url
stream=True,
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 488, in get
return self.request('GET', url, **kwargs)
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/download.py", line 386, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 47, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/opt/dims/envs/dimsenv/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 485, in send
raise ProxyError(e, request=request)
ProxyError: HTTPSConnectionPool(host='source.devops.develop', port=443): Max retries exceeded with url: /source/python_dimscli-0.26.0-py2.py3-none-any.whl (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',)))
zip_safe flag not set; analyzing archive contents...
rpc.rpc_common: module MAY be using inspect.getouterframes
/opt/dims/envs/dimsenv/lib/python2.7/site-packages/setuptools/dist.py:341: UserWarning: Normalizing '1.0.0-dev' to '1.0.0.dev0'
normalized_version,
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
zip_safe flag not set; analyzing archive contents...
TASK [python-virtualenv : Run unit test for Python virtualenv] ****************
. . .
PLAY RECAP ********************************************************************
dimsdemo1.devops.develop : ok=50 changed=10 unreachable=0 failed=1
The highlighted lines show the problem, which is a proxy failure. This is typically due to
the Docker container used as a local squid-deb-proxy
to optimize Vagrant installations
being hung and non-responsive. (To resolve this, see restartProxy in the Appendix.)
A final example here of a bats
unit test being used to avoid hidden problems
resulting from subtle errors in Ansible playbooks is the unit test
ansible-yaml
. This test is intended to perform validations checks of
YAML syntax and Ansible module requirements.
Perhaps the most important test (and the only shown here) has to do
with avoiding the way mode
attributes to modules like copy
and template
are used. Ansible is very powerful, but it has some
pedantic quirks related to the use of YAML to create Python data
structures that are used to run Unix command line programs. This is
perhaps nowhere more problematic than file permissions (which can
horribly break things on a Unix system). The problem has to do
with the way Unix modes (i.e., file permissions) were historically
defined using bit-maps expressed in numeric form using octal
(i.e., base-8) form, not decimal (i.e., base-10) form
that is more universally used for expressing numbers. An octal
value is expressed using the digits 0
through 7
. Many
programming languages assume that if a given string representing a
numeric value starts with an alpha-numeric character in the range
[1..9]
, that string represents a decimal value, while a
string starting with a 0
instead represents an octal value.
Since YAML and JSON are text representations of data structures
that are interpretted to create binary data structures used
internally to Python
Note
Two other numeric bases used commonly in programming are
binary (base-2) and hexadecimal (base-16). Binary is used
so rarely that we can ignore it here. Because hexadecimal
goes beyond 9
, using the letters in the range [A..F]
,
it doesn’t have the same conflict in being recognized as
decimal vs. octal, so its values typically start with 0x
followed by the number (e.g., 0x123ABCD
).
You can find issues describing this decimal vs. octal/string vs. number problem and related discussion of ways to deal with it in many places:
The ansible-dims-playbooks
repository uses the convention of expressing
modes using the 0o
format to explicitly ensure proper octal modes, e.g.:
- name: Ensure /etc/rsyslog.d present
file:
path: /etc/rsyslog.d
state: directory
mode: 0o644
become: yes
when: ansible_os_family != "Container Linux by CoreOS"
tags: [ base, config, rsyslogd ]
Running the ansible-yaml
unit test will detect when modes deviate from this
standard policy, as seen here:
$ test.runner unit/ansible-yaml
[+] Running test unit/ansible-yaml
✓ [U][EV] Modes in tasks under $PBR use valid '0o' notation
✗ [U][EV] Modes in tasks under $DIMS_PRIVATE use valid '0o' notation
(in test file unit/ansible-yaml.bats, line 30)
`[ -z "$DIMS_PRIVATE" ] && skip 'DIMS_PRIVATE is not defined'' failed
/home/dittrich/dims/git/private-develop/inventory/all.yml: mode: 755
2 tests, 1 failure
Note
Exceptions to this check will likely need to be made, since the test
is for strings of the form mode: 0o
or mode=0o
(followed by
a few more numbers). These exceptions are put into a whitelist file
that the test uses to ignore them. To update the whitelist, add
any false positive failure strings to the variable
yaml_mode_whitelist
in group_vars/all/dims.yml
.