Part of the drive to create Bash Builder and its associated Bash Libs was to provide a toolset to bring bash scripting closer to “real” development practices. One of those concerns is having code that is tested, by which I mean, has unit tests that prove that the code does what it claims to.
The libraries in fact were pretty broken back in the 4.x days. After attending a Python programming course I came to realize that having unit tests that sanity-check the code is a plus, and I started writing such for some of the more convoluted libs and hoo-boy was I in for a nasty surprise… Nearly every one of them had easy-to-spot corner case bugs, some even didn’t work at all as expected! I’ve been progressively thus adding more and more tests to as much of the libs as I can, though retrofitting them is proving to be hard work.
One of the problems I’ve encountered in trying to bring everything through to full testability comes in the form of the “everything is a library” approach to bash. For example, how do I write meaningful tests for the
guiout.sh library? Answer is, you don’t, you just write a single test to check if it is available.
Another is when libraries depend on session or terminal state. You can’t test all cases of
isroot.sh without shimming the library itself, because the check relies on the value of
$UID which is a read-only variable. I coudl change the library to read
$UID into a custom variable prior to the function being called, and then adjusting that custom variable during test but that would just add another possibilty of a bug should the variable somehow get overwritten. Should I do this for the sake of tests? It adds more complexity than needed, but then improves the coverage… it’s a simple example, but probably one that is not isolated.
Another issue is the inability to pass values back and forth easily apart from through “echo” statements. I only recently discovered name references (kinda-pointers) and have had to decide when to and not to retrofit or even altogether rewrite some libs to use this feature. The
arrays.sh lib for example in fact necessitated it, to provide a guarantee on data integrity. Passing back data with trailing
\n characters did not truly survive being returned, properly. I wrote a new function for that so that the old API would remain consistent… but for some libraries it wouldn’t make sense to do this, rather changing the behaviour altogether of the existing functions might be better (I do not have a specific example to hand…)
There’s hope …
At this point, most of the libraries have meaningful quantities of tests, though this does need further checking. Running
./verify.sh inside the bash-libs folder prints out a heap ton of green tests, but then proceeds to tell us that there are 11 failures. I added an extra item to list the files with no tests, and this evidences that the 11 failures are due to these missing tests, and the libs in question are fairly trivial - all apart fromthe
log.sh one which continues to give me problems to write comprehensive tests for… I might just be overcimplicating them though.
So for the most part, even if the tests show a number of “failures” I’m pretty confident they all work as they are meant to. I’d be happy to see tests that prove me wrong though. Nobdy likes latent software bugs…
EDIT: as of 2.1.13 (2019-05-15) pretty much all common libraries are tested and pass. This means that there is a safeguard against regressions at a later date, but does not in itself mean that everything is perfect:
- Even though all the tests I’ve defined pass, there are likely many corner cases for which I have not written tests
- There are a handful of libraries there which I do not know how to write tests for in the first place, as they do not use stdin/stdout/stderr to communicate.
- Tests never mean “all OK” ; they only mean “does what we expected” (as opposed to what we never thought to expect), and accessorily “haven’t changed output behaviour since we last checked” (the end result is the same, but does not guarantee that the itnernals are fully sane and not switching things in a different place)