June 24, 2010 posted by Antti Kantee
In a software project as large as NetBSD the interactions between
different software components are not always immediately obvious
to even the most skilled programmers. Tests help ensure that the
system functions according to the desired criteria. Periodic
automated runs of these tests with results visible on the web
ensures both that tests are run in a regular fashion and that the
results are available to all interested parties.
This short article explains the NetBSD test strategies and provides
a brief overview of the enabling technologies. It also details
how effortless it is to run the test suite and why doing so is in
every developer's, patch submitter's and system administrator's
best interest. The intended audience is people with a keen interest
in testing and quality assurance, and a desire to reduce personal
headache. The article is written against NetBSD-current as of June
2010 and applies to what will eventually become NetBSD 6.
Automated Testing Framework (ATF)
Julio Merino's Automated
Testing Framework (ATF) unifies the interface for running tests,
enables customizable test report formats and provides a standard
interface for implementing tests. ATF also provides a mechanism
for tests to determine if the feature under test, such as hardware,
is present in the system and skip the test instead of incorrectly
failing it. The goal is to make the tests run conveniently in
batch mode without human supervision -- hence the name automated.
ATF is shipped with NetBSD and all new NetBSD tests should be
written against ATF.
ATF tests are found under /usr/tests in a standard NetBSD installation.
As pointed out on the ATF website, this is done to
enable a system administrator to run the NetBSD test suite
for the specific hardware/software setup with minimal effort.
Executing the tests should be viewed as insurance for a particular
any test failures immediately may save a lot of head scratching
down the road.
The tests can be run with the atf-run command
in the appropriate subdirectory for a partial set of tests or on
the top level of /usr/tests for the entire NetBSD test suite.
Since the output of atf-run is meant to be post-processed by other
tools, the idiomatic command for creating a human-readable report
includes a pipe to the report generator:
atf-run | atf-report
This gives a verdict for all the tests. Also, a summary like the
following one is presented:
Summary for 25 test programs:
83 passed test cases.
0 failed test cases.
0 expected failures.
2 skipped test cases.
Further documentation for running the tests and controlling the report
format is available from the ATF manual pages
Automated NetBSD Installation and Test Application (anita)
The Automated NetBSD
Installation and Test Application (anita) is a tool written by
Andreas Gustafsson. When anita is run, a URL to release set
binaries is given as an argument. Anita downloads the release
sets, creates a disk image, boots the downloaded release in a
virtual machine and installs the release. For example, the following
command will download and install NetBSD/i386 5.0.2, and boot the
resulting installation to a command prompt:
anita interact ftp://ftp.NetBSD.org/pub/NetBSD/NetBSD-5.0.2/i386/
Currently, anita supports only QEMU
and the i386 port, although there has been interest in
adding support for other virtual containers and NetBSD ports.
Since installation is done in a virtual machine, the environment
is theoretically the same regardless of the host the command is
run on. This is both a blessing and a curse: different anita runs
are comparable regardless of where they are executed, but features
specific to certain machine configurations are not exercised.
Nevertheless, if an anita install is successful, there is reasonably
high confidence that the release it was executed for works.
What makes anita especially effective for testing installation is
that it uses "screen scrape" with the sysinst installer. This
means the display output of sysinst is read and interpreted by
anita, and commands are given as response to the output. This
tests that the installation works like a human would be doing it
and sets it apart from testers which use various machine scripts
to perform test installations.
Test Reports On The Web
In addition to anita, Andreas has written a set of tools which
fetch the current sources from cvs, build a release, and use anita
to install the release and run the ATF tests. The results are currently
available on his
The source revisions committed between each build/install/test run
are available behind the "Details" links on the page. Furthermore,
if the system build is broken, the tools make an effort to hunt
down the exact guilty commit before publishing the result as to
when the build was broken.
If despite testing efforts a regression does slip through, the logs
from the runs make it easy to track down which commit introduced
the regression, even days after it was introduced -- although
hopefully it will not take so long to correct things. Once enough
logs have been accumulated, they can also provide a source to figure
out what breaks often, due to what, and for how long. This
information can be used to prevent similar problems for occurring in
Running The Test Suite With Anita
The results mentioned in the previous section are used as a reference
point to determine the current health of NetBSD. Developers and
users submitting patches are encouraged to repeat the anita test
run to make sure changes do not have unwanted side-effects.
Additionally, developers are committed to not causing long-term
regressions in the anita test runs -- a clean test report for a
submitted patch may further convince that the patch was sufficiently
tested and should be included in NetBSD. Nevertheless, common
sense is allowed as to when this is necessary.
Although writing tests is not covered in this article, it is also
highly recommended that new features are submitted with the relevant
tests in the same package. This is also to the advantage of the
submitter since, as mentioned above, developers are required to
make sure [future] changes do not cause existing tests to fail.
Anita can be found in pkgsrc
As of writing this, pkgsrc-current (and what will become pkgsrc-2010Q2)
is required. Also, QEMU version 0.12nb3 or later is required due
to a bugfix in the CPU emulation -- test runs will hang indefinitely
without this bugfix.
A full anita install/boot/test cycle including the ATF test report
is accomplished by using the test option. For example, when
is used to build release sets into /objs/obj.i386/releasedir/i386,
the following command would be used to run an install/boot/test
anita test /objs/obj.i386/releasedir/i386/
If the development host runs NetBSD and is sufficiently up-to-date, it is possible
to simply run the ATF tests there. However, due to reasons already
mentioned, the results might or might not reflect
the anita run. The recommended "no brains necessary" method to
make sure that a change does not cause a regression in the anita
run is to do an anita run. Since it does not require
additional hardware or even disrupting current work due to having
to reboot, there is little excuse for not doing so. On the flip-side,
the anita run does not test the system configuration of the
development host, so ultimately the best choice is to run the tests
in both environments. This arguably doubles the amount of necessary
command lines from one to two, but still leaves little excuse for
not executing both.
This short article presented NetBSD testing options and provided a brief
introduction to the tools to make it happen. NetBSD testing is done periodically
with results available from a web page, but manual execution, either
piecemeal or wholesale, is possible and highly recommended.
Ultimately tests help ensure the quality of NetBSD, and it is in
everyone's personal interest to run tests on their local machines
and to include test cases along code submissions.
Thanks to Andreas and Julio for their work on these awesome tools
and for comments on a draft of this article.