Automated testing in NetBSD 5.0

With the release of NetBSD 5.0, a new testing infrastructure for the operating system will get wide exposure. This testing infrastructure is based on the Automated Testing Framework (ATF), a project that was started as part of the Google Summer of Code 2007 program, and that provides a platform-independent framework to easily write and automatically exercise test cases. As of this release, only a few NetBSD-specific tests are available as ATF-based tests, but they are a good preview of what the future will look like.

Starting from this release, you will see a new tests.tgz distribution set during the installation of NetBSD. If you choose to install it, sysinst will populate /usr/tests with a new collection of test programs that are based on the ATF framework. Once installed, edit /etc/atf/NetBSD.conf to suit your system preferences and then, to run the tests, do:

# cd /usr/tests
# atf-run | atf-report

The whole idea of providing the test programs as an installable distribution set is that your specific combination of hardware and software is not available to system developers, so only you can make sure that the system behaves as it should. Furthermore, by successfully running the tests, you can have a good feeling that everything is working as expected!

As far as I know, there are some tests in this release that are broken, but I'm not sure if this is because the tests themselves are broken or because the features under test are broken. Help is welcome in this area!

There is still a lot of work to do in the automated testing area... but the fact that ATF is now bundled in a formal release of NetBSD raises my willingness to work on it. If only I had enough time... At the very least, expect many more ATF-based test cases in NetBSD 6.0.