March 01, 2013 posted by Julio Merino
I am very happy to announce that the initial import of Kyua into the NetBSD source tree is complete! I am sure there are some rough edges and here is where you come into play.
First of all, let me clarify that all the integration changes are gated
by the MKKYUA build setting, which still defaults to no. Unless you
explicitly set MKKYUA=yes in /etc/mk.conf or in the build.sh command
line, you will not get a system with Kyua. However, once you set that
flag, you will transition to the full new setup:
- Kyua will be installed as /usr/bin/kyua. See kyua(1) to get started.
- The old atf-run and atf-report tools will become compatibility
wrappers. These should be a reasonable drop-in replacement for most
use cases, but they are probably not perfect.
- Your /usr/tests tree will be populated by a bunch of Kyuafiles.
So what's new compared to the ATF tools? Here are some highlights:
- Ability to write and run ATF-less tests. It has been a common desire
around here to develop test programs that do not rely on the ATF
libraries. With this change, this becomes possible. See kyuafile(5)
- Direct HTML report generation. There is no need to set up a complex
XML toolchain any more to convert the ATF test reports into HTML
pages. Kyua can generate plain HTML directly, so it will be feasible
to serve such content straight from NetBSD's built-in httpd.
- Historical data. As seen in the various test beds that have appeared
around ATF, there is a desire to maintain historical data of the test
results. Kyua does that natively, by recording the results of the
execution in a SQLite database. Reports can later be extracted from
this database. There is still a lot of room for improvement here.
- More flexible metadata and configuration. While this does not provide
a real advantage today, as soon as the old atf-run and atf-report
tools are gone we can trivially fix some long-standing issues
(e.g. the inability to customize test deadlines).
- Less complexity during test case execution. As you may have noticed
over the years, the code in atf-run to capture the output of tests and
deal with interrupts is not particularly robust. There have been
several problems in this area, and I'm not convinced that they are all
fixed. The new code works in a different manner and has been more
carefully thought around these edges.
- Independent testers. The code that implements the isolation of test
cases and their controlled execution has been split into a set of
"testers" that live in /usr/libexec/kyua-*-tester. These tools
provide scriptable interfaces to interact with tests, with the idea
that the kyua(1) frontend should end up being pretty straightforward.
Should you want to write your own trivial script to run tests without
kyua(1), you could pretty easily do that by interacting with the
How can you help? Easy. Just rebuild your system with MKKYUA=yes, read
through kyua(1), start using it to stress-test your system and report
any problems you may encounter!
My immediate next steps include addressing your feedback and working
with our major test runners to add support to their systems to use the
new tools (for example, change anita to support running tests with
Enjoy and thanks for reading.