GSoC Reports: Benchmarking NetBSD, first evaluation report

This report was written by Apurva Nandan as part of Google Summer of Code 2020.

My GSoC project under NetBSD involves developing an automated regression and performance test framework for NetBSD that offers reproducible benchmarking results with detailed history and logs across various hardware & architectures.

To achieve this performance testing framework, I am using the Phoronix Test Suite (PTS) which is an open-source, cross-platform automated testing/benchmarking software for Linux, Windows and BSD environments. It allows the creation of new tests using simple XML files and shell scripts and integrates with revision control systems for per-commit regression testing.

About PTS

PTS core

PTS core is the engine of the Phoronix Test Suite and handles the following jobs:

• Regularly updating the test profiles, test suites, and repository index.
• Downloading, compilation, installation and evaluation of test packages.
• Test result management and uploading results and user-defined test-profiles/suites to OpenBenchmarking.org

Test-profiles/Suites & OpenBenchmarking

Test-profiles are light-weight XMLs and shell scripts that allow easy installation of the benchmarking packages and their evaluation. Test-profiles generally contains the following files:

• downloads.xml: Stores the links of the benchmarking packages, required additional packages, patches to be applied, etc. with their hash sums.
• install.sh: Actual compilation, installation, and test evaluation shell script. Also handles the task of applying patches.
• results-definition.xml: Meta-data to define the information for test result data (e.g. result data units, LIB or HIB, etc.)
• test-definition.xml: Meta-data to specify test-profile details, such as compatible OS, external dependencies, test arguments, etc.

A simple test-profile C-Ray-1.2.0 can be seen to get an idea of the XMLs and shell scripts.

Test-suites are bundles of related test-profiles or more suites, focusing on a subsystem or certain category of tests e.g. Disk Test Suite, Kernel, CPU suite, etc.

OpenBenchmarking.org serves as a repository for storing test profiles, test suites, hence allowing new/updated tests to be seamlessly obtained via PTS. OpenBenchmarking.org also provides a platform to store the test result data openly and do a comparison between any test results stored in the OpenBenchmarking.org cloud.

Usage

Test installation

The command:

$ phoronix-test-suite install test-profile-name

Fetches the sources of the tests related to the the test-profile in ~/.phoronix-test-suite/installed-tests, applies patches and carries out compilation of test sources for generating the executables to run the tests.

Test execution

The command:

$ phoronix-test-suite run test-profile-name

Performs installation of the test (similar to $ phoronix-test-suite install test-profile-name) followed by exectution the binary from the compiled sources of the test in ~/.phoronix-test-suite/installed-tests and finally reports the tests results/outcome to the user and provides an option to upload results to OpenBenchmarking.org.

Progress in the first phase of GSoC

pkgsrc-wip

The first task performed by me was upgrading the Phoronix Test Suite from version 8.8 to the latest stable version 9.6.1 in pkgsrc-wip and is available as wip/phoronix-test-suite. You can have a look at the PTS Changelog to know about the improvements between these two versions.

Major Commits:

• wip/phoronix-test-suite: update phoronix-test-suite to 9.6.1
• Added required PHP dependencies & removed TODO file

Currently, the PTS upgrade to 9.6.1 is subject to more modifications and fixes before it gets finally merged to the pkgsrc upstream. To test the Phoronix Test Suite 9.6.1 on NetBSD till then, you can setup pkgsrc-wip on your system via:

$ cd /usr/pkgsrc
$ git clone git://wip.pkgsrc.org/pkgsrc-wip.git wip

To learn more about pkgsrc-wip please see The pkgsrc-wip project homepage.

After setting up pkgsrc-wip, use the following command for installation of PTS 9.6.1:

$ cd /usr/pkgsrc/wip/phoronix-test-suite
$ make install

If any new build/installation errors are encountered, please document them in wip/phoronix-test-suite/TODO file and/or contact me.

Testing & Debugging

As we now know, having a look over OpenBenchmarking.org’s available test-profiles section or using $ phoronix-test-suite list-available-tests, there are 309 test-profiles available on PTS for Linux, and only 166 of 309 tests have been ported to NetBSD (can be differentiated by a thunder symbol on the test profile on the website). These 166 test-profiles can be fetch, build and executed on NetBSD using just a single command $ phoronix-test-suite run test-profile-name. I am ignoring the graphics ones in both Linux & NetBSD as GPU benchmarking wasn’t required in the project.

Many of the test profiles available on NetBSD have installation, build, or runtime errors i.e. they don’t work out of the box using the command $ phoronix-test-suite run test-profile-name. I ran 90 of these test profiles on a NetBSD-current x86_64 QEMU VM (took a lot of time due to the heavy tests) and have reported the status in the sheet: NetBSD GSoC: Test Porting Progress Report

You can have a look at how a PTS test-profile under action looks like!

Aircrack-NG test-profile demonstration

Aircrack-ng is a tool for assessing WiFi/WLAN network security.

The PTS test-profile is available at: https://openbenchmarking.org/test/pts/aircrack-ng

For further information, complete results can be seen at https://openbenchmarking.org/result/2007116-NI-AIRCRACKN51

AOBench test-profile demonstration

AOBench is a lightweight ambient occlusion renderer, written in C.

The PTS test-profile is available at: https://openbenchmarking.org/test/pts/aircrack-ng

For further information, complete results can be seen at: https://openbenchmarking.org/result/2007148-NI-AOBENCHTE94

Debugging and fixing non-working test-profiles

After testing these test-profiles, I debugged the following build errors in the following test-profiles:

• pts/t-test1-1.0.1: memalign() not available on NetBSD
• pts/coremark-1.0.0: calling bmake instead of gmake
• pts/apache-siege-1.0.4: Missing signal.h header
• pts/mbw-1.0.0: mempcpy() not available on NetBSD

Fixes to the above bugs can be found in the sheet or in the GitHub repositories shared in the next paragraph.

The modified test-profiles have been pushed to pts-test-profiles-dev and the fix patches to pts-test-profiles-patches. These patches are automatically downloaded by the debugged test-profiles and automatically applied before building of tests. The steps to install the debugged test-profiles have been added in the README.md of pts-test-profiles-dev.

These patches have been added in the downloads.xml of the modified test-profiles, and hence they get fetched during the test installation. The install.sh script in these modified test-profiles applies them on the benchmarking packages if the operating system is detected as NetBSD, thereby removing the build errors.

coremark test-profile demonstration

You can have a look at the patched coremark-1.0.0 test-profile under execution:

The patched PTS test-profile is available at: https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/coremark-1.0.0

This is a test of EEMBC CoreMark processor benchmark.

For further information, complete results can be seen at: https://openbenchmarking.org/result/2007148-NI-LOCALCORE64

Porting more test-profiles to NetBSD

Attempts were made to port new test-profiles from Linux to NetBSD, but the major incompatibility issue was missing external dependencies in NetBSD. Hence, this may reduce the number of test-profiles we can actually port, but more sincere efforts will be made in this direction in the next phase.

Future Plans

My next steps would involve porting the non-available tests to NetBSD i.e. the non-graphics ones available on Linux but unavailable on NetBSD, and fix the remaining test-profiles for NetBSD.

After gaining an availability of a decent number of test-profiles on NetBSD, I will use Phoromatic, a remote tests management system for the PTS (allows the automatic scheduling of tests, remote installation of new tests, and the management of multiple test systems) to host the live results of the automated benchmarking framework on benchmark.NetBSD.org, thereby delivering a functional performance and regression testing framework.

[0 comments]