Bookmarks

Feeds

GSoC Reports: Benchmarking NetBSD, second evaluation report


August 12, 2020 posted by Leonardo Taccari

This report was written by Apurva Nandan as part of Google Summer of Code 2020.

This blog post is in continuation of GSoC Reports: Benchmarking NetBSD, first evaluation report blog and describes my progress in the second phase of GSoC 2020 under The NetBSD Foundation.

In this phase, I worked on the automation of the regression suite made using Phoronix Test Suite (PTS) and its integration with Anita.

The automation framework consists of two components Phoromatic server, provided by Phoronix Test Suite in pkgsrc, and Anita, a Python tool for automating NetBSD installation.

About Phoromatic

Phoromatic is a remote management system for the Phoronix Test Suite, which allows the automatic scheduling of tests, remote installation of new tests, and the management of multiple test systems through a web interface. Tests can be scheduled to run on a routine basis across multiple test systems automatically. Phoromatic can also interface with revision control systems to offer support for issuing new tests on a context-basis, such as whenever a Git commit has been pushed. The test results are then available from the web interface.

Phoromatic client-server architecture

Phoromatic server-client architecture

The Phoromatic server relies upon a PHP/HHVM built-in web server process and a PTS-hosted WebSocket server. The web server process handles the web UI and the responsibilities of the Phoromatic server.

Phoromatic clients are testing machines installed with PTS that connect to the Phoromatic web server through the HTTP port of the server.

Phoromatic Setup

To start the Phoromatic server, Phoromatic server HTTP port and web server socket port needs to be set in ~/.phoronix-test-suite/user-config.xml as shown:

...
<Server>
      <RemoteAccessPort>8640</RemoteAccessPort>
      <Password></Password>
      <WebSocketPort>8642</WebSocketPort>
      <AdvertiseServiceZeroConf>TRUE</AdvertiseServiceZeroConf>
      <AdvertiseServiceOpenBenchmarkRelay>TRUE</AdvertiseServiceOpenBenchmarkRelay>
      <PhoromaticStorage>~/.phoronix-test-suite/phoromatic/</PhoromaticStorage>
</Server>

Phoromatic Usage

To start the Phoromatic web server for controlling local Phoronix Test Suite client systems:

$ phoronix-test-suite start-phoromatic-server

The Phoromatic web server will be hosted at localhost:8640 and will require a local account creation on the server.

Phoromatic Clients

The Phoromatic client is used for connecting to a Phoromatic server to facilitate the automatic running of tests on that client.

Phoromatic clients can be created and connected to the server using the following command:

$ phoronix-test-suite phoromatic.connect SERVER_IP:SERVER_HTTP_PORT/ACCOUNT_ID

Phoromatic server interacts with the Phoromatic clients through the HTTP port specified in the ~/.phoronix-test-suite/user-config.xml.

Phoromatic Test-schedules

A test schedule is used to facilitate automatically running a set of test(s)/suite(s) on either a routine timed basis or whenever triggered by an external script or process, e.g. Git/VCS commit, manually triggered, etc. Phoromatic provides an option for pre-install, pre-run, post-install and post-run shell scripts that are executed on the Phoromatic clients. Test-schedules can be configured to run any tests on any specific systems.

About Anita

Anita is a tool for automated testing of the NetBSD operating system. Using Anita, we can download a NetBSD distribution and install it in a virtual machine in a fully automated fashion. Anita is written in Python and uses the pexpect module to “screen scrape” the sysinst output over an emulated serial console and script the installation procedure.

Installation

Anita can be installed on NetBSD, Linux and macOS systems using the following:

$ pip install pexpect
$ git clone https://github.com/gson1703/anita/
$ python setup.py install

Phoromatic-Anita Integration

I would like to describe the workflow here briefly:

  • A test-schedule was created on the Phoromatic server meant to run pts/idle-1.2.0 test on the host machine that contains the phoromatic-anita-integration.sh as a pre-run script.
  • The script performs the following:
    • Creates a mountable disk image with an executable script for setting up Phoronix Test Suite and Phoromatic client creation on the benchmarking VM systems.
    • Invokes Anita with the appropriate command-line options for configurations and network setup and mounts the image to run the configuration script on the VM.
    • Configuration script performs hostname change, DHCP setup, NFS setup, PKG_PATH setup, PTS installation, its configuration and connecting it to the Phoromatic server through a network bridge.
  • Once the benchmarking VM systems get connected to the Phoromatic server, Phoromatic server identifies the benchmarking VM systems with their IP address, hostname and MAC address.
  • After the identification, Phoromatic initiates the pending tests on VM (test-profiles are downloaded on the go in the VM and executed) and maintains a history of the test result data.

Few points to be noted:

  • I have used a local PKG_PATH with a NFS server setup as PTS 9.6.1 is available in wip and recompiling it would be a wastage of time. Later I have planned to use the binary shard by Joyent: https://pkgsrc.joyent.com/packages/NetBSD/9.99.69/amd64/All/ once the updated PTS gets upstreamed.
  • The host machine needs some one-time manual setup like installation of QEMU, Anita, pexpect, term, cvs, etc., initial user registration on Phoromatic server, Phoromatic port setup, network bridge setup. Apart from this, the rest of the framework does not require user supervision.
VM configuration script

The following script is used as a pre-run script in the test-schedules for invoking Anita and setting up the VMs:

https://gist.github.com/apurvanandan1997/48b54402db1df3723920735f85fc7934

Networking Setup

A bridged networking mode configuration of QEMU has been used in Anita as multiple VMs will be able to accommodate with a single bridge (created on the host machine, one-time setup) using dhcpcd(8), without complicated host forwarding setup (Phoromatic server requires HTTP port forwarding).

In order to enable bridged networking for your QEMU guests, you must first create and configure a bridge interface on your host.

# ifconfig virbr0 create

Next, you must specify the newly-created bridge interface in /etc/qemu/bridge.conf:

$ sudo mkdir /etc/qemu
$ sudo touch /etc/qemu/bridge.conf && sudo chmod 644 /etc/qemu/bridge.conf
$ sudo sh -c "echo 'allow virbr0' >> /etc/qemu/bridge.conf"

Finally, in order for non-privileged processes to be able to invoke qemu-bridge-helper, you must set the setuid bit on the utility:

$ sudo chmod u+s /usr/local/libexec/qemu-bridge-helper

For more details on the bridged mode networking setup in QEMU, please refer to the following guides:

Reproducing the framework

To reproduce the framework, you need to have Phoronix Test Suite, QEMU, Anita, pexpect, cvs, xterm, makefs installed on your host machine.

For example on NetBSD:

# pkg_add qemu
# pkg_add py37-anita
$ cd pkgsrc/wip/phoronix-test-suite
$ make install

The step-by-step process to get the framework after installing PTS, including the one-time manual setup, can be summarized as follows: All control and configuration of the Phoromatic Server is done via the web-based interface when the Phoromatic Server is active.

  • Configure the port of Phoromatic server as 8640 and web socket as 8642 as described above.
  • Start the Phoromatic server using the command stated above. Phoromatic login page image
  • Create your user account on the Phoromatic server using the web interface GUI. Phoromatic default page image
  • Disable client system approval for new system addition from the settings menu in the web interface.
  • Connect the host machine as a Phoromatic client to the Phoromatic server using the command stated above.
  • Create a test-schedule for the host machine with the pre-run script as specified above and pts/idle-1.2.0 as the test-profile. Phoromatic create test-schedule image
  • Execute the test-schedule or assign it on a timed-schedule and watch it running! Phoromatic Anita integration test-scehdule
  • New VM systems with the latest NetBSD-current binaries and packages will be created and identified by Phoromatic server automatically.
  • Once for all, we need to specify what benchmarking test-profiles need to be run on the VM systems in the test-schedules section and it will be taken care of by Phoromatic. Phoromatic benchmarking system test-schedule image
  • The result history can also be viewed from Phoromatic web interface.

You can have a look at the video to get a clearer picture of how to setup the framework:

Future Plans

The regression suite is complete and final tasks of deploying it on benchmark.NetBSD.org and upstreaming the wip of Phoronix Test Suite will be done in the final phase of my GSoC project. I want to thank my mentors for their constant support.

[0 comments]

 

« GSoC 2020 Second... | Main | GSoC 2020: Report-2:... »


Post a Comment:
Comments are closed for this entry.