December 14, 2015 posted by Thomas Klausner
Last week I met with about 40 other developers from various projects
(mostly Debian, but also Arch Linux, FreeBSD, Guix, Homebrew,
and some others) in Athens for a three day conference
about reproducible builds
, i.e. the task of getting the same binaries
from the same source on a particular platform.
The advantages are better verifyability that the source code
matches the binaries, thus addressing one of the many steps one has to
check before trusting the software one runs.
We discussed various topics during the conference in small groups:
- technical aspects (how to achieve this, how to cooperate over
- social aspects (how to argue for it with programmers, managers, lay
financial aspects (how to get funding for such work)
- lots of other stuff :)
For NetBSD, there are two parts:
Making the base system reproducible: a big part of the work for this
has already been done, but there a number of open issues, visible e.g.
in Debian's regularly scheduled test builds, up to the fact
that this is not the default yet.
Making pkgsrc reproducible: This will be a huge task, since pkgsrc
targets so many and diverse platforms. On the other hand, we have a
very good framework below that that should help.
For giggles, I've compared the binary packages for png built on
7.99.22 and 7.99.23 (in my chrooted pbulk only though) and found that
most differences were indeed only timestamps. So there's probably a
lot of low-hanging fruit in this area as well.
If you want to help, here are some ideas:
- fix the MKREPRO bugs (like PRs 48355, 48637, 48638, 50119, 50120,
- check https://reproducible.debian.net/netbsd/netbsd.html for more issues,
or do your own tests
- discuss turning on MKREPRO by default
- starting working on reproducibility in pkgsrc:
- remove gzip time stamps from binary packages
- use a fixed time stamp for files inside binary packages (perhaps
depending on newest file in sources, or latest change in pkgsrc
files for the pkg)
- identify more of the issues, like how to get symbols ordered
reproducible in binaries (look at shells/bash)
Thanks to the NetBSD developers who already worked on this before, and
to TNF for funding the travel and the Linux Foundation for funding the
accomodation for my participation in the conference, and Holger Levsen for inviting me.
June 08, 2014 posted by Martin Husemann
We all know that NetBSD-current is bloated compared to VMS of the early 80s. But we still can boot on some VAXes, so why not make it as easy as possible to install?[Read More
March 06, 2014 posted by Martin Husemann
NetBSD starts deploying gcc 4.8[Read More
October 11, 2013 posted by Thomas Klausner
The following report is by Manuel Wiesinger:
First of all, I like to thank the NetBSD Foundation for enabling me to successfully complete this Google Summer of Code. It has been a very valuable experience for me.
My project is a defragmentation tool for FFS. I want to point out at the beginning that it is not ready for use yet.
What has been done:
Fragment analysis + reordering. When a file is smaller or equal than the file system's fragment size, it is stored as a fragment. One can think of a fragment as a block. It can happen that there are many small files that occupy a fragment. When the file systems changes over time it can happen that there are many blocks containing fewer fragments than they can hold. The optimization my tool does is to pack all these fragments into fewer blocks. This way the system may get a little more free space.
Directory optimization. When a directory gets deleted, the space for that directory and its name are appended to the previous directory. This can be imagined like a linked list. My tool reads that list and writes all entries sequentially.
Non-contiguous files analysis + reordering strategy. This is what most other operating systems call defragmentation - a reordering of blocks, so that blocks belonging to the same file or directory can be read sequentially.
September 23, 2013 posted by Martin Husemann
Just a small update on the previous post
about firefox on sparc64: after a bit more work, the brand new version 24 ESR builds straight from pkgsrc (so should be included in the next set of binary pkgs).
All open issues (wrong colours on scaled images, failing https, ...) have been resolved.
Here is a new screeenshot:
May 26, 2013 posted by Martin Husemann
New firefox will be available for NetBSD/sparc64 again starting with the import of the official version 22 release into pkgsrc.[Read More
March 01, 2013 posted by Julio Merino
I am very happy to announce that the initial import of Kyua into the NetBSD source tree is complete! I am sure there are some rough edges and here is where you come into play.
First of all, let me clarify that all the integration changes are gated
by the MKKYUA build setting, which still defaults to no. Unless you
explicitly set MKKYUA=yes in /etc/mk.conf or in the build.sh command
line, you will not get a system with Kyua. However, once you set that
flag, you will transition to the full new setup:
- Kyua will be installed as /usr/bin/kyua. See kyua(1) to get started.
- The old atf-run and atf-report tools will become compatibility
wrappers. These should be a reasonable drop-in replacement for most
use cases, but they are probably not perfect.
- Your /usr/tests tree will be populated by a bunch of Kyuafiles.
So what's new compared to the ATF tools? Here are some highlights:
- Ability to write and run ATF-less tests. It has been a common desire
around here to develop test programs that do not rely on the ATF
libraries. With this change, this becomes possible. See kyuafile(5)
- Direct HTML report generation. There is no need to set up a complex
XML toolchain any more to convert the ATF test reports into HTML
pages. Kyua can generate plain HTML directly, so it will be feasible
to serve such content straight from NetBSD's built-in httpd.
- Historical data. As seen in the various test beds that have appeared
around ATF, there is a desire to maintain historical data of the test
results. Kyua does that natively, by recording the results of the
execution in a SQLite database. Reports can later be extracted from
this database. There is still a lot of room for improvement here.
- More flexible metadata and configuration. While this does not provide
a real advantage today, as soon as the old atf-run and atf-report
tools are gone we can trivially fix some long-standing issues
(e.g. the inability to customize test deadlines).
- Less complexity during test case execution. As you may have noticed
over the years, the code in atf-run to capture the output of tests and
deal with interrupts is not particularly robust. There have been
several problems in this area, and I'm not convinced that they are all
fixed. The new code works in a different manner and has been more
carefully thought around these edges.
- Independent testers. The code that implements the isolation of test
cases and their controlled execution has been split into a set of
"testers" that live in /usr/libexec/kyua-*-tester. These tools
provide scriptable interfaces to interact with tests, with the idea
that the kyua(1) frontend should end up being pretty straightforward.
Should you want to write your own trivial script to run tests without
kyua(1), you could pretty easily do that by interacting with the
How can you help? Easy. Just rebuild your system with MKKYUA=yes, read
through kyua(1), start using it to stress-test your system and report
any problems you may encounter!
My immediate next steps include addressing your feedback and working
with our major test runners to add support to their systems to use the
new tools (for example, change anita to support running tests with
Enjoy and thanks for reading.
July 30, 2012 posted by Mike M. Volokhov
Demand for a cheap yet powerful computer is pretty high in geek circles. Remember those days when you could take a ZX Spectrum, or Commodore 64, or something like that and attach it to your TV to fill next few days with a great fun. Since then, PCs pushed out these tiny home computers. But today, Raspberry Pi — a little credit-card sized sweet computer — is here.
The Raspberry Pi is built on Broadcom BCM2835 system on chip which contains ARM1176 core running at 700MHz, with VideoCore 4 GPU, and has 256 MB of RAM on board. It provides two USB ports, 100Mb/s Ethernet, HDMI and composite video; audio output presented as well. Operating system loaded from SD flash.
Yet more fun is that the initial NetBSD support was just committed to the NetBSD source tree, and the Raspberry Pi can boot into multiuser now. Currently work on device drivers is in progress. USB and Ethernet are planned to be supported next.
Generally, NetBSD support for the ARM1176 core is in very good shape. The largest parts of the code I committed were the update to the plcom serial driver and the infrastructure to support the low level parts of the bcm2835 (interrupt controller, bus_space, and timer) — says Nick Hudson, who did the bringup. However, the porting has some hard nuts to crack:
The graphics part is a bit of a challenge. But I hope to get dumb framebuffer support relatively soon. There is a publicly available datasheet for part of bcm2835, but certainly not the video controller.
Matt Thomas, the port-evbarm portmaster, was very knowledgeable and helpful in answering all my questions. I'd like to also thank Stephen Borrill for getting me an RPI in the first place. Stephen spoke to Eben Upton in Cambridge and soon after an RPI arrived — adds Nick.
So, there is a boot log from the very first NetBSD driven RPI board, and work on the device is in full swing. Many people exposed interest, and eventually the Raspberry Pi could became a good alternative for number of well supported, but aging single board computers.
April 01, 2012 posted by Michael Lorenz
Over the last few months, many of NetBSD's graphical console drivers have gained the ability to use anti-aliased fonts. This was done mostly as a way to allow relatively easy access to a vast number of fonts, but they also look a lot better than plain monochrome fonts. With the genfb(4) driver this feature is available on almost all supported hardware which supports graphics. Some specialized drivers also support it (such as radeonfb(4), r128fb(4), voodoofb(4), ffb(4) and voyagerfb(4) ) and more are being added over time.
In order to improve overall usability and to give NetBSD a more unified look across different platforms, we are going to use a single default font and colour scheme on all platforms. Since studies prove its superiour readability we are proud to announce that Microsoft has granted permission to use the Comic Sans font in all future NetBSD releases.
February 26, 2012 posted by Martin Husemann
The posix_spawn kernel implementation, mostly developed by Charles Zhang during Google Summer of Code 2011, has now been committed.
September 05, 2010 posted by Julio Merino
With this post, I would like to announce the availability of the atf tutorial published on the wiki!
This new document intends to aid NetBSD developers in writing atf-based tests for the NetBSD source tree. It does so by providing a description of the basic concepts behind the atf design and listing what the necessary steps are to write an atf-based test for the NetBSD source tree. The tutorial also includes several code snippets ready to be copy/pasted into your own code, some reference sections for common functions and a short FAQ section for common concerns raised by users of the framework.
The tutorial comes to fill an important gap in achieving wide acceptance of atf among NetBSD developers. Some of us have been trying to migrate old regress tests to the new atf framework for a while, but such efforts are futile if other developers keep submitting new tests to the obsolete regress tree. However, such developers have been doing so because the atf documentation is not yet up to the task to guide a new developer into writing a test program for NetBSD. I expect this new tutorial to cover this gap and I hope you enjoy reading it as much as I enjoyed writing it. Comments will be highly appreciated!
Lastly, I want to give special thanks to Antti Kantee for the multiple rounds of reviews and suggestions on the text.
August 11, 2010 posted by Thomas Klausner
Kristaps Dzonsons successfully implemented PostScript and PDF output for mandoc for his Google Summer of Code project.