NetBSD Blog

The NetBSD Foundation will participate in Google Summer of Code 2026!

2026-02-23T11:32:46+00:00

We are happy to announce that The NetBSD Foundation will participate in Google Summer of Code 2026!

Would you like to learn how to contribute to open source? Google Summer of Code is a great chance to contribute to NetBSD and/or pkgsrc!

You can find a list of possible projects at Google Summer of Code project page. Please do not limit yourself to the project list... If have any cool idea/project about NetBSD and/or pkgsrc please also propose your one!

Please reach us via #netbsd-code IRC channel on Libera.Chat and/or via mailing lists.

If you are more interested about Google Summer of Code, please also check the homepage at g.co/gsoc.

Looking forward to a great Summer!

Google Summer of Code 2025 Reports: Using bubblewrap to add sandboxing to NetBSD

2025-11-08T15:00:19+00:00

This report was written by Vasyl Lanko as part of Google Summer of Code 2025.

Introduction

As of the time of writing, there is no real sandboxing technique available to NetBSD. There is chroot, which can be considered a weak sandbox because it modifies the root directory of the process, effectively restricting the process' view of the file system, but it doesn't isolate anything else, so all networking, IPC, and mounts inside this restricted file system are the same as of the system, and are accessible.

There has already been some research on implementing kernel-level isolation in NetBSD with tools like gaols, mult and netbsd-sandbox, but they haven't been merged to NetBSD. Other operating systems have their own ways to isolate programs, FreeBSD has jails, and Linux has namespaces.

Project Goals

The goal of this project is to bring a new way of sandboxing to NetBSD. More specifically, we want to implement a mechanism like Linux namespaces. These namespaces allow the isolation of parts of the system from a namespace, or, as the user sees it, from an application.

NetBSD has compat_linux to run Linux binaries on NetBSD systems, and the implementation of namespaces can also be utilized to emulate namespace-related functionality of Linux binaries.

A simple example to visualize our intended result is to consider an application running under an isolated UTS namespace that modifies the hostname. From the system's view, the hostname remains the same old hostname, but from the application's view it sees the modified hostname.

Project Implementation

Linux has 8 namespace types, in this project we will focus on only 2 of them:

UTS namespace, it is the simplest so we can focus on building the general namespace infrastructure with little namespace-specific details
mount namespace, it is a prerequisite to most other namespace types because UNIX follows the philosophy of "everything is a file", so we need a separate mount namespace to have different configuration files on the same location as the system.

Linux creates namespaces via the unshare or clone system calls, and it will also be our way of calling the namespace creation logic.

We setup the base for implementing Linux namespaces in the NetBSD kernel using kauth, the subsystem managing all authorization requests inside the kernel. It associates credentials with objects, and because the namespace lifecycle management is related to the credential lifecycle it handles all the credential inheritance and reference counting for us. (Thanks kauth devs!)

We separate the implementation of each namespace in a different secmodel, resulting in a similar framework to Linux which allows the isolation of a single namespace type. Our implementation also allows users to pick whether they want to have namespace support, and of what kind, via compilation flags, just like in Linux.

UTS namespace

UTS stands for UNIX Timesharing System, because it allows multiple users to share a single computer system. Isolating the utsname can be useful to give users the illusion that they have control over the system's hostname, and also, for example, to give different hostnames to virtual servers.

The UTS namespace stores the namespace's hostname, domain name, and their lengths. To isolate the utsname we need to first create a copy of the current UTS information, plus we need a variable containing the number of credentials referencing this namespace, or, in simpler terms, the reference count of this namespace.

This namespace specific information needs to be saved somewhere, and for that we use the credential's private_data field, so we can use a UTS_key to save and retrieve UTS related information from the secmodel. The key specifies the type of information we want to retrieve from the private_data, hence using a UTS_key for the UTS namespace. The key for each namespace is a fixed value (we don't create a new key for every credential), but the retrieved value for that key from different credentials may be different.

We had to modify kernel code that was directly accessing the hostname and domainname variables, to instead call get_uts(), which retrieves the UTS struct for the namespace of the calling process. We didn't modify occurrences in kernel drivers because drivers are not part of any namespace, so they should still access the system's resources directly.

MNT namespace

The MNT namespace isolates mounts across namespaces. It is used to have different versions of mounted filesystems across namespaces, meaning a user inside a mount namespace can mount and unmount whatever they want without affecting or even breaking the system.

The mount namespace structure in Linux is fairly complicated. To have something similar in NetBSD we need to be able to control the mounts accessed by each namespace, and for that we need to control what is each namespace's mountlist, this is also enough for unmounting file systems, because in practice we can just hide them.

For the mount_namespace, mountlist structure and the number of credentials using the mount namespace are stored in the credential's private data with the MNT_key. Similarly to the UTS namespace, we had to modify kernel code to not directly access the mountlist, but instead go through a wrapper called get_mountlist() which returns the correct mountlist for the namespace the calling process resides in.

Implementation for the mount namespace is immensely more complex than for the UTS namespace, it involves having a good understanding of both Linux and NetBSD behaviour, and I would frequently find myself wondering how to implement something after reading the Linux man pages, which would lead to me looking for it in the Linux source code, understanding it, then going back to NetBSD source code, trying to implement it, and seeing it's too different to implement in the same way.

Project Status

You can find all code written during this project in GitHub at maksymlanko/netbsd-src gsoc-bubblewrap branch. Because I intend to continue this work outside of GSoC, I want to reinforce that this was the last commit still during GSoC on gsoc-bubblewrap branch and this was the last one for the mnt_ns still WIP branch.

The link includes implementation of general namespace code via secmodels, implementation of the UTS namespace and related ATF-tests, and the work-in-progress implementation of mount namespaces.

The mount namespace functionality is not finished as it would require much more work than the time available for this project. To complete it, it would be required invasive and non-trivial changes to the original source code, and, of course, more time.

Future Work

As previously mentioned, Linux has 8 namespace types, it is important to see which of the missing namespaces are considered useful and feasible to implement.

I believe that after mount namespaces it would be interesting to implement PID namespaces as this in combination with mount namespaces would permit process isolation from this sandbox. Afterwards, implementing user namespaces would allow users to get capabilities similar to root in the namespace, giving them sudo permissions while still restricting system-wide actions like shutting down the machine.

A lower hanging fruit is to implement the namespace management functionality, which in Linux is lsns to list existing namespaces, and setns to move the current process to an already existing namespace.

Challenges

Semantics. Did you know the unmount system call with MNT_FORCE flag in Linux (usually) returns EBUSY, and in NetBSD it forces the unmounting? One of them makes it easier to implement mount namespaces.
The behaviour of namespaces is not fully specified in the man pages. If something is not clear from the man pages you need to read the source code.
Unexpected need to learn a lot of VFS concepts and their differences in NetBSD and Linux.
There was a much bigger research component than I anticipated.

In the end, Linux and NetBSD are different operating systems, implemented in different ways. Linux is complex and it is not trivial to port namespaces to NetBSD.

Notes

The project is called "Using bubblewrap to add sandboxing to NetBSD" and was initially projected to emulate the unshare system call into compat_linux, but, seeing that having namespaces could be useful for NetBSD, and that it would be easy to add to compat_linux afterwards, we decided to instead implement namespaces directly in the NetBSD kernel. Implementing other system calls necessary to make the bwrap linux binary work correctly also wouldn't be as satisfying as implementing namespaces directly into NetBSD, so this was why the project was initially called "Using bubblewrap to add sandboxing to NetBSD" but nowadays it would be more accurate to call it "Sandboxing in NetBSD with Linux-like namespaces".

Thanks

I am very grateful to Google for Google Summer of Code, because without it I wouldn't have learned so much this summer, wouldn't have met with smart and interesting people, and for sure wouldn't have tried to contribute to a project like NetBSD, even if I always wanted to write operating systems code... But, the biggest thing I will take with me from this project is the confidence to be able to contribute to NetBSD and other open source projects.

I would also like to thank the members of the NetBSD organization for helping me throughout this project, and more specifically:

Taylor R. Campbell, Harold Gutch and Nia Alarie from IRC, for helping me fix a nasty LD_LIBRARY_PATH bug I had on my system which wouldn't let me finish compiling NetBSD, and general GSoC recomendations.
Emmanuel Dreyfus from tech-kern, with whom I discussed ideas for projects and proposal suggestions, and in the end inspired the namespaces project.
Christoph Badura and Leonardo Taccari who volunteered to be my mentors. They took time to research and answer my questions, anticipated possible problems in my approaches, and always pointed me in the right direction, daily, during all of GSoC's period. This project is from the 3 of us.

Google Summer of Code 2025 Mentor Summit in Munich, Germany: travel notes

2025-10-31T11:41:23+00:00

I just came back home from Google Summer of Code 2025 Mentor Summit. We were 185 mentors from 133 organizations and it was amazing!

Google Summer of Code (GSoC) is a program organized by Google with the focus to bring new developers to open source projects.

The NetBSD Foundation has been participating in GSoC since 2005.

After nearly a decade being part of GSoC for The NetBSD Foundation, first as student and then as mentor and org admin, I finally attended my first GSoC Mentor Summit! That was a fantastic, very intense and fun experience! I met with a lot of new folks and learned about a lot of other cool open source projects.

Let's share my travel notes!

Wednesday 22, October 2025: arriving in Munich

Going to Munich is relatively doable by train from my hometown. I departed from Urbisaglia-Sforzacosta at 6:59 in the morning. I had around 25 minutes to wait for the change from Ancona to Bologna. I arrived in Bologna at around 11:30 where I met Andrea, my friend and favorite music pusher since childhood. We had lunch together, eating tasty miso veggie ramen, drank some hot sake and then we had coffee. He then accompanied me back to the station where I had the train to Munich Central Station at 13:50.

The scenery from the train was really nice. Near Trento and Bolzano/Bozen, full of vineyards and apple orchards with mountains in the background. It was cloudy for most of the travel but starting from Bressanone/Brixen I began to see «beautiful blue skies and golden sunshine». After Bressanone the scenery was more uncontaminated with light green grazing lands. Unfortunately when reaching Brennero/Brenner (last Italian city before Austria) it started to get dark and I had not enjoyed the rest of the scenery in Austria and Germany. I arrived in Munich at 20:50 and checked in at my hotel which was around 1km from the station.

For this journey I was not alone! Also Christoph Badura (<bad@>) was a delegate for Google Summer of Code and we had been in touch to get dinner and beers together. Christoph had some train delays but at 21:40 we were able to meet and went for a walk a bit to the south-east to find some places to eat and drink. I had done my homework for the beer places (obviously!) but the place in my TODO list to visit on Wednesday did not have a lot of food so we decided to first go to a restaurant and we found Ha Veggie - Vietnamese Cuisine. I had some Edamame and Bò xào sả ớt, a delicious dish with seitan, vegetables, lemongrass and chili pepper.

We then stopped at Frisches Bier, a bit too late, but the publican was kind enough and she permitted us a last round. I had a pint of a refreshing Hoppebräu Wuide Hehna Session IPA.

We then took a walk back to our hotels, talked a bit and went to sleep.

Thursday 23, October 2025: 1st day of Mentor Summit

Thursday was the first day of the Mentor Summit. The summit was in Munich Marriott Hotel, more in the north of the city, around 5km from the central station.

I checked out of my hotel and walked to the city center in Marienplatz and nearby. I also stopped in a couple of shops to grab some souvenirs for my family and friends and then took a long walk in the direction of Munich Marriott Hotel, to hopefully be there at 13:00 sharp for the start of check-in of GSoC Mentor Summit.

Rear of the Siegestor showing its inscription that can be translated to "Dedicated to victory, destroyed by war, urging peace"

I walked through Ludwigstraße and enjoyed the architecture around me, walking near LMU University and Siegestor. I then proceeded through Leopoldstraße and Ungererstraße and then arrived to the Munich Marriott Hotel.

Chris was already in the lobby and he had already checked in. We talked a bit and then I checked in as well. The room was huge and comfy! I quickly went back down to the lobby. We then checked in for Mentor Summit and I finally met Stephanie, Mary and Lucy, the GSoC Program Admins. I also took with me from home some classical and specialty Italian chocolate (Cioccolato di Modica) for the chocolate room (more about that later!) and left the bars in the chocolate room.

The time from 13:00-17:00 was reserved to actually permit mentors to arrive. At 13:00 there were still not a lot of mentors around so with Chris we decided to have lunch. We had lunch in a trattoria where we had an antipasto of grilled vegetables, penne all'arrabbiata with red wine from Montepulciano.

While eating Chris talked about the Tantris but we were already full. We had not tasted the haute cuisine but walked there to just look at the restaurant building. O:)

Tables full of chocolate and sweets from the chocolate room

When we came back to Munich Marriott Hotel, I went to the chocolate room to taste some chocolate/sweets. In GSoC Mentor Summit it is a tradition to bring great quality chocolate - or other sweets for places where chocolate is less usual - so folks can taste sweets from all over the world. That's a very nice initiative! I was curious more about non-chocolate sweets completely new to me so I had some Laddu and Kaju katli, both delicious!

I spent the rest of the afternoon down at the Champions Bar socializing with other mentors.

We had dinner at around 19:00 with good food accompanied by a couple of glasses of Primitivo.

At 20:15 we had the Opening Session. Stephanie, Mary and Robert warmly welcomed us. They shared the schedule and introduced to the unconference format of the sessions. We then had dessert and played the GSoC 2025 Mentor Summit Scavenger Hunt. The Scavenger Hunt is a game where you can meet and find 25 different folks with something that could be common (e.g. «prefers spaces (not tabs)») to something pretty rare (e.g. «has jumped out of a helicopter»). This game was nice because it was also a great conversation starter. I met a lot of mentors both of open source software that I regularly use but also learned about new interesting open source software and organizations while doing that!

We had time until Friday 12:30 and 10 lucky mentors who completed it (at the end around 60 of 185 were able to complete) got randomly selected and they got special prizes.

I stayed up until probably 1am or so, socializing a bit more in the lobby and then went back to my room to have some sleep, knowing that Friday was completely packed with Lightning talks and sessions!

Friday 24, October 2025: 2nd day of Mentor Summit

I had breakfast around 8:20 at the Green's Restaurant. I sat at a table together with other folks and after a minute I saw a known name in front of me: Lourival Pereira Vieira Neto <lneto@>! I was very happy to meet another NetBSD developer and that was a complete surprise. He was there as a mentor of the LabLua organization.

GSoC Program Admins welcomed us for the day and recapped the schedule for Friday and Saturday.

Lightning talks, round 1

The lightning talks consisted of mentors presenting their best GSoC 2025 projects. The format was fast and fun: a maximum of 3 minutes for the talk and a maximum of 4 slides! We had presentations from 18 different mentors and orgs and all of them were able to stay under 3 minutes!

That was a great occasion also to learn about open source projects, new orgs and the experiences shared were interesting too.

GSoC Feedback Session

After the 1st round of Lightning Talks I attended the GSoC Feedback Session. That was a Q&A session with program admins and org admins/mentors.

Hot topics this year were AI usage and spam applications that were not discussed as part of this session because there were two other separate sessions regarding that later.

If I only have one sentence to summarize this session... I should quote Robert sharing that Google Summer of Code is about the journey for the contributors and mentors to get involved in open source. The coding is only the means to an end.

"Hallway track", lunch and group photo

After the first session I decided to take a break and instead stay in the "hallway track" where I met new folks and socialized a bit. Another funny and at the same time a bit embarrassing for me thing of GSoC is that I often met someone and after a couple of minutes of talk I can associate the face with a name and I figured out that I'm an avid user / pkgsrc MAINTAINER of the software they contribute to! :)

At 12:30 we had lunch at Green's Restaurant and then at 13:40 we had a group photo and it was pretty tricky to put around 200 folks (program admins and mentors) on the stage of the Ballroom! :)

Let's talk about improving diversity + inclusion in FOSS!

In the afternoon I joined the session about improving diversity. In open source unfortunately there are a lot of underrepresented groups and we should fix that.

There were a lot of experiences shared from several orgs, food for thought for me! Only to name few topics: Outreachy, how to know and create safe spaces, importance of localization in software and documentation, be sure to make underrepresented folk as part of key people and also try to take the burden of other tasks off them.

GSoC and AI

Artificial Intelligence (AI), in particular Generative AI (GenAI) has been a hot topic since project proposals opened this year!

Some people consider it a speed-up for researching but at the same time it impedes learning.

In NetBSD - according to our Commit Guidelines - code generated by large language model (LLM) or similar technologies is considered tainted code because such models can be trained on copyrighted materials and such resulting code can then violate copyright.

More than 80% of GSoC contributors who filled an anonymous survey used AI, mainly for code generation, code completion, text generation, debugging and error detection.

Most mentors are usually not happy with the outcomes of AI with code often resulting in buggy/vulnerable and poor quality, violating copyright and some mentors also pointed out that as part of mentoring we should also make contributors aware of environmental/ecological impact of such use.

However, both contributors' and mentors' surveys on AI are relatively small dataset (around 90 mentors and 90 contributors).

Lightning talks, round 2

At 16:00 we had the 2nd and last round of Lightning talks. That was another great opportunity to learn more about more projects and organizations!

Asynchronous I/O by Ethan Miller

Christoph Badura (<bad@>) presenting his lightning talk

Christoph Badura (<bad@>) did a lightning talk too and he presented work done by Ethan Miller. Ethan also blogged about his work, please read Google Summer of Code 2025 Reports: Asynchronous I/O Framework if you missed it!

This code was also imported by Christos Zoulas (<christos@>), thanks Christos!, and is now part of -current and it will be in NetBSD 12.0.

GSoC spammy proposals

After the Lightning talks there was a break and then at 17:30 I joined the session about GSoC spammy proposals.

This year most organizations received many more proposals, mostly due contributors starting to massively use GenAI.

A lot of suggestions and tips were shared to make the mentor review job smooth and easy as possible.

The most important suggestion is that mentors must do a 1:1 conversation with potential contributors before accepting them. The weight of the project proposal is like 2/10 and the actual 8/10 weight is on conversations between mentors and contributor.

Dinner and social event

Around 19:00 we had dinner, desserts and socialized. Stephanie also did a final talk recapping GSoC 2025 and thanking all mentors for making that possible.

We then had drinks and it then started the karaoke session (and there were a lot of pro folks doing that, very nice!).

The karaoke session at Ballroom closed with the waiter singing Closing Time (not the one by Tom Waits that is mostly instrumental, but the one by Semisonics!, I did not know it but that's melancholic as well for me, just smells a little bit less of whisky and cigs compared to the Tom Waits one ;)).

We went downstairs to the Champions Bar, had two rounds of some good Higgins Ale Works IPA and socialized a bit more. Time passed pretty quickly and also the barman there at Champions Bar started singing Closing Time!

We went a bit outside and in the lobby talking with other mentors and then I went back to my room to get some sleep for the last day of the summit.

Saturday 25, October 2025: 3rd day of Mentor Summit

I had breakfast around 8:00, a bit earlier, given that on Saturday the first session started at 9:00.

Porting & Packaging

At 9:00 I joined a session about porting and packaging. We had both FreeBSD porters, pkgsrc maintainers and other package systems maintainers on one side. On the other side there were also a lot of upstreams.

We shared do-s and don't-s on packaging.

How to get students to engage with the community

Christoph Badura (<bad@>) proposed a session to share experiences on how to get contributors to engage with the community and a lot of mentors provided a lot of great suggestions.

One thing that most mentors agreed on and worked well was to invite contributors regular (or less regular, to avoid putting too much pressure on contributors) blog posts / status updates.

Some organizations also did that as part of their weekly / bi-weekly updates that are often video meetings. In that case they reserved a slot for the contributor so that they can share their status updates.

These are great opportunities for the contributor to get in touch with the community.

Open source tools for supply chain security

I then joined a session about open source tools for supply chain security.

We discussed about Software Bill of Materials (SBOM) and its importance in the context of regulations like EU Cyber Resilience Act (CRA).

We also discussed Common Platform Enumeration (CPE) and Package URL (PURL) schemas

We talked about vulnerability management and I shared a bit my experience in pkgsrc-security@ and how often the metadata in CVEs (like vendor, product and versions affected) is not that good. Most package systems have their own workflows and usually add extra metadata only for their vulnerability DB.

I also learned about Vulnerability Exploitability eXchange (VEX) that some package systems use.

There were also mentors from AboutCode and SW360, projects that looks very interesting and I should learn more about them!

Vintage computing

Before lunch I joined the Vintage computing session.

Everyone presented themselves and talked about the most vintage computer they had, how running old machines is both fun and productive and we also talked about old Unix-es and The Unix Heritage Society.

Lunch

I had lunch at Green's Restaurant with other mentors and then socialized a bit outside Ballroom.

Waitlisted Lightning talk and funny stories

At 14:00 we had the last sessions. I went to the waitlisted Lightning talk, that can be considered the Lightning talk, round 3! :)

Mentors from different organizations shared interesting projects.

After the lightning talks several of us shared funny stories/hacks. Like learning languages in interesting ways, taking photographs of garbage to train and realize a robot that cleans it, a sort of Tinder for food... And much more! :)

Closing session

At 15:30 we had the closing session.

Some of us stayed for another 1 or 2 hours and talked, socialized a bit more and said see you soon to each other.

Going near Munich East Station and dinner with Chris

Around 17:00 I left the Munich Marriott hotel to check in to my hotel for Saturday night near Munich East Station. It was raining for most of the morning and afternoon but luckily around 17:00 it stopped and the sky seemed fine. I decided to take a walk - a bit more than 6km - to reach the hotel. Also that time I'm happy that I took a long walk because I was able to stay for most of my walk in the Englischer Garten.

Glass of Camba Island (NEIPA) beer and list of draft beers

I checked in at my hotel and then went to Tap-House where with Christoph we planned to have dinner and a couple of beers. Tap-House had a really huge choice of craft beers with 40 drafts! We had a pinsa marinara, a flatbread similar to pizza and I had a couple of small IPAs from Camba Bavaria and Yankee & Kraut Pure HBC 630, a DNEIPA.

We decided to take a walk and went to BrewsLi for some last good night beers. BrewsLi was a very nice brew pub. We sat at a table and near us there were several board games. Chris took one of this board game Mensch ärgere Dich nicht and explained to me the rules. A lot of aleatory is involved but there is also some strategy and it was funny to play and we probably played for 40 minutes or so because most of our game pieces returned to the "out" section. I took a walk with Chris back to the nearest metro station and I came back to my hotel around 2:00.

Mensch ärgere Dich nicht board game

Sunday 26, October 2025: stop in Bolzano/Bozen

On Sunday I took a train from Munich East Station to Bolzano/Bozen because it was unfeasible to go back home by train to Marche region without a stop somewhere in Italy.

View of Bolzano from St. Oswald Promenade

I went for a walk Passeggiata di Sant'Osvaldo (St. Oswald Promenade) uphill to be able to enjoy a view of the city from the top until sunset.

I had a simple but very tasty onion soup and a Gose (really good, one of the best Gose I've drunk!) and Session IPA at Batzen Häusl.

I was not able to visit anything else in the night because I was pretty tired so I went to sleep earlier.

Monday 27, October 2025: back home

In the morning I took a walk to Walther Square and nearby. I got some souvenirs for the family and then I took the trains back home and spent most of my day on the trains.

Conclusion

Google Summer of Code 2025 Mentor Summit was an amazing experience!

I had the chance to participate in very interesting talks, sessions and discussions. I met a lot of mentors from all over the world and learned about new open source projects and organizations. All the folks were also extremely positive, easy to talk to and I had a lot of fun.

Thanks to program admins and all the mentors who made this possible! Thanks a lot also to Google for organizing it and thanks to The NetBSD Foundation that permitted me to go!

NetBSD hand-written logo on the GSoC guest book

If you are new to open source, consider applying for it! If you are a seasoned open source contributor, consider participating as a mentor! You can learn more about GSoC at Google Summer of Code (GSoC) website.

Google Summer of Code 2025 Reports: Enhancing Support for NAT64 Protocol Translation in NetBSD, part 2

2025-10-20T14:19:28+00:00

This report was written by Dennis Onyeka as part of Google Summer of Code 2025.

This is the 2nd blog post about his work. If you have missed the first blog post please read Google Summer of Code 2025 Reports: Enhancing Support for NAT64 Protocol Translation in NetBSD.

Overview

Typical rules looks like:

map wm0 algo "nat64" 64:ff9b:2a4:: -> 192.0.2.33
map wm0 algo nat64 plen 96 64:ff9b::8c52:7903 <- 140.82.121.3

This tells NPF to translate outgoing IPv6 packets using the prefix 64:ff9b:2a4::/96, rewriting them to use the IPv4 address 192.0.2.33. When the packet returns and hits NPF, it changes source from GitHub's IPv4 to GitHub's IPv6 address and then it rewrites the header.

When an IPv6 packet from a client hits the NPF machine

The NAT64 translation routine (npf_nat64_rwrheader()) rewrites the IPv6 header to an IPv4 header.
- The source becomes the host's IPv4 address or any pool of IPv4 addresses.
- The destination becomes the IPv4 address of github.com extracted from IPv4 embedded IPv6 address defined in the rule configuration. (e.g. 140.82.121.3 from 64:ff9b::8c52:7903).
TCP/UDP/ICMP checksums are recalculated using in4_cksum() or in6_cksum().
The packet is then routed out to the IPv4 network.

When a reply packet comes back from the IPv4 server

NPF performs the reverse translation, embedding the IPv4 address inside the NAT64 prefix to form a valid IPv6 address using npf_embed_ipv4()
The IPv6 packet is then delivered back to the IPv6 client.

Project Accomplishments

Core Translation Path: Implemented IPv6 -> IPv4 header and reverse rewriting (npf_nat64_rwrheader() routines).
Address Mapping: Added functions for embedding and extracting IPv4 addresses within IPv6 prefixes.
Checksum Recalculation: Integrated checksum updates for IPv4/IPv6 and transport layers.
Rule Parsing: Extended npf.conf(5) syntax and parser to accept NAT64 configuration parameters.
Userland and Kernel Integration: Updated kernel headers, userland utilities, and rule constructors.
Testing: Verified translation with ping, curl and dig, observing packets using tcpdump and Wireshark.

Summary

This project successfully integrates NAT64 along with a separate DNS64 configuration into NPF, enabling IPv6-only clients to reach IPv4-only servers through seamless translation. Although there's a need for additional changes and implementation.

This is indeed the end of my GSoC Program, it was indeed an exciting moment working with system developers and certainly I'd be an active contributor to the NetBSD codebase.

Source code of the Google Summer of Code project can be found at the following branch.

Google Summer of Code 2025 Reports: Asynchronous I/O Framework

2025-08-30T17:56:23+00:00

This report was written by Ethan Miller as part of Google Summer of Code 2025.

Introduction

The goal is to improve the capabilities of asynchronous IO within NetBSD. Originally the project espoused a model that pinned a single worker thread to each process. That thread would iterate over pending jobs and complete blocking IO. From this, the logical next step was to support an arbitrary number of worker threads. Each process now has a pool of workers recycled from a freelist, and jobs are grouped per-file so that we do not thrash multiple threads on the same vnode which would inevitably lock. This grouping also opens the door for future optimisations in concurrency. The guiding principle is to keep submission cheap, coalesce work sensibly, and only spawn threads when the kernel would otherwise block.

Project Details and Status

We pin what is referred to as a service pool to each process, with each service pool capable of spawning and managing service threads. When a job is enqueued it is distributed to its respective service thread. For regular files we coalesce jobs that act on the same vnode into one thread. If we fall back to the synchronous IO path within the kernel it would lock anyway, but this approach is prudent because if more advanced concurrency optimisations such as VFS bypass are implemented later this is precisely the model that would be required. At present, since that solution is not yet in place, all IO falls back to the synchronous pipeline. Even so there are performance gains when working with different files, since synchronous IO can still run on separate vnodes at the same time.

Through the traditional VFS read/write path, requests eventually reach bread/bwrite and block upon a cache miss until completion. This kills concurrency. I considered a solution that bypassed the normal vnode read/write path by translating file offsets to device LBAs with VOP_BMAP, constructing block IO at the buffer and device layer, submitting with B_ASYNC, and deferring the wait to the AIO layer with biodone bookkeeping instead of calling biowait at submission. This keeps submission short and releases higher level locks before any device wait. The assumptions are that filesystem metadata is frequently accessed therefore cached so VOP_BMAP usually does not block, that block pointers for an inode mostly remain stable for existing data, and that truncation does not rewrite past data. For the average case this would provide concurrency on the same file. In practice, however, it was exceptionally difficult to implement because the block layer lacks the necessary abstractions.

This is, however, exactly the solution espoused by FreeBSD, and they make it work well because struct bio is an IO token independent of the page and buffer cache. GEOM can split or clone a bio, queue them to devices, collect child completions, and run the parent callback. Record locks are treated as advisory so once a bio is in flight the block layer completes it even if the advisory state changes. NetBSD has no equivalent token. Struct buf is both a cache object and an IO token tied to UBC and drivers through biodone and biowait. For now the implementation of service pools and service threads lays the groundwork for asynchronous IO. Once the BIO layer reaches adequate maturity, integrating a bio-like abstraction will be straightforward and yield immediate improvements for concurrency on the same vnode. The logical next step is to design and port something comparable to FreeBSDs struct bio which would map very cleanly onto the current POSIX AIO framework.

My Development Environment

My development setup is optimised for building and testing quickly. I use scripts to cross-build the kernel and boot it under QEMU with a small FFS root. The kernel boots directly with the QEMU option -kernel without any supporting bootloader. Early on I tested against a custom init dropped onto an FFS image. Now I do the same except init simply launches a shell which allows me to run ATF tests without a full distribution. This makes it possible to compile a new kernel and run tests within seconds.

Lessons

One lesson I have taken away is that progress never happens overnight. It takes enormous effort to get even a few thousand lines of highly multi-threaded race-prone code to behave consistently under all conditions. Precision in implementation is absolutely required. My impression of NetBSD is that it is a fascinating project with an abundance of seemingly low-hanging fruit. In reality none of it is truly low-hanging or simple, but compared to Linux there remains a great deal of work to be done. It is not easy work but the problems are visible and the path forward is clearer.

I also want to note that I intend on providing long term support for this code in the case that any issues may arise.

The code written as part of this project can be found here.

The NetBSD Foundation 2025 Annual General Meeting summary and logs

2025-05-28T17:46:58+00:00

On May 17, 21:00 UTC we had The NetBSD Foundation Annual General Meeting on #netbsd-agm IRC channel on Libera.Chat.

We had presentations from:

board (billc)
secteam (billc)
releng (martin)
core (riastradh)
finance-exec (riastradh)
membership-exec (martin, christos)
pkgsrc-pmc (wiz)
pkgsrc-security (tm, leot)
gnats (dh)

At the end we also had a Q&A session open to anyone.

If you have missed it you can find the IRC logs here.

Welcome to Google Summer of Code 2025 contributors!

2025-05-08T19:09:56+00:00

We are happy to announce that The NetBSD Foundation will participate in Google Summer of Code 2025 with 3 projects!

Here the list of the projects and contributors:

For the next 3 weeks mentors and contributors will get in touch for the community bonding period. Mentors will help contributors to get started with the project, introduce them to the community and get more familiar with the codebase and adjusting deliverables for the the project.

Welcome Dennis, Ethan and Vasyl!

The NetBSD Foundation will participate in Google Summer of Code 2025!

2025-02-27T19:10:49+00:00

We are happy to announce that The NetBSD Foundation will participate in Google Summer of Code 2025!

Would you like to contribute to NetBSD and/or pkgsrc in the next months? Google Summer of Code is a great chance for that!

You can find a list of possible projects at Google Summer of Code project page. Of course, you can also propose your own!

Please reach us via #netbsd-code IRC channel on Libera.Chat and/or via mailing lists.

If you are more interested about Google Summer of Code please also check official homepage at g.co/gsoc.

Looking forward to a great Summer!

X.Org on NetBSD - the state of things

2024-05-04T12:05:20+00:00

A few years ago, I wrote a "state of things" blog post about Wayland on NetBSD. It's only natural that I should do one about X11, which is used by far more people to get a graphical environment on NetBSD.

There are a lot of differences from how NetBSD and the typical distributor ship X.Org. For one, we ship it as an optional monolithic package rather than separate individual packages. This means every driver is included on every system, rather than as an optional module. Sometimes, this means we need to fine-tune driver selection to ensure the correct drivers are loaded on the correct hardware, since multiple conflicting drivers can claim a video output. We also want sensible fallbacks, since if you're using a GPU from the future with an old OS version, you probably want X to seamlessly fall back to a regular framebuffer.

Secondly, the way our "xsrc" repository is set up, it's effectively functioning as a fork of X.Org that regularly pulls from upstream freedesktop.org (but does not push back). This allows X development to happen as part of NetBSD.

Thirdly, we use our own build system based purely on BSD makefiles, not X.Org's based on GNU autotools. This fits well with our build.sh cross-compilation system.

We have a number of drivers which have not made their way upstream. Perhaps the most ubiquitous of these is xf86-input-ws, a driver which came from OpenBSD, targets an API from NetBSD, and continues to be developed in both. This is a generic input driver that can support any pointing device that the kernel supports. Unlike xf86-input-mouse, it doesn't assume the device is a mouse, and can support advanced touchpad and touchscreen features. Other NetBSD exclusives include xf86-video-pnozz, xf86-video-mgx, and xf86-video-crime. While these all share the "xf86" name inherited from the historical XFree86 distribution, none of them are exclusively for x86.

There are a number of drivers that are accelerated when used in NetBSD, but the acceleration support is missing upstream. This is mostly due to the work of macallan@, who has diligently worked on drivers for accelerators found on SPARC and PowerPC hardware.

X.Org has historically supported two 2D acceleration modes, XAA and EXA. XAA seems complicated - according to the X.Org Foundation it supports accelerating "patterned fills and Bresenham lines" (eh?). XAA was removed from the X.Org server in 2012, and many old drivers were not updated to support the newer and simpler EXA model, except in NetBSD, over a time period of several years.

Did you know that Nvidia used to have an open source graphics driver? It supported 2D acceleration for a range of cards. In NetBSD, it's retained for platforms that included embedded Nvidia chips and aren't capable of (or predate, or don't want) the modern novueau driver. Six years ago, it was updated in NetBSD to support EXA acceleration.

There are a few ways our X integration could be improved. While lots of attention has been paid to the server, less has been paid to clients (programs). Did you know that X includes a text editor, and that text editor supports syntax highlighting and spell checking?

NetBSD includes its own command-line spell checker and associated dictionaries, spell(1), inherited from the BSD UNIX of yore. It's pretty basic, and only supports variations of English. To get spell checking to work in xedit, you need to install ispell (another command-line spell checker) from pkgsrc, install a dictionary, then set some Xresources (or create symlinks) to make sure xedit finds ispell. This could surely be streamlined by teaching xedit about spell.

We also ship every program that has been included with historical X.Org distributions. This includes well-known things like xterm, slightly less well-known things like xbiff(1), and obscurities like bitmap(1) (apparently a 1-bit-per-pixel alternative to MS Paint). A while ago, we removed some libraries which are no longer used by the modern X server, and maybe we should evaluate whether we need all of these programs too. xmh(1) is a frontend for a mail system that isn't included in base. Together, bitmap and xmh are around 300 kilobytes.

We include fonts, bitmaps and scalable, for a wide range of computing devices. In the latest versions of NetBSD, the font size will automatically scale with the screen size to support HiDPI displays as well as small mobile devices. However, we don't ship a scalable cursor theme at the moment. We're also missing high-resolution fonts for Japanese, a shame considering the popularity of NetBSD in Japan. Koruri looks interesting and is suitably small, maybe we should import it.

While we have many useful simple programs by default (a clock, a calculator, an editor, a window manager, a compositor, a terminal emulator...), we're notably missing a screen locking program for X in the default install, although we have lock(1) for the tty.

The big question - does all this have a future? The good news is that all new hardware has generic support in X. Someone writes either a modesetting kernel driver or a classical wsdisplay kernel driver and they will be automatically supported by the associated drivers in X. The bad news is that to have applications running we require access to a larger open source ecosystem, and that ecosystem has a lot of churn and is easily distracted by shiny new squirrels. The process of upstreaming stuff to X.Org is an ongoing process, but it's likely we'll run into things that will never be suitable for upstream.

Of course, on NetBSD, you also have the option of trying vanilla modular X.Org from pkgsrc, or using something else entirely.

The Geeks way of checking what the outside wheather is like

2022-09-24T18:58:31+00:00

Prologue

When I bought my house in 2004 I went shopping for a outside thermometer - and ended up with a full weather-station instead (a WS2300). When I unpacked it I found a serial cable inside...

Long story short - I was still in the process of recabling the house (running ethernet to every room) and added a serial cable from the machine room to the WS2300, and then did some pkgsrc work and got misc/open2300 and misc/open2300-mysql. I used those to log the data from the weather-station to a mysql database, and later moved that (via misc/open2300-pgsql) to a postgres database.

Now sometime this year the machine running that database had to be replaced (should have done that earlier, it was power hungry and wasteful). The replacement was an aarch64 SoC (a Pine64 Quartz64 model A) - and it had no real com ports (of course) any more. I had experimented with USB serial adapters and the WS2300 before, but for unclear reasons this time I had no luck and couldn't get it to work. Since some of the outdoor sensors of the old weather-station had started failing, I decided to replace it.

New Weather-Station, new Sensors

I picked a WS3500 because it comes with a nice remote sensor arrangement:

I attached it to a satellite dish mount about 1.2m above my garage and ran a two wire cable through the mount to supply it with 3V and get rid of any batteries. It does not have a connector for that, but the battery compartment had enough space for a 330µF elco and soldering that and the cable directly to the battery contacts was easy.

The sensors report to the weather-station via a proprietary protocol in the 868 MHz band.

New Weather-Station, new Reporting

The weather-station can connect to a wifi network but does not offer any services itself. The app used to configure the station offers several predefined weather collection services.

I found the idea a bit strange to have my local weather data logged to some server somewhere else in the cloud and then get it back via my browser, but for others this is a good thing. I found this article that describes exactly the remote-only, no machines required on-site setup. I used that article as inspiration for the data collection (but that part turned out to be quite trivial, see below) and copied a lot of the presentation site from it (also more details below).

So in my setup I created web servers on two dedicated ports of my tiny machine running the postgres server. One is used by the weather-station for reporting the data, the other is used to query the database.

The configuration of the weather-station for a custom server was easy:

I tested the ecowitt protocol first. It uses a post to a fixed URL and the form data has nearly identical data as we get with the solution I ended up with - only a few names (of form fields) are slightly different.

The blacked items "StationID" and "StationKey" appear verbatim in the reported data, you can set them to whatever you want - the scripts below do not check them.

The weather underground protocol does a simple http GET and provides all data as query parameters (I had to add the trailing question mark in the configuration). This makes it very easy to extract the data in a script on the server side.

But lets get there step by step. NetBSD comes with a http/https server in base, originally called "bozohttpd". It is very lightweight, but it can run various types of scripts - I picked the plain old simple CGI and /bin/sh as language, using a bit of awk to convert units.

First I added two users, so I could separate file access rights. This is how they look like in vipw:

weatherupdate:*************:1004:1004::0:0:Weather Update Service:/weather/home:/sbin/nologin
weatherquery:*************:1005:1004::0:0:Weather Query Service:/weather/query:/sbin/nologin

and two httpd instances for them /etc/inetd entry to collect the incoming data:

88		stream	tcp	nowait:600	weatherupdate	/usr/libexec/httpd	httpd -q -c /weather/cgi /weather/files
89		stream	tcp	nowait:600	weatherquery	/usr/libexec/httpd	httpd -q -c /weather/cgi -M .js "text/javascript" - - /weather/files

The document root (/weather/files) would not be used for the instance on port 88, but httpd needs one. Note that these lines use the quiet flag ("-q") which is only available in netbsd-current. You can replace it with "-s" for older versions.

The home directories of both users are mostly empty, besides a .pgpass file that contains the password for this user connection to the postgres server. They look like this:

127.0.0.1:5432:weatherhistory:open2300:xxxxxxxxxxxxxx

where "weatherhistory" is the datebase and "open2300" is the name of the postgres user for the update script and the password is x-ed out. The other file looks very similar:

127.0.0.1:5432:weatherhistory:weatherquery:xxxxxxxxxxx

At the postgres level the user "weatherquery" needs to have SELECT privilege on the table "weather", and "open2300" needs to have INSERT privilege. The table schema (output of "pg_dump -s") looks like this:

--
-- Name: weather; Type: TABLE; Schema: public; Owner: weathermaster
--

CREATE TABLE public.weather (
    "timestamp" timestamp without time zone DEFAULT '1970-01-01 00:00:00'::timestamp without time zone NOT NULL,
    temp_in double precision DEFAULT '0'::double precision NOT NULL,
    temp_out double precision DEFAULT '0'::double precision NOT NULL,
    dewpoint double precision DEFAULT '0'::double precision NOT NULL,
    rel_hum_in integer DEFAULT 0 NOT NULL,
    rel_hum_out integer DEFAULT 0 NOT NULL,
    windspeed double precision DEFAULT '0'::double precision NOT NULL,
    wind_angle double precision DEFAULT '0'::double precision NOT NULL,
    wind_chill double precision DEFAULT '0'::double precision NOT NULL,
    rain_1h double precision DEFAULT '0'::double precision NOT NULL,
    rain_24h double precision DEFAULT '0'::double precision NOT NULL,
    rain_total double precision DEFAULT '0'::double precision NOT NULL,
    rel_pressure double precision DEFAULT '0'::double precision NOT NULL,
    wind_gust double precision DEFAULT 0 NOT NULL,
    light double precision DEFAULT 0 NOT NULL,
    uvi double precision DEFAULT 0 NOT NULL
);

ALTER TABLE public.weather OWNER TO weathermaster;

--
-- Name: weather weather_pkey; Type: CONSTRAINT; Schema: public; Owner: weathermaster
--
ALTER TABLE ONLY public.weather
    ADD CONSTRAINT weather_pkey PRIMARY KEY ("timestamp");

--
-- Name: TABLE weather; Type: ACL; Schema: public; Owner: weathermaster
--
GRANT INSERT ON TABLE public.weather TO open2300;
GRANT SELECT ON TABLE public.weather TO weatherquery;

As noted above, I carried this database over (with minor modifications) from previous instances of the whole setup - so it may not be optimal or elegant. One thing that needs special attention is the "timestamp" column - it carries date/time in UTC and has no timezone associated. This looked like a natural choice, but has some unexpected consequences. When querying data in JSON format, "timestamp" will not get the JavaScript marker for "UTC", a "Z" suffix. So in the JavaScript code in the web pages you will find quite a few places that cover up for this.

Now when the weather station sends data to the configured server, inetd(8) runs httpd(8) and that invokes a shell script /weather/cgi/update.cgi as the "weatherupdate" user. This script uses awk(1) to do a few unit conversions and output a SQL command to insert the data into the "weather" table. This SQL command is then piped to psql(1) with the connection string passed on the command line. The corresponding password is found in ~/.pgpass of the "weatherupdate" user.

The script looks like this:

#! /bin/sh

TZ=UTC; export TZ

awk -v $( echo "$QUERY_STRING" | sed 's/\&/ -v /g' ) 'BEGIN {

temp=(tempf-32)/1.8;
indoortemp=(indoortempf-32)/1.8;
dewpt=(dewptf-32)/1.8;
windchill=(windchillf-32)/1.8;
windspeed=windspeedmph*1.609344;
windgust=windgustmph*1.609344;
rain=rainin*25.4;
dailyrain=dailyrainin*25.4;
totalrain=totalrainin*25.4;
rel_preasure=baromin/0.029529980164712;

printf("INSERT INTO weather VALUES ('"'"'%s'"'"', %f, %f, %f, %d, %d, %f, %d, %f, %f, %f, %f, %f, %f, %f, %f);\n",
	strftime("%F %T"),
	indoortemp,
	temp,
	dewpt,
	indoorhumidity,
	humidity,
	windspeed,
	winddir,
	windchill,
	rain, dailyrain, totalrain,
	rel_preasure,
	windgust,
	solarradiation, UV);

}' | psql "hostaddr='127.0.0.1'dbname='weatherhistory'user='open2300'" > /dev/null 2>&1

Note that it explicitly sets the timezone to UTC. The input data comes (as defined by CGI) via the QUERY_STRING environment variable, as a set of "field=value" items, separated by &. They are converted to sets of "-v" args for the awk invocation via a simple sed script.

With this in place, the weather-station adds a record every five minutes to the database, and it was fun to check it via SQL, but for reasons not quite clear to me most of the rest of the family did not like that kind of access very much.

psql (14.5)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

weatherhistory=> select min(temp_out), max(temp_out) from weather;
  min  | max  
-------+------
 -18.1 | 80.9
(1 row)

I initially thought the 80.9°C were measured while I was soldering the power cable, but apparently they were fallout from the sometimes failing sensors of the old station. The database has 2840 rows with temp_out > 40°C and all of them are 80.something. I should replace them with an average of the neighbor records.

Presenting the data

So I needed an internal web site. Which needs access to the data. The above setup already paved the way for that, via the second port I set up. I wanted to show all the current data in one page, and variable history data on another - which meant two CGI scripts to query the data. The /weather/cgi/latest.cgi script just fetches the last record logged and creates a JSON from it, and also uses pom(6) and the sunwait(1) program from pkgsrc to supply some site and date specific data:

#! /bin/sh

PATH=/usr/games:/usr/pkg/bin:$PATH

GEOPOS="51.505554N 0.075278W"	# geographic position of this weather station
UPDATE=300			# seconds between updates

# This script uses psql(1) from pkgsrc/databases/postgresql14-client,
# pom(6) from the NetBSD games set and pkgsrc/misc/sunwait.

# collect global site data: sunrise and friends
eval $( sunwait report ${GEOPOS} | awk -F": " '
	/Sun directly north/	{
		printf("zenith=\"%s\"\n", $2);
	}
	/Daylight:/		{
		split($2,v," to ");
		printf("sunrise=\"%s\"\nsunset=\"%s\"\n", v[1], v[2]);
	}
	/with Civil twilight:/	{
		split($2,v," to ");
		printf("dawn=\"%s\"\ndusk=\"%s\"\n", v[1], v[2]);
	}
	/It is: Day/ {
		printf("day=true\n");
	}
	/It is: Night/ {
		printf("day=false\n");
	}
' )

# moon phase
eval $( pom | awk '-F('	'
	/The Moon is Full/	{ printf("moontrend=\"-\"\nmoon=100\n"); }
	/The Moon is New/	{ printf("moontrend=\"+\"\nmoon=0\n"); }
	/First Quarter/		{ printf("moontrend=\"+\"\nmoon=50\n"); }
	/Last Quarter/		{ printf("moontrend=\"-\"\nmoon=50\n"); }
	/Waxing/		{
		a=$0;
		sub(/^.*\(/, "", a);
		sub(/%.*$/, "", a);
		printf("moontrend=\"+\"\nmoon=%d\n", a+0);
	}
	/Waning/		{
		a=$0;
		sub(/^.*\(/, "", a);
		sub(/%.*$/, "", a);
		printf("moontrend=\"-\"\nmoon=%d\n", a+0);
	}
' )

# start the json output
printf "\n\n{ \"site\": { \"updates\": ${UPDATE},
	\"dawn\": \"${dawn}\", \"sunrise\": \"${sunrise}\",
	\"zenith\": \"${zenith}\", \"day\": ${day},
	\"sunset\": \"${sunset}\", \"dusk\": \"${dusk}\",
	\"moon\": { \"trend\": \"${moontrend}\", \"percent\": ${moon} }\n}, \"weather\":\n"

# fill database results
printf "WITH t AS ( SELECT * FROM weather ORDER BY timestamp DESC LIMIT 1 ) SELECT row_to_json(t) FROM t;\n" |
	psql --tuples-only --no-align "hostaddr='127.0.0.1'dbname='weatherhistory'user='weatherquery'"

# terminate json
printf "\n}\n"

As you can see, if you would restrict output to plain data from the database, the script would be only four or five lines long. But I like the additional spicing.

The /weather/cgi/history.cgi script fetches rows between two timestamps passed to it (in JSON timestamp format) and answers with a JSON containing an array of all the data in the requested time window:

#! /bin/sh

COND=$( echo "${QUERY_STRING}" | tr '&' '\n'| sed -e 's/%22/\"/g' -e 's/%3A/:/g' | awk '
	/from=/	{ v=$0; sub(/^[^"]*\"/, "", v); sub(/\".*$/, "", v); arg_from=v; }
	/to=/	{ v=$0; sub(/^[^"]*\"/, "", v); sub(/\".*$/, "", v); arg_to=v; }
	END	{
		if (arg_from && arg_to) {
			printf("timestamp >= '"'"'%s'"'"' AND timestamp <= '"'"'%s'"'"'\n",
			    arg_from, arg_to);
		}
	}
' )

if [ -z "${COND}" ]; then
	# printf "could not parse: ${QUERY_STRING}\n" >> /tmp/sql.log
	exit 0;
fi

# start output
printf "\n\n"

# printf "${COND}\n" >> /tmp/sql.log

# fill database results
printf "WITH t AS ( SELECT * FROM weather WHERE ${COND} ORDER by timestamp ASC ) SELECT json_agg(t) FROM t;\n" |
	psql --tuples-only --no-align "hostaddr='127.0.0.1'dbname='weatherhistory'user='weatherquery'" # 2&>> /tmp/sql.err

Fetching this data now is easy in JavaScript.

We have a request URL defined as a const, like this:

const queryURL = 'http://weatherhost.duskware.de:89/cgi-bin/history.cgi?';

and then add (if needed) the paramaters for the query, like in this example function that gets passed a from-date and a to-date:

function showData(fromD, toD)
{
        var url = new URL(queryURL);
        url.searchParams.append("from", '"'+fromD.toJSON()+'"');
        url.searchParams.append("to", '"'+toD.toJSON()+'"');
        fetch(url).then(function(response) {
                return response.json();
        }).then(function(data) {
                makeGraphs(data);
                updateButtons();
        }).catch(function(error) {
                console.error(error)
        });   
}

When the answer from the server arrives, it is decoded as JSON and returned as input data to the next function that makes some graphs from the data array. Finally a few buttons are updated (in this example the time window is put into a start and a end date control.

Inspired by the post mentioned above I used canvas gauges for the display of the latest data and dygraphs for the display of historic data.

Here is an example of how the latest display looks:

And here is how the history display looks:

I have put an archive of the cgi scripts and web pages here, and also for the curious who just want to peek at the full glory of my web design skills the start page (showing the latest weather data) and the history page.

Besides those files, you will need

a /weather/files/favicon.ico if you like.
download gauge.min.js from canvas gauges and put it into /weather/files/.
download dygraph.css, dygraph.min.js from dygraph, plus synchronizer.js from the dygraph extras/ directory and put it also into /weather/files/.

Then you should be ready to go - easy, isn't it? And no heavy weight dependencies or pkgs needed.

What about other weather stations?

There are quite a few similar weather stations out there now that seem to run "related" firmware and have similar capabilities. Most likely the update script (and details in the presentation pages) will need adjustements for other types.

If you start with a different device, just log all the data it sends and adjust the cgi scripts/database/JavaScript accordingly. For protocol analyzis there are several easy means:

Remove the "-q" flag in the httpd command (in /etc/inetd.conf) and check /var/log/xferlog for the quey paramaters sent by the weather station (when using the weather underground protocol).
Make the station log to a debug.cgi first to capture everything (including form data posted). This works for the ecowitt protocoll.
All this stations seem to use http only (not https), so you can sniff the traffic. Use tcpdump -w on the server to capture the data and analyze it with net/wireshark from pkgsrc.

Here is what a debug.cgi script could look like:

#! /bin/sh
env > /tmp/debug.env
printf "\n\nOK\n"
cat > /tmp/debug.input &

This allows you to see the form input in /tmp/debug.input and the CGI environment in /tmp/debug.env.

Announcing Google Summer of Code 2022 projects

2022-05-22T12:20:02+00:00

The NetBSD Foundation has finalized the list of projects for this year’s Google Summer of Code. The contributors and projects are the following:

Brian Schnepp - Raspberry Pi GPU Driver
Arjun Bemarkar - inetd enhancements
Piyush Sachdeva - Emulating missing linux syscalls
Vihas Makwana - Introduce a new Wi-Fi driver
Vivek Kumar Sah - Automating donor acknowledgement and information storage

The community bonding period has already started (from May 20) and it will last until June 12. During this time, the contributors are expected to coordinate with their mentors and community.

This will be immediately followed by the coding period from June 13 to September 4. After which, the contributors are expected to submit their final work, evaluate their mentors, and get evaluated by their mentors as well. Results will be announced on September 20.

For more information about the Google Summer of Code 2022 kindly refer to the official GSoC website.

We would like to express our gratitude to Google for organizing the yearly GSoC, and to The NetBSD Foundation mentors and administrators for their efforts and hardwork!

Let us welcome all the contributors to our growing NetBSD community!

The NetBSD Foundation is a mentoring organization at Google Summer of Code 2022

2022-03-16T17:02:15+00:00

We are happy to announce that The NetBSD Fundation is a mentoring organization at Google Summer of Code 2022!

Would you like to contribute to NetBSD or pkgsrc during the summer? Please give a look to NetBSD wiki Google Summer of Code page with possible ideas list and/or please join #NetBSD-code IRC channel on libera or get in touch with us via mailing lists to propose new projects!

Please note that unlike past years where Google Summer of Code was opened only to university students since this year if you are 18 or older you can be a GSoC contributor.

For more information about Google Summer of Code please give a look to the official GSoC website.

Looking forward to have a nice Google Summer of Code!

Making RockPro64 a NetBSD Server

2022-03-09T23:57:54+00:00

The time has come to upgrade my SunBlade 2500s to something more power friendly and faster. I'd already removed one CPU and thus half the ram from two of these systems to reduce their power consumption, but it's still much higher than it could be.

After much searching, I've decided on Pine64's RockPro64 4GiB ram model (technically, only 3.875GiB ram.) Pine64 make SBCs, laptops, phones, and various other mostly ARM gadgets, and the RockPro64 has the fastest CPU they ship (Rockchip RK3399), and can use a small "NAS Case", that is sufficient to house 2 HDDs and, at a stretch, upto 6 SSDs (cooling would become quite an issue at this point.)

In my SATA setup, I have 3 SSDs with a JMicron 585 card in the PCIe slot, two SSDs in the NAS case SSD region, and the third is in the HDD region with an adapter. I have used two SATA power splitters to convert the NAS case's 2 SATA power ports into 4, with the 4th one also powering a Noctua case fan. The cabling is not great with this, with enough SATA power cabling for 6 devices to lay. Probably a bespoke power cable to connect to the RockPro64 would make this nicer and probably improve cooling slightly, but I'm just using off-the-shelf components for now.

In the last year or so I've been working on making NetBSD/arm64 big-endian more featureful. In particular, I've added support for:

other-endian access disklabels, FFS file-systems in the NetBSD libsa
the "look 64 sectors later" for RAIDFrame partitions in MI efiboot (the x86 specific efiboot has more extensive support for this that should be ported over)
other-endian access to RAIDFrame labels in the kernel
updated the U-Boot package and featureset to include AHCI/SATA support, workaround some bugs and fix the newer U-Boot SPI loader location, and figured out how to default loading from either SATA or NVMe

There are not too many special actions needed for this sort of setup compared to a normal NetBSD or Arm system. While I built my installations by hand, the standard NetBSD Arm images are suitable for this task. It's easiest to start from an SD card with the RockPro64 u-boot installed. There are two U-Boot images available, one for SD/eMMC, and one for SPI (there is an odd problem with the early SPI loader that requires a portion of the image to be different.) The pkgsrc package for sysutils/u-boot-rockpro64 version 2022.01 has these suggested methods for installing the U-Boot image (this package should be buildable on any supported pkgsrc platform).

To install U-Boot into the SD/eMMC card (can run on any system, the image must be written at 32KiB into the device):

# dd if=/usr/pkg/share/u-boot/rockpro64/rksd_loader.img seek=64 of=/dev/rld0c

To install U-Boot into the SPI flash (must be run on the host, and lives at the very start of the device:

# dd if=/usr/pkg/share/u-boot/rockpro64/rkspi_loader.img bs=64k of=/dev/spiflash0

When booting from NVMe or SATA, one must drop to the U-Boot prompt and adjust the "boot_targets" value to put scsi* (SATA) or nvme* ahead of the mmc* and usb* options.

The original value for me:

=> printenv boot_targets
boot_targets=mmc1 usb0 mmc0 nvme0 scsi0 pxe dhcp sf0

Which is then adjusted and saved with eg:

=> setenv boot_targets nvme0 scsi0 mmc1 usb0 mmc0 pxe dhcp sf0
=> saveenv
Saving Environment to SPIFlash... Erasing SPI flash...Writing to SPI flash...done
OK

(In this list, mmc1 is the SD slot, mmc0 is the eMMC card, pxe and dhcp are netbooting, and sf0 attempts to load further U-Boot scripts from SPI.)

There are some minor issues with the RockPro64. It has no ability to use ECC memory. It only comes with 1 PCIe 4x slot, and Rockchip errata limited this to PCIe 1.x (though no NetBSD users encounted any issues with PCIe 2.x enabled, and this can be forced back on via a DTS patch.) It is possible to use a PCIe bridge (I have never done this, though I would like to try in the future) to enable more devices for booting, or to enable both a network and storage device.

Public NetBSD IRC chat channels moved to Libera

2021-05-30T18:23:29+00:00

Hi everyone,

Due to the unfortunate situation regarding changes in administration on freenode.net, and the resulting chaos, we have decided to move the public NetBSD IRC chat channels from freenode to irc.libera.chat.

This includes:

#NetBSD - general discussion
#netbsd-code - development discussion
#pkgsrc - pkgsrc (primarily development) discussion

You can find information on connecting to Libera at https://libera.chat/

Allen K. Briggs Memorial Scholarship

2020-12-21T11:37:59+00:00

Allen Briggs was one of the earliest members of the NetBSD community, pursuing his interest in macBSD, and moving to become a NetBSD developer when the two projects merged. Allen was known for his quiet and relaxed manner, and always brought a keen wisdom with him; allied with his acute technical expertise, he was one of the most valued members of the NetBSD community.

He was a revered member of the NetBSD core team, and keenly involved in many aspects of its application; from working on ARM chips to helping architect many projects, Allen was renowned for his expertise. He was a distinguished engineer at Apple, and used his NetBSD expertise there to bring products to market.

Allen lived in Blacksburg Virginia with his wife and twin boys and was active with various community volunteer groups. His family touched the families of many other NetBSD developers and those friendships have endured beyond his passing.

We have received the following from Allen's family and decided to share it with the NetBSD community. If you can, we would ask you to consider contributing to his Memorial Scholarship.

https://www.ncssm.edu/donate/distance-education/allen-k-briggs-88-memorial-scholarship

The Allen K. Briggs Memorial Scholarship is an endowment to provide scholarships in perpetuity for summer programs at the North Carolina School of Science & Math, which Allen considered to be a place that fundamentally shaped him as a person. We would love to invite Allen's friends and colleagues from the BSD community to donate to this cause so that we can provide more scholarships to students with financial need each year. We are approximately halfway to our goal of $50K with aspirations to exceed that target and fund additional scholarships.

Two quick notes on donating: Important! When donating, you must select "Allen K. Briggs Memorial Scholarship" under designation for the donation to be routed to the scholarship If you have the option to use employer matching (i.e., donating to NCSSM through an employer portal to secure a match from your employer), please email the NCSSM Foundation's Director of Development, April Horton (april.horton@ncssm.edu), after donating to let her know you want your gift and employer match to go to the Allen K. Briggs Memorial Scholarship Thanks in advance for your help. I'd be happy to answer any questions you or any others have about this.

Default window manager switched to CTWM in NetBSD-current

2020-09-28T08:33:28+00:00

For more than 20 years, NetBSD has shipped X11 with the "classic" default window manager of twm. However, it's been showing its age for a long time now.

In 2015, ctwm was imported, but after that no progress was made. ctwm is a fork of twm with some extra features - the primary advantages are that it's still incredibly lightweight, but highly configurable, and has support for virtual desktops, as well as a NetBSD-compatible license and ongoing development. Thanks to its configuration options, we can provide a default experience that's much more usable to people experienced with other operating systems.

Recently, I've been installing NetBSD with some people in real life and was inspired by their reactions to the default twm to improve the situation, so I played with ctwm, wrote a config, and used it myself for a week. It's now the default in NetBSD-current.

We gain some nice features like an auto-generated application menu (that will fill up as packages are installed to /usr/pkg), and a range of useful keyboard shortcuts including volume controls - the default config should be fully usable without a mouse. It should also work at a range of screen resolutions. We can add HiDPI support after some larger bitmap fonts are imported - another advantage of ctwm is that we can support very slow and very fast hardware with one config.

If you're curious about ctwm, check out the ctwm website. It's also included in previous NetBSD releases, though not as the default window manager and not with this config.

GSoC Reports: Benchmarking NetBSD, third evaluation report

2020-09-12T08:29:06+00:00

This report was written by Apurva Nandan as part of Google Summer of Code 2020.

Introduction

This blog post is in continuation of GSoC Reports: Benchmarking NetBSD, first evaluation report and GSoC Reports: Benchmarking NetBSD, second evaluation report blogs, and describes my progress in the final phase of GSoC 2020 under The NetBSD Foundation.

In the third phase, I upgraded to the latest stable version Phoronix Test Suite (PTS) 9.8.0 in pkgsrc-wip, resolved the TODOs and created patches for more test-profiles to fix their installation and runtime errors on NetBSD-current.

Progress in the third phase of GSoC

wip/phoronix-test-suite TODO and update

As a newer stable version of the Phoronix Test Suite was available in upstream, I upgraded the Phoronix Test Suite from version 9.6.1 to 9.8.0 in pkgsrc-wip and is available as wip/phoronix-test-suite. You can have a look at the PTS Changelog to know about the improvements between these two versions.

To get the package ready for merge in pkgsrc upstream, I also resolved the pkgsrc-wip TODOs.

pkgsrc-wip commits:

If any new problems are encountered, please document them in wip/phoronix-test-suite/TODO file and/or contact me.

Testing of automated benchmarking framework

I had been assigned a remote testing machine having Intel 6138 dual processor, 40 cores, 80 threads, 192GB of RAM. I spent time reproducing my automated framework i.e., Phoromatic-Anita Integration on the machine. I was able to reproduce the integration framework working without networking configuration, but the network bridge needs to be setup on the remote machine and the integration script to be tested with it. I shall continue this task in the post-GSoC period.

Benchmarking Results

I also performed benchmarking of NetBSD-9 amd64 native installation by running 50 test-profiles on a remote machine assigned to me by mentors and uploaded the benchmark results to OpenBenchmarking.org at:

https://openbenchmarking.org/result/2008287-NE-3966268951

Test-profile debugging

I then continued the task of maintaining/porting test-profiles and fixed the following test-profiles:

Timed FFmpeg Compilation

This test times how long it takes to build FFmpeg. This test is part of Processor Test category.

Compile Bench

Compilebench tries to age a filesystem by simulating some of the disk IO common in creating, compiling, patching, stating and reading kernel trees. It indirectly measures how well filesystems can maintain directory locality as the disk fills up and directories age. This test is part of Disk Test category.

Original Test-profile:

https://openbenchmarking.org/test/pts/compilebench

Patched Test-profile:

https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/compilebench-1.0.2

Commit:

Replaced python2 with NetBSD style naming python2.7

Timed MAFFT Alignment

This test performs an alignment of 100 pyruvate decarboxylase sequences. This test is part of Processor Test category.

Original Test-profile:

https://openbenchmarking.org/test/pts/mafft

Patched Test-profile:

https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/mafft-1.5.0

Commits:

Future Plans

This officially summarizes my GSoC project: Benchmark NetBSD, and my end goal of the project that is to integrate Phoronix Test Suite with NetBSD and Anita for automated benchmarking is complete and its deployment on benchmark.NetBSD.org will be continued to be worked on with the coordination of moderators and merging the wip of Phoronix Test Suite 9.8.0 will be done by the pkgsrc maintainers in next days.

I want to thank my mentors and the NetBSD community without whose constant support I wouldn't have achieved the goals.

GSoC Reports: Benchmarking NetBSD, second evaluation report

2020-08-12T10:09:16+00:00

This report was written by Apurva Nandan as part of Google Summer of Code 2020.

This blog post is in continuation of GSoC Reports: Benchmarking NetBSD, first evaluation report blog and describes my progress in the second phase of GSoC 2020 under The NetBSD Foundation.

In this phase, I worked on the automation of the regression suite made using Phoronix Test Suite (PTS) and its integration with Anita.

The automation framework consists of two components Phoromatic server, provided by Phoronix Test Suite in pkgsrc, and Anita, a Python tool for automating NetBSD installation.

About Phoromatic

Phoromatic is a remote management system for the Phoronix Test Suite, which allows the automatic scheduling of tests, remote installation of new tests, and the management of multiple test systems through a web interface. Tests can be scheduled to run on a routine basis across multiple test systems automatically. Phoromatic can also interface with revision control systems to offer support for issuing new tests on a context-basis, such as whenever a Git commit has been pushed. The test results are then available from the web interface.

Phoromatic client-server architecture

The Phoromatic server relies upon a PHP/HHVM built-in web server process and a PTS-hosted WebSocket server. The web server process handles the web UI and the responsibilities of the Phoromatic server.

Phoromatic clients are testing machines installed with PTS that connect to the Phoromatic web server through the HTTP port of the server.

Phoromatic Setup

To start the Phoromatic server, Phoromatic server HTTP port and web server socket port needs to be set in ~/.phoronix-test-suite/user-config.xml as shown:

...
<Server>
      <RemoteAccessPort>8640</RemoteAccessPort>
      <Password></Password>
      <WebSocketPort>8642</WebSocketPort>
      <AdvertiseServiceZeroConf>TRUE</AdvertiseServiceZeroConf>
      <AdvertiseServiceOpenBenchmarkRelay>TRUE</AdvertiseServiceOpenBenchmarkRelay>
      <PhoromaticStorage>~/.phoronix-test-suite/phoromatic/</PhoromaticStorage>
</Server>

Phoromatic Usage

To start the Phoromatic web server for controlling local Phoronix Test Suite client systems:

$ phoronix-test-suite start-phoromatic-server

The Phoromatic web server will be hosted at localhost:8640 and will require a local account creation on the server.

Phoromatic Clients

The Phoromatic client is used for connecting to a Phoromatic server to facilitate the automatic running of tests on that client.

Phoromatic clients can be created and connected to the server using the following command:

$ phoronix-test-suite phoromatic.connect SERVER_IP:SERVER_HTTP_PORT/ACCOUNT_ID

Phoromatic server interacts with the Phoromatic clients through the HTTP port specified in the ~/.phoronix-test-suite/user-config.xml.

Phoromatic Test-schedules

A test schedule is used to facilitate automatically running a set of test(s)/suite(s) on either a routine timed basis or whenever triggered by an external script or process, e.g. Git/VCS commit, manually triggered, etc. Phoromatic provides an option for pre-install, pre-run, post-install and post-run shell scripts that are executed on the Phoromatic clients. Test-schedules can be configured to run any tests on any specific systems.

About Anita

Anita is a tool for automated testing of the NetBSD operating system. Using Anita, we can download a NetBSD distribution and install it in a virtual machine in a fully automated fashion. Anita is written in Python and uses the pexpect module to “screen scrape” the sysinst output over an emulated serial console and script the installation procedure.

Installation

Anita can be installed on NetBSD, Linux and macOS systems using the following:

$ pip install pexpect
$ git clone https://github.com/gson1703/anita/
$ python setup.py install

Phoromatic-Anita Integration

I would like to describe the workflow here briefly:

A test-schedule was created on the Phoromatic server meant to run pts/idle-1.2.0 test on the host machine that contains the phoromatic-anita-integration.sh as a pre-run script.
The script performs the following:
- Creates a mountable disk image with an executable script for setting up Phoronix Test Suite and Phoromatic client creation on the benchmarking VM systems.
- Invokes Anita with the appropriate command-line options for configurations and network setup and mounts the image to run the configuration script on the VM.
- Configuration script performs hostname change, DHCP setup, NFS setup, PKG_PATH setup, PTS installation, its configuration and connecting it to the Phoromatic server through a network bridge.
Once the benchmarking VM systems get connected to the Phoromatic server, Phoromatic server identifies the benchmarking VM systems with their IP address, hostname and MAC address.
After the identification, Phoromatic initiates the pending tests on VM (test-profiles are downloaded on the go in the VM and executed) and maintains a history of the test result data.

Few points to be noted:

I have used a local PKG_PATH with a NFS server setup as PTS 9.6.1 is available in wip and recompiling it would be a wastage of time. Later I have planned to use the binary shard by Joyent: https://pkgsrc.joyent.com/packages/NetBSD/9.99.69/amd64/All/ once the updated PTS gets upstreamed.
The host machine needs some one-time manual setup like installation of QEMU, Anita, pexpect, term, cvs, etc., initial user registration on Phoromatic server, Phoromatic port setup, network bridge setup. Apart from this, the rest of the framework does not require user supervision.

VM configuration script

The following script is used as a pre-run script in the test-schedules for invoking Anita and setting up the VMs:

https://gist.github.com/apurvanandan1997/48b54402db1df3723920735f85fc7934

Networking Setup

A bridged networking mode configuration of QEMU has been used in Anita as multiple VMs will be able to accommodate with a single bridge (created on the host machine, one-time setup) using dhcpcd(8), without complicated host forwarding setup (Phoromatic server requires HTTP port forwarding).

In order to enable bridged networking for your QEMU guests, you must first create and configure a bridge interface on your host.

# ifconfig virbr0 create

Next, you must specify the newly-created bridge interface in /etc/qemu/bridge.conf:

$ sudo mkdir /etc/qemu
$ sudo touch /etc/qemu/bridge.conf && sudo chmod 644 /etc/qemu/bridge.conf
$ sudo sh -c "echo 'allow virbr0' >> /etc/qemu/bridge.conf"

Finally, in order for non-privileged processes to be able to invoke qemu-bridge-helper, you must set the setuid bit on the utility:

$ sudo chmod u+s /usr/local/libexec/qemu-bridge-helper

For more details on the bridged mode networking setup in QEMU, please refer to the following guides:

Reproducing the framework

To reproduce the framework, you need to have Phoronix Test Suite, QEMU, Anita, pexpect, cvs, xterm, makefs installed on your host machine.

For example on NetBSD:

# pkg_add qemu
# pkg_add py37-anita
$ cd pkgsrc/wip/phoronix-test-suite
$ make install

The step-by-step process to get the framework after installing PTS, including the one-time manual setup, can be summarized as follows: All control and configuration of the Phoromatic Server is done via the web-based interface when the Phoromatic Server is active.

Configure the port of Phoromatic server as 8640 and web socket as 8642 as described above.
Start the Phoromatic server using the command stated above.
Create your user account on the Phoromatic server using the web interface GUI.
Disable client system approval for new system addition from the settings menu in the web interface.
Connect the host machine as a Phoromatic client to the Phoromatic server using the command stated above.
Create a test-schedule for the host machine with the pre-run script as specified above and pts/idle-1.2.0 as the test-profile.
Execute the test-schedule or assign it on a timed-schedule and watch it running!
New VM systems with the latest NetBSD-current binaries and packages will be created and identified by Phoromatic server automatically.
Once for all, we need to specify what benchmarking test-profiles need to be run on the VM systems in the test-schedules section and it will be taken care of by Phoromatic.
The result history can also be viewed from Phoromatic web interface.

You can have a look at the video to get a clearer picture of how to setup the framework:

Future Plans

The regression suite is complete and final tasks of deploying it on benchmark.NetBSD.org and upstreaming the wip of Phoronix Test Suite will be done in the final phase of my GSoC project. I want to thank my mentors for their constant support.

GSoC Reports: Benchmarking NetBSD, first evaluation report

2020-07-16T09:38:58+00:00

This report was written by Apurva Nandan as part of Google Summer of Code 2020.

My GSoC project under NetBSD involves developing an automated regression and performance test framework for NetBSD that offers reproducible benchmarking results with detailed history and logs across various hardware & architectures.

To achieve this performance testing framework, I am using the Phoronix Test Suite (PTS) which is an open-source, cross-platform automated testing/benchmarking software for Linux, Windows and BSD environments. It allows the creation of new tests using simple XML files and shell scripts and integrates with revision control systems for per-commit regression testing.

About PTS

PTS core

PTS core is the engine of the Phoronix Test Suite and handles the following jobs:

Regularly updating the test profiles, test suites, and repository index.
Downloading, compilation, installation and evaluation of test packages.
Test result management and uploading results and user-defined test-profiles/suites to OpenBenchmarking.org

Test-profiles/Suites & OpenBenchmarking

Test-profiles are light-weight XMLs and shell scripts that allow easy installation of the benchmarking packages and their evaluation. Test-profiles generally contains the following files:

downloads.xml: Stores the links of the benchmarking packages, required additional packages, patches to be applied, etc. with their hash sums.
install.sh: Actual compilation, installation, and test evaluation shell script. Also handles the task of applying patches.
results-definition.xml: Meta-data to define the information for test result data (e.g. result data units, LIB or HIB, etc.)
test-definition.xml: Meta-data to specify test-profile details, such as compatible OS, external dependencies, test arguments, etc.

A simple test-profile C-Ray-1.2.0 can be seen to get an idea of the XMLs and shell scripts.

Test-suites are bundles of related test-profiles or more suites, focusing on a subsystem or certain category of tests e.g. Disk Test Suite, Kernel, CPU suite, etc.

OpenBenchmarking.org serves as a repository for storing test profiles, test suites, hence allowing new/updated tests to be seamlessly obtained via PTS. OpenBenchmarking.org also provides a platform to store the test result data openly and do a comparison between any test results stored in the OpenBenchmarking.org cloud.

Usage

Test installation

The command:

$ phoronix-test-suite install test-profile-name

Fetches the sources of the tests related to the the test-profile in ~/.phoronix-test-suite/installed-tests, applies patches and carries out compilation of test sources for generating the executables to run the tests.

Test execution

The command:

$ phoronix-test-suite run test-profile-name

Performs installation of the test (similar to $ phoronix-test-suite install test-profile-name) followed by exectution the binary from the compiled sources of the test in ~/.phoronix-test-suite/installed-tests and finally reports the tests results/outcome to the user and provides an option to upload results to OpenBenchmarking.org.

Progress in the first phase of GSoC

pkgsrc-wip

The first task performed by me was upgrading the Phoronix Test Suite from version 8.8 to the latest stable version 9.6.1 in pkgsrc-wip and is available as wip/phoronix-test-suite. You can have a look at the PTS Changelog to know about the improvements between these two versions.

Major Commits:

Currently, the PTS upgrade to 9.6.1 is subject to more modifications and fixes before it gets finally merged to the pkgsrc upstream. To test the Phoronix Test Suite 9.6.1 on NetBSD till then, you can setup pkgsrc-wip on your system via:

$ cd /usr/pkgsrc
$ git clone git://wip.pkgsrc.org/pkgsrc-wip.git wip

To learn more about pkgsrc-wip please see The pkgsrc-wip project homepage.

After setting up pkgsrc-wip, use the following command for installation of PTS 9.6.1:

$ cd /usr/pkgsrc/wip/phoronix-test-suite
$ make install

If any new build/installation errors are encountered, please document them in wip/phoronix-test-suite/TODO file and/or contact me.

Testing & Debugging

As we now know, having a look over OpenBenchmarking.org’s available test-profiles section or using $ phoronix-test-suite list-available-tests, there are 309 test-profiles available on PTS for Linux, and only 166 of 309 tests have been ported to NetBSD (can be differentiated by a thunder symbol on the test profile on the website). These 166 test-profiles can be fetch, build and executed on NetBSD using just a single command $ phoronix-test-suite run test-profile-name. I am ignoring the graphics ones in both Linux & NetBSD as GPU benchmarking wasn’t required in the project.

Many of the test profiles available on NetBSD have installation, build, or runtime errors i.e. they don’t work out of the box using the command $ phoronix-test-suite run test-profile-name. I ran 90 of these test profiles on a NetBSD-current x86_64 QEMU VM (took a lot of time due to the heavy tests) and have reported the status in the sheet: NetBSD GSoC: Test Porting Progress Report

You can have a look at how a PTS test-profile under action looks like!

Aircrack-NG test-profile demonstration

Aircrack-ng is a tool for assessing WiFi/WLAN network security.

The PTS test-profile is available at: https://openbenchmarking.org/test/pts/aircrack-ng

For further information, complete results can be seen at https://openbenchmarking.org/result/2007116-NI-AIRCRACKN51

AOBench test-profile demonstration

AOBench is a lightweight ambient occlusion renderer, written in C.

The PTS test-profile is available at: https://openbenchmarking.org/test/pts/aircrack-ng

For further information, complete results can be seen at: https://openbenchmarking.org/result/2007148-NI-AOBENCHTE94

Debugging and fixing non-working test-profiles

After testing these test-profiles, I debugged the following build errors in the following test-profiles:

pts/t-test1-1.0.1: memalign() not available on NetBSD
pts/coremark-1.0.0: calling bmake instead of gmake
pts/apache-siege-1.0.4: Missing signal.h header
pts/mbw-1.0.0: mempcpy() not available on NetBSD

Fixes to the above bugs can be found in the sheet or in the GitHub repositories shared in the next paragraph.

The modified test-profiles have been pushed to pts-test-profiles-dev and the fix patches to pts-test-profiles-patches. These patches are automatically downloaded by the debugged test-profiles and automatically applied before building of tests. The steps to install the debugged test-profiles have been added in the README.md of pts-test-profiles-dev.

These patches have been added in the downloads.xml of the modified test-profiles, and hence they get fetched during the test installation. The install.sh script in these modified test-profiles applies them on the benchmarking packages if the operating system is detected as NetBSD, thereby removing the build errors.

coremark test-profile demonstration

You can have a look at the patched coremark-1.0.0 test-profile under execution:

The patched PTS test-profile is available at: https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/coremark-1.0.0

This is a test of EEMBC CoreMark processor benchmark.

For further information, complete results can be seen at: https://openbenchmarking.org/result/2007148-NI-LOCALCORE64

Porting more test-profiles to NetBSD

Attempts were made to port new test-profiles from Linux to NetBSD, but the major incompatibility issue was missing external dependencies in NetBSD. Hence, this may reduce the number of test-profiles we can actually port, but more sincere efforts will be made in this direction in the next phase.

Future Plans

My next steps would involve porting the non-available tests to NetBSD i.e. the non-graphics ones available on Linux but unavailable on NetBSD, and fix the remaining test-profiles for NetBSD.

After gaining an availability of a decent number of test-profiles on NetBSD, I will use Phoromatic, a remote tests management system for the PTS (allows the automatic scheduling of tests, remote installation of new tests, and the management of multiple test systems) to host the live results of the automated benchmarking framework on benchmark.NetBSD.org, thereby delivering a functional performance and regression testing framework.

Announcing Google Summer of Code 2020 projects

2020-05-07T10:16:58+00:00

We are very happy to announce The NetBSD Foundation Google Summer of Code 2020 projects:

Apurva Nandan - Benchmark NetBSD
Jain Naman - Curses library automated testing
Nikita Gillmann - Make system(3) and popen(3) use posix_spawn(3) internally
Ayushi Sharma - Enhance the syzkaller support for NetBSD
Aditya Vardhan Padala - Rumpkernel Syscall Fuzzing
Nisarg Joshi - Fuzzing the network stack of NetBSD in a rumpkernel environment
Jason High - Extending the functionality of the netpgp suite

The community bonding period - where students get in touch with mentors and community - started on May 4 and will go on until June 1. The coding period will be June 1 to August 24.

Please welcome all our students and a big good luck to students and mentors!

A big thank you to Google and The NetBSD Foundation organization mentors and administrators!

Looking forward to having another nice Google Summer of Code!

Porting wine to amd64 on NetBSD, third evaluation report

2019-08-21T19:11:41+00:00

This report was written by Naveen Narayanan as part of Google Summer of Code 2019.

This report encompasses the progress of the project during the third coding period. You can make sense of the overall progress of the project by going through the first evaluation report and second evaluation report.

WINE on amd64

Wine-4.4 (released on Mar 2019) is working fine on amd64 and i386. I have been able to use a script as a workaround for the problem of setting LD_LIBRARY_PATH. My patch for setting guard size to 0 and hence, precluding Wine from segfaulting, that got upstreamed, can be found here. I have updated the package to the latest development version of Wine which is Wine-4.13 (released on Aug 2019). I have added support to Wine pkgsrc packages to run tests using make test, and at the time of writing, they are failing. I have also noticed them fail on Linux non-pkgsrc environment and hence, will require further investigation. Initially, they were disabled owing to pkgsrc setting FORTIFY_SOURCE which is a macro that provides support for detecting buffer overflows. In pkgsrc, the wip/wine* packages honor PKGSRC_USE_FORTIFY variable passing _FORTIFY_SOURCE macro accordingly. Programs compiled with FORTIFY_SOURCE substitute wrappers for commonly used libc functions that don't do bounds checking regularly, but could in some cases. Wine unconditionally disables that via their configure script because for some platforms that triggered false positives in the past. However, in my experience, no false positive were found.

Running tests on Wine-4.13 throws errors as you can see below:

mode64# make test
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M adsldp.dll -p adsldp_test.exe.so sysinfo && touch sysinfo.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
002a:err:winediag:SECUR32_initNTLMSP ntlm_auth was not found or is outdated. Make sure that ntlm_auth >= 3.0.25 is in your path. Usually, you can find it in the winbind package of your distribution.
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f150,0x00000001,0x22f120) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EEA0 000000000002B028 000000000022EE9C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f0e0,0x00000001,0x22f0b0) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EE30 000000000002B318 000000000022EE2C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f0e0,0x00000001,0x22f0b0) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EE30 000000000002B318 000000000022EE2C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:adsldp:sysinfo_get_UserName 0000000000021670,000000000022F0C8: stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f980,0x00000001,0x22f950) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022F6D0 000000000002B318 000000000022F6CC): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f980,0x00000001,0x22f950) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022F6D0 000000000002B318 000000000022F6CC): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:adsldp:sysinfo_get_UserName 0000000000021670,000000000022FB48: stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so cred && touch cred.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
002e:fixme:cred:CredReadW unhandled type -1
002e:fixme:cred:CredReadW unhandled flags 0xdeadbeef
002e:err:cred:CredWriteW bad username L"winetest"
002e:err:cred:CredWriteW bad username (null)
002e:err:cred:CredWriteW bad username (null)
002e:fixme:cred:CredDeleteW unhandled type -1
002e:fixme:cred:CredDeleteW unhandled flags 0xdeadbeef
002e:fixme:cred:CredReadDomainCredentialsW (0x1b1d0, 0x0, 0x22fa50, 0x22f908) stub
002e:fixme:cred:CredReadDomainCredentialsW (0x1b1d0, 0x0, 0x22fa50, 0x22f908) stub
002e:fixme:cred:CredReadDomainCredentialsW (0x1b3c0, 0x0, 0x22fa50, 0x22f908) stub
cred.c:820: Tests skipped: CRED_TYPE_DOMAIN_VISIBLE_PASSWORD credentials are not supported or are disabled. Skipping
002e:fixme:cred:CredIsMarshaledCredentialW BinaryBlobCredential not checked
002e:fixme:cred:CredIsMarshaledCredentialW BinaryBlobCredential not checked
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt && touch crypt.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_lmhash && touch crypt_lmhash.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_md4 && touch crypt_md4.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_md5 && touch crypt_md5.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_sha && touch crypt_sha.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so eventlog && touch eventlog.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),(null)) stub
0046:fixme:advapi:OpenEventLogW (L"IDontExist",(null)) stub
0046:fixme:advapi:OpenEventLogW (L"IDontExist",L"deadbeef") stub
0046:fixme:advapi:OpenEventLogW Remote server not supported
0046:fixme:advapi:OpenEventLogW ((null),L"deadbeef") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW (L"",L"Application") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:GetEventLogInformation (0x0, 1, 0x0, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0x0, 0, 0x0, 0, 0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x0, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x0, 0, 0x22fb40) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 0, 0x22fb40) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 8, 0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x0) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x22fb40) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x0) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x22fb40) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:BackupEventLogW (0x0,(null)) stub
0046:fixme:advapi:BackupEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,(null)) stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:BackupEventLogW (0x0,L"backup2.evt") stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),(null)) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"idontexist.evt") stub
0046:fixme:advapi:OpenBackupEventLogW (L"IDontExist",(null)) stub
0046:fixme:advapi:OpenBackupEventLogW (L"IDontExist",L"idontexist.evt") stub
0046:fixme:advapi:OpenBackupEventLogW Remote server not supported
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
eventlog.c:546: Tests skipped: We don't have a backup eventlog to work with
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x0,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x22fa88,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x0,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000000,0x0,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000000,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000000,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000001,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000002,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x0000000d,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x0000000e,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000007,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa84) stub
eventlog.c:479: Tests skipped: No records in the 'Application' log
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:ClearEventLogW (0x0,(null)) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup2.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,(null)) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0020,0x0000,0x00000000,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000005,0x00000000,0x1b1e0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ClearEventLogW (0xcafe4242,(null)) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"Wine"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"Wine"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0004,0x0001,0x00000001,0x0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0004,0x0001,0x00000001,0x0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0001,0x00000002,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc1"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc1"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0010,0x0001,0x00000003,0x0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0010,0x0001,0x00000003,0x0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc20") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0001,0x0001,0x00000004,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc300"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc300"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0001,0x00000005,0x0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0002,0x0001,0x00000005,0x0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0000,0x0002,0x00000006,0x1b3d0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0000,0x0002,0x00000006,0x1b3d0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0010,0x0002,0x00000007,0x1b3d0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc1") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0008,0x0002,0x00000008,0x1b3d0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0008,0x0002,0x00000008,0x1b3d0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc20"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc20"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0002,0x00000009,0x1b3d0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc300") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0001,0x0002,0x0000000a,0x1b3d0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0001,0x0002,0x0000000a,0x1b3d0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:err:eventlog:ReportEventW L"First string"
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
eventlog.c:889: Tests skipped: No events were written to the eventlog
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "this name is too long", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x0) stub
0046:fixme:advapi:StartTraceA (0x0, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:ControlTraceA (cafe4242, "wine", 0x1b1e0, 1) stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so lsa && touch lsa.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
004a:fixme:advapi:LsaOpenPolicy ((null),0x22f960,0x000f0fff,0x22f930) stub
004a:fixme:security:GetWindowsAccountDomainSid (000000000022F690 0000000000019078 000000000022F68C): semi-stub
004a:fixme:advapi:LsaEnumerateAccountRights (0xcafe,0x22fa20,0x22f990,0x22f92c) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb10,0x000f0fff,0x22fae8) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb40,0x00000800,0x22fb00) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb40,0x00000800,0x22fb08) stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so registry && touch registry.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
registry.c:2850: Tests skipped: HKCR key merging not supported
registry.c:3146: Tests skipped: HKCR key merging not supported
004e:fixme:winspool:PerfOpen (null): stub
004e:fixme:winspool:PerfCollect L"Global", 0x22ead8, 0x22eabc, 0x22eac0: stub
004e:fixme:winspool:PerfClose stub
004e:fixme:winspool:PerfOpen (null): stub
004e:fixme:winspool:PerfCollect L"invalid counter name", 0x22ead8, 0x22eabc, 0x22eac0: stub
004e:fixme:winspool:PerfClose stub
registry.c:4032: Test failed: [ 9] expected 1168, got 2
registry.c:4032: Test failed: [10] expected 1814, got 2
*** Error code 2

Stop.
make[1]: stopped in /usr/pkgsrc/wip/wine64-dev/work/wine-4.13/wine64/dlls/advapi32/tests
*** Error code 1

Stop.
make: stopped in /usr/pkgsrc/wip/wine64-dev/work/wine-4.13/wine64

Running programs on Wine

You can find obligatory screenshots of Wine-4.4/4.13 on amd64/i386 below:

010 Editor (Professional Text/Hex Editor) on Wine-4.13 (amd64)

Notepad++ on Wine-4.4 (i386)

Pinball on Wine-4.13 (amd64)

How to run Wine on NetBSD/amd64

Compile kernel with USER_LDT enabled.
Install kernel.
Clone wip repo.
cd /usr/pkgsrc/wip/wine64; make install (for 4.4)
cd /usr/pkgsrc/wip/wine64-dev; make install (for 4.13)
wine notepad

Future Plans

Wine requires the kernel option USER_LDT to be able to run 32-bit applications on amd64 - facilitated by WoW64. Presently, this feature isn't enabled by default on NetBSD and hence, the kernel has to be compiled with USER_LDT enabled. I will work on getting the tests to pass and finding a better approach to deal with LD_LIBRARY_PATH issue. In addition to that, I shall work on incorporating a NetBSD VM to Wine Test Bot infrastructure so we can preclude Wine from getting out of shape on NetBSD in the future (work in progress).

Summary

Presently, Wine-4.4 and Wine-4.13 are working fine on amd64/i386. I have been able to meet all the goals as per my original plan. I would like to attribute the success to the support provided by my mentors @leot, @maya and @christos. I would also like to thank my mentor @maxv for solving the conflict between SVS and USER_LDT kernel options. I intend to maintain the packages wine64/wine64-dev/wine32 for the foreseeable future. As stated above, I shall try to set up a NetBSD VM as Wine Test Bot for purpose of CI. Once again, thanks to Google for enabling me to do what I love.

Source Code

Porting wine to amd64 on NetBSD, second evaluation report

2019-08-01T11:48:34+00:00

This report was written by Naveen Narayanan as part of Google Summer of Code 2019.

This report encompasses the progress of the project during the second coding period.

WINE on amd64

As getting Wine to work with WoW64 support was of foremost importance, my focus was on compat32 dependency packages without which Wine's functionality would be limited and more importantly untestable. Initially, being unaware of what to expect, I just wanted Wine to run, at the earliest. So, with outmost support from mentors, the consensus was to install libs from 32-bit packages to ${PREFIX}/lib/32 and ignore everything else that came with the respective packages.

I had most of the compat32 packages ready after a couple of days. And it was time we gave Wine a whirl. Well, the build was successful. However, I had problems with 32-bit Xorg. The applications which came packaged with Wine worked fine, but, other Microsoft Windows applications like notepad++, Mario etc had a hard time running. Additionally, I noticed that fontconfig went wild and crashed spewing errors symptomatic of Wine (32-bit) not playing nice with the fontconfig lib that came with 32-bit Xorg package. On top of this, I found that build failed on certain machines due to unavailability of headers. This made us reconsider our decision to install 32-bit libs to ${PREFIX}/lib/32 and ignore everything else which included headers and binaries.

Ultimately, we realized that it was time we found a proper solution to the problem of where 32-bit packages should be installed on NetBSD amd64 and then, we eventually settled on ${PREFIX}/emul/netbsd32/. It seemed logical, and I got on with the task of adapting the respective Makefiles of dependency packages to install to ${PREFIX}/emul/netbsd32/. Additionally, I packaged fontconfig (32-bit) in high hopes that Wine would behave appropriately with the former. Wine build was successful. However, I noticed that Wine wasn't linking against the fontconfig libs from my package but against the ones which came with 32-bit Xorg package. Later, I realized, after consulting with mentors, that pkgsrc doesn't search for pkgconfig files (*.pc) in ${PREFIX}/emul/netbsd32/lib by default. pkgsrc sets _PKG_CONFIG_LIBDIR appropriately based on ${LIBABISUFFIX}. As you can see, it doesn't search for .pc files in ${PREFIX}/emul/netbsd32/lib and hence, pkgconfig couldn't get the right flags for fontconfig package. On the other hand, Wine configure script found fontconfig libs provided by 32-bit Xorg package against which it linked wine. A kludgy workaround was to use configure args for the respective libs thereby, obviating configure from finding the wrong libs. Finally, with the help of aforementioned kludgy workarounds, we were able to build Wine and successfully run Mario and Lua (32-bit binaries).

How to run Wine on NetBSD -current

Build -current distribution.
Compile -current kernel with USER_LDT enabled and SVS disabled.
Install kernel and distribution.
Clone wip repo.
cd /usr/pkgsrc/wip/wine64; make install
export LD_LIBRARY_PATH=/usr/pkg/emul/netbsd32/lib; wine mario.exe

Future Plans

Wine requires the kernel option USER_LDT to be able to run 32-bit applications on amd64 - facilitated by WoW64. Presently, this feature isn't enabled by default on NetBSD and hence, the kernel has to be compiled with USER_LDT enabled. This also warrants the kernel option SVS to be disabled currently owing to compatability issues. Work is being done to resolve the conflict which would apparently allow USER_LDT to be enabled by default in the GENERIC kernel.

pkgsrc can be made to search for pkgconf config files in ${PREFIX}/emul/netbsd32/lib by setting _PKG_CONFIG_LIBDIR to ${BUILDLINK_DIR}${LIBABIPREFIX}/lib${LIBABISUFFIX} as per @maya's suggestion. It might take some more thought and time before we finalize it.

Summary

Presently, Wine on amd64 is in test phase. It seems to work fine with caveats like LD_LIBRARY_PATH which has to be set as 32-bit Xorg libs don't have ${PREFIX}/emul/netbsd32/lib in its rpath section. The latter is due to us extracting 32-bit libs from tarballs in lieu of building 32-bit Xorg on amd64. As previously stated, pkgsrc doesn't search for pkgconfig files in ${PREFIX}/emul/netbsd32/lib which might have inadvertent effects that I am unaware of as of now. I shall be working on these issues during the final coding period. I would like to thank @leot, @maya and @christos for saving me from shooting myself in the foot many a time. I, admittedly, have had times when multiple approaches, which all seemed right at that time, perplexed me. I believe those are times when having a mentor counts, and I have been lucky enough to have really good ones. Once again, thanks to Google for this wonderful opportunity.

Implementation of DRM ioctl Support for NetBSD kernel

2019-07-08T18:33:04+00:00

This report was prepared by Surya P as a part of Google Summer of Code 2019

What is DRM ioctl ?

Ioctls are input/output control system calls and DRM stands for direct rendering manager The DRM layer provides several services to graphics drivers, many of them driven by the application interfaces it provides through libdrm, the library that wraps most of the DRM ioctls. These include vblank event handling, memory management, output management, framebuffer management, command submission & fencing, suspend/resume support, and DMA services.

Native DRM ioctl calls

NetBSD was able to make native DRM ioctl calls with hardware rendering once xorg and proper mesa packages where installed. We used the glxinfo and glxgears applications to test this out.

DRM ioctl calls from emulation

In order to make sure DRM ioctl calls where also made from the linux emulation layer of NetBSD . We used rpm and suse131 packages In specific base,compat,X11,libdrm,libglx,libexpat packages where used. To my surprise the applications kept segfaulting . I used glxgears and glxinfo rpm packages for this test .when I analyzed the segfault and traced the process , I was able to identify the cause of the segfault which was caused due to broken suse131 libdrm package which did not support nouveau based cards. To further make user that the problem was with the suse packages , I downgraded to suse121 and as expected the glxinfo and glxgears rpm packages ran, but it was using software rendering instead of hardware rendering but nevertheless we were still able to see the DRM ioctl calls made by the emulation layer hence we added some print statements in the kernel source to identify the calls made.

Summary

Fixing the Suse131 package and enabling hardware rendering from emulation is of highest priority , I have also planned to port steam and its dependencies to NetBSD to incorporate some gaming on NetBSD! And finally conversion between 32bit DRM ioctl calls 64bit DRM ioctl calls will be implemented.

Last but not the least I would like to thank my mentor @christos , @maya , @leot for helping me out and guiding me throughout the process and Google for providing me with such a wonderful opportunity to work with NetBSD community.

Porting NetBSD to HummingBoard Pulse, Part 1

2019-06-30T13:05:27+00:00

This report was written by Saurav Prakash as part of Google Summer of Code 2019.

My venture into the first phase of The Google Summer of Code is nearing an end. The experience was enriching in every dimension, and the learning exposure I was subjected to was genuinely worthwhile. Here is a brief report on the work I have performed during this coding period.

About HummingBoard Pulse

HummingBoard Pulse is an arm64 board developed by SolidRun, with iMX8 System-on-Chip, Dual/Quad core Cortex A53 processor. This project involves extending the GENERIC64 kernel to support iMX8 SoC.

Device Tree Files

We compare the compatible property of the root node in the device tree with the list of ARM platforms compiled into the kernel. We add the .dts file for imx8 which is compatible with the imx8mq.dtsi file adopted from the Linux mainline 5.1.4. At this early stage, nodes for uart, iomux and clock only were created.

Board Platform Code

The platform code provides SoC specific code needed early at boot. The arm_platform structure for imx8 is initialised here. It contains function pointers like .ap_attach_init_args, .ap_device_register, .ap_reset, .ap_delay, .ap_uart_freq, .ap_mpstart and a, platform_early_putchar_function so that we can send a character to UART.

Clock Driver

During the booting stage we only need to enable the uart clocks (IMX8MQ_CLK_UART*_ROOT) and its parents. This includes writing drivers for fixed clocks (OSC_25M, OSC_27M, OSC_32K, and more), fixed-factor clocks, divider clocks, mux clocks, composite clocks (IMX8MQ_CLK_UART*), gate clocks (IMX8MQ_CLK_UART*_ROOT), sccg-pll clocks, and frac-pll clocks.

UART Driver

The imx drivers are separated between "core driver" and "bus glue". <arch/arm/imx/imxuart.c> is a "core driver", and we need to write a fdt based "bus glue" for the custom bus, this non-fdt based imx code currently uses.

You can checkout the code from here.

What's Next

The sccg-pll, frac-pll, composite clocks aren't complete yet. So the focus in the next weeks would be to complete the unfinished work as required for the board to be booting till root. Finalising the UART driver code. Modifying the drivers written now, to suit the needs later. And then moving on further to write iomux driver, and then a USB driver for a storage device to boot from.

Lastly I would like to thank my mentors, @jmcneill, @martin @Cryo for being a constant support and non-flickering guidance throughout the process.

Porting Wine to amd64 on NetBSD, first evaluation report

2019-06-30T11:31:31+00:00

This report was written by Naveen Narayanan as part of Google Summer of Code 2019.

I have been working on porting Wine to amd64 on NetBSD as a GSoC 2019 project. Wine is a compatibility layer which allows running Microsoft Windows applications on POSIX-complaint operating systems. This report provides an overview of the progress of the project during the first coding period.

WINE on i386

Initially, when I started working on getting Wine-4.4 to build and run on NetBSD i386 the primary issue that I faced was Wine displaying black windows instead of UI, and this applied to any graphical program I tried running with Wine.

I suspected it , as it is related to graphics, to be an issue with the graphics driver or Xorg. Subsequently, I tried building modular Xorg, and I tried running Wine on it only to realize that Xorg being modular didn't affect it in the least. After having tried a couple of configurations, I realized that trying to hazard out every other probability is going to take an awful lot of time that I didn't have. This motivated me to bisect the repo using git, and find the first version of Wine which failed on NetBSD.

I started with the last version of Wine that worked on NetBSD which is 1.9.18. After two days of git bisect, I had the culprit. It was a commit that introduced a single clipboard manager thread per window station in Wine. Later, I found that pthread_create(3) calls were borked and didn't return anything. I tried to walk through the code to know where in pthreads lib was the program getting stuck at. Finally, I got a segfault.

I realized the address at which it segfaulted was identical to the address at which Wine has been throwing an unhandled page fault. I had a hunch that these two issues were some how correlated. After learning more about memory mapping and /proc, I was sure black windows was an effect of this unhandled page fault.

Eventually, I found that pthread_attr_setstack(3) was setting the guard size to 65536 bytes even though the man page said otherwise. And Wine relied on it not being set. This resulted in out-of-bound access which caused the unhandled page fault. After setting the guard size to 0 using pthread_attr_setguardsize(3), Wine started behaving fine.

I discussed about the patch with Wine devs, and they were happy to upstream it as long as it didn't cause any inadvertent issues on other platforms. Of course, who wouldn't want to play mario now?

WINE on amd64

Compiling Wine with 32 bit support is a bit tricky on amd64. I proceeded with the chroot approach as I wanted to know if Wine worked as promised on NetBSD, and if it didn't, to what degree, it required patching. So, as the first step, I compiled Wine on amd64 and it ran fine. The subsequent step was to build a chroot environment for i386. I corresponded with my mentors to learn more about compat32_netbsd and successfully built an i386 chroot. I had to compile Wine twice under i386 to have it inject the 32 bit components to 64 bit Wine for WoW64 support. And to my amazement, it worked!

Summary

I think Wine-4.4 is in pretty good shape as of right now, but packaging it is tricky especially since chroot isn't an option, as it is privileged. I have been writing compat32 packages for dependencies of Wine to have them crosscompiled on amd64. I shall be working on getting Wine crosscompiled for 32 bit support on amd64 during the next coding period. On a different note and a very important one, I would like to thank my mentors @christos, @leot, @maya, and @maxv for their valuable suggestions. From where I see it, I wouldn't have reached this phase if it weren't for them. I would also like to thank @kamil for his feedback and input.

LLDB from trunk is running on NetBSD once again!

2019-03-02T18:52:08+00:00

Upstream describes LLDB as a next generation, high-performance debugger. It is built on top of LLVM/Clang toolchain, and features great integration with it. At the moment, it primarily supports debugging C, C++ and ObjC code, and there is interest in extending it to more languages.

Originally, LLDB was ported to NetBSD by Kamil Rytarowski. However, multiple upstream changes and lack of continuous testing have resulted in decline of support. So far we haven't been able to restore the previous state.

In February, I have started working on LLDB, as contracted by the NetBSD Foundation. My four first goals as detailed in the previous report were:

Restore tracing in LLDB for NetBSD (i386/amd64/aarch64) for single-threaded applications.

Restore execution of LLDB regression tests, unless there is need for a significant LLDB or kernel work, mark detected bugs as failing or unsupported ones.

Enable execution of LLDB regression tests on the buildbot in order to catch regressions.

Upstream NetBSD (i386/amd64) core(5) support. Develop LLDB regression tests (and the testing framework enhancement) as requested by upstream.

Of those tasks, I consider running regression tests on the buildbot the highest priority. Bisecting regressions post-factum is hard due to long build times, and having continuous integration working is going to be very helpful to maintaining the code long-term.

In this report, I'd like to summarize what I achieved and what technical difficulties I met.

The kqueue interoperability issues

Given no specific clue as to why LLDB was no longer able to start processes on NetBSD, I've decided to start by establishing the status of the test suites. More specifically, I've started with a small subset of LLDB test suite — unittests. In this section, I'd like to focus on two important issues I had with them.

Firstly, one of the tests was hanging indefinitely. As I established, the purpose of the test was to check whether the main loop implementation correctly detects and reports when all the slaves of a pty are disconnected (and therefore the reads on master would fail). Through debugging, I've came to the conclusion that kevent() is not reporting this particular scenario.

I have built a simple test case (which is now part of kqueue ATF tests) and confirmed it. Afterwards, I have attempted to establish whether this behavior is correct. While kqueue(2) does not mention ptys specifically, it states the following for pipes:

Fifos, Pipes

Returns when there is data to read; data contains the number of bytes available.

When the last writer disconnects, the filter will set EV_EOF in flags. This may be cleared by passing in EV_CLEAR, at which point the filter will resume waiting for data to become available before returning.

Furthermore, my test program indicated that FreeBSD exhibits the described EV_EOF behavior. Therefore, I have decided to write a kernel patch adding this functionality, submitted it to review and eventually committed it after applying helpful suggestions from Robert Elz ([PATCH v3] kern/tty_pty: Fix reporting EOF via kevent and add a test case). I have also disabled the test case temporarily since the functionality is non-critical to LLDB (r353545).

Secondly, a few gdbserver-based tests were flaky — i.e. unpredictably passed and failed every iteration. I've started debugging this with a test whose purpose was to check verbose error messages support in the protocol. To my surprise, it seemed as if gdbserver worked fine as far as error message exchange was concerned. This packet was followed by a termination request from client — and it seemed that the server sometimes replies to it correctly, and sometimes terminates just before receiving it.

While working on this particular issue, I've noticed a few deficiencies in LLDB's error handling. In this case, this involved two major issues:

gdbserver ignored errors from main loop. As a result, if kevent() failed, it silently exited with a successful status. I've fixed it to catch and report the error verbosely instead: r354030.
Main loop reported meaningless return value (-1) from kevent(). I've established that most likely all kevent() implementation use errno instead, and made the function return it: r354029.

After applying those two fixes, gdbserver clearly indicated the problem: kevent() returned due to EINTR (i.e. the process receiving a signal). Lacking correct handling for this value, the main loop implementation wrongly treated it as fatal error and terminated the program. I've fixed this via implementing EINTR support for kevent() in r354122.

This trivial fix not only resolved most of the flaky tests but also turned out to be the root cause for LLDB being unable to start processes. Therefore, at this point tracing for single-threaded processes was restored on amd64. Testing on other platforms is pending.

Now, for the moral: working error reporting can save a lot of time.

Socket issues

The next issue I hit while working on the unittests is rather curious, and I have to admit I haven't managed to neither find the root cause or build a good reproducer for it. Nevertheless, I seem to have caught the gist of it and found a good workaround.

The test in question focuses on the high-level socket API in LLDB. It is rather trivial — it binds a server in one thread, and tries to connect to it from a second thread. So far, so good. Most of the time the test works just fine. However, sometimes — especially early after booting — it hangs forever.

I've debugged this thoroughly and came to the following conclusion: the test binds to 127.0.0.1 (i.e. purely IPv4) but tries to connect to localhost. The latter results in the client trying IPv6 first, failing and then succeeding with IPv4. The connection is accepted, the test case moves forward and terminates successfully.

Now, in the failing case, the IPv6 connection attempt succeeds, even though there is no server bound to that port. As a result, the client part is happily connected to a non-existing service, and the server part hangs forever waiting for the connection to come.

I have attempted to reproduce this with an isolated test case, reproducing the use of threads, binding to port zero, the IPv4/IPv6 mixup and I simply haven't been able to reproduce this. However, curiously enough my test case actually fixes the problem. I mean, if I start my test case before LLDB unit tests, they work fine afterwards (until next reboot).

Being unable to make any further progress on this weird behavior, I've decided to fix the test design instead — and make it connect to the same address it binds to: r353868.

Getting the right toolchain for live testing

The largest problem so far was getting LLDB tests to interoperate with NetBSD's clang driver correctly. On other systems, clang either defaults to libstdc++, or has libc++ installed as part of the system (FreeBSD, Darwin). The NetBSD driver wants to use libc++ but we do not have it installed by default.

While this could be solved via installing libc++ on the buildbot host, I thought it would be better to establish a solution that would allow LLDB to use just-built clang — similarly to how other LLVM projects (such as OpenMP) do. This way, we would be testing the matching libc++ revision and users would be able to run the tests in a single checkout out of the box.

Sadly, this is non-trivial. While it could be all hacked into the driver itself, it does not really belong there. While it is reasonable to link tests into the build tree, we wouldn't want regular executables built by user to bind to it. This is why normally this is handled via the test system. However, the tests in LLDB are an accumulation of at least three different test systems, each one calling the compiler separately.

In order to establish a baseline for this, I have created wrappers for clang that added the necessary command-line options. The state-of-art wrapper for clang looked like the following:

#!/usr/bin/env bash

topdir=/home/mgorny/llvm-project/build-rel-master
cxxinc="-cxx-isystem $topdir/include/c++/v1"
lpath="-L $topdir/lib"
rpath="-Wl,-rpath,$topdir/lib"
pthread="-pthread"
libs="-lunwind"

# needed to handle 'clang -v' correctly
[ $# -eq 1 ] && [ "$1" = -v ] && exec $topdir/bin/clang-9-real "$@"
exec $topdir/bin/clang-9-real $cxxinc $lpath $rpath "$@" $pthread $libs

The actual executable I renamed to clang-9-real, and this wrapper replaced clang and a similar one replaced clang++. clang-cl was linked to the real executable (as it wasn't called in wrapper-relevant contexts), while clang-9 was linked to the wrapper.

After establishing a baseline of working tests, I've looked into migrating the necessary bits one by one to the driver and/or LLDB test system, removing the migrated parts and verifying whether tests pass the same.

My proposal so far involves, appropriately:

Replacing -cxx-isystem with libc++ header search using path relative the compiler executable: D58592.
Integrating -L and -Wl,-rpath with the LLDB test system: D58630.
Adding NetBSD to list of platforms needing -pthread: r355274.
The need for -lunwind is solved via switching the test failing due to the lack of it to use libc++ instead of libstdc++: r355273.

The reason for adjusting libc++ header search in the driver rather than in LLDB tests is that the path is specific to building against libc++, and the driver makes it convenient to adjust the path conditionally to standard C++ library being used. In other words, it saves us from hard-relying on the assumption that tests will be run against libc++ only.

I've went for integrating -L in the test system since we do not want to link arbitrary programs to the libraries in LLVM's build directory. Appending this path unconditionally should be otherwise harmless to LLDB's tests, so that is the easier way to go.

Originally I wanted to avoid appending RPATHs. However, it seems that the LD_LIBRARY_PATH solution that works for Linux does not reliably work on NetBSD with LLDB. Therefore, passing -Wl,-rpath along with -L allowed me to solve the problem simpler.

Furthermore, those design solutions match other LLVM projects. I've mentioned OpenMP before — so far we had to pass -cxx-isystem to its tests explicitly but it passed -L for us. Those patches render passing -cxx-isystem unnecessary, and therefore make LLDB follow the suit of OpenMP.

Finishing touches

Having a reasonably working compiler and major regressions fixed, I have focused on establishing a baseline for running tests. The goal is to mark broken tests XFAIL or skip them. With all tests marked appropriately, we would be able to start running tests on the buildbot and catch regressions compared to this baseline. The current progress on this can be see in D58527.

Sadly, besides failing tests there is still a small number of flaky or hanging tests which are non-trivial to detect. The upstream maintainer, Pavel Labath is very helpful and I hope to be able to finally get all the flaky tests either fixed or covered with his help.

Other fixes not worth a separate section include:

fixing compiler warnings about empty format strings: r354922,
fixing two dlopen() based test cases not to link -ldl on NetBSD: r354617,
finishing Kamil's patch for core file support: r354466, followup fix in r354483,
removing dead code in main loop: r354050,
fixing stand-alone builds after they've been switched to LLVMConfig.cmake: r353925,
skipping lldb-mi tests when Python support (needed by lldb-mi) is disabled: r353700,
fixing incorrect initialization of sigset_t (not actually used right now): r353675.

Buildbot updates

The last part worth mentioning is that the NetBSD LLVM buildbot has seen some changes. Notably, zorg r354820 included:

fixing the bot commit filtering to include all projects built,
renaming the bot to shorter netbsd-amd64,
and moving it to toolchain category.

One of the most useful functions of buildbot is that it associated every successive build with new commits. If the build fails, it blames the authors of those commits and reports the failure to them. However, for this to work buildbot needs to be aware which projects are being tested.

Our buildbot configuration has been initially based on one used for LLDB, and it assumed LLVM, Clang and LLDB are the only projects built and tested. Over time, we've added additional projects but we failed to update the buildbot configs appropriately. Finally, with the help of Jonas Hahnfeld, Pavel Labath and Galina Kistanova we've managed to update the list and make the bot blame all projects correctly.

While at it, we were suggested to rename the bot. The previous name was lldb-amd64-ninja-netbsd8, and others suggested that the developers may ignore failures in other projects seeing lldb there. Kamil Rytarowski also pointed out that the version number confuses users to believe that we're running separate bots for different versions. The new name and category mean to clearly indicate that we're running a single bot instance for multiple projects.

Quick summary and future plans

At this point, the most important regressions in LLDB have been fixed and it is able to debug simple programs on amd64 once again. The test suite patches are still waiting for review, and once they're approved I still need to work on flaky tests before we can reliably enable that on the buildbot. This is the first priority.

The next item on the TODO list is to take over and finish Kamil's patch for core files with thread. Most notably, the patch requires writing tests, and verifying whether there are no new bugs affecting it.

On a semi-related note, LLVM 8.0.0 will be released in a few days and I will be probably working on updating src to the new version. I will also try to convince Joerg to switch from unmaintained libcxxrt to upstream libc++abi. Kamil also wanted to change libc++ include path to match upstream (NetBSD is dropping /v1 suffix at the moment).

Once this is done, the next big step is to fix threading support. Testing on non-amd64 arches is deferred until I gain access to some hardware.

This work is sponsored by The NetBSD Foundation

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:

http://netbsd.org/donations/#how-to-donate

The NetBSD Foundation participating in Google Summer of Code 2019

2019-02-27T16:34:42+00:00

For the 4th year in a row and for the 13th time The NetBSD Foundation will participate in Google Summer of Code 2019!

If you are a student and would like to learn more about Google Summer of Code please go to the Google Summer of Code homepage.

You can find a list of projects in Google Summer of Code project proposals in the wiki.

Do not hesitate to get in touch with us via #netbsd-code IRC channel on Freenode and via NetBSD mailing lists!

Looking forward to have a great summer!

Announcing Google Summer of Code 2018 projects

2018-05-01T10:15:52+00:00

We are very happy to announce The NetBSD Foundation Google Summer of Code 2018 projects:

Harsh Khatore - Modern cryptographic algorithms to netpgp, netpgpverify
Nizar Benshaqi - SQL Database for ATF tests results with online query and statistics page
Marwa Desouky - Tickless Kernel with high-resolution timers
Harry Pantazis - Kernel Undefined Behavior SANitizer
Does025 - Porting FreeBSD Atheros driver to NetBSD
Saad Mahmood - Machine-independent EFI bootloader
Yang Zheng - Integrate libFuzzer With the Basesystem
Keivan Motavalli - configuration files versioning in pkgsrc
R3x - Implementing Kernel Address Sanitizer (KASan) in the NetBSD kernel

We have started the community bonding period where students get in touch with mentors and community and get more confident with documentation and the code base. The coding period will start from May 14 until August 6.

Please welcome all our students and a big good luck to students and mentors!

A big thank you also to Google and The NetBSD Foundation Organization Administrators!

Looking forward to a nice Google Summer of Code!

Google Summer of Code 2017

2017-10-18T16:43:40+00:00

NetBSD participated in the 2017 edition of Google of Summer of Code with 3 students. All of the students finished their projects successfully. The following links report about their activities:

Congratulations to the students for finishing their projects successfully, and thanks to Google for sponsoring!

Announcing NetBSD and the Google Summer of Code Projects 2017

2017-05-05T19:16:02+00:00

We are very happy to announce that the selection process in this year's Summer of Code with its bargaining of slots and what student gets assigned to which project is over. As a result, the following students will take on their projects:

Leonardo Taccari will work add multi-packages support to pkgsrc.
Maya Rashish will work on the LFS cleanup.
Utkarsh Anand will make Anita support multiple virtual machine systems and more architectures within them to improve testing coverage.

What follows now is a community bonding period until May 30th, followed by a coding period over the summer (it's Summer of Code, after all ) until August 21st, evaluations, code submission and an announcement of the results on September 6th 2017.

Good luck to all our students and their mentors - we look forward to your work results, and welcome you to The NetBSD Project!