NetBSD Bloghttps://blog.netbsd.org/tnf/feed/entries/atom2024-03-17T12:20:22+00:00Apache Roller (incubating)https://blog.netbsd.org/tnf/entry/netbsd_10_0_rc6_availableNetBSD 10.0 RC6 available!martin2024-03-13T23:10:11+00:002024-03-13T23:10:11+00:00<p>
The NetBSD project is pleased to announce the sixth
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC6/">release candidate</a> of the upcoming 10.0 release, please help testing!<br />
See the <a href="https://www.NetBSD.org/releases/formal-10/NetBSD-10.0.html">release announcement</a> for details.
</p><p>
The NetBSD project is pleased to announce the sixth
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC6/">release candidate</a> of the upcoming 10.0 release, please help testing!<br />
See the <a href="https://www.NetBSD.org/releases/formal-10/NetBSD-10.0.html">release announcement</a> for details.
</p>
<p>The netbsd-10 release branch is more than a year old now, so it is high time the 10.0 release makes it to the front stage. This matches the long time it took for the development branch to get ready for branching, a lot of development went into this new release.
</p>
<p>
This also caused the release announcement to be one of the longest we ever did.
</p>
<p>
Since RC1 there have been numerous changes, including major updates to external software included in the release: Postfix, OpenSSH, and the firmware used for Raspberry PI devices. Various issues with RC1 have been fixed, including installer (sysinst) crashes. Lots of architecture specific fixes happend, e.g. various toolchain changes for VAX (so it is now finaly self-hosting again), and kernel changes for macppc, netwinder, and alpha.
</p>
<p>
For RC3 only few (relatively) minor changes were made, including https certificate verification in libfetch (which is used by pkg_ad(1)), and also improvements to the EFI bootloader to better deal with booting from CD (or in virtual machines ISO images), plus lots of various bug fixes.
</p>
<p>
RC4 became necessary as a few very important DRM/KMS issues especially for Intel GPUs have been resolved.
And as an (unexpected) bonus support for the Nintendo Wii has been added to the evbppc port.
</p>
<p>
RC5 has a few important security related updates of third party components (named, nsd, unbound, wpa_supplicant).
</p>
<p>
RC6 fixes a few issues with the new named/bind imported for RC5 plus several minor issues.
</p>
<p>Especially on amd64 machines please notes that we got a new DRM/KMS subsystem version, and this may lead to fallout on
some hardware. Unfortunately not all known bugs from the release engineering
<a href="https://wiki.NetBSD.org/releng/netbsd-10/">pre-release task list</a> could be fixed in time for this release - we will continue to improve the current state and hope to have more of them solved for the next (10.1) release.
</p>
<p>
If you want to test 10.0 RC6 please check the <a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC6/amd64/INSTALL.html">installation notes</a> for your architecture and download the preferred
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC6/images/">install image</a> from the CDN or if you are using an ARM based device from the netbsd-10 builds from the <a href="https://armbsd.org">bootable ARM images</a> page.
</p>
<p>
If you have any issues with installation or run into issues with the system during use, please contact us on one of the
<a href="https://mail-index.NetBSD.org">mailing lists</a> or file a <a href="https://www.netbsd.org/cgi-bin/sendpr.cgi?gndb=netbsd">problem report</a>.
</p>https://blog.netbsd.org/tnf/entry/netbsd_10_0_rc5_availableNetBSD 10.0 RC5 available!martin2024-02-28T19:37:19+00:002024-02-28T19:37:19+00:00<p>
The NetBSD project is pleased to announce the fourth (and probably last)
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC5/">release candidate</a> of the upcoming 10.0 release, please help testing!<br />
See the <a href="https://www.NetBSD.org/releases/formal-10/NetBSD-10.0.html">release announcement</a> for details.
</p><p>
The NetBSD project is pleased to announce the fifth (and probably last)
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC5/">release candidate</a> of the upcoming 10.0 release, please help testing!<br />
See the <a href="https://www.NetBSD.org/releases/formal-10/NetBSD-10.0.html">release announcement</a> for details.
</p>
<p>The netbsd-10 release branch is more than a year old now, so it is high time the 10.0 release makes it to the front stage. This matches the long time it took for the development branch to get ready for branching, a lot of development went into this new release.
</p>
<p>
This also caused the release announcement to be one of the longest we ever did.
</p>
<p>
Since RC1 there have been numerous changes, including major updates to external software included in the release: Postfix, OpenSSH, and the firmware used for Raspberry PI devices. Various issues with RC1 have been fixed, including installer (sysinst) crashes. Lots of architecture specific fixes happend, e.g. various toolchain changes for VAX (so it is now finaly self-hosting again), and kernel changes for macppc, netwinder, and alpha.
</p>
<p>
For RC3 only few (relatively) minor changes were made, including https certificate verification in libfetch (which is used by pkg_ad(1)), and also improvements to the EFI bootloader to better deal with booting from CD (or in virtual machines ISO images), plus lots of various bug fixes.
</p>
<p>
RC4 became necessary as a few very important DRM/KMS issues especially for Intel GPUs have been resolved.
And as an (unexpected) bonus support for the Nintendo Wii has been added to the evbppc port.
</p>
<p>
RC5 has a few important security related updates of third party components (named, nsd, unbound, wpa_supplicant).
</p>
<p>Especially on amd64 machines please notes that we got a new DRM/KMS subsystem version, and this may lead to fallout on
some hardware. Unfortunately not all known bugs from the release engineering
<a href="https://wiki.NetBSD.org/releng/netbsd-10/">pre-release task list</a> could be fixed in time for this release - we will continue to improve the current state and hope to have more of them solved for the next (10.1) release.
</p>
<p>
If you want to test 10.0 RC5 please check the <a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC5/amd64/INSTALL.html">installation notes</a> for your architecture and download the preferred
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC5/images/">install image</a> from the CDN or if you are using an ARM based device from the netbsd-10 builds from the <a href="https://armbsd.org">bootable ARM images</a> page.
</p>
<p>
If you have any issues with installation or run into issues with the system during use, please contact us on one of the
<a href="https://mail-index.NetBSD.org">mailing lists</a> or file a <a href="https://www.netbsd.org/cgi-bin/sendpr.cgi?gndb=netbsd">problem report</a>.
</p>https://blog.netbsd.org/tnf/entry/netbsd_10_0_rc4_availableNetBSD 10.0 RC4 available!martin2024-02-07T15:58:51+00:002024-02-07T15:58:52+00:00<p>
The NetBSD project is pleased to announce the fourth (and probably last)
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC4/">release candidate</a> of the upcoming 10.0 release, please help testing!<br />
See the <a href="https://www.NetBSD.org/releases/formal-10/NetBSD-10.0.html">release announcement</a> for details.
</p><p>
The NetBSD project is pleased to announce the fourth (and probably last)
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC4/">release candidate</a> of the upcoming 10.0 release, please help testing!<br />
See the <a href="https://www.NetBSD.org/releases/formal-10/NetBSD-10.0.html">release announcement</a> for details.
</p>
<p>The netbsd-10 release branch is more than a year old now, so it is high time the 10.0 release makes it to the front stage. This matches the long time it took for the development branch to get ready for branching, a lot of development went into this new release.
</p>
<p>
This also caused the release announcement to be one of the longest we ever did.
</p>
<p>
Since RC1 there have been numerous changes, including major updates to external software included in the release: Postfix, OpenSSH, and the firmware used for Raspberry PI devices. Various issues with RC1 have been fixed, including installer (sysinst) crashes. Lots of architecture specific fixes happend, e.g. various toolchain changes for VAX (so it is now finaly self-hosting again), and kernel changes for macppc, netwinder, and alpha.
</p>
<p>
For RC3 only few (relatively) minor changes were made, including https certificate verification in libfetch (which is used by pkg_ad(1)), and also improvements to the EFI bootloader to better deal with booting from CD (or in virtual machines ISO images), plus lots of various bug fixes.
</p>
<p>
RC4 became necessary as a few very important DRM/KMS issues especially for Intel GPUs have been resolved.
And as an (unexpected) bonus support for the Nintendo Wii has been added to the evbppc port.
</p>
<p>Especially on amd64 machines please notes that we got a new DRM/KMS subsystem version, and this may lead to fallout on
some hardware. Unfortunately not all known bugs from the release engineering
<a href="https://wiki.NetBSD.org/releng/netbsd-10/">pre-release task list</a> could be fixed in time for this release - we will continue to improve the current state and hope to have more of them solved for the next (10.1) release.
</p>
<p>
If you want to test 10.0 RC4 please check the <a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC4/amd64/INSTALL.html">installation notes</a> for your architecture and download the preferred
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC4/images/">install image</a> from the CDN or if you are using an ARM based device from the netbsd-10 builds from the <a href="https://armbsd.org">bootable ARM images</a> page.
</p>
<p>
If you have any issues with installation or run into issues with the system during use, please contact us on one of the
<a href="https://mail-index.NetBSD.org">mailing lists</a> or file a <a href="https://www.netbsd.org/cgi-bin/sendpr.cgi?gndb=netbsd">problem report</a>.
</p>https://blog.netbsd.org/tnf/entry/netbsd_10_0_rc3_availableNetBSD 10.0 RC3 available!martin2024-01-17T18:11:06+00:002024-01-17T18:11:06+00:00<p>
The NetBSD project is pleased to announce the third (and probably last)
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC3/">release candidate</a> of the upcoming 10.0 release, please help testing!<br />
See the <a href="https://www.NetBSD.org/releases/formal-10/NetBSD-10.0.html">release announcement</a> for details.
</p><p>
The NetBSD project is pleased to announce the third (and probably last)
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC3/">release candidate</a> of the upcoming 10.0 release, please help testing!<br />
See the <a href="https://www.NetBSD.org/releases/formal-10/NetBSD-10.0.html">release announcement</a> for details.
</p>
<p>The netbsd-10 release branch is more than a year old now, so it is high time the 10.0 release makes it to the front stage. This matches the long time it took for the development branch to get ready for branching, a lot of development went into this new release.
</p>
<p>
This also caused the release announcement to be one of the longest we ever did.
</p>
<p>
Since RC1 there have been numerous changes, including major updates to external software included in the release: Postfix, OpenSSH, and the firmware used for Raspberry PI devices. Various issues with RC1 have been fixed, including installer (sysinst) crashes. Lots of architecture specific fixes happend, e.g. various toolchain changes for VAX (so it is now finaly self-hosting again), and kernel changes for macppc, netwinder, and alpha.
</p>
<p>
For RC3 only few (relatively) minor changes were made, including https certificate verification in libfetch (which is used by pkg_ad(1)), and also improvements to the EFI bootloader to better deal with booting from CD (or in virtual machines ISO images), plus lots of various bug fixes.
<p>
Especially on amd64 machines please notes that we got a new DRM/KMS subsystem version, and this may lead to fallout on
some hardware. Unfortunately not all known bugs from the release engineering
<a href="https://wiki.NetBSD.org/releng/netbsd-10/">pre-release task list</a> could be fixed in time for this release - we will continue to improve the current state and hope to have more of them solved for the next (10.1) release.
</p>
<p>
If you want to test 10.0 RC3 please check the <a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC3/amd64/INSTALL.html">installation notes</a> for your architecture and download the preferred
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC3/images/">install image</a> from the CDN or if you are using an ARM based device from the netbsd-10 builds from the <a href="https://armbsd.org">bootable ARM images</a> page.
</p>
<p>
If you have any issues with installation or run into issues with the system during use, please contact us on one of the
<a href="https://mail-index.NetBSD.org">mailing lists</a> or file a <a href="https://www.netbsd.org/cgi-bin/sendpr.cgi?gndb=netbsd">problem report</a>.
</p>https://blog.netbsd.org/tnf/entry/netbsd_10_0_rc2_availableNetBSD 10.0 RC2 available!martin2024-01-04T08:21:36+00:002024-01-04T08:21:36+00:00<p>
The NetBSD project is pleased to announce the second (and probably last)
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC2/">release candidate</a> of the upcoming 10.0 release, please help testing!<br />
See the <a href="https://www.NetBSD.org/releases/formal-10/NetBSD-10.0.html">release announcement</a> for details.
</p><p>
The NetBSD project is pleased to announce the second (and probably last)
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC2/">release candidate</a> of the upcoming 10.0 release, please help testing!<br />
See the <a href="https://www.NetBSD.org/releases/formal-10/NetBSD-10.0.html">release announcement</a> for details.
</p>
<p>The netbsd-10 release branch is more than a year old now, so it is high time the 10.0 release makes it to the front stage. This matches the long time it took for the development branch to get ready for branching, a lot of development went into this new release.
</p>
<p>
This also caused the release announcement to be one of the longest we ever did.
</p>
<p>
Since RC1 there have been numerous changes, including major updates to external software included in the release: Postfix, OpenSSH, and the firmware used for Raspberry PI devices. Various issues with RC1 have been fixed, including installer (sysinst) crashes. Lots of architecture specific fixes happend, e.g. various toolchain changes for VAX (so it is now finaly self-hosting again), and kernel changes for macppc, netwinder, and alpha.
</p>
<p>
Especially on amd64 machines please notes that we got a new DRM/KMS subsystem version, and this may lead to fallout on
some hardware. Unfortunately not all known bugs from the release engineering
<a href="https://wiki.NetBSD.org/releng/netbsd-10/">pre-release task list</a> could be fixed in time for this release - we will continue to improve the current state and hope to have more of them solved for the next (10.1) release.
</p>
<p>
If you want to test 10.0 RC2 please check the <a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC2/amd64/INSTALL.html">installation notes</a> for your architecture and download the preferred
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC2/images/">install image</a> from the CDN or if you are using an ARM based device from the netbsd-10 builds from the <a href="https://armbsd.org">bootable ARM images</a> page.
</p>
<p>
If you have any issues with installation or run into issues with the system during use, please contact us on one of the
<a href="https://mail-index.NetBSD.org">mailing lists</a> or file a <a href="https://www.netbsd.org/cgi-bin/sendpr.cgi?gndb=netbsd">problem report</a>.
</p>https://blog.netbsd.org/tnf/entry/netbsd_10_0_rc1_availableNetBSD 10.0 RC1 available!martin2023-11-11T13:56:20+00:002023-11-11T13:56:20+00:00<p>
The NetBSD project is pleased to announce the first
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC1/">release candidate</a> of the upcoming 10.0 release, please help testing!<br />
See the <a href="https://www.NetBSD.org/releases/formal-10/NetBSD-10.0.html">release anouncement</a> for details.
</p><p>
The NetBSD project is pleased to announce the first
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC1/">release candidate</a> of the upcoming 10.0 release, please help testing!<br />
See the <a href="https://www.NetBSD.org/releases/formal-10/NetBSD-10.0.html">release anouncement</a> for details.
</p>
<p>The netbsd-10 release branch is more than a year old now, so it is high time the 10.0 release makes it to the front stage. This matches the long time it took for the developement branch to get ready for branching, a lot of developement went into this new release.
</p>
<p>
This also caused the release anouncement to be one of the longest we ever did.
</p>
<p>
Especially on amd64 machines please notes that we got a new DRM/KMS subsystem version, and this may lead to fallout on
some hardware. Unfortunately not all known bugs from the release engineering
<a href="https://wiki.NetBSD.org/releng/netbsd-10/">pre-release task list</a> could be fixed in time for this release - we will continue to improve the current state and hope to have more of them solved for the next (10.1) release.
</p>
<p>
If you want to test 10.0 RC1 please check the <a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC1/amd64/INSTALL.html">installation notes</a> for your architecture and download the prefered
<a href="https://cdn.NetBSD.org/pub/NetBSD/NetBSD-10.0_RC1/images/">install image</a> from the CDN or if you are using an ARM based device from the netbsd-10 builds from the <a href="https://armbsd.org">bootable ARM images</a> page.
</p>
<p>
If you have any issues with installation or run into issues with the system during use, please contact us on one of the
<a href="https://mail-index.NetBSD.org">mailing lists</a> or file a <a href="https://www.netbsd.org/cgi-bin/sendpr.cgi?gndb=netbsd">problem report</a>.
</p>https://blog.netbsd.org/tnf/entry/fosdem_2023FOSDEM 2023Pierre Pronchery2023-02-08T17:25:39+00:002023-02-08T17:25:39+00:00FOSDEM took place last week-end, as an offline-first event again for the first time since 2020. It was located as usual at the university campus of the ULB in Brussels. It was packed with developers, users, passionate and professionals of Open Source software, and while NetBSD did not have a booth this year, its presence could be felt on Saturday morning at the BSD DevRoom.<p><a href="https://fosdem.org">FOSDEM</a> took place last week-end, as an offline-first event again for the first time since 2020. It was located as usual at the university campus of the ULB in Brussels. It was packed with developers, users, passionate and professionals of Open Source software, and while <a href="https://www.NetBSD.org">NetBSD</a> did not have a booth this year, its presence could be felt on Saturday morning at the <a href="https://fosdem.org/2023/schedule/track/bsd/">BSD DevRoom</a> thanks to the many <a href="https://www.NetBSD.org/people/developers.html">developers</a> who made it to the conference.</p>
<p>Together with Rodrigo Osorio of the <a href="https://www.FreeBSD.org">FreeBSD project</a>, I had the pleasure to help manage the DevRoom, have a front seat for the talks, and even start the session by introducing the <a href="https://www.NetBSD.org/gallery/presentations/#2023">BSD Driver Harmony initiative</a>.</p>
<p>The staff and respective speakers are currently busy uploading slides and <a href="https://video.fosdem.org">reviewing videos</a>, so keep in mind to <a href="https://twitter.com/fosdem">check again</a> for <a href="https://fosstodon.org/@fosdem">new content</a> in the coming few days and weeks if you missed anything or need to dig further into any event from this awesome conference!</p>
<p>Finally, I would like to thank the <a href="https://www.NetBSD.org/foundation/">NetBSD Foundation</a> for sponsoring me to manage the room and attend the <a href="https://summerofcode.withgoogle.com/programs/2022/organizations/the-netbsd-foundation">GSoC meet-up</a>.</p>https://blog.netbsd.org/tnf/entry/reproducible_builds_summit_venice_2022Reproducible Builds Summit Venice 2022Pierre Pronchery2023-01-02T06:22:08+00:002023-01-02T06:23:02+00:00<p>The sixth Reproducible Builds Summit took place exactly two months ago in Venice, Italy. These three days of workshops were filled with a succession of interactive sessions, where everyone attending had the opportunity to present or learn about anything related to Build Reproducibility. This included the status of specific Open Source projects, techniques to locate, analyse, and understand issues, or also how to explain and communicate better around this topic.</p><p>The <a href="https://reproducible-builds.org/events/venice2022/">sixth Reproducible Builds Summit</a> took place exactly <a href="https://reproducible-builds.org/events/venice2022/">two months ago</a> in Venice, Italy. These <b>three days of workshops</b> were filled with a <a href="https://reproducible-builds.org/events/venice2022/agenda/">succession of interactive sessions</a>, where everyone attending had the opportunity to present or learn about anything related to <b>Build Reproducibility</b>. This included the status of specific Open Source projects, techniques to locate, analyse, and understand issues, or also how to explain and communicate better around this topic.</p>
<a href="http://blog.netbsd.org/tnf/mediaresource/b9caa1e5-f797-4299-9eab-9932f7138322"><img src="http://blog.netbsd.org/tnf/mediaresource/b9caa1e5-f797-4299-9eab-9932f7138322?t=true" alt="rb.svg"></img></a>
<h3>But what is this about?</h3>
<p>Reproducible Builds are a set of software development practices that create an <b>independently-verifiable path from source to binary code</b>.</p>
<h3>Why is this important?</h3>
<p>Anyone may inspect the source code of Free and Open Source Software for correctness or vulnerabilities. However, <b>most software is distributed pre-compiled</b>, with no method to confirm whether it actually corresponds to the source code published. <b>This allows attacks</b> in a number of different situations, from a malicious developer to network attacks, or the compromise of build infrastructure.</p>
<h3>What can be done about it?</h3>
<p>The purpose of Reproducible Builds is therefore to <b>allow the verification</b> that no vulnerabilities or backdoors have been introduced during the compilation process. By promising identical results for a given source, Build Reproducibility allows multiple third-parties to compare “correct” results, and to <b>flag any deviations as suspect</b> and worthy of scrutiny.</p>
<h3>How is NetBSD doing in this regard?</h3>
<p>The base system of <b>NetBSD can be built reproducibly</b> since its <a href="https://www.netbsd.org/releases/formal-8/NetBSD-8.0.html">8.0 release</a>! It can be enabled in <a href="https://man.netbsd.org/NetBSD-9.3/i386/mk.conf.5">mk.conf</a> when <a href="https://www.netbsd.org/docs/guide/en/chap-build.html">building NetBSD</a> for instance.</p>
<h3>And in pkgsrc?</h3>
<p>A first step has been implemented, when using GCC on NetBSD to build packages. Some important tools have been packaged, such as <a href="https://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc/sysutils/py-diffoscope/">diffoscope</a>. However, <b>further aspects of build reproducibility are not covered in pkgsrc yet</b>, and <b>we welcome contributions</b> to improve this situation! This would help bring this additional security mitigation to the NetBSD community as well as to other systems and users of pkgsrc.</p>
<h3>Summary and conclusion</h3>
<p>If not already, you should definitely consider Build Reproducibility for your environment or software projects. It also applies to firmware, when sources are available. Thankfully NetBSD offers this ability for the base system already, but more work is required for packages.</p>
<p>As for myself, it was an honour and a pleasure to attend the Summit, <a href="https://reproducible-builds.org/reports/2022-11/">keep in touch with the community</a>, participate to the event, learn from everyone attending, and obviously to represent the NetBSD Foundation there. I am looking forward to the <a href="https://reproducible-builds.org/events/">next Summit</a>, which should take place in Hamburg from October 30<sup>th</sup> to November 2<sup>nd</sup> of 2023.</p>
<p>In the meantime, do not hesitate to <a href="https://netbsd.org/community/">get in touch</a>, including to <a href="https://netbsd.org/foundation/">the NetBSD Foundation</a> or to <a href="https://pkgsrc.org/#index2h1">the pkgsrc community</a> specifically, if you want to get involved with any aspect of Build Reproducibility or represent the NetBSD or pkgsrc projects for the <a href="https://reproducible-builds.org">Reproducible Builds community</a>.</p>https://blog.netbsd.org/tnf/entry/netbsd_10_0_beta_availableNetBSD 10.0 BETA availableNia Alarie2022-12-20T18:19:04+00:002022-12-21T09:08:32+00:00<p>After nearly <a href="https://www.youtube.com/watch?v=fNlxFh43V-w">3 whole years</a> of development (work started on NetBSD 10 in late 2019), BETA snapshots have finally been published for interested users to test. More changes will be backported from the development branch over the next few months before we tag a final release, so the BETA images will keep getting updated.</p><p>After nearly <a href="https://www.youtube.com/watch?v=fNlxFh43V-w">3 whole years</a> of development (work started on NetBSD 10 in late 2019), BETA snapshots have finally been published for interested users to test. More changes will be backported from the development branch over the next few months before we tag a final release, so the BETA images will keep getting updated.</p>
<h3>What to expect</h3>
<p>While NetBSD 10.0 is expected to be a major milestone on performance, especially on multi-core systems, currently the BETA builds have some extra kernel diagnostics enabled that may reduce performance somewhat.</p>
<p>Among the features you can expect to find in NetBSD 10 are reworked cryptography, including compatibility with WireGuardⓇ, automatic swap encryption, new disk encryption methods, and CPU acceleration in the kernel. In hardware support, there are updated GPU drivers from Linux 5.6, support for more ARM hardware (including Rockchip RK356X, NXP i.MX 8M, Amlogic G12, <a href="https://wiki.netbsd.org/ports/evbarm/apple/">Apple M1</a>, and Raspberry Pi 4), support for new security features found in the latest ARM CPUs, and support for Realtek 2.5 gigabit and new Intel 10/25/40 gigabit ethernet adapters. compat_linux has been ported to AArch64 and DTrace has been ported to MIPS. For retrocomputing enthusiasts, there's improved multiprocessor support on Alpha, and more iMac G5 support. The Xen hypervisor support has received a major rework. There are various new userspace programs, including blkdiscard(8) to manually TRIM a disk, aiomixer(1) to control audio volume, realpath(1), and fsck_udf(8). And loads more...</p>
<p>There are many little details that might be relevant to admins when upgrading from NetBSD 9, so <strong>wait and read the final release announcement before you upgrade any production systems</strong>. Please note that networking setups using tap(4) as a bridge endpoint must be modified to use vether(4) instead, and compat_linux is no longer built into the kernel for security reasons (load it as a module instead). DisplayPort and HDMI audio is now enabled in the default x86 kernel, so you might need to change the default audio device with audiocfg(1) if you're not getting any sound output. blacklistd(8) was renamed to blocklistd(8).</p>
<h3>Downloads</h3>
<ul>
<li> <a href="https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-10/latest/images/NetBSD-10.0_BETA-amd64-install.img.gz">Installation / USB stick image for 64-bit x86 (hybrid UEFI/BIOS image)</a>. Use gunzip and dd to write an .img.gz file on Unix, or Rawrite32 on Windows.</li>
<li> <a href="https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-10/latest/images/NetBSD-10.0_BETA-amd64-bios-install.img.gz">Installation / USB stick image for 64-bit x86 (legacy BIOS only)</a></li>
<li><a href="https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-10/latest/images/NetBSD-10.0_BETA-i386-install.img.gz">Installation / USB stick image for 32-bit x86</a></li>
<li><a href="https://armbsd.org">For images for various ARM devices, see armbsd.org</a>. All Arm boards without UFEI or Raspberry Pi firmware also require U-Boot to be written the SD card separately when using a generic image.</li>
<li> <a href="https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-10/latest/evbarm-earmv6hf/binary/gzimg/rpi.img.gz">Live image for Raspberry Pi 0 or 1</a></li>
<li> <a href="https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-10/latest/evbmips-mips64eb/binary/gzimg/octeon.img.gz">Live image for Cavium Octeon</a> (also needs U-Boot)</li>
<li> <a href="https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-10/latest/images/NetBSD-10.0_BETA-amd64.iso">CD/DVD image for 64-bit x86</a> (for actual CD/DVDs and virtual machines only)</li>
<li> <a href="https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-10/latest/images/NetBSD-10.0_BETA-i386.iso">CD/DVD image for 32-bit x86</a></li>
<li> <a href="https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-10/latest/images/NetBSD-10.0_BETA-evbarm-aarch64.iso">CD/DVD image for 64-bit Arm virtual machines</a></li>
<li> <a href="https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-10/latest/images/NetBSD-10.0_BETA-sparc64.iso">CD/DVD image for SPARC64</a></li>
<li> <a href="https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-10/latest/images/NetBSD-10.0_BETA-alpha.iso">CD/DVD image for Alpha</a></li>
<li> <a href="https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-10/latest/images/">More? (e.g. Apple PowerPC/68K, Amiga, Dreamcast, PA-RISC, SGI MIPS, SPARC32, VAX...)</a></li>
</ul>
<h3>Packages</h3>
<p>Third-party software is available from <a href="https://pkgsrc.org">pkgsrc</a>, as ever. <a href="http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/10.0_2022Q3/">Binary packages are available for amd64</a>, and I hope to publish binaries for i386 shortly.</p>
<h3>Reporting bugs</h3>
<p><a href="http://www.netbsd.org/cgi-bin/sendpr.cgi?gndb=netbsd"><strong>You can report bugs here!</strong></a></p>https://blog.netbsd.org/tnf/entry/netbsd_arm_on_oracle_cloudNetBSD Arm on Oracle Cloudjmcneill2022-10-15T20:05:05+00:002022-10-15T20:05:05+00:00<p>Support for running NetBSD on Oracle Cloud Arm-Based Compute Instances has been added to NetBSD -current.</p>
<p>A build of NetBSD/evbarm64 after 2022-10-15 will generate a bootable image (arm64.img.gz) that can be converted to a Custom Image that can run on Oracle Cloud. <i>See full post for details.</i></p><p>Support for running NetBSD on <a href="https://cloud.oracle.com">Oracle Cloud</a> Arm-Based Compute Instances has been added to NetBSD -current.</p>
<p>A build of NetBSD/evbarm64 after 2022-10-15 will generate a bootable image (arm64.img.gz) that can be converted to a Custom Image that can run on Oracle Cloud.</p>
<p>To get started, the image needs to be converted to QCOW2 format:</p>
<pre>
$ gunzip arm64.img.gz
$ qemu-img convert -f raw -O qcow2 arm64.img netbsd.qcow2
</pre>
<p>Next, upload the image to an Oracle Cloud storage bucket.</p>
<p>Once the QCOW2 file has been uploaded, switch to Compute / Custom Images and click <i>Import image</i>. Set an image name, make sure the <i>Operating system</i> field is set to Linux, and select the bucket and object name for your uploaded image. Make sure to select QCOW2 as the <i>Image type</i>. Set the mode to Paravirtualized mode.</p>
<p>After the image is imported, click <i>Edit details</i> and clear all checkboxes except for <i>VM.Standard.A1.Flex</i>. You could also try <i>BM.Standard.A1.160</i> (bare metal instance) but this is untested. Once the compatible shapes have been updated, click <i>Save changes</i>.</p>
<p>Now click <i>Edit image capabilities</i>, and under the <i>Firmware</i> heading, uncheck BIOS and click <i>Save changes</i>.</p>
<p>Finally, to create an instance, click the <i>Create instance</i> button. Make sure to either provide SSH keys, or download the generated private key in the <i>Add SSH keys</i> section. Click the <i>Create</i> button to start the instance.</p>
<p>The <i>Instance details page</i> will assign you a public IP address. Once the instance has started, you can ssh to it with the SSH key used during image creation as user <i>opc</i>.</p>
<pre>
$ ssh -i ssh-key-2022-10-15.key opc@x.x.x.x
Last login: Sat Oct 15 18:50:51 2022 from y.y.y.y
NetBSD 9.99.101 (GENERIC64) #9: Sat Oct 15 15:35:49 ADT 2022
Welcome to NetBSD!
This is a development snapshot of NetBSD for testing -- user beware!
Bug reports: https://www.NetBSD.org/support/send-pr.html
Donations to the NetBSD Foundation: https://www.NetBSD.org/donations/
-- UNSAFE KEYS WARNING:
The ssh host keys on this machine have been generated with
not enough entropy configured, so may be predictable.
To fix, follow the "Adding entropy" section in the entropy(7)
man page and after this machine has enough entropy, re-generate
the ssh host keys by running:
sh /etc/rc.d/sshd keyregen
instance-20221015-1520$ sysctl machdep.dmi
machdep.dmi.system-vendor = QEMU
machdep.dmi.system-product = KVM Virtual Machine
machdep.dmi.system-version = virt-4.2
machdep.dmi.chassis-vendor = QEMU
machdep.dmi.chassis-type = QEMU
machdep.dmi.chassis-version = virt-4.2
machdep.dmi.chassis-asset-tag = OracleCloud.com
machdep.dmi.processor-vendor = QEMU
machdep.dmi.processor-version = virt-4.2
machdep.dmi.processor-frequency = 2000 MHz
</pre>https://blog.netbsd.org/tnf/entry/the_geeks_way_of_checkingThe Geeks way of checking what the outside wheather is likemartin2022-09-24T18:58:31+00:002022-09-24T18:58:31+00:00<p>I recently had to replace my oldish WS2300 weather station, which was connected via a <strong>long</strong> serial cable (running from my kitchen to the machine room in the basement) with a modern device, a WS3500. This now connects to my wifi network and logs data to a postgres server running on a tiny aarch64 SoC, which also provides a website to query the data.</p>
<p>This all was done with minimal base systems means, plus very few additional pkgs from pkgsrc: in my case the postgres server, obviously, (or at least databases/postgresql14-client, if a postgres server already runs somewhere) and misc/sunwait used for a few site related calculations I found interesting to display. The only other suprising component used is pom(6) from the games set, used to calculate the phase of moon. The weather station displays this on its console, but it is not part of the reported weather data - but easy to recalculate.</p>
<p>Part of this work was to analyze details of the ecowitt or the weather underground protocol and extracting data from it.</p>
<p>The other part was creating two websites that display the current weather or some parts of the weather history.</p>
<p>For the two last parts I took inspiration from previous work done on this by others, and overall it turned out to be straight forward.</p>
<h1>Prologue</h1>
<p>
When I bought my house in 2004 I went shopping for a outside thermometer - and ended up with a full weather-station instead (a WS2300).
When I unpacked it I found a serial cable inside...</p>
<p>
Long story short - I was still in the process of recabling the house (running ethernet to every room) and added a serial cable from the machine room to the WS2300,
and then did some pkgsrc work and got misc/open2300 and misc/open2300-mysql. I used those to log the data from the weather-station to a mysql database, and later
moved that (via misc/open2300-pgsql) to a postgres database.</p>
<p>
Now sometime this year the machine running that database had to be replaced (should have done that earlier, it was power hungry and wasteful). The replacement was
an aarch64 SoC (a Pine64 Quartz64 model A) - and it had no real com ports (of course) any more. I had experimented with USB serial adapters and the WS2300 before,
but for unclear reasons this time I had no luck and couldn't get it to work. Since some of the outdoor sensors of the old weather-station had started failing, I decided
to replace it.</p>
<h1>New Weather-Station, new Sensors</h1>
<p>I picked a WS3500 because it comes with a nice remote sensor arrangement:</p>
<a href="//www.NetBSD.org/~martin/weather/ext_sensor.jpg"><img src="//www.NetBSD.org/~martin/weather/ext_sensor.jpg" style="max-width:80%;"/></a>
<p>
I attached it to a satellite dish mount about 1.2m above my garage and ran a two wire cable through the mount to supply it with 3V and get rid of any batteries.
It does not have a connector for that, but the battery compartment had enough space for a 330µF elco and soldering that and the cable directly to the battery contacts was easy.</p>
<p>The sensors report to the weather-station via a proprietary protocol in the 868 MHz band.</p>
<h1>New Weather-Station, new Reporting</h1>
<p>The weather-station can connect to a wifi network but does not offer any services itself.
The app used to configure the station offers several predefined weather collection services.</p>
<p>
I found the idea a bit strange to have my local weather data logged to some server somewhere else in the cloud and then get it back via my browser, but for others this is a good thing.
I found this <a href="https://www.linkedin.com/pulse/collecting-presenting-weather-sensor-data-using-ecowitts-jonas-frantz">article</a> that describes exactly the remote-only, no machines required on-site setup.
I used that article as inspiration for the data collection (but that part turned out to be quite trivial, see below) and copied a lot of the presentation site from it (also more details below).</p>
<p>
So in my setup I created web servers on two dedicated ports of my tiny machine running the postgres server.
One is used by the weather-station for reporting the data, the other is used to query the database.</p>
<p>
The configuration of the weather-station for a custom server was easy:</p>
<a href="http://netbsd.org/~martin/weather/screenshot1_app.jpg"><img src="//netbsd.org/~martin/weather/screenshot1_app.jpg" style="max-width:80%;"/></a>
<a href="http://netbsd.org/~martin/weather/screenshot2_app.jpg"><img src="//netbsd.org/~martin/weather/screenshot2_app.jpg" style="max-width:80%;"/></a>
<p>
I tested the ecowitt protocol first. It uses a post to a fixed URL and the form data has nearly identical data as we get with the solution I ended up with - only a few names (of form fields)
are slightly different.
</p>
<p>
The blacked items "StationID" and "StationKey" appear verbatim in the reported data, you can set them to whatever you want - the scripts below do not check them.</p>
<p>
The weather underground protocol does a simple http GET and provides all data as query parameters (I had to add the trailing question mark in the configuration).
This makes it very easy to extract the data in a script on the server side.</p>
<p>
But lets get there step by step. NetBSD comes with a http/https server in base, originally called "bozohttpd". It is very lightweight, but it can run various types of scripts - I picked
the plain old simple CGI and /bin/sh as language, using a bit of awk to convert units.</p>
<p>
First I added two users, so I could separate file access rights. This is how they look like in vipw:
<pre style="background-color:lightgoldenrodyellow;">
weatherupdate:*************:1004:1004::0:0:Weather Update Service:/weather/home:/sbin/nologin
weatherquery:*************:1005:1004::0:0:Weather Query Service:/weather/query:/sbin/nologin
</pre>
and two httpd instances for them
<code>/etc/inetd</code> entry to collect the incoming data:</p>
<pre style="background-color:lightgoldenrodyellow;">
88 stream tcp nowait:600 weatherupdate /usr/libexec/httpd httpd -q -c /weather/cgi /weather/files
89 stream tcp nowait:600 weatherquery /usr/libexec/httpd httpd -q -c /weather/cgi -M .js "text/javascript" - - /weather/files
</pre>
<p>
The document root (<code>/weather/files</code>) would not be used for the instance on port 88, but httpd needs one.
Note that these lines use the quiet flag ("-q") which is only available in netbsd-current. You can replace it with "-s" for older versions.</p>
<p>
The home directories of both users are mostly empty, besides a <code>.pgpass</code> file that contains the password for this
user connection to the postgres server. They look like this:</p>
<pre style="background-color:lightgoldenrodyellow;">
127.0.0.1:5432:weatherhistory:open2300:xxxxxxxxxxxxxx
</pre>
<p>where "weatherhistory" is the datebase and "open2300" is the name of the postgres user for the update script and the password is x-ed out. The other file looks very similar:</p>
<pre style="background-color:lightgoldenrodyellow;">
127.0.0.1:5432:weatherhistory:weatherquery:xxxxxxxxxxx
</pre>
<p>
At the postgres level the user "weatherquery" needs to have SELECT privilege on the table "weather", and "open2300" needs to have INSERT privilege.
The table schema (output of "pg_dump -s") looks like this:</p>
<pre style="background-color:lightgoldenrodyellow;">
--
-- Name: weather; Type: TABLE; Schema: public; Owner: weathermaster
--
CREATE TABLE public.weather (
"timestamp" timestamp without time zone DEFAULT '1970-01-01 00:00:00'::timestamp without time zone NOT NULL,
temp_in double precision DEFAULT '0'::double precision NOT NULL,
temp_out double precision DEFAULT '0'::double precision NOT NULL,
dewpoint double precision DEFAULT '0'::double precision NOT NULL,
rel_hum_in integer DEFAULT 0 NOT NULL,
rel_hum_out integer DEFAULT 0 NOT NULL,
windspeed double precision DEFAULT '0'::double precision NOT NULL,
wind_angle double precision DEFAULT '0'::double precision NOT NULL,
wind_chill double precision DEFAULT '0'::double precision NOT NULL,
rain_1h double precision DEFAULT '0'::double precision NOT NULL,
rain_24h double precision DEFAULT '0'::double precision NOT NULL,
rain_total double precision DEFAULT '0'::double precision NOT NULL,
rel_pressure double precision DEFAULT '0'::double precision NOT NULL,
wind_gust double precision DEFAULT 0 NOT NULL,
light double precision DEFAULT 0 NOT NULL,
uvi double precision DEFAULT 0 NOT NULL
);
ALTER TABLE public.weather OWNER TO weathermaster;
--
-- Name: weather weather_pkey; Type: CONSTRAINT; Schema: public; Owner: weathermaster
--
ALTER TABLE ONLY public.weather
ADD CONSTRAINT weather_pkey PRIMARY KEY ("timestamp");
--
-- Name: TABLE weather; Type: ACL; Schema: public; Owner: weathermaster
--
GRANT INSERT ON TABLE public.weather TO open2300;
GRANT SELECT ON TABLE public.weather TO weatherquery;
</pre>
<p>
As noted above, I carried this database over (with minor modifications) from previous instances of the whole setup - so it may not be optimal or elegant.
One thing that needs special attention is the "timestamp" column - it carries date/time in UTC and has no timezone associated.
This looked like a natural choice, but has some unexpected consequences.
When querying data in JSON format, "timestamp" will not get the JavaScript marker for "UTC", a "Z" suffix. So in the JavaScript code in the web pages
you will find quite a few places that cover up for this.</p>
<p>
Now when the weather station sends data to the configured server, inetd(8) runs httpd(8) and that invokes a shell script <code>/weather/cgi/update.cgi</code>
as the "weatherupdate" user.
This script uses awk(1) to do a few unit conversions and output a SQL command to insert the data into the "weather" table.
This SQL command is then piped to psql(1) with the connection string passed on the command line.
The corresponding password is found in <code>~/.pgpass</code> of the "weatherupdate" user.</p>
<p>
The script looks like this:</p>
<pre style="background-color:lightgoldenrodyellow;">
#! /bin/sh
TZ=UTC; export TZ
awk -v $( echo "$QUERY_STRING" | sed 's/\&/ -v /g' ) 'BEGIN {
temp=(tempf-32)/1.8;
indoortemp=(indoortempf-32)/1.8;
dewpt=(dewptf-32)/1.8;
windchill=(windchillf-32)/1.8;
windspeed=windspeedmph*1.609344;
windgust=windgustmph*1.609344;
rain=rainin*25.4;
dailyrain=dailyrainin*25.4;
totalrain=totalrainin*25.4;
rel_preasure=baromin/0.029529980164712;
printf("INSERT INTO weather VALUES ('"'"'%s'"'"', %f, %f, %f, %d, %d, %f, %d, %f, %f, %f, %f, %f, %f, %f, %f);\n",
strftime("%F %T"),
indoortemp,
temp,
dewpt,
indoorhumidity,
humidity,
windspeed,
winddir,
windchill,
rain, dailyrain, totalrain,
rel_preasure,
windgust,
solarradiation, UV);
}' | psql "hostaddr='127.0.0.1'dbname='weatherhistory'user='open2300'" > /dev/null 2>&1
</pre>
<p>
Note that it explicitly sets the timezone to UTC.
The input data comes (as defined by CGI) via the QUERY_STRING environment variable, as a set of "field=value" items, separated by &.
They are converted to sets of "-v" args for the awk invocation via a simple sed script.</p>
<p>
With this in place, the weather-station adds a record every five minutes to the database, and it was fun to check it via SQL,
but for reasons not quite clear to me most of the rest of the family did not like that kind of access very much.</p>
<pre style="background-color:lightgoldenrodyellow;">
psql (14.5)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
weatherhistory=> select min(temp_out), max(temp_out) from weather;
min | max
-------+------
-18.1 | 80.9
(1 row)
</pre>
<p>
I initially thought the 80.9°C were measured while I was soldering the power cable, but apparently they were fallout from the sometimes failing
sensors of the old station. The database has 2840 rows with temp_out > 40°C and all of them are 80.something. I should replace them with an average of the neighbor
records.</p>
<h1>Presenting the data</h1>
<p>
So I needed an internal web site. Which needs access to the data.
The above setup already paved the way for that, via the second port I set up.
I wanted to show all the current data in one page, and variable history data on another - which meant two CGI scripts to query the data.
The <code>/weather/cgi/latest.cgi</code> script just fetches the last record logged and creates a JSON from it, and also uses pom(6) and the sunwait(1) program
from pkgsrc to supply some site and date specific data:</p>
<pre style="background-color:lightgoldenrodyellow;">
#! /bin/sh
PATH=/usr/games:/usr/pkg/bin:$PATH
GEOPOS="51.505554N 0.075278W" # geographic position of this weather station
UPDATE=300 # seconds between updates
# This script uses psql(1) from pkgsrc/databases/postgresql14-client,
# pom(6) from the NetBSD games set and pkgsrc/misc/sunwait.
# collect global site data: sunrise and friends
eval $( sunwait report ${GEOPOS} | awk -F": " '
/Sun directly north/ {
printf("zenith=\"%s\"\n", $2);
}
/Daylight:/ {
split($2,v," to ");
printf("sunrise=\"%s\"\nsunset=\"%s\"\n", v[1], v[2]);
}
/with Civil twilight:/ {
split($2,v," to ");
printf("dawn=\"%s\"\ndusk=\"%s\"\n", v[1], v[2]);
}
/It is: Day/ {
printf("day=true\n");
}
/It is: Night/ {
printf("day=false\n");
}
' )
# moon phase
eval $( pom | awk '-F(' '
/The Moon is Full/ { printf("moontrend=\"-\"\nmoon=100\n"); }
/The Moon is New/ { printf("moontrend=\"+\"\nmoon=0\n"); }
/First Quarter/ { printf("moontrend=\"+\"\nmoon=50\n"); }
/Last Quarter/ { printf("moontrend=\"-\"\nmoon=50\n"); }
/Waxing/ {
a=$0;
sub(/^.*\(/, "", a);
sub(/%.*$/, "", a);
printf("moontrend=\"+\"\nmoon=%d\n", a+0);
}
/Waning/ {
a=$0;
sub(/^.*\(/, "", a);
sub(/%.*$/, "", a);
printf("moontrend=\"-\"\nmoon=%d\n", a+0);
}
' )
# start the json output
printf "\n\n{ \"site\": { \"updates\": ${UPDATE},
\"dawn\": \"${dawn}\", \"sunrise\": \"${sunrise}\",
\"zenith\": \"${zenith}\", \"day\": ${day},
\"sunset\": \"${sunset}\", \"dusk\": \"${dusk}\",
\"moon\": { \"trend\": \"${moontrend}\", \"percent\": ${moon} }\n}, \"weather\":\n"
# fill database results
printf "WITH t AS ( SELECT * FROM weather ORDER BY timestamp DESC LIMIT 1 ) SELECT row_to_json(t) FROM t;\n" |
psql --tuples-only --no-align "hostaddr='127.0.0.1'dbname='weatherhistory'user='weatherquery'"
# terminate json
printf "\n}\n"
</pre>
<p>
As you can see, if you would restrict output to plain data from the database, the script would be only four or five lines long.
But I like the additional spicing.</p>
<p>
The <code>/weather/cgi/history.cgi</code> script fetches rows between two timestamps passed to it (in JSON timestamp format)
and answers with a JSON containing an array of all the data in the requested time window:</p>
<pre style="background-color:lightgoldenrodyellow;">
#! /bin/sh
COND=$( echo "${QUERY_STRING}" | tr '&' '\n'| sed -e 's/%22/\"/g' -e 's/%3A/:/g' | awk '
/from=/ { v=$0; sub(/^[^"]*\"/, "", v); sub(/\".*$/, "", v); arg_from=v; }
/to=/ { v=$0; sub(/^[^"]*\"/, "", v); sub(/\".*$/, "", v); arg_to=v; }
END {
if (arg_from && arg_to) {
printf("timestamp >= '"'"'%s'"'"' AND timestamp <= '"'"'%s'"'"'\n",
arg_from, arg_to);
}
}
' )
if [ -z "${COND}" ]; then
# printf "could not parse: ${QUERY_STRING}\n" >> /tmp/sql.log
exit 0;
fi
# start output
printf "\n\n"
# printf "${COND}\n" >> /tmp/sql.log
# fill database results
printf "WITH t AS ( SELECT * FROM weather WHERE ${COND} ORDER by timestamp ASC ) SELECT json_agg(t) FROM t;\n" |
psql --tuples-only --no-align "hostaddr='127.0.0.1'dbname='weatherhistory'user='weatherquery'" # 2&>> /tmp/sql.err
</pre>
<p>
Fetching this data now is easy in JavaScript.</p>
<p>
We have a request URL defined as a const, like this:</p>
<pre style="background-color:lightgoldenrodyellow;">
const queryURL = 'http://weatherhost.duskware.de:89/cgi-bin/history.cgi?';
</pre>
<p>and then add (if needed) the paramaters for the query, like in this example function that gets passed a from-date and a to-date:</p>
<pre style="background-color:lightgoldenrodyellow;">
function showData(fromD, toD)
{
var url = new URL(queryURL);
url.searchParams.append("from", '"'+fromD.toJSON()+'"');
url.searchParams.append("to", '"'+toD.toJSON()+'"');
fetch(url).then(function(response) {
return response.json();
}).then(function(data) {
makeGraphs(data);
updateButtons();
}).catch(function(error) {
console.error(error)
});
}
</pre>
<p>
When the answer from the server arrives, it is decoded as JSON and returned as input data to the next function that makes some graphs from the data array.
Finally a few buttons are updated (in this example the time window is put into a start and a end date control.</p>
<p>
Inspired by the <a href="https://www.linkedin.com/pulse/collecting-presenting-weather-sensor-data-using-ecowitts-jonas-frantz">post</a>
mentioned above I used <a href="https://canvas-gauges.com/">canvas gauges</a> for the display of the latest data
and <a href="https://dygraphs.com/">dygraphs</a> for the display of historic data.</p>
<p>
Here is an example of how the latest display looks:</p>
<a href="//www.NetBSD.org/~martin/weather/website_weather.png"><img src="//www.NetBSD.org/~martin/weather/website_weather.png" style="max-width:80%;"/></a>
<p>
And here is how the history display looks:</p>
<a href="//www.NetBSD.org/~martin/weather/website_weatherhistory.png"><img src="//www.NetBSD.org/~martin/weather/website_weatherhistory.png" style="max-width:80%;"/></a>
<p>
I have put an archive of the cgi scripts and web pages <a href="//www.NetBSD.org/~martin/weather/weather.tgz">here</a>, and also for the curious who just
want to peek at the full glory of my web design skills the <a href="//www.NetBSD.org/~martin/weather/index.html">start page</a> (showing the latest weather data)
and the <a href="//www.NetBSD.org/~martin/weather/history.html">history page</a>.
</p>
<p>
Besides those files, you will need</p>
<ul>
<li>a <code>/weather/files/favicon.ico</code> if you like.</li>
<li>download <code>gauge.min.js</code> from canvas gauges and put it into <code>/weather/files/</code>.</li>
<li>download <code>dygraph.css</code>, <code>dygraph.min.js</code> from dygraph, plus <code>synchronizer.js</code> from the dygraph <code>extras/</code>
directory and put it also into <code>/weather/files/</code>.</li>
</ul>
<p>
Then you should be ready to go - easy, isn't it? And no heavy weight dependencies or pkgs needed.</p>
<h1>What about other weather stations?</h1>
<p>
There are quite a few similar weather stations out there now that seem to run "related" firmware and have similar capabilities.
Most likely the update script (and details in the presentation pages) will need adjustements for other types.</p>
<p>
If you start with a different device, just log all the data it sends and adjust the cgi scripts/database/JavaScript accordingly.
For protocol analyzis there are several easy means:</p>
<ul>
<li>Remove the "-q" flag in the httpd command (in <code>/etc/inetd.conf</code>) and check <code>/var/log/xferlog</code> for the quey paramaters sent by the
weather station (when using the weather underground protocol).</li>
<li>Make the station log to a <code>debug.cgi</code> first to capture everything (including form data posted). This works for the ecowitt protocoll.</li>
<li>All this stations seem to use http only (not https), so you can sniff the traffic. Use <code>tcpdump -w</code> on the server to capture the data and
analyze it with net/wireshark from pkgsrc.</li>
</ul>
<p>Here is what a debug.cgi script could look like:</p>
<pre style="background-color:lightgoldenrodyellow;">
#! /bin/sh
env > /tmp/debug.env
printf "\n\nOK\n"
cat > /tmp/debug.input &
</pre>
<p>
This allows you to see the form input in /tmp/debug.input and the CGI environment in /tmp/debug.env.</p>https://blog.netbsd.org/tnf/entry/eurobsdcon_2022EuroBSDCon 2022Nia Alarie2022-09-20T09:21:16+00:002022-09-20T09:38:48+00:00<p>After two years of trying, we managed to have a EuroBSDCon in Vienna. Here's how it went...</p><p>No videos are available yet to provide much-needed context to presentations, but we'll keep you posted.</p>
<h2>Day -2 - Arrival in Vienna</h2>
<p>After being thoroughly delayed by Deutsche Bahn, I hopped off an InterCity Express train to check out the hotel room for people speaking at EuroBSDCon, which was An Experience in itself. There was a mural of a shirtless man with a sword covered in snakes next to my bed, what else do you need in life? Lots of coffee, obviously.</p>
<p>Begin the march to the conference to listen to Marshall Kirk McKusick lecture on schedulers.</p>
<h2>Day -1 - NetBSD Developer Summit</h2>
<p>Around 16 NetBSD developers gathered in a room for the first time in two years. I was a little bit distracted and late due to Marshall Kirk McKusick's very detailed lecture on filesystems melting my brain somewhat, but we had the opportunity to present various informal presentations, after we'd finished showing off suspend/resume support on our ThinkPad laptops.</p>
<p><strong>Benny Siegert</strong> opened with a presentation on the <strong>state of the Go programming language on NetBSD</strong> (and whether it is "in trouble"), covering various problems with instability being detected inside the Go test suite. Go is particularly interesting (and maybe error-prone) because it mostly bypasses NetBSD libc, which is unusual for software running on NetBSD, instead preferring to implement its own wrappers around the kernel's system calls.</p>
<p>A few problems had been narrowed down to being (likely) AMD CPU bugs, others weren't reproducible in production (outside of the test suite) at all, and others <em>may</em> have been fixed in NetBSD 9.1 - the NetBSD machines running tests for Go do need to be updated. If you're from AMD, please get in touch.</p>
<p>We've got a very impressive test suite for NetBSD itself, but outside tests are always useful for identifying problems that we can't catch... that said, they do require a lot of work to maintain, and a lack of patience is understandable. We'd love any help we can get with this.</p>
<p>I pointed out that we get occasional failures bootstrapping Go in pkgsrc, and better debug output would be nice -- Benny was able to arrange this within the day, and we should get nice detailed bootstrapping logs for Go now.</p>
<p><strong>Pierre Pronchery</strong> (khorben@) discussed cross-BSD collaboration on synchronizing our device driver code bases, including his recent NetBSD Foundation-supported work on the <a href="https://man.netbsd.org/emuxki.4">emuxki(4)</a> sound card driver, where other BSDs have taken the same code base but improvements had not yet been universal. We all agreed that collaboration and keeping drivers in sync is important. We talked about the on-going project to synchronize NetBSD Wi-Fi drivers with FreeBSD.</p>
<p><strong>Martin Kjellstrand</strong> then gave us a very nice demonstration of his <a href="https://github.com/madworx/docker-netbsd">NetBSD docker images</a>, and how easy it is to spin up NetBSD on-demand to run a command (this also has wide potential for being useful for testing). In turn, I rambled a bit about my own experiments of <a href="https://github.com/alarixnia/mkimg-netbsd">dynamically creating NetBSD images</a>. This would lead to a later discussion about whether we need to prioritize improving the <a href="https://man.netbsd.org/resize_ffs.8">resize_ffs(8)</a> command's support for new filesystems.</p>
<p>The theme of creating NetBSD images "for the cloud" continued, with <strong>Benny Siegert</strong> presenting again about <a href="https://www.netbsd.org/gallery/presentations/bsiegert/netbsd-on-gce.pdf">NetBSD on Google Compute Engine</a>.</p>
<p><strong>Stephen Borrill</strong> then stepped up to give us an incredibly detailed history of the British computer company <a href="https://en.wikipedia.org/wiki/Acorn_Computers">Acorn Computers</a>, complete with his personal experiences servicing Acorn machines in the early 90s. We discussed the history of the ARM CPU, and <a href="https://wiki.netbsd.org/ports/acorn32/">NetBSD/acorn32</a>.</p>
<p><strong>Nia Alarie</strong> (surprise) finished up with a very short unplanned demonstration of some of the projects she's been working on lately - using NetBSD as a professional digital audio workstation, improving the default graphical experience of NetBSD with dynamically generated menus, and (again) creating customized micro-images of NetBSD. We discussed support for MIDI devices (I'd later chat with some of the FreeBSD people about collaborating on JACK MIDI).</p>
<p>We then retired to Thomas Klausner (wiz@)'s favorite ramen restaurant and discussed, among other things, <a href="https://en.wikipedia.org/wiki/Studio_Ghibli">Studio Ghibli</a> films, and trains. Trains would be a recurring theme.</p>
<h2>Day 0 - start of talks</h2>
<p>We began the day with two NetBSD presentations scheduled back-to-back. This mostly meant that I got to talk about some of <a href="https://netbsd.org/gallery/presentations/nia/eurobsdcon2022/releng.html">NetBSD 10's upcoming features, and why it's taking so long</a> to a small crowd of interested people who didn't have much prior experience with NetBSD, while in another room <strong>Taylor R. Campbell</strong> (riastradh@) discussed his very dedicated efforts to make <a href="https://netbsd.org/gallery/presentations/riastradh/eurobsdcon2022/opendetach.pdf">suddenly disappearing devices</a> more reliable and not crash the kernel (we're still waiting for a live demonstration).</p>
<p>Next, <strong>Pierre Pronchery</strong> (khorben@) discussed the power of <a href="https://pkgsrc.org">pkgsrc</a> for creating consistent environments across platforms for software developers, serving as a nice portable, classic Unix alternative to technologies like Docker and Nix.</p>
<p>The final presentation of the day was riastradh@ again, this time providing a live lecture (from Emacs!) about <a href="https://netbsd.org/gallery/presentations/riastradh/eurobsdcon2022/membars.txt">memory barriers in the kernel</a>. We all learned to appreciate the nice abstractions technologies like mutexes provide to stop CPUs from re-ordering code on multi-processor machines in inexplicable ways.</p>
<h2>Day 1 - final talks</h2>
<p>The second day of EuroBSDCon presentations was mostly devoid of anything NetBSD-focused, so we had a nice opportunity for cross-pollination and to learn and collaborate with other BSD projects. I chatted a bit with an OpenBSD Ports developer about the challenge technologies like Rust pose to developing a cross-architecture packaging system, and with a FreeBSD person about the state of professional audio on our respective platforms. Michael Dexter finished the day of presentations with a very passionate speech about why we all need BSD in our lives, regardless of our preferred flavour.</p>
<p>More topics were discussed in the various break periods, including whether our newest update to the GPU drivers is stable enough to include in a release (verdict: works for me).</p>
<p>We then watched as various BSD t-shirts and boxes of chocolates were auctioned away to support a local refugee center. The organizing committee forgot to include the NetBSD Foundation on the list of sponsors, but we forgive them.</p>
<h2>Other news from the Project</h2>
<p>I've recently made sure the <a href="https://netbsd.org/changes/changes-10.0.html">NetBSD 10 changelog</a> is up to date with all the new goodness, so you should check that out.</p>https://blog.netbsd.org/tnf/entry/netbsd_9_3_releasedNetBSD 9.3 releasedNia Alarie2022-08-06T13:55:55+00:002022-08-06T13:55:55+00:00<p>The NetBSD Project is pleased to announce NetBSD 9.3, the third release from the NetBSD 9 stable branch.</p>
<p>It represents a selected subset of fixes deemed important for security or stability reasons since the release of NetBSD 9.2 in May 2021, as well some enhancements backported from the development branch. It is fully compatible with NetBSD 9.0. Users running 9.2 or an earlier release are strongly recommended to upgrade.</p><p>The NetBSD Project is pleased to announce NetBSD 9.3, the third release from the NetBSD 9 stable branch.</p>
<p>It represents a selected subset of fixes deemed important for security or stability reasons since the release of NetBSD 9.2 in May 2021, as well some enhancements backported from the development branch. It is fully compatible with NetBSD 9.0. Users running 9.2 or an earlier release are strongly recommended to upgrade.</p>
<p>Aside from many bug fixes, 9.3 includes backported <strong>improvements to suspend and resume support</strong>, various minor additions of new hardware to existing device drivers, compatibility with <strong>UDF file systems created on Windows 10</strong>, enhanced support for newer Intel Gigabit Ethernet chipsets, better support for <strong>new Intel and AMD Zen 3 chipsets</strong>, support for <strong>configuring connections to Wi-Fi networks using sysinst(8)</strong>, support for <strong>wsfb-based X11 servers on the Commodore Amiga</strong>, and minor <strong>performance improvements for the Xen hypervisor</strong>.</p>
<p>The general NetBSD community is very excited about NetBSD 10.0, but it was deemed necessary to make this bug fix release available while we wait for the resolution of some compatibility problems in NetBSD-current concerning FFS Access Control Lists preventing the <code>netbsd-10</code> release.</p>
<p><strong><a href="http://netbsd.org/releases/formal-9/NetBSD-9.3.html">Full release notes, including download links</a></strong></p>https://blog.netbsd.org/tnf/entry/announcing_google_summer_of_code3Announcing Google Summer of Code 2022 projectsAndrius Varanavicius2022-05-22T12:20:02+00:002022-05-22T12:20:02+00:00<p>
<a href='https://summerofcode.withgoogle.com/'><img alt="Google Summer of Code logo" src='//www.NetBSD.org/~leot/blog-posts/imgs/GSoC-icon-192.png' style="width: 80px; height: 80px; float: right;" /></a>
The NetBSD Foundation has finalized the list of projects for this year’s <a href="https://summerofcode.withgoogle.com/programs/2022/organizations/the-netbsd-foundation">Google Summer of Code</a>. The contributors and projects are the following:
</p>
<ul>
<li>Brian Schnepp - <a href="https://summerofcode.withgoogle.com/programs/2022/projects/esA98HIB">Raspberry Pi GPU Driver</a></li>
<li>Arjun Bemarkar - <a href="https://summerofcode.withgoogle.com/programs/2022/projects/LDY5asp0">inetd enhancements</a></li>
<li>Piyush Sachdeva - <a href="https://summerofcode.withgoogle.com/programs/2022/projects/37Q8OZNU">Emulating missing linux syscalls</a></li>
<li>Vihas Makwana - <a href="https://summerofcode.withgoogle.com/programs/2022/projects/yO4fTbNn">Introduce a new Wi-Fi driver</a></li>
<li>Vivek Kumar Sah - <a href="https://summerofcode.withgoogle.com/programs/2022/projects/UPlZRXS6">Automating donor acknowledgement and information storage</a></li>
</ul>
<p>
The community bonding period has already started (from May 20) and it will last until June 12. During this time, the contributors are expected to coordinate with their mentors and community.
</p>
<p>
This will be immediately followed by the coding period from June 13 to September 4. After which, the contributors are expected to submit their final work, evaluate their mentors, and get evaluated by their mentors as well.
Results will be announced on September 20.
</p>
<p>
For more information about the Google Summer of Code 2022 kindly refer to the official <a href="https://summerofcode.withgoogle.com/">GSoC website</a>.
</p>
<p>
We would like to express our gratitude to Google for organizing the yearly GSoC, and to The NetBSD Foundation mentors and administrators for their efforts and hardwork!
</p>
<p>
Let us welcome all the contributors to our growing NetBSD community!
</p>https://blog.netbsd.org/tnf/entry/the_netbsd_foundation_is_aThe NetBSD Foundation is a mentoring organization at Google Summer of Code 2022Leonardo Taccari2022-03-16T17:02:15+00:002022-03-16T17:07:32+00:00<p>
<a href='https://summerofcode.withgoogle.com/'><img alt="Google Summer of Code logo" src='https://developers.google.com/open-source/gsoc/resources/downloads/GSoC-Horizontal.png' /></a>
</p>
<p>
We are happy to announce that The NetBSD Fundation is a <a href="https://summerofcode.withgoogle.com/programs/2022/organizations/the-netbsd-foundation">mentoring organization</a> at <a href="https://summerofcode.withgoogle.com/">Google Summer of Code 2022</a>!
</p>
<p>
Would you like to contribute to NetBSD or pkgsrc during the summer? Please give a look to <a href="https://wiki.NetBSD.org/projects/gsoc/">NetBSD wiki Google Summer of Code page</a> with possible ideas list and/or please join <a href="https://web.libera.chat/#NetBSD-code">#NetBSD-code</a> IRC channel on <a href="https://libera.chat/">libera</a> or get in touch with us via <a href="https://www.NetBSD.org/mailinglists/">mailing lists</a> to propose new projects!
</p>
<p>
Please note that unlike past years where Google Summer of Code was opened only to university students since this year if you are 18 or older you can be a GSoC contributor.
</p>
<p>
For more information about Google Summer of Code please give a look to the official <a href="https://summerofcode.withgoogle.com/">GSoC website</a>.
</p>
<p>
Looking forward to have a nice Google Summer of Code!
</p>https://blog.netbsd.org/tnf/entry/making_rockpro64_a_netbsd_serverMaking RockPro64 a NetBSD Servermatthew green2022-03-09T23:57:54+00:002022-03-09T23:57:54+00:00<p>The time has come to upgrade my SunBlade 2500s to something more power friendly and faster. I'd already removed one CPU and thus half the ram from two of these systems to reduce their power consumption, but it's still much higher than it could be.</p>
<p>After much searching, I've decided on <a href="https://wiki.pine64.org/wiki/ROCKPro64">Pine64's RockPro64</a> 4GiB ram model (technically, only 3.875GiB ram.) Pine64 make SBCs, laptops, phones, and various other mostly ARM gadgets, and the RockPro64 has the fastest CPU they ship (Rockchip RK3399), and can use a small "NAS Case", that is sufficient to house 2 HDDs and, at a stretch, upto 6 SSDs (cooling would become quite an issue at this point.)</p>
<p>In my SATA setup, I have 3 SSDs with a JMicron 585 card in the PCIe slot, two SSDs in the NAS case SSD region, and the third is in the HDD region with an adapter. I have used two SATA power splitters to convert the NAS case's 2 SATA power ports into 4, with the 4th one also powering a Noctua case fan. The cabling is not great with this, with enough SATA power cabling for 6 devices to lay. Probably a bespoke power cable to connect to the RockPro64 would make this nicer and probably improve cooling slightly, but I'm just using off-the-shelf components for now.</p>
<p>In the last year or so I've been working on making NetBSD/arm64 big-endian more featureful. In particular, I've added support for:
<ul><li>other-endian access disklabels, FFS file-systems in the NetBSD libsa</li>
<li>the "look 64 sectors later" for RAIDFrame partitions in MI efiboot (the x86 specific efiboot has more extensive support for this that should be ported over)</li>
<li>other-endian access to RAIDFrame labels in the kernel</li>
<li>updated the U-Boot package and featureset to include AHCI/SATA support, workaround some bugs and fix the newer U-Boot SPI loader location, and figured out how to default loading from either SATA or NVMe</li>
</ul></p>
<p>There are not too many special actions needed for this sort of setup compared to a normal NetBSD or Arm system. While I built my installations by hand, the standard NetBSD Arm images are suitable for this task. It's easiest to start from an SD card with the RockPro64 u-boot installed. There are two U-Boot images available, one for SD/eMMC, and one for SPI (there is an odd problem with the early SPI loader that requires a portion of the image to be different.) The pkgsrc package for sysutils/u-boot-rockpro64 version 2022.01 has these suggested methods for installing the U-Boot image (this package should be buildable on any supported pkgsrc platform).</p>
<p>To install U-Boot into the SD/eMMC card (can run on any system, the image must be written at 32KiB into the device):
<pre>
# dd if=/usr/pkg/share/u-boot/rockpro64/rksd_loader.img seek=64 of=/dev/rld0c
</pre>
<p>
To install U-Boot into the SPI flash (must be run on the host, and lives at the very start of the device:
<pre>
# dd if=/usr/pkg/share/u-boot/rockpro64/rkspi_loader.img bs=64k of=/dev/spiflash0
</pre>
</p>
<p>When booting from NVMe or SATA, one must drop to the U-Boot prompt and adjust the "boot_targets" value to put scsi* (SATA) or nvme* ahead of the mmc* and usb* options.</p>
<p>The original value for me:
<pre>
=> printenv boot_targets
boot_targets=mmc1 usb0 mmc0 nvme0 scsi0 pxe dhcp sf0
</pre>
</p>
<p>Which is then adjusted and saved with eg:
<pre>
=> setenv boot_targets nvme0 scsi0 mmc1 usb0 mmc0 pxe dhcp sf0
=> saveenv
Saving Environment to SPIFlash... Erasing SPI flash...Writing to SPI flash...done
OK
</pre></p>
<p>(In this list, mmc1 is the SD slot, mmc0 is the eMMC card, pxe and dhcp are netbooting, and sf0 attempts to load further U-Boot scripts from SPI.)</p>
<p>There are some minor issues with the RockPro64. It has no ability to use ECC memory. It only comes with 1 PCIe 4x slot, and Rockchip errata limited this to PCIe 1.x (though no NetBSD users encounted any issues with PCIe 2.x enabled, and this can be forced back on via a DTS patch.) It is possible to use a PCIe bridge (I have never done this, though I would like to try in the future) to enable more devices for booting, or to enable both a network and storage device.</p>
https://blog.netbsd.org/tnf/entry/project_report_add_support_forProject Report: Add support for chdir(2) support in posix_spawn(3)martin2021-11-22T18:01:54+00:002021-11-23T16:38:20+00:00<p>Piyush Sachdeva finished the "add chdir support to posix_spawn(3)" project and reports about his work and experience.
His code is already in -current and will be part of NetBSD 10.</p>
<p>Originally submitted as a proposal for GSoC, but unfortunately (due to low slot allocations) this project was not part of GSoC.</p>
<p>The NetBSD Foundation decided to nevertheless run the project and funded it.</p><h3>This post was written by Piyush Sachdeva:</h3>
<h2>Abstract</h2>
<p>The primary goal of the project was to extend posix_spawn(3) to include chdir(2)
for the newly created child process. Two functions were supposed to be implemented,
namely posix_spawn_file_actions_addchdir() and
posix_spawn_file_actions_addfchdir(), to support both chdir(2) and
fchdir(2) respectively.
posix_spawn() is a POSIX standard method
responsible for creating and executing new child processes.</p>
<h2>Implementation</h2>
<p>The original code can be found at <a href="https://github.com/cosmologistPiyush/posix_spawn-chdir/tree/trunk">my github tree</a>.</p>
<p>The implementation plan was discussed and made with the guidance of both my
mentors Martin Husemann and Joerg Sonnenberger. The plan was divided into
three phases each corresponding to the specific part of The NetBSD code-base which is
supposed to be touched:</p>
<h3>User-Land</h3>
<p>The following actions were performed in the user-land to set things up for the
kernel-space.
<ul>
<li>Add another member to the posix_spawn_file_actions_t struct i.e a union,
which would hold the path to chdir to.</li>
</li>
<li>Implement the two functions posix_spawn_file_actions_addchdir()
and posix_spawn_file_actions_addfchdir(). These functions would:
<ol>
<li>allocate memory for another posix_spawn_file_actions_t object in the
posix_spawn_file_actions_t array.</li>
<li>take the path/file descriptor from the user as an argument and make the
relative field of the newly allocated file actions object, point to it.</li>
</ol>
</li>
<li>The final step was to add the prototypes for the two new functions to the
`src/include/spawn.h' header file
</li>
</ul>
<p>
Once the aforementioned changes were made, the only thing left to do was to make the
kernel support these two new functions.</p>
<h3>Kernel-Space</h3>
<p>The following actions were performed inside the kernel space.</p>
<ul>
<li>The three functions in the `src/sys/kern_exec.c' file which correspond to the
posix_spawn_file_actions were edited:
<ul>
<li>posix_spawn_fa_alloc() was adjusted to make sure that the path passed to
posix_spawn_file_actions_addchdir() gets copied from the user-land
to the kernel-space.
</li>
<li>Similarly posix_spawn_fa_free() was adjusted to make sure that the
memory allocated in case of FAE_CHDIR gets freed as well.
</li>
<li>Finally, two new cases FAE_CHDIR & FAE_FCHDIR were added in the
handle_posix_spawn_file_actions(). In each of the cases,
a call to one of the two newly created functions (discussed in the next point)
do_sys_chdir() and do_sys_fchdir() was made respectively.
</li>
</ul>
<p>Note: At the time of code integration, a helper function was written by
Christos Zoulas. This function aimed to reduce the amount of repeated code
in both posix_spawn_fa__free() and posix_spawn_fa_alloc()</p>
</li>
<li>Two new functions, similar to the already present sys_chdir() and sys_fchdir() in
`src/sys/vfs_syscalls.c' were created. Namely do_sys_chdir() and do_sys_chdir
were written with two specific thoughts in mind:
<ul>
<li>By default sys_chdir() and sys_fchdir() took syscallargs as a parameter.
The purpose of the new functions was to replace this with
const char * and an int type parameter respectively.</li>
<li>The do_sys_chdir() also replaced UIO_USERSPACE with
UIO_SYSSPACE. This was done because the chdir path passed to this
function already resided in the Kernel-space due to the change made in
posix_spawn_fa_alloc().</li>
</ul></li>
<li>Finally, the prototypes for the newly written functions were added to the
`src/sys/sys/vfs_syscalls.h' file and this file was also included in the
'sys/kern/kern_exec.c'.</li>
</ul>
<p>Note: Similar to the above changes of user-land and kernel-space, a few tweaks were also made to
`src/sys/compat/netbsd/netbsd32.h' and `netbsd32_execve.c'. This was required to help COMPAT_NETBSD32
deal with the new file actions member. However, these changes were made at the time of integration
by Martin Husemann.</p>
<p>With most of addition of new features being done, all that remained was testing and documentation.</p>
<h3>Testing & Documentation</h3>
<ul>
<li>A total of ten new test cases have been added to the
`src/tests/lib/libc/gen/posix_spawn/t_spawn.c' file.</li>
<li>Three utility functions were also used to aid in testing. Out of the three,
one new function was written and two existing functions (filesize()
and empty_outfile()) from `t_fileactions.c' were used.
To make sure that the 2 existing functions were shared between both the files i.e `t_spawn.c'
and `t_fileactions.c' a new header and C file was created, namely `fa_spawn_utils.h' and
`fa_spawn_utils.c'.
Following this, the bodies of both the functions were moved from
`t_fileactions.c' to `fa_spawn_utils.c' and their prototypes were added to the
corresponding header file.</li>
<li>The general approach that was taken to all test cases was to make
posix_spawn() execute ``/bin/pwd'' and write the output to a file.
Then read the file and do string comparison. The third function i.e. check_succes()
was written for just this purpose.</li>
<li>The ten test cases cover the following scenarios:
<ul>
<li>Absolute path test - for both chdir and fchdir.</li>
<li>Relative path test - for both chdir and fchdir.</li>
<li>Trying to open a file instead of directory - for both chdir and fchdir.</li>
<li>Invalid path/file descriptor (fd=-1) - for both chdir and fchdir.</li>
<li>Trying to open a directory without access permissions for chdir.</li>
<li>Opening a closed file descriptor for fchdir.</li>
</ul></li>
<li>The first 8 test cases had a lot of repetitive code. Therefore, at the time of integration,
another function was created i.e spawn_chdir().
This function included a huge chunk of the common code and it did all the heavy lifting
for those first 8 test cases.</li>
</ul>
<h4>Documentation:</h4>
<p>In this matter, a complete man page is written which explains both
posix_spawn_file_actions_addchdir() and posix_spawn_file_actions_addfchdir() in great detail.
The content of the manual page is taken from the POSIX documentation provided to us by Robert Elz.
</p>
<h2>Issues</h2>
<p>Since the project was well planned from the beginning, it resulted in few issues.</p>
<ul>
<li>The user-land was the most straight forward part of the project and I had no
trouble sailing through it.</li>
<li>Kernel space was where things got a bit complicated, as I had to add functionality
to pre-existing functions.</li>
<li>I was completely new to using atf(7) and groff(1). Therefore, it took me some time
to understand the respective man pages and become comfortable with testing and
documentation part.</li>
</ul>
<p>
Most of the issues faced were generally logistical. As it was my first time doing a
kernel project, I was new to building from source, Virtual Machines and other things like SSH.
But luckily, I had great help from my mentors and the entire NetBSD community.</p>
<h2>Thanks</h2>
<p>I would like to express my heartfelt gratitude to The NetBSD Foundation for
giving me this opportunity and sponsoring the Project.
This project would not have been possible without the constant support and
encouragement of both my mentors Martin Husemann and Joerg Sonnenberger.
My gratitude to Christos Zoulas who worked on the crucial part of integrating the code.
A special mention to all of the other esteemed NetBSD developers,
who have helped me navigate through the thick and thin of this project and
have answered even my most trivial questions.</p>
https://blog.netbsd.org/tnf/entry/wifi_project_status_updatewifi project status updatemartin2021-08-26T17:46:53+00:002021-08-26T17:46:53+00:00<p>About a year ago the <a href="/tnf/entry/wifi_renewal_restarted">wifi renewal project</a> got restarted. A lot things happened, but the high hopes of a quick breakthrough and fast merge to mainline did not come true.</p>
<p>Here is where we are today, what needs to be done and how things are planned to move on...</p><p>After initial work on the <a href="https://wiki.NetBSD.org/Wifi_renewal_on_hg/">wifi renewal branch</a> went quite fast and smooth, things have slowed down a bit in the last few months.</p>
<p>Most of the slow down was due to me not being available for this type of work for unexpectedly long times - a problem that should be fixed now.</p>
<p>However, there were other obstacles and unexpected issues on the way:</p>
<ul>
<li>bpf taps are handled differently in the new stack and some slightly obscure site conditions of this had been overlooked in the initial conversion. To make everything work, changes to our bpf framework were needed (and have landed in -current some time ago now).</li>
<li>Many wifi drivers seem to be in a, let's say, slightly fragile state. When testing the random collection of wifi hardware that I acquired during this project in -current, many drivers did not work at first try and often I was able to provoke kernel panics quickly.
This is not a happy base to start converting drivers from.</li>
<li>After the great success of usbnet(9) for USB ethernet drivers, core and I agreed to do the same for wifi - the result is called usbwifi(9) and makes conversion of usb drivers a lot easier than other wifi drivers. See <a href="https://wiki.NetBSD.org/tutorials/converting_usb_drivers_to_usbwifi__40__9__41__/">the conversion instructions</a> for more details. usbwifi(9) is both quite similar but also quite different to usbnet(9), mostly for two reasons: it interfaces to a totally different stack, and many usb wlan chipsets are more complex than ethernet chipsets (e.g. have support for multiple send queues with different priorities). Developing usbwifi did cost quit some time (initially unplanned), but is expected to amortize over the next few drivers and quickly end up as a net win.</li>
<li>I have been hitting a bug in the urtwn(4) driver used for inial usbwifi(9) development and still not found it (as of today). It seems to hit randomly and not be caused by the usbwifi(9) conversion - a fact that I found out only recently. So for now I will put this driver aside (after spending *way* too much time on it) and instead work on other usb drivers, returning to the bug every now and then and see if I can spot it. Maybe I can borrow a USB analyzer and get more insight that way.</li>
</ul>
<p>The current state of driver conversion and what drivers are still open are listed in <a href="https://wiki.NetBSD.org/Driver_state_matrix/">the wifi driver conversion matrix</a>.</p>
<p>Next steps ahead are:</p>
<ul>
<li><s>make another pass over documentation and improve things / fixup for recent changes</s> (done before this blog post got published)</li>
<li>sync the branch with HEAD and keep tracking it more closely</li>
<li>convert run(4) to usbwifi</li>
<li>revisit rtwn(4) and decide if/how it should be merged with urtwn(4)</li>
<li>revisit iwm(4) and make it work fully</li>
<li>convert all other drivers, starting with the ones I have hardware for</li>
</ul>
<p>Currently it is not clear if this branch can be merged to HEAD before branching for netbsd-10. We will not delay the netbsd-10 branch for this.</p>
https://blog.netbsd.org/tnf/entry/support_for_chdir_2_inSupport for chdir(2) in posix_spawn(3)martin2021-06-10T15:28:05+00:002021-06-10T15:28:05+00:00<p>Piyush Sachdeva is working on an extension to NetBSD's posix_spawn system call implementation and library support.</p>
<p>He applied as a GSoC student, but unfortunately we only got a single slot from Google this year, so The NetBSD Foundation
offered Piyush to work on it by TNF funding outside of the official GSoC.</p>
<p>In this post Piyush introduces himself and the project. He already started with the work... </p><h3>This post was written by Piyush Sachdeva:</h3>
<p>What really happens when you double click an icon on your desktop?</p>
<h1>Support for chdir(2) in posix_spawn(3)</h1>
<p>Processes are the bread and butter of your operating system. The moment
you double click an icon, that particular program gets loaded in your
Random Access Memory (RAM) and your operating system starts to run it. At
this moment the program becomes a process. Though you can only see the execution
of your process, the operating system (the Kernel) is always running a lot
of processes in the background to facilitate you.</p>
<p>From the moment you hit that power button, everything that happens on the
screen is the result of some or the other process. In this post we are
going to talk about one such interface which helps in creation of your
programs.</p>
<h2>The fork() & exec() shenanigans</h2>
<p>The moment a computer system comes alive, it launches a bunch of
processes. For the purpose of this blog let’s call them, ‘the master
processes’. These processes run in perpetuity, provided the computer is
switched on. One such process is <a href="#note1">init/systemd/launchd (depending on your OS)</a>.
This ‘init’ master process owns all the other processes in the computer,
either directly or indirectly.</p>
<p>Operating systems are elegant, majestic software that work
seamlessly under the hood. They do so much without even breaking a sweat
(unless it’s Windows). Let's consider a scenario where you have decided to
take a trip down memory lane and burst open those old photos. The ‘init
master process’ just can’t terminate itself and let you look at your
photos. What if you unknowingly open a malicious file, which corrupts all
your data? So init doesn’t just exit, rather it employs fork() and exec()
to start a new process. The fork() function is used to create child
processes which are an exact copy of their parents. Whichever process calls
fork, gets duplicated. The newly created process becomes the child of the
original running process and the original running process is called the
parent. Just how parents look after their kids, the parent process makes
sure that the child process doesn't do any mischief. So now you have two
exactly similar processes running in your computer.</p>
<img id="flowchar", src="//www.NetBSD.org/~martin/flowchart.jpg", alt="Flowchart of fork() + exec()", width="960", height="362", object-fit="contain">
<p>One might think that the newly created child process doesn’t really help
us. But actually, it does. Now exec() comes into the picture. What exec()
does is, it replaces any process which calls it. So what if we replace the child
process, the one we just thought to be useless, with our photos? That's
exactly what we are going to do indeed. This will result in replacement of
the fork() created child process with your photos. Therefore, the master
init process is still running and you can also enjoy your photos with no
threat to your data.</p>
<cite>
“Neither abstraction nor simplicity is a substitute for getting it right.
Sometimes, you just have to do the right thing, and when you do, it is way
better than the alternatives. There are lots of ways to design APIs for
process creation; however, the combination of fork() and exec() is simple
and immensely powerful. Here, the UNIX designers simply got it right.”
</cite> <a href="#note2">Lampson’s Law - Getting it Right</a>
<p>Now you could ask me, `But what about the title, some ‘posix_spawn()’
thing?´ Don’t worry, that’s next.</p>
<h2>posix_spawn()</h2>
<p>posix_spawn() is an alternative to the fork() + exec() routine. It
implements fork() and exec(), but not directly (as that would make it
slow, and we all need everything to be lightning fast). What actually
happens is that posix_spawn() only implements the functionality of the
fork() + exec() routines, but in one single call. However, because fork() +
exec() is a combination of two different calls, there is a lot of room for
customization. Whatever software you are running on your computer, calls
these routines on its own and does the necessary. Meanwhile a lot is
cooking in the background. Between the call to fork() and exec() there is
plenty of leeway for tweaking different aspects of the exec-ing process.
But posix_spawn doesn’t bear this flexibility and therefore has a lot of
limitations. It does take a lot of parameters to give the caller some
flexibility, but it is not enough.</p>
<p>Now the question before us is,
“If fork() + exec() is so much more powerful, then why have,
or use the posix_spawn() routine?” The answer to that is, that
<a href="#note3">fork() and exec()</a> are UNIX system routines.
They are not present in operating systems that are not a derivative of UNIX.
Eg- Windows implements a family of spawn functions.<br/>
There is another issue with fork() (not exec() ), which in reality is one
of the biggest reasons behind the growth of posix_spawn(). The outline of
the issue is that, creating child processes in multi-threaded programs is a
whole another ball game altogether.</p>
<p>Concurrency is one of those disciplines in operating systems where the
order in which the cards are going to unravel is not always how you expect
them to. Multi-threading in a program is a way to do different and independent tasks of a
program simultaneously, to save time. No matter how jazzy or intelligent the above statement looks,
multi-threaded programs require an eagle’s eye as they can often have a lot
of holes. Though the “tasks” are different and independent, they often
share a few common attributes. When these different tasks due to
concurrency start running in parallel, a data race begins to access those
shared attributes. To not wreak havoc, there are mechanisms through which,
when modifying/accessing these common attributes (Critical Section) we can
provide a sort of mutual exclusion (locks/conditional variables) - only
letting one of the processes modify the shared attribute at a time. Here
when things are already so intricate due to multithreading, and to top it
off, we start creating child processes. Complications are bound to arise.
When one of the threads from the multi-threaded program calls fork() to
create a child process, fork() does clone everything (variables, their
states, functions, etc) but it fails to clone other threads (though this is not
required at all times).</p>
<p>The child process now knows only about that one thread which called fork().
But all the other attributes of the child that were inherited from
the parent (locks, mutexes) are set from the parent’s address space
(considering multiple threads). So there is no way for the child process to
know which attributes conform to which parts of the parent. Also, those
mechanisms that we used to provide mutual exclusion, like locks and
conditional variables, need to be reset. This reset step is essential
in letting the parent access it’s attributes. Failing this reset can cause deadlocks.
To put it simply, you can see how difficult things
have become all of a sudden. The posix_spawn() call is free from these
limitations of fork() encountered in multi-threaded programs. However, as
mentioned by me earlier, there needs to be enough rope to meet all the
requirements before posix_spawn() does the implicit exec().</p>
<h2>About my Project</h2>
<p>Hi, I am Piyush Sachdeva and I am going to start a project which will focus
on relaxing one limitation of posix_spawn - changing the current directory
of the child process, before the said call to exec() is made. This is not
going to restrict it to the parent’s current working directory. Just
passing the new directory as one of the parameters will do the trick.
Resolving all the impediments would definitely be marvelous. Alas! That is
not possible. Every attempt to resolve even a single hindrance can create
plenty of new challenges.</p>
<p>As already mentioned by me, posix_spawn() is a POSIX standard. Hence the
effect of my project will probably be reflected in the next POSIX release.
I came across this project through Google Summer of Code 2021. It was being
offered by The NetBSD Foundation Inc. However, as the slots for
Google Summer of Code were numbered, my project didn’t make the selection.
Nevertheless, the Core Team at The NetBSD Foundation offered me to work on
the project and even extended a handsome stipend. I will forever be
grateful to The NetBSD Foundation for this opportunity.</p>
<h2>Notes</h2>
<ul>
<li>init, systemd & launchd are system daemon processes. init is the
historical process present since System III UNIX. systemd was the replacement
for the authentic init which was written for the Linux kernel.
launchd is the MacOS alternative of init/systemd.</li>
<li>This is taken from Operating Systems: The Three Easy Pieces book by
Andrea C. Arpaci-Dusseau and Remzi H. Arpaci-Dusseau.</li>
<li>UNIX is the original AT&T UNIX operating system developed at the Bell
Labs research center, headed by Ken Thompson and Dennis Ritchie.</li>
</ul>
<h2>References</h2>
<ol>
<li> <a name="note1"> Operating Systems: Three Easy Pieces by Andrea C. Arpaci-Dusseau and
Remzi H. Arpaci-Dusseau.</a></li>
<li> <a name="note2"> Advanced Programming in the UNIX Environment by W. Richard Stevens
and Stephen A. Rago.</a></li>
<li> <a name="note3">UNIX and Linux System Administration Handbook by Evi Nemeth, Garth
Synder, Trent R. Hein, Ben Whaley and Dan Mackin.</a></li>
</ol>https://blog.netbsd.org/tnf/entry/public_netbsd_irc_channels_movedPublic NetBSD IRC chat channels moved to LiberaNia Alarie2021-05-30T18:23:29+00:002021-05-30T18:24:49+00:00<pHi everyone,</p>
<p>Due to the unfortunate situation regarding changes in administration on
freenode.net, and the resulting chaos, we have decided to move the public
NetBSD IRC channels from freenode to irc.libera.chat.</p>
<p>This includes:</p>
<ul>
<li><a href="https://web.libera.chat/#NetBSD">#NetBSD</a> - general discussion</li>
<li><a href="https://web.libera.chat/#netbsd-code">#netbsd-code</a> - development discussion</li>
<li><a href="https://web.libera.chat/#pkgsrc">#pkgsrc</a> - pkgsrc (primarily development) discussion</li>
</ul>
<p>You can find information on connecting to Libera at <a href="https://libera.chat">https://libera.chat/</a></p><p>Hi everyone,</p>
<p>Due to the unfortunate situation regarding changes in administration on
freenode.net, and the resulting chaos, we have decided to move the public
NetBSD IRC chat channels from freenode to irc.libera.chat.</p>
<p>This includes:</p>
<ul>
<li><a href="https://web.libera.chat/#NetBSD">#NetBSD</a> - general discussion</li>
<li><a href="https://web.libera.chat/#netbsd-code">#netbsd-code</a> - development discussion</li>
<li><a href="https://web.libera.chat/#pkgsrc">#pkgsrc</a> - pkgsrc (primarily development) discussion</li>
</ul>
<p>You can find information on connecting to Libera at <a href="https://libera.chat">https://libera.chat/</a></p>https://blog.netbsd.org/tnf/entry/netbsd_9_2_releasedNetBSD 9.2 releasedNia Alarie2021-05-17T14:06:03+00:002021-05-17T14:37:04+00:00<p> The NetBSD Project is pleased to announce NetBSD 9.2 "Nakatomi Socrates", the second update of the NetBSD 9 release branch.</p>
<p>As well as the usual bug, stability, and security fixes, this release includes: support for exporting ZFS filesystems over NFS, various updates to the bozotic HTTP daemon, improvements to ARM 32-bit and Linux compatibility, <code>fread()</code> performance improvements, support for the TP-Link TL-WN821N V6 wireless adapter, support for the Allwinner H5 cryptographic accelerator, Pinebook Pro display brightness fixes, new defaults for <code>kern.maxfiles</code>, and accessibility improvements for the default window manager configuration.</p>
<p><a href="//www.NetBSD.org/releases/formal-9/NetBSD-9.2.html">Release notes and download links for NetBSD 9.2</a></p><p> The NetBSD Project is pleased to announce NetBSD 9.2 "Nakatomi Socrates", the second update of the NetBSD 9 release branch.</p>
<p>As well as the usual bug, stability, and security fixes, this release includes: support for exporting ZFS filesystems over NFS, various updates to the bozotic HTTP daemon, improvements to ARM 32-bit and Linux compatibility, <code>fread()</code> performance improvements, support for the TP-Link TL-WN821N V6 wireless adapter, support for the Allwinner H5 cryptographic accelerator, Pinebook Pro display brightness fixes, new defaults for <code>kern.maxfiles</code>, and accessibility improvements for the default window manager configuration.</p>
<p><a href="//www.NetBSD.org/releases/formal-9/NetBSD-9.2.html">Release notes and download links for NetBSD 9.2</a></p>https://blog.netbsd.org/tnf/entry/aiomixer_x_open_curses_andaiomixer, X/Open Curses and ncurses, and other newsNia Alarie2021-05-12T13:35:14+00:002021-05-12T14:48:40+00:00<p>aiomixer, X/Open Curses and ncurses, and other news</p><p>
aiomixer is an application that I've been maintaining outside of NetBSD for a
few years. It was available as a package, and was a "graphical" (curses,
terminal-based) mixer for NetBSD's audio API, inspired by programs like alsamixer.
For some time I've thought that it should be integrated into the NetBSD
base system - it's small and simple, very useful, and many developers
and users had it installed (some told me that they would install it on all
of their machines that needed audio output).
For my particular use case, as well as my NetBSD laptop, I have some small
NetBSD machines around the house plugged into speakers that I play music from.
Sometimes I like to SSH into them to adjust the playback volume, and it's often
easier to do visually than with
<a href="https://man.NetBSD.org/mixerctl.1">mixerctl(1)</a>.
</p>
<p>
However, there was one problem: when I first wrote aiomixer 2 years ago,
I was intimidated by the curses API, so opted to use the
<a href="https://invisible-island.net/cdk/">Curses Development Kit</a>
instead.
This turned out to be a mistake, as not only was CDK inflexible for an
application like aiomixer, it introduced a hard dependency on ncurses.
</p>
<h2>X/Open Curses and ncurses</h2>
<p>
Many people think ncurses is the canonical way to develop terminal-based
applications for Unix, but it's actually an implementation of the
<a href="https://pubs.opengroup.org/onlinepubs/7908799/xcurses/curses.h.html">X/Open Curses specification</a>.
There's a few other Curses implementations:
</p>
<ul>
<li><a href="https://man.netbsd.org/curses.3">NetBSD libcurses</a></li>
<li><a href="https://docs.oracle.com/cd/E36784_01/html/E36880/curses-3curses.html">Solaris libcurses</a></li>
<li><a href="https://en.wikipedia.org/wiki/PDCurses">PDCurses</a>, used on Windows</li>
</ul>
<p>
NetBSD curses is descended from the original BSD curses, but contains
many useful extensions from ncurses as well. We use it all over the
base system, and for most packages in pkgsrc.
It's also been
<a href="https://github.com/sabotage-linux/netbsd-curses">ported to other operating systems</a>,
including Linux.
As far as I'm aware, NetBSD is one of the last operating systems left
that doesn't primarily depend on ncurses.
</p>
<p>
There's one crucial incompatibility, however: ncurses exposes its
internal data structures, NetBSD libcurses keeps them opaque.
Since CDK development is very tied to ncurses development (they have
the same maintainer), CDK peeks into those structures, and can't
be used with NetBSD libcurses.
There are also a few places where ncurses breaks with X/Open Curses,
like
<a href="https://github.com/irssi/irssi/pull/1305">this case I recently fixed in irssi</a>.
</p>
<h2>Rewriting aiomixer</h2>
<p>
I was able to rewrite aiomixer in a few days using only my free time
and NetBSD libcurses. It's now been imported to the base system.
It was a good lesson in why Curses isn't actually that intimidating -
while there are many functions, they're mostly variations on the
same thing. Using Curses directly resulted in a much lighter and
more usable application, and provided a much better fit for the
types of widgets I needed.
</p>
<p>
Many people also provided testing, and I learned a lot about
how different terminal attributes should be used in the process.
NetBSD is probably one of the few communities where you'll get
easy and direct feedback on how to not only make your software
work well in a variety of terminal emulators, but also old school
hardware terminals. During development, I was also able to find
a strange bug in the curses library's window resizing function.
</p>
<p>
The API support was also improved, and the new version of aiomixer
should work better with a wider variety of sound hardware drivers.
</p>
<a href="https://cdn.netbsd.org/pub/NetBSD/misc/nia/aiomixer2.png"><img src="https://cdn.netbsd.org/pub/NetBSD/misc/nia/aiomixer2.png" width=400" /></a>
<h2>Other happenings</h2>
<p>
Since I'm done plugging my own work, I thought I might talk
a bit about some other recent changes to CURRENT.
</p>
<ul>
<li>Most ports of NetBSD now build with GCC 10, thanks to work by mrg.
The new version of GCC introduced many new compiler warnings. By
default, since NetBSD is compiled with a fixed toolchain version,
we use <code>-Werror</code>. Many minor warnings and actual-bugs
were uncovered and fixed with the new compiler.</li>
<li>On the ARM front, support for the Allwiner V3S system-on-a-chip
was introduced thanks to work by Rui-Xiang Guo. This is an
older model Allwinner core, primarily used on small embedded
devices. It's of likely interest to hardware hackers because it
comes in an easily soldered package. A development board is
available, the Lichee Pi Zero. Also in the Allwinner world,
support for the H3 SoC (including the NanoPi R1) was added
to the
<a href="https://man.NetBSD.org/sun8icrypto.4">sun8icrypto(4)</a>
driver by bad.</li>
<li>Support for RISC-V is progressing, including an UEFI
bootloader for 64-bit systems, and an in-kernel disassembler.
Some NetBSD developers have recently obtained Beagle-V development
boards.</li>
<li>On the SPARC64 front, support for sun4v is progressing thanks
to work by palle. The
sun4v architecture includes most newer SPARC servers that are
based on the
<a href="https://en.wikipedia.org/wiki/Oracle_VM_Server_for_SPARC">Logical Domains</a>
architecture - virtualization is
implemented at the hardware/firmware level, and operating systems
get an abstracted view of the underlying hardware. With other
operating systems are discussing removing support for SPARC64, there's
an interest among NetBSD developers in adding and maintaining
support for this very interesting hardware from Oracle, Fujitsu,
and Sun in an open source operating system, not just Oracle
Solaris.</li>
<li>A kernel-wide audit and rework of the auto-configuration
APIs was completed by thorpej.</li>
<li>Various new additions and fixes have been made to the
networking stack's
<a href="https://man.NetBSD.org/pppoe.4">PPP over Ethernet</a>
support by yamaguchi.</li>
<li>A new API was introduced by macallan that allows adding
a <code>-l</code> option to the
<a href="https://man.NetBSD.org/wsfontload.8">wsfontload(8)</a>
command, allowing easy viewing of the tty fonts currently
loaded into the kernel.</li>
<li>... OK, I'm not done plugging my own work: I recently
wrote new documentation on
using
<a href="https://www.netbsd.org/docs/guide/en/chap-virt.html">NetBSD for virtualization</a>,
<a href="https://www.netbsd.org/docs/guide/en/chap-power.html">Power Management</a>,
and rewrote the
<a href="https://www.netbsd.org/docs/guide/en/index.html">NetBSD Guide</a>'s sections on
<a href="https://www.netbsd.org/docs/guide/en/chap-net-practice.html">Networking in Practice</a>
and
<a href="https://www.netbsd.org/docs/guide/en/chap-audio.html">Audio</a>.
I also recently added support for the
<a href="https://en.wikipedia.org/wiki/Neo_(keyboard_layout)">Neo 2 keyboard layout</a>
to NetBSD's console system - Neo 2 is a Dvorak-like
optimized layout for German and other languages based on
multiple layers for alphabetical characters, navigation,
and symbols.</li>
</ul>
https://blog.netbsd.org/tnf/entry/gsoc_reports_make_system_31GSoC Reports: Make system(3), popen(3) and popenve(3) use posix_spawn(3) internally (Final report)Nikita Ronja Gillmann2021-03-30T11:11:15+00:002022-02-24T10:36:27+00:00<i>This report was prepared by Nikita Ronja Gillmann as a part of Google Summer of Code 2020</i>
<p>This is my second and final report for the Google Summer of Code project I am working on for NetBSD.</p>
<p>
My code can be found at <a href="https://github.com/teknokatze/src/">github.com/teknokatze/src</a> in the gsoc2020 branch, at the time of writing some of it is still missing. The test facilities and logs can be found in <a href="https://github.com/teknokatze/gsoc2020/">github.com/teknokatze/gsoc2020</a>. A diff can be found at <a href="https://github.com/NetBSD/src/compare/trunk...teknokatze:gsoc2020">github</a> which will later be split into several patches before it is sent to QA for merging.
</p>
<p>
The initial and defined goal of this project was to <a href="https://blog.netbsd.org/tnf/entry/gsoc_reports_make_system_3">make system(3) and popen(3) use posix_spawn(3) internally</a>, which had been completed in June.
For the second part I was given the task to replace fork+exec calls in our standard shell (sh) in one scenario. Similar to the previous goal we determine through implementation if the initial motivation, to get performance improvements, is correct otherwise we collect metrics for why posix_spawn() in this case should be avoided. This second part meant in practice that I had to add and change code in the kernel, add a new public libc function, and understand shell internals.
</p>
<i>This report was prepared by Nikita Ronja Gillmann as a part of Google Summer of Code 2020</i>
<p>This is my second and final report for the Google Summer of Code project I am working on for NetBSD.</p>
<p>
My code can be found at <a href="https://github.com/nikicoon/src/">github.com//src</a> in the <i>gsoc2020</i> branch, at the time of writing some of it is still missing.
The test facilities and logs can be found in <a href="https://github.com/nikicoon/gsoc2020/">github.com/nikicoon/gsoc2020</a>.
A diff can be found at <a href="https://github.com/NetBSD/src/compare/trunk...nikicoon:gsoc2020">github</a> which will later be split into several patches before it is sent to QA for merging.
</p>
<p>
The initial and defined goal of this project was to <a href="https://blog.netbsd.org/tnf/entry/gsoc_reports_make_system_3">make system(3) and popen(3) use posix_spawn(3) internally</a>, which had been completed in June.
For the second part I was given the task to replace fork+exec calls in our standard shell (sh) in one scenario. Similar to the previous goal we determined through implementation if the initial motivation, to get performance improvements, is correct otherwise we collect metrics for why posix_spawn() in this case should be avoided.
This second part meant in practice that I had to add and change code in the kernel, add a new public libc function, and understand shell internals.
</p>
<h2>Summary of part 1</h2>
<p>
Prior work: In GSoC 2012 Charles Zhang <a href="https://blog.netbsd.org/tnf/entry/posix_spawn_syscall_added">added the posix_spawn syscall</a> which according to <a href="https://web.archive.org/web/20150905210045/http://netbsd-soc.sourceforge.net/projects/posix_spawn/">its SF repository</a> at the time (maybe even now, I have not looked very much into comparing all other systems and libcs + kernels) is an in-kernel implementation of posix_spawn which provides performance benefits compared to FreeBSD and other systems which had a userspace implementation (in 2012).
</p>
<p>
After 1 week of reading POSIX and writing code, 2 weeks of coding and another 1.5 weeks of bugfixes I have successfully implemented posix_spawn in usage in system(3) and popen(3) internally.</p><p>The biggest challenge for me was to understand POSIX, to read the standard. I am used to reading more formal books, but I can't remember working with POSIX Standard directly before.
</p>
<h3>system(3)</h3>
<p>system(3) was changed to use posix_spawnattr_ (where we used sigaction before) and posix_spawn (which replaced execve + vfork calls).</p>
<h3>popen(3) and popenve(3)</h3>
<p>
Since the popen and popenve implementation in NetBSD's libc use a couple of shared helper functions, I was able to change both functions while keeping the majority of the changes focused on (some of ) the helper functions (pdes_child).
</p>
<p>
pdes_child, an internal function in popen.c, now takes one more argument (<i>const char *cmd</i>) for the command to pass to posix_spawn which is called in pdes_child.
</p>
<p>
On a high level what happens in pdes_child() and popen is that we first lock the pidlist_mutex. Then we create a file file action list for all concurrent popen() / popenve() instances and the side of the pipe not necessary, and the move to stdin/stdout. We unlock the pidlist_mutex. Finally we return the list and destroy.
</p>
<p>
In the new version of this helper function which now handles the majority of what popen/popenve did, we have to initialize a file_actions object which by default contains no file actions for posix_spawn() to perform. Since we have to have error handling and a common return value for the functions calling pdes_child() and deconstruction, we make use of <i>goto</i> in some parts of this function.
</p>
<p>
The close() and dup2() actions now get replaced by corresponding file_actions syscalls, they are used to specify a series of actions to be performed by a posix_spawn operation.</p>
<p>After this series of actions, we call _readlockenv(), and call posix_spawn with the file_action object and the other arguments to be executed. If it succeeds, we return the pid of the child to popen, otherwise we return -1, in both cases we destroy the file_action object before we proceed.</p>
<p>
In popen and popenve our code has been reduced to the <i>pid == -1</i> branch, everything else happens in pdes_child() now.
</p>
<p>
After readlockenv we call pdes_child and pass it the command to execute in the posix_spawn'd child process; if pdes_child returns -1 we run the old error handling code. Likewise for popenve.</p>
<p>
The outcome of the first part is, that thanks to how we implement posix_spawn in NetBSD we reduced the syscalls being made for popen and system.
A full test with proper timing should indicate this, my reading was based on comparing old and new logs with ktrace and kdump.
</p>
<h2>sh, posix_spawn actions, libc and kernel - Part 2</h2>
<h3>Motivation</h3>
<p>
The main goal of part 2 of this project was to change sh(1) to
determine which simple cases of (v)fork + exec I could replace, and to
replace them with posix_spawn where it makes sense.</p>
<p>
fork needs to create a new address space by cloning the address space,
or in the case of vfork update at least some reference counts.
posix_spawn can avoid most of this as it creates the new address space from scratch.
</p>
<h3>Issues</h3>
<p>
The current posix_spawn as defined in POSIX has no good way to do tcsetpgrp, and we found
that <a href="https://github.com/fish-shell/fish-shell/issues/360">fish just avoids posix_spawn for foreground processes</a>.
</p>
<h3>Implementation</h3>
<p>
Since, roughly speaking, modern BSDs handle "#!" execution in the kernel (probably since before the 1990s, systems which didn't handle this started to disappear most likely in the mid to late 90s), our main concern so far was in the evalcmd function the default cmd switch case ('NORMALCMD').</p><p>After adjusting the function to use posix_spawn, I hit an issue in the execution of the curses application htop where htop would run but input would not be accepted properly (keysequences pressed are visible).
In pre-posix_spawn sh, every subprocess that sh (v)forked runs forkchild() to set up the subprocess's environment.
With posix_spawn, we need to arrange posix_spawn actions to do the same thing.
</p>
<p>
The intermediate resolution was to switch FORK_FG processes to fork+exec again. For foreground processes with job control we're in an interactive shell, so the performance benefit is small enough in this case to be negligible. It's really only for shell scripts that it matters.</p><p>Next I implemented a posix_spawn file_action, with the prototype
</p>
<pre>
<code>int posix_spawn_file_actions_addtcsetpgrp(posix_spawn_file_actions_t *fa, int fildes)</code>
</pre>
<p>
The kernel part of this was implemented inline in sys/kern/kern_exec.c, in the function handle_posix_spawn_file_actions() for the new case 'FAE_TCSETPGRP'.</p><p>The new version of the code is still in testing and debugging phase and at the time of writing not included in my repository (it will be published after Google Summer of Code when I'm done moving).
</p>
<h3>Future steps</h3>
<h4>posix_spawnp kernel implementation</h4>
<p>
According to a conversation with kre@, the posix_spawnp() implementation we have is just itterating over $PATH calling posix_spawn until it succeeds.
For some changes we might want a kernel implementation of posix_spawnp(), as the path search is supposed to happen in the kernel so the file actions are only ever run once:
</p>
<pre>
<code>
some of the file actions may be "execute once only",
they can't be repeated (eg: handling "set -C; cat foo >file" - file
can only be created once, that has to happen before the exec (as the fd
needs to be made stdout), and then the exec part of posix_spawn is
attempted - if that fails, when it can't find "cat" in $HOME/bin (or
whatever is first in $PATH) and we move along to the next entry (maybe /bin
doesn't really matter) then the repeated file action fails, as file now
exists, and "set -C" demands that we cannot open an already existing file
(noclobber mode). It would be nice for this if there were "clean up on
failure" actions, but that is likely to be very difficult to get right,
and each would need to be attached to a file action, so only those which
had been performed would result in cleanup attempts.
</code>
</pre>
<h4>Replacing all of fork+exec in sh</h4>
<p>
Ideally we could replace all of (v)fork + exec with posix_spawn.
According to my mentors there is pmap synchronisation as an impact of
constructing the vm space from scratch with (v)fork.
Less IPIs (inter-processor interrupts) matter for small processes too.
</p>
<h4>posix_spawn_file_action_ioctl</h4>
<p>
Future directions could involve a posix_spawn action for an arbitrary ioctl.
</p>
<h2>Thanks</h2>
<p>
My thanks go to fellow NetBSD developers for answering questions, most recently kre@ for sharing invaluable sh knowledge, Riastradh and Jörg as the mentors I've interacted with most of the time and for their often in-depth explanations as well as allowing me to ask questions I sometimes felt were too obvious. My friends, for sticking up with my "weird" working schedule. Lastly would like to thank the Google Summer of Code program for continuing through the ongoing pandemic and giving students the chance to work on projects full-time.
</p>https://blog.netbsd.org/tnf/entry/hitting_donation_milestone_financial_reportHitting donation milestone, financial report for 2020Maya Rashish2021-03-29T09:17:29+00:002021-03-29T10:32:22+00:00<p>
We nearly hit our donation milestone set after the release of 9.0 of $50,000.<br />
These donations have enabled us to fund significant paid work on NetBSD in 2020.
</p><p>
We nearly hit our 2020 donation milestone set after the release of 9.0 of $50,000.
These donations have enabled us to fund significant work on NetBSD in 2020 such as:
<ul>
<li>an aarch64 package build server, <a href="http://victory.netbsd.org/pkgsrc/packages/reports/HEAD/evbarm64-9.0/">victory.netbsd.org</a>. Thanks to Western Washington University for hosting this machine.</li>
<li><a href="https://blog.netbsd.org/tnf/entry/wifi_renewal_restarted">Modernizing wi-fi network stack</a> and release engineering work by Martin Husemann</li>
<li><a href="https://blog.netbsd.org/tnf/entry/lldb_work_concluded">LLDB support</a> by Michał Górny</li>
<li><a href="https://blog.netbsd.org/tnf/entry/accomplishment_of_porting_ptrace_2">ptrace and GDB improvements</a> by Kamil Rytarowski</li>
</ul>
</p>
<p>
If you are interested in seeing more work like this, please consider donating <a href="https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=paypal%40NetBSD.org¤cy_code=USD&source=url">via PayPal</a> or <a href="https://github.com/sponsors/NetBSD">GitHub sponsors</a>.
</p>
<p>
The <a href="https://www.netbsd.org/foundation/reports/financial/2020.html">financial report for 2020</a> is also available.
</p>
<p>
Note: We realize that this data is inconsistent with the website indicator of donations. This is due to the fact the website is updated manually in an error-prone process as the donations are processed. The financial report (just completed) is prepared separately using <a href="https://www.ledger-cli.org/">ledger</a>.
</p>https://blog.netbsd.org/tnf/entry/allen_k_briggs_memorial_scholarshipAllen K. Briggs Memorial ScholarshipLeonardo Taccari2020-12-21T11:37:59+00:002020-12-21T11:37:59+00:00<p>
Allen Briggs was one of the earliest members of the NetBSD community,
pursuing his interest in macBSD, and moving to become a NetBSD
developer when the two projects merged. Allen was known for his
quiet and relaxed manner, and always brought a keen wisdom with
him; allied with his acute technical expertise, he was one of the
most valued members of the NetBSD community.
</p>
<p>
He was a revered member of the NetBSD core team, and keenly involved
in many aspects of its application; from working on ARM chips to
helping architect many projects, Allen was renowned for his expertise.
He was a distinguished engineer at Apple, and used his NetBSD
expertise there to bring products to market.
</p>
<p>
Allen lived in Blacksburg Virginia with his wife and twin boys and
was active with various community volunteer groups. His family
touched the families of many other NetBSD developers and those
friendships have endured beyond his passing.
</p>
<p>
We have received the following from Allen's family and decided to
share it with the NetBSD community. If you can, we would ask you
to consider contributing to his Memorial Scholarship.
</p>
<p>
<a href="https://www.ncssm.edu/donate/distance-education/allen-k-briggs-88-memorial-scholarship">https://www.ncssm.edu/donate/distance-education/allen-k-briggs-88-memorial-scholarship</a>
</p>
<p>
The Allen K. Briggs Memorial Scholarship is an endowment to provide
scholarships in perpetuity for summer programs at the North Carolina
School of Science & Math, which Allen considered to be a place that
fundamentally shaped him as a person. We would love to invite
Allen's friends and colleagues from the BSD community to donate to
this cause so that we can provide more scholarships to students
with financial need each year. We are approximately halfway to our
goal of $50K with aspirations to exceed that target and fund
additional scholarships.
</p>
<p>
Two quick notes on donating: <strong>Important!</strong> When donating, you must
select "Allen K. Briggs Memorial Scholarship" under designation
for the donation to be routed to the scholarship If you have the
option to use employer matching (i.e., donating to NCSSM through
an employer portal to secure a match from your employer), please
email the NCSSM Foundation's Director of Development, April Horton
(<code>april.horton@ncssm.edu</code>), after donating to let her know you want
your gift and employer match to go to the Allen K. Briggs Memorial
Scholarship Thanks in advance for your help. I'd be happy to answer
any questions you or any others have about this.
</p>
https://blog.netbsd.org/tnf/entry/netbsd_9_1_releasedNetBSD 9.1 releasedmartin2020-10-21T04:19:23+00:002020-10-21T04:19:23+00:00<p>NetBSD 9.1, the first maintenance update for the NetBSD 9 branch, has been released</p><p>After a small delay<super><a href="#footnote_delay">*</a></super>, the NetBSD Project is pleased to announce <a href="https://mail-index.NetBSD.org/netbsd-announce/2020/10/21/msg000321.html">NetBSD 9.1</a>, the first feature and stability maintenance release of the netbsd-9 stable branch.</p>
<p>
The new release features (among various other changes) many bug fixes,
a few performance enhancements, stability improvements for ZFS and LFS
and support for USB security keys in a mode easily usable in Firefox
and other applications.</p>
<p>
For more details and instructions see the <a href="https://www.netbsd.org/releases/formal-9/NetBSD-9.1.html">9.1 announcement</a>.</p>
<p>
Get NetBSD 9.1 from our <a href="https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.1/">CDN</a> (provided by <a href="http://www.fastly.com/">fastly</a>) or one of the ftp mirrors.</p>
<p>
Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, and other services may be found at <a href="https://www.NetBSD.org/mirrors/">https://www.NetBSD.org/mirrors/</a>.</p>
<p style="font-size: 0.8em" name="footnote_delay">* for the delay: let us say there was a minor hickup and we took the opportunity to provide up to date timezone files for NetBSD users in Fiji.</p>https://blog.netbsd.org/tnf/entry/google_summer_of_code_20202Google Summer of Code 2020: [Final Report] Enhancing Syzkaller support for NetBSDKamil Rytarowski2020-10-19T13:20:28+00:002020-10-19T13:20:28+00:00This report was written by Ayushu Sharma as part of Google Summer of Code 2020.
<p>This post is a follow up of the <a href="https://blog.netbsd.org/tnf/entry/gsoc_reports_enhancing_syzkaller_support">first report</a> and <a href="https://blog.netbsd.org/tnf/entry/gsoc_reports_enhancing_syzkaller_support1">second report</a>. Post summarizes the work done during the third and final coding period for the Google Summer of Code (GSoc’20) project - <a href="https://wiki.netbsd.org/projects/project/syzkaller/">Enhance Syzkaller support for NetBSD</a></p>This report was written by Ayushu Sharma as part of Google Summer of Code 2020.
<p>This post is a follow up of the <a href="https://blog.netbsd.org/tnf/entry/gsoc_reports_enhancing_syzkaller_support">first report</a> and <a href="https://blog.netbsd.org/tnf/entry/gsoc_reports_enhancing_syzkaller_support1">second report</a>. Post summarizes the work done during the third and final coding period for the Google Summer of Code (GSoc’20) project - <a href="https://wiki.netbsd.org/projects/project/syzkaller/">Enhance Syzkaller support for NetBSD</a></p>
<h2>Sys2syz</h2>
<p>Sys2syz would give an extra edge to Syzkaller for NetBSD. It has a potential of efficiently automating the conversion of syscall definitions to syzkaller’s grammar. This can aid in increasing the number of syscalls covered by Syzkaller significantly with the minimum possibility of manual errors. Let’s delve into its internals.</p>
<h2>A peek into Syz2syz Internals</h2>
<p>This tool parses the source code of device drivers present in C to a format which is compatible with grammar customized for syzkaller. Here, we try to cull the details of the target device by compiling, and then collocate the details with our python code. For further details about proposed design for the tool, refer to <a href="https://blog.netbsd.org/tnf/entry/gsoc_reports_enhancing_syzkaller_support1">previous post</a>.<p>
<p>Python code follows 4 major steps:<p>
<ul>
<li><a href="https://github.com/ais2397/sys2syz/blob/master/core/Extractor.py"><b>Extractor.py</b></a> - Extraction of all ioctl commands of a given device driver along with their arguments from the header files.</li>
<li><a href="https://github.com/ais2397/sys2syz/blob/master/core/Bear.py"><b>Bear.py</b></a> - Preprocessing of the device driver's files using compile_commands.json generated during the setup of tool using Bear.</li>
<li><a href="https://github.com/ais2397/sys2syz/blob/master/core/C2xml.py"><b>C2xml.py</b></a> - XML files are generated by running c2xml on preprocessed device files. This eases the process of fetching the information related to arguments of commands</li>
<li><a href="https://github.com/ais2397/sys2syz/blob/master/core/Description.py"><b>Description.py</b></a> - Generates descriptions for the ioctl commands and their arguments (builtin-types, arrays, pointers, structures and unions) using the XML files.</li>
</ul>
<h3>Extraction:</h3>
<p>This step involves fetching the possible ioctl commands for the target device driver and getting the files which have to be included in our dev_target.txt file. We have already seen all the commands for device drivers are defined in a specific way. These commands defined in the header files need to be grepped along with the major details, regex comes in as a rescue for this</p>
<pre>
<code>
io = re.compile("#define\s+(.*)\s+_IO\((.*)\).*")
iow = re.compile("#define\s+(.*)\s+_IOW\((.*),\s+(.*),\s+(.*)\).*")
ior = re.compile("#define\s+(.*)\s+_IOR\((.*),\s+(.*),\s+(.*)\).*")
iowr = re.compile("#define\s+(.*)\s+_IOWR\((.*),\s+(.*),\s+(.*)\).*")
</code>
</pre>
<p>
Code scans through all the header files present in the target device folder and extracts all the commands along with their details using compiled regex expressions. Details include the direction of buffer(null, in, out, inout) based on the types of Ioctl calls(_IO, _IOR, _IOW, _IOWR) and the argument of the call. These are stored in a file named ioctl_commands.txt at location out/<target_name>.
Example output:
<pre>
<code>
out, I2C_IOCTL_EXEC, i2c_ioctl_exec_t
</code>
</pre>
<h3>Preprocessing:</h3>
<p>Preprocessing is required for getting XML files, about which we would look in the next step. Bear plays a major role when it comes to preprocessing C files. It records the commands executed for building the target device driver. This step is performed when setup.sh script is executed.</p>
<p>Extracted commands are modified with the help of parse_commands() function to include ‘-E’ and ‘-fdirectives’ flags and give it a new output location. Commands extracted by this function are then used by the compile_target function which filters out the unnecessary flags and generates preprocessed files in our output directory.</p>
<h3>Generating XML files</h3>
<p>Run C2xml on the preprocessed files to fetch XML files which stores source code in a tree-like structure, making it easier to collect all the information related to each and every element of structures, unions etc. For eg: </p>
<pre>
<code>
<symbol type="struct" id="_5970" file="am2315.i" start-line="13240" start-col="16" end-line="13244" end-col="11" bit-size="96" alignment="4" offset="0">
<symbol type="node" id="_5971" ident="ipending" file="am2315.i" start-line="13241" start-col="33" end-line="13241" end-col="41" bit-size="32" alignment="4" offset="0" base-type-builtin="unsigned int"/<
<symbol type="node" id="_5972" ident="ilevel" file="am2315.i" start-line="13242" start-col="33" end-line="13242" end-col="39" bit-size="32" alignment="4" offset="4" base-type-builtin="int"/>
<symbol type="node" id="_5973" ident="imasked" file="am2315.i" start-line="13243" start-col="33" end-line="13243" end-col="40" bit-size="32" alignment="4" offset="8" base-type-builtin="unsigned int"/>
</symbol>
<symbol type="pointer" id="_5976" file="am2315.i" start-line="13249" start-col="14" end-line="13249" end-col="25" bit-size="64" alignment="8" offset="0" base-type-builtin="void"/>
<symbol type="array" id="_5978" file="am2315.i" start-line="13250" start-col="33" end-line="13250" end-col="39" bit-size="288" alignment="4" offset="0" base-type-builtin="unsigned int" array-size="9"/>
</code>
</pre>
<p>We would further see how attributes like - idents, id, type, base-type-builtin etc conveniently helps us to analyze code and generate descriptions in a trouble-free manner .
</p>
<h3>Descriptions.py</h3>
<p>Final part, which offers a txt file storing all the required descriptions as its output. Here, information from the xml files and ioctl_commands.txt are combined together to generate descriptions of ioctl commands and their arguments.</p>
<p>Xml files for the given target device are parsed to form trees, </p>
<pre>
<code>
for file in (os.listdir(self.target)):
tree = ET.parse(self.target+file)
self.trees.append(tree)
</code>
</pre>
<p>We then traverse through these trees to search for the arguments of a particular ioctl command (particularly _IOR, _IOW, _IOWR commands) by the name of the argument. Once an element with the same value for ident attribute is found, attributes of the element are further examined to get its type. Possible types for these arguments are - struct, union, enum, function, array, pointer, macro and node. Using the type information we determine the way to define the element in accordance with syzkaller’s grammar syntax.
</p>
<p>Building structs and unions involves defining their elements too, XML makes it easier. Program analyses each and every element which is a child of the root (struct/union) and generates its definitions. A dictionary helps in tracking the structs/unions which have been already built. Later, the dictionary is used to pretty print all the structs and union in the output file. Here is a code snippet which depicts the approach</p>
<pre>
<code>
name = child.get("ident")
if name not in self.structs_and_unions.keys():
elements = {}
for element in child:
elem_type = self.get_type(element)
elem_ident = element.get("ident")
if elem_type == None:
elem_type = element.get("type")
elements[element.get("ident")] = elem_type
element_str = ""
for element in elements:
element_str += element + "\t" + elements[element] + "\n"
self.structs_and_unions[name] = " {\n" + element_str + "}\n"
return str(name)
</code>
</pre>
<p>Task of creating descriptions for arrays is made simpler due to the attribute - `array-size`.
When it comes to dealing with pointers, syzkaller needs the user to fill in the direction of the pointer. This has already been taken care of while analyzing the ioctl commands in Extractor.py. The second argument with in/out/inout as its possible value depends on ‘fun’ macros - _IOR, _IOW, _IOWR respectively. </p>
<p>There is another category named as nodes which can be distinguished using the base-type-builtin and base-type attributes.
</p>
<h2>Result</h2>
<p>Once the setup script for sys2syz is executed, sys2syz can be used for a certain target_device file by executing the python wrapper script (<a href="https://github.com/ais2397/sys2syz/blob/master/sys2syz.py">sys2syz.py</a>) with :</p>
<pre>
<code>#bin/sh
python sys2syz.py -t <absolute_path_to_device_driver_source> -c compile_commands.json -v
</code>
</pre>
<p>This would generate a dev_<device_driver>.txt file in the out directory. An example description file autogenerated by sys2syz for i2c device driver.</p>
<pre>
<code>
#Autogenerated by sys2syz
include <i2c_io.h>
resource fd_i2c[fd]
syz_open_dev$I2C(dev ptr[in, string["/dev/i2c"]], id intptr, flags flags[open_flags]) fd_i2c
ioctl$I2C_IOCTL_EXEC(fd fd_i2c, cmd const[I2C_IOCTL_EXEC], arg ptr[out, i2c_ioctl_exec])
i2c_ioctl_exec {
iie_op flags[i2c_op_t_flags]
iie_addr int16
iie_buflen len[iie_buf, intptr]
iie_buf buffer[out]
iie_cmdlen len[iie_cmd, intptr]
iie_cmd buffer[out]
}
</code>
</pre>
<h2>Future Work</h2>
<p>Though we have a basic working structure of this tool, yet a lot has to be worked upon for leveling it up to make the best of it. Perfect goals would be met when there would be least of manual labor needed. Sys2syz still looks forward to automating the detection of macros used by the flag types in syzkaller. List of to-dos also includes extending syzkaller’s support for generation of description of syscalls.</p>
<p>Some other yet-to-be-done tasks include-
<ul>
<li> Generating descriptions for function type </li>
<li> Calculating attributes for structs and unions </li>
</ul>
<h2>Summary</h2>
<p>We have surely reached closer to our goals but the project needs active involvement and incremental updates to scale it up to its full potential. Looking forward to much more learning and making more contribution to NetBSD community.</p>
<p>Atlast, a word of thanks to my mentors William Coldwell, Siddharth Muralee, Santhosh Raju and Kamil Rytarowski as well as the NetBSD organization for being extremely supportive. Also, I owe a big thanks to Google for giving me such a glaring opportunity to work on this project.</p>https://blog.netbsd.org/tnf/entry/the_gnu_gdb_debugger_and4The GNU GDB Debugger and NetBSD (Part 5) Kamil Rytarowski2020-10-07T17:16:53+00:002020-10-07T17:24:34+00:00The NetBSD developers maintain two copies of GDB:
<ul>
<li>One in the base-system that includes a significant set of local patches.</li>
<li>Another one in pkgsrc whose patching is limited to mostly build fixes.</li>
</ul>
<p>
The base-system version of GDB (GPLv3) still relies on local patching to work.
I have set a goal to reduce the number of custom patches to bare minimum, ideally achieving the state of GDB working without any local modifications at all.The NetBSD developers maintain two copies of GDB:
<ul>
<li>One in the base-system that includes a significant set of local patches.</li>
<li>Another one in pkgsrc whose patching is limited to mostly build fixes.</li>
</ul>
<p>
The base-system version of GDB (GPLv3) still relies on local patching to work.
I have set a goal to reduce the number of custom patches to bare minimum, ideally achieving the state of GDB working without any local modifications at all.
<p>
<h2>GDB changes</h2>
<p>
Last month, the NetBSD/amd64 support was merged into gdbserver.
This month, the gdbserver target support was extended to
<a href="https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=8b667faedf6012048f1f6e71785b1ac1412b8a9c">NetBSD/i386</a> and
<a href="https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=8e1d09292902cff8325b08a64fa5a918c7f9aa4f">NetBSD/aarch64</a>.
The gdbserver and gdb code was cleaned up, refactored and made capable of introducing even more NetBSD targets.
<p>
Meanwhile, the NetBSD/i386 build of GDB was
<a href="https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=1eb6eb795fd3479c97d8aadc4f70d6afad5f8511">fixed</a>.
The missing include of x86-bsd-nat.h as a common header was added to i386-bsd-nat.h.
The i386 GDB code for BSD contained a runtime assert that verified whether the locally hardcoded
<var>struct sigcontext</var> is compatible with the system headers. In reality, the system headers are no longer using
this structure since 2003, after the switch to ucontext_t, and the validating code was no longer
effective. After the switch to newer GCC, this was reported as a unused local variable by the compiler.
I have decided to <a href="https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=6ff330351e7741774c4b7ac1214cf7d73c7eac4d">remove</a> the check on NetBSD entirely.
This was followed up by a small <a href="https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=064280be25f2ff27477ce649f01a70d42ddad2ae">build fix</a>.
<p>
The NetBSD team has noticed that the GDB's agent.cc code contains a portability bug and prepared a local fix.
The traditional behavior of the BSD kernel is that passing random values of sun_len (part of sockaddr_un)
can cause failures. In order to prevent the problems, the sockaddr_un structure is now zeroed before use.
I've reimplemented the fix and successfully
<a href="https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=e2a2a24a8e78427ff8667d625f5befbe88c328bb">upstreamed it</a>.
<p>
In order to easily resolve the issue with environment hardening enforced by PaX MPROTECT,
I've <a href="https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=91e5e8db334b9a87c54f03982dfa0c88e3c9d7a1">introduced
a runtime warning whenever byte transfers betweeen the debugee and debugger occur with the EACCES errno code</a>.
<p>
<h2>binutils changes</h2>
<p>
I've added support for NetBSD/aarch64 upstream, in
<a href="https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=c0b313441717b65569edb01bf9984d2066d899de">GNU BFD</a> and
<a href="https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=cc8b27f89cc8d0fde4532134c19c40c47a023abd">GNU GAS</a>.
NetBSD still carries local patches for the GNU binutils components, and GNU ld does not build out of the box on NetBSD/aarch64.
<p>
<h2>Summary</h2>
<p>
The NetBSD support in GNU binutils and GDB is improving promptly, and the most popular
platforms of amd64, i386 and aarch64 are getting proper support out of the box, without downstream patches.
The remaining patches for these CPUs include: streamlining kgdb support,
adding native GDB support for aarch64,
upstreaming local modifications from the GNU binutils components (especially BFD and ld) and introducing portability enhancements
in the dependent projects like libiberty and gnulib.
Then, the remaining work is to streamline support for the remaining CPUs (Alpha, VAX, MIPS, HPPA, IA64, SH3, PPC, etc.),
to develop the missing generic features (such as listing open file descriptors for the specified process) and
to fix failures in the regression test-suite.https://blog.netbsd.org/tnf/entry/wayland_on_netbsd_trials_andWayland on NetBSD - trials and tribulationsNia Alarie2020-09-28T17:34:22+00:002020-09-28T17:34:22+00:00<p>After I posted about the <a href="https://blog.netbsd.org/tnf/entry/default_window_manager_switched_to">new default window manager</a> in NetBSD I got a few questions, including "when is NetBSD switching from X11 to Wayland?", Wayland being X11's "new" rival. In this blog post, hopefully I can explain why we aren't yet!</p><p>After I posted about the <a href="https://blog.netbsd.org/tnf/entry/default_window_manager_switched_to">new default window manager</a> in NetBSD I got a few questions, including "when is NetBSD switching from X11 to Wayland?", Wayland being X11's "new" rival. In this blog post, hopefully I can explain why we aren't yet!</p>
<p>Last year (and early this year) I was responsible for porting the first working Wayland compositor to NetBSD - <a href="https://github.com/michaelforney/swc">swc</a>. I chose it because it looked small and hackable. You can try it out by installing the <code>velox</code> window manager from pkgsrc.</p>
<a href="http://ftp.netbsd.org/pub/NetBSD/misc/nia/images/wayland-laptop.jpg"><img src="http://ftp.netbsd.org/pub/NetBSD/misc/nia/images/wayland-laptop.jpg" style="max-width: 750px;"
alt="A Wayland compositor running on my NetBSD laptop, with a few applications like Luakit and Dungeon Crawl Stone Soup open."
title="A Wayland compositor running on my NetBSD laptop, with a few applications like Luakit and Dungeon Crawl Stone Soup open."></a>
<h2>Difficulties</h2>
<p>In a Wayland system, the "compositor" (display server) is responsible for managing displays, input, and window management. Generally, this means a lot of OS-specific code is contained there.</p>
<p>Wayland does not define protocols for features X11 users expect, like screenshots, screen locking, or window management. Either you implement these inside the compositor (lots of work that has to be redone), or you define your own protocol extension.</p>
<p>The Wayland "reference implementation" is a small set of libraries that can be used to build a compositor or a client application. These libraries currently have hard dependencies on Linux kernel APIs like <code>epoll</code>. In pkgsrc we've patched the libraries to add <a href=https://man.netbsd.org/kqueue.2>kqueue(2)</a> support, but the patches haven't been accepted upstream. Wayland is written with the assumption of Linux to the extent that every client application tends to <code>#include <linux/input.h></code> because Wayland's designers didn't see the need to define a OS-neutral way to get mouse button IDs.</p>
<p>So far, all Wayland compositors but swc have a hard dependency on libinput, which only supports Linux's input API (also cloned in FreeBSD). In NetBSD we have an entirely different input API - <a href="https://man.netbsd.org/wscons.4">wscons(4).</a> wscons is actually fairly simple to write code for, someone just needs to go out there and do it. You can use my code in swc as a reference. :)</p>
<p>In general, Wayland is moving away from the modularity, portability, and standardization of the X server.</p>
<h2>Is it ready for production?</h2>
<p>No, but you can play with it.</p>
<ul>
<li>swc has some remaining bugs and instability.</li>
<li>swc is incompatible with key applications like Firefox, but others like Luakit work, as do most things that use Qt5, GTK3, or SDL2. Not being able to run X11 applications currently is quite limiting.</li>
<li>Other popular compositors are not yet available. Alternatively, someone could write some new ones.</li>
<li>You need a supported GPU or SoC with kernel modesetting, since safe software fallbacks don't work here. So far, I've only tested this with Intel GPUs.</li>
</ul>
<h2>Task list</h2>
<ul>
<li>Adding support for wscons to more Wayland compositors and persuading developers to accept the patches.</li>
<li>Persuading developers not to add hard dependencies on <code>epoll</code> and instead use an abstraction layer like libevent.</li>
<li>Updating the NetBSD kernel DRM/KMS stack. This is a difficult undertaking that involves porting code from the Linux kernel (a very fast moving target).
<ul>
<li>Getting support for newer DRM versions</li>
<li>Getting support for atomic modesetting</li>
<li>Getting support for Glamor X servers (for running X11 applications inside wayland, etc)</li>
<li>Newer AMDGPU drivers, etc</li>
</ul>
</li>
<li>Adding support for basic (non-DRMKMS) framebuffers to a Wayland compositor. X11 can run from a basic unaccelerated NetBSD framebuffer, but this isn't yet possible in any Wayland compositor.</li>
<li>Extending swc to add more features and fix bugs.</li>
</ul>
<p>I've decided to take a break from this, since it's a fairly huge undertaking and uphill battle.
Right now, X11 combined with a compositor like picom or xcompmgr is the more mature option.</p>
https://blog.netbsd.org/tnf/entry/default_window_manager_switched_toDefault window manager switched to CTWM in NetBSD-currentNia Alarie2020-09-28T08:33:28+00:002020-09-28T17:42:32+00:00<p>
For more than 20 years, NetBSD has shipped X11 with the "classic" default window manager of twm. However, it's been showing its age for a long time now.
</p>
<p>
In 2015, ctwm was imported, but after that no progress was made. ctwm is a fork of twm with some extra features - the primary advantages are that it's still incredibly lightweight, but highly configurable, and has support for virtual desktops, as well as a NetBSD-compatible license and ongoing development. Thanks to its configuration options, we can provide a default experience that's much more usable to people experienced with other operating systems.
</p><p>
For more than 20 years, NetBSD has shipped X11 with the "classic" default window manager of twm. However, it's been showing its age for a long time now.
</p>
<p>
In 2015, ctwm was imported, but after that no progress was made. ctwm is a fork of twm with some extra features - the primary advantages are that it's still incredibly lightweight, but highly configurable, and has support for virtual desktops, as well as a NetBSD-compatible license and ongoing development. Thanks to its configuration options, we can provide a default experience that's much more usable to people experienced with other operating systems.
</p>
<p>
Recently, I've been installing NetBSD with some people in real life and was inspired by their reactions to the default twm to improve the situation, so I played with ctwm, wrote a config, and used it myself for a week. It's now the default in NetBSD-current.
</p>
<a href="https://ftp.netbsd.org/pub/NetBSD/misc/nia/images/desktop.png"><img src="https://ftp.netbsd.org/pub/NetBSD/misc/nia/images/desktop.png" alt="" style="max-width: 750px;"></a>
<p>
We gain some nice features like an auto-generated application menu (that will fill up as packages are installed to /usr/pkg), and a range of useful keyboard shortcuts including volume controls - the default config should be fully usable without a mouse. It should also work at a range of screen resolutions. We can add HiDPI support after some larger bitmap fonts are imported - another advantage of ctwm is that we can support very slow and very fast hardware with one config.
</p>
<p>
If you're curious about ctwm, check out the <a href="https://www.ctwm.org/index.html">ctwm website</a>. It's also included in previous NetBSD releases, though not as the default window manager and not with this config.
</p>