NetBSD Bloghttps://blog.netbsd.org/tnf/feed/entries/atom2024-03-17T12:20:22+00:00Apache Roller (incubating)https://blog.netbsd.org/tnf/entry/the_geeks_way_of_checkingThe Geeks way of checking what the outside wheather is likemartin2022-09-24T18:58:31+00:002022-09-24T18:58:31+00:00<p>I recently had to replace my oldish WS2300 weather station, which was connected via a <strong>long</strong> serial cable (running from my kitchen to the machine room in the basement) with a modern device, a WS3500. This now connects to my wifi network and logs data to a postgres server running on a tiny aarch64 SoC, which also provides a website to query the data.</p>
<p>This all was done with minimal base systems means, plus very few additional pkgs from pkgsrc: in my case the postgres server, obviously, (or at least databases/postgresql14-client, if a postgres server already runs somewhere) and misc/sunwait used for a few site related calculations I found interesting to display. The only other suprising component used is pom(6) from the games set, used to calculate the phase of moon. The weather station displays this on its console, but it is not part of the reported weather data - but easy to recalculate.</p>
<p>Part of this work was to analyze details of the ecowitt or the weather underground protocol and extracting data from it.</p>
<p>The other part was creating two websites that display the current weather or some parts of the weather history.</p>
<p>For the two last parts I took inspiration from previous work done on this by others, and overall it turned out to be straight forward.</p>
<h1>Prologue</h1>
<p>
When I bought my house in 2004 I went shopping for a outside thermometer - and ended up with a full weather-station instead (a WS2300).
When I unpacked it I found a serial cable inside...</p>
<p>
Long story short - I was still in the process of recabling the house (running ethernet to every room) and added a serial cable from the machine room to the WS2300,
and then did some pkgsrc work and got misc/open2300 and misc/open2300-mysql. I used those to log the data from the weather-station to a mysql database, and later
moved that (via misc/open2300-pgsql) to a postgres database.</p>
<p>
Now sometime this year the machine running that database had to be replaced (should have done that earlier, it was power hungry and wasteful). The replacement was
an aarch64 SoC (a Pine64 Quartz64 model A) - and it had no real com ports (of course) any more. I had experimented with USB serial adapters and the WS2300 before,
but for unclear reasons this time I had no luck and couldn't get it to work. Since some of the outdoor sensors of the old weather-station had started failing, I decided
to replace it.</p>
<h1>New Weather-Station, new Sensors</h1>
<p>I picked a WS3500 because it comes with a nice remote sensor arrangement:</p>
<a href="//www.NetBSD.org/~martin/weather/ext_sensor.jpg"><img src="//www.NetBSD.org/~martin/weather/ext_sensor.jpg" style="max-width:80%;"/></a>
<p>
I attached it to a satellite dish mount about 1.2m above my garage and ran a two wire cable through the mount to supply it with 3V and get rid of any batteries.
It does not have a connector for that, but the battery compartment had enough space for a 330µF elco and soldering that and the cable directly to the battery contacts was easy.</p>
<p>The sensors report to the weather-station via a proprietary protocol in the 868 MHz band.</p>
<h1>New Weather-Station, new Reporting</h1>
<p>The weather-station can connect to a wifi network but does not offer any services itself.
The app used to configure the station offers several predefined weather collection services.</p>
<p>
I found the idea a bit strange to have my local weather data logged to some server somewhere else in the cloud and then get it back via my browser, but for others this is a good thing.
I found this <a href="https://www.linkedin.com/pulse/collecting-presenting-weather-sensor-data-using-ecowitts-jonas-frantz">article</a> that describes exactly the remote-only, no machines required on-site setup.
I used that article as inspiration for the data collection (but that part turned out to be quite trivial, see below) and copied a lot of the presentation site from it (also more details below).</p>
<p>
So in my setup I created web servers on two dedicated ports of my tiny machine running the postgres server.
One is used by the weather-station for reporting the data, the other is used to query the database.</p>
<p>
The configuration of the weather-station for a custom server was easy:</p>
<a href="http://netbsd.org/~martin/weather/screenshot1_app.jpg"><img src="//netbsd.org/~martin/weather/screenshot1_app.jpg" style="max-width:80%;"/></a>
<a href="http://netbsd.org/~martin/weather/screenshot2_app.jpg"><img src="//netbsd.org/~martin/weather/screenshot2_app.jpg" style="max-width:80%;"/></a>
<p>
I tested the ecowitt protocol first. It uses a post to a fixed URL and the form data has nearly identical data as we get with the solution I ended up with - only a few names (of form fields)
are slightly different.
</p>
<p>
The blacked items "StationID" and "StationKey" appear verbatim in the reported data, you can set them to whatever you want - the scripts below do not check them.</p>
<p>
The weather underground protocol does a simple http GET and provides all data as query parameters (I had to add the trailing question mark in the configuration).
This makes it very easy to extract the data in a script on the server side.</p>
<p>
But lets get there step by step. NetBSD comes with a http/https server in base, originally called "bozohttpd". It is very lightweight, but it can run various types of scripts - I picked
the plain old simple CGI and /bin/sh as language, using a bit of awk to convert units.</p>
<p>
First I added two users, so I could separate file access rights. This is how they look like in vipw:
<pre style="background-color:lightgoldenrodyellow;">
weatherupdate:*************:1004:1004::0:0:Weather Update Service:/weather/home:/sbin/nologin
weatherquery:*************:1005:1004::0:0:Weather Query Service:/weather/query:/sbin/nologin
</pre>
and two httpd instances for them
<code>/etc/inetd</code> entry to collect the incoming data:</p>
<pre style="background-color:lightgoldenrodyellow;">
88 stream tcp nowait:600 weatherupdate /usr/libexec/httpd httpd -q -c /weather/cgi /weather/files
89 stream tcp nowait:600 weatherquery /usr/libexec/httpd httpd -q -c /weather/cgi -M .js "text/javascript" - - /weather/files
</pre>
<p>
The document root (<code>/weather/files</code>) would not be used for the instance on port 88, but httpd needs one.
Note that these lines use the quiet flag ("-q") which is only available in netbsd-current. You can replace it with "-s" for older versions.</p>
<p>
The home directories of both users are mostly empty, besides a <code>.pgpass</code> file that contains the password for this
user connection to the postgres server. They look like this:</p>
<pre style="background-color:lightgoldenrodyellow;">
127.0.0.1:5432:weatherhistory:open2300:xxxxxxxxxxxxxx
</pre>
<p>where "weatherhistory" is the datebase and "open2300" is the name of the postgres user for the update script and the password is x-ed out. The other file looks very similar:</p>
<pre style="background-color:lightgoldenrodyellow;">
127.0.0.1:5432:weatherhistory:weatherquery:xxxxxxxxxxx
</pre>
<p>
At the postgres level the user "weatherquery" needs to have SELECT privilege on the table "weather", and "open2300" needs to have INSERT privilege.
The table schema (output of "pg_dump -s") looks like this:</p>
<pre style="background-color:lightgoldenrodyellow;">
--
-- Name: weather; Type: TABLE; Schema: public; Owner: weathermaster
--
CREATE TABLE public.weather (
"timestamp" timestamp without time zone DEFAULT '1970-01-01 00:00:00'::timestamp without time zone NOT NULL,
temp_in double precision DEFAULT '0'::double precision NOT NULL,
temp_out double precision DEFAULT '0'::double precision NOT NULL,
dewpoint double precision DEFAULT '0'::double precision NOT NULL,
rel_hum_in integer DEFAULT 0 NOT NULL,
rel_hum_out integer DEFAULT 0 NOT NULL,
windspeed double precision DEFAULT '0'::double precision NOT NULL,
wind_angle double precision DEFAULT '0'::double precision NOT NULL,
wind_chill double precision DEFAULT '0'::double precision NOT NULL,
rain_1h double precision DEFAULT '0'::double precision NOT NULL,
rain_24h double precision DEFAULT '0'::double precision NOT NULL,
rain_total double precision DEFAULT '0'::double precision NOT NULL,
rel_pressure double precision DEFAULT '0'::double precision NOT NULL,
wind_gust double precision DEFAULT 0 NOT NULL,
light double precision DEFAULT 0 NOT NULL,
uvi double precision DEFAULT 0 NOT NULL
);
ALTER TABLE public.weather OWNER TO weathermaster;
--
-- Name: weather weather_pkey; Type: CONSTRAINT; Schema: public; Owner: weathermaster
--
ALTER TABLE ONLY public.weather
ADD CONSTRAINT weather_pkey PRIMARY KEY ("timestamp");
--
-- Name: TABLE weather; Type: ACL; Schema: public; Owner: weathermaster
--
GRANT INSERT ON TABLE public.weather TO open2300;
GRANT SELECT ON TABLE public.weather TO weatherquery;
</pre>
<p>
As noted above, I carried this database over (with minor modifications) from previous instances of the whole setup - so it may not be optimal or elegant.
One thing that needs special attention is the "timestamp" column - it carries date/time in UTC and has no timezone associated.
This looked like a natural choice, but has some unexpected consequences.
When querying data in JSON format, "timestamp" will not get the JavaScript marker for "UTC", a "Z" suffix. So in the JavaScript code in the web pages
you will find quite a few places that cover up for this.</p>
<p>
Now when the weather station sends data to the configured server, inetd(8) runs httpd(8) and that invokes a shell script <code>/weather/cgi/update.cgi</code>
as the "weatherupdate" user.
This script uses awk(1) to do a few unit conversions and output a SQL command to insert the data into the "weather" table.
This SQL command is then piped to psql(1) with the connection string passed on the command line.
The corresponding password is found in <code>~/.pgpass</code> of the "weatherupdate" user.</p>
<p>
The script looks like this:</p>
<pre style="background-color:lightgoldenrodyellow;">
#! /bin/sh
TZ=UTC; export TZ
awk -v $( echo "$QUERY_STRING" | sed 's/\&/ -v /g' ) 'BEGIN {
temp=(tempf-32)/1.8;
indoortemp=(indoortempf-32)/1.8;
dewpt=(dewptf-32)/1.8;
windchill=(windchillf-32)/1.8;
windspeed=windspeedmph*1.609344;
windgust=windgustmph*1.609344;
rain=rainin*25.4;
dailyrain=dailyrainin*25.4;
totalrain=totalrainin*25.4;
rel_preasure=baromin/0.029529980164712;
printf("INSERT INTO weather VALUES ('"'"'%s'"'"', %f, %f, %f, %d, %d, %f, %d, %f, %f, %f, %f, %f, %f, %f, %f);\n",
strftime("%F %T"),
indoortemp,
temp,
dewpt,
indoorhumidity,
humidity,
windspeed,
winddir,
windchill,
rain, dailyrain, totalrain,
rel_preasure,
windgust,
solarradiation, UV);
}' | psql "hostaddr='127.0.0.1'dbname='weatherhistory'user='open2300'" > /dev/null 2>&1
</pre>
<p>
Note that it explicitly sets the timezone to UTC.
The input data comes (as defined by CGI) via the QUERY_STRING environment variable, as a set of "field=value" items, separated by &.
They are converted to sets of "-v" args for the awk invocation via a simple sed script.</p>
<p>
With this in place, the weather-station adds a record every five minutes to the database, and it was fun to check it via SQL,
but for reasons not quite clear to me most of the rest of the family did not like that kind of access very much.</p>
<pre style="background-color:lightgoldenrodyellow;">
psql (14.5)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
weatherhistory=> select min(temp_out), max(temp_out) from weather;
min | max
-------+------
-18.1 | 80.9
(1 row)
</pre>
<p>
I initially thought the 80.9°C were measured while I was soldering the power cable, but apparently they were fallout from the sometimes failing
sensors of the old station. The database has 2840 rows with temp_out > 40°C and all of them are 80.something. I should replace them with an average of the neighbor
records.</p>
<h1>Presenting the data</h1>
<p>
So I needed an internal web site. Which needs access to the data.
The above setup already paved the way for that, via the second port I set up.
I wanted to show all the current data in one page, and variable history data on another - which meant two CGI scripts to query the data.
The <code>/weather/cgi/latest.cgi</code> script just fetches the last record logged and creates a JSON from it, and also uses pom(6) and the sunwait(1) program
from pkgsrc to supply some site and date specific data:</p>
<pre style="background-color:lightgoldenrodyellow;">
#! /bin/sh
PATH=/usr/games:/usr/pkg/bin:$PATH
GEOPOS="51.505554N 0.075278W" # geographic position of this weather station
UPDATE=300 # seconds between updates
# This script uses psql(1) from pkgsrc/databases/postgresql14-client,
# pom(6) from the NetBSD games set and pkgsrc/misc/sunwait.
# collect global site data: sunrise and friends
eval $( sunwait report ${GEOPOS} | awk -F": " '
/Sun directly north/ {
printf("zenith=\"%s\"\n", $2);
}
/Daylight:/ {
split($2,v," to ");
printf("sunrise=\"%s\"\nsunset=\"%s\"\n", v[1], v[2]);
}
/with Civil twilight:/ {
split($2,v," to ");
printf("dawn=\"%s\"\ndusk=\"%s\"\n", v[1], v[2]);
}
/It is: Day/ {
printf("day=true\n");
}
/It is: Night/ {
printf("day=false\n");
}
' )
# moon phase
eval $( pom | awk '-F(' '
/The Moon is Full/ { printf("moontrend=\"-\"\nmoon=100\n"); }
/The Moon is New/ { printf("moontrend=\"+\"\nmoon=0\n"); }
/First Quarter/ { printf("moontrend=\"+\"\nmoon=50\n"); }
/Last Quarter/ { printf("moontrend=\"-\"\nmoon=50\n"); }
/Waxing/ {
a=$0;
sub(/^.*\(/, "", a);
sub(/%.*$/, "", a);
printf("moontrend=\"+\"\nmoon=%d\n", a+0);
}
/Waning/ {
a=$0;
sub(/^.*\(/, "", a);
sub(/%.*$/, "", a);
printf("moontrend=\"-\"\nmoon=%d\n", a+0);
}
' )
# start the json output
printf "\n\n{ \"site\": { \"updates\": ${UPDATE},
\"dawn\": \"${dawn}\", \"sunrise\": \"${sunrise}\",
\"zenith\": \"${zenith}\", \"day\": ${day},
\"sunset\": \"${sunset}\", \"dusk\": \"${dusk}\",
\"moon\": { \"trend\": \"${moontrend}\", \"percent\": ${moon} }\n}, \"weather\":\n"
# fill database results
printf "WITH t AS ( SELECT * FROM weather ORDER BY timestamp DESC LIMIT 1 ) SELECT row_to_json(t) FROM t;\n" |
psql --tuples-only --no-align "hostaddr='127.0.0.1'dbname='weatherhistory'user='weatherquery'"
# terminate json
printf "\n}\n"
</pre>
<p>
As you can see, if you would restrict output to plain data from the database, the script would be only four or five lines long.
But I like the additional spicing.</p>
<p>
The <code>/weather/cgi/history.cgi</code> script fetches rows between two timestamps passed to it (in JSON timestamp format)
and answers with a JSON containing an array of all the data in the requested time window:</p>
<pre style="background-color:lightgoldenrodyellow;">
#! /bin/sh
COND=$( echo "${QUERY_STRING}" | tr '&' '\n'| sed -e 's/%22/\"/g' -e 's/%3A/:/g' | awk '
/from=/ { v=$0; sub(/^[^"]*\"/, "", v); sub(/\".*$/, "", v); arg_from=v; }
/to=/ { v=$0; sub(/^[^"]*\"/, "", v); sub(/\".*$/, "", v); arg_to=v; }
END {
if (arg_from && arg_to) {
printf("timestamp >= '"'"'%s'"'"' AND timestamp <= '"'"'%s'"'"'\n",
arg_from, arg_to);
}
}
' )
if [ -z "${COND}" ]; then
# printf "could not parse: ${QUERY_STRING}\n" >> /tmp/sql.log
exit 0;
fi
# start output
printf "\n\n"
# printf "${COND}\n" >> /tmp/sql.log
# fill database results
printf "WITH t AS ( SELECT * FROM weather WHERE ${COND} ORDER by timestamp ASC ) SELECT json_agg(t) FROM t;\n" |
psql --tuples-only --no-align "hostaddr='127.0.0.1'dbname='weatherhistory'user='weatherquery'" # 2&>> /tmp/sql.err
</pre>
<p>
Fetching this data now is easy in JavaScript.</p>
<p>
We have a request URL defined as a const, like this:</p>
<pre style="background-color:lightgoldenrodyellow;">
const queryURL = 'http://weatherhost.duskware.de:89/cgi-bin/history.cgi?';
</pre>
<p>and then add (if needed) the paramaters for the query, like in this example function that gets passed a from-date and a to-date:</p>
<pre style="background-color:lightgoldenrodyellow;">
function showData(fromD, toD)
{
var url = new URL(queryURL);
url.searchParams.append("from", '"'+fromD.toJSON()+'"');
url.searchParams.append("to", '"'+toD.toJSON()+'"');
fetch(url).then(function(response) {
return response.json();
}).then(function(data) {
makeGraphs(data);
updateButtons();
}).catch(function(error) {
console.error(error)
});
}
</pre>
<p>
When the answer from the server arrives, it is decoded as JSON and returned as input data to the next function that makes some graphs from the data array.
Finally a few buttons are updated (in this example the time window is put into a start and a end date control.</p>
<p>
Inspired by the <a href="https://www.linkedin.com/pulse/collecting-presenting-weather-sensor-data-using-ecowitts-jonas-frantz">post</a>
mentioned above I used <a href="https://canvas-gauges.com/">canvas gauges</a> for the display of the latest data
and <a href="https://dygraphs.com/">dygraphs</a> for the display of historic data.</p>
<p>
Here is an example of how the latest display looks:</p>
<a href="//www.NetBSD.org/~martin/weather/website_weather.png"><img src="//www.NetBSD.org/~martin/weather/website_weather.png" style="max-width:80%;"/></a>
<p>
And here is how the history display looks:</p>
<a href="//www.NetBSD.org/~martin/weather/website_weatherhistory.png"><img src="//www.NetBSD.org/~martin/weather/website_weatherhistory.png" style="max-width:80%;"/></a>
<p>
I have put an archive of the cgi scripts and web pages <a href="//www.NetBSD.org/~martin/weather/weather.tgz">here</a>, and also for the curious who just
want to peek at the full glory of my web design skills the <a href="//www.NetBSD.org/~martin/weather/index.html">start page</a> (showing the latest weather data)
and the <a href="//www.NetBSD.org/~martin/weather/history.html">history page</a>.
</p>
<p>
Besides those files, you will need</p>
<ul>
<li>a <code>/weather/files/favicon.ico</code> if you like.</li>
<li>download <code>gauge.min.js</code> from canvas gauges and put it into <code>/weather/files/</code>.</li>
<li>download <code>dygraph.css</code>, <code>dygraph.min.js</code> from dygraph, plus <code>synchronizer.js</code> from the dygraph <code>extras/</code>
directory and put it also into <code>/weather/files/</code>.</li>
</ul>
<p>
Then you should be ready to go - easy, isn't it? And no heavy weight dependencies or pkgs needed.</p>
<h1>What about other weather stations?</h1>
<p>
There are quite a few similar weather stations out there now that seem to run "related" firmware and have similar capabilities.
Most likely the update script (and details in the presentation pages) will need adjustements for other types.</p>
<p>
If you start with a different device, just log all the data it sends and adjust the cgi scripts/database/JavaScript accordingly.
For protocol analyzis there are several easy means:</p>
<ul>
<li>Remove the "-q" flag in the httpd command (in <code>/etc/inetd.conf</code>) and check <code>/var/log/xferlog</code> for the quey paramaters sent by the
weather station (when using the weather underground protocol).</li>
<li>Make the station log to a <code>debug.cgi</code> first to capture everything (including form data posted). This works for the ecowitt protocoll.</li>
<li>All this stations seem to use http only (not https), so you can sniff the traffic. Use <code>tcpdump -w</code> on the server to capture the data and
analyze it with net/wireshark from pkgsrc.</li>
</ul>
<p>Here is what a debug.cgi script could look like:</p>
<pre style="background-color:lightgoldenrodyellow;">
#! /bin/sh
env > /tmp/debug.env
printf "\n\nOK\n"
cat > /tmp/debug.input &
</pre>
<p>
This allows you to see the form input in /tmp/debug.input and the CGI environment in /tmp/debug.env.</p>https://blog.netbsd.org/tnf/entry/announcing_google_summer_of_code3Announcing Google Summer of Code 2022 projectsAndrius Varanavicius2022-05-22T12:20:02+00:002022-05-22T12:20:02+00:00<p>
<a href='https://summerofcode.withgoogle.com/'><img alt="Google Summer of Code logo" src='//www.NetBSD.org/~leot/blog-posts/imgs/GSoC-icon-192.png' style="width: 80px; height: 80px; float: right;" /></a>
The NetBSD Foundation has finalized the list of projects for this year’s <a href="https://summerofcode.withgoogle.com/programs/2022/organizations/the-netbsd-foundation">Google Summer of Code</a>. The contributors and projects are the following:
</p>
<ul>
<li>Brian Schnepp - <a href="https://summerofcode.withgoogle.com/programs/2022/projects/esA98HIB">Raspberry Pi GPU Driver</a></li>
<li>Arjun Bemarkar - <a href="https://summerofcode.withgoogle.com/programs/2022/projects/LDY5asp0">inetd enhancements</a></li>
<li>Piyush Sachdeva - <a href="https://summerofcode.withgoogle.com/programs/2022/projects/37Q8OZNU">Emulating missing linux syscalls</a></li>
<li>Vihas Makwana - <a href="https://summerofcode.withgoogle.com/programs/2022/projects/yO4fTbNn">Introduce a new Wi-Fi driver</a></li>
<li>Vivek Kumar Sah - <a href="https://summerofcode.withgoogle.com/programs/2022/projects/UPlZRXS6">Automating donor acknowledgement and information storage</a></li>
</ul>
<p>
The community bonding period has already started (from May 20) and it will last until June 12. During this time, the contributors are expected to coordinate with their mentors and community.
</p>
<p>
This will be immediately followed by the coding period from June 13 to September 4. After which, the contributors are expected to submit their final work, evaluate their mentors, and get evaluated by their mentors as well.
Results will be announced on September 20.
</p>
<p>
For more information about the Google Summer of Code 2022 kindly refer to the official <a href="https://summerofcode.withgoogle.com/">GSoC website</a>.
</p>
<p>
We would like to express our gratitude to Google for organizing the yearly GSoC, and to The NetBSD Foundation mentors and administrators for their efforts and hardwork!
</p>
<p>
Let us welcome all the contributors to our growing NetBSD community!
</p>https://blog.netbsd.org/tnf/entry/the_netbsd_foundation_is_aThe NetBSD Foundation is a mentoring organization at Google Summer of Code 2022Leonardo Taccari2022-03-16T17:02:15+00:002022-03-16T17:07:32+00:00<p>
<a href='https://summerofcode.withgoogle.com/'><img alt="Google Summer of Code logo" src='https://developers.google.com/open-source/gsoc/resources/downloads/GSoC-Horizontal.png' /></a>
</p>
<p>
We are happy to announce that The NetBSD Fundation is a <a href="https://summerofcode.withgoogle.com/programs/2022/organizations/the-netbsd-foundation">mentoring organization</a> at <a href="https://summerofcode.withgoogle.com/">Google Summer of Code 2022</a>!
</p>
<p>
Would you like to contribute to NetBSD or pkgsrc during the summer? Please give a look to <a href="https://wiki.NetBSD.org/projects/gsoc/">NetBSD wiki Google Summer of Code page</a> with possible ideas list and/or please join <a href="https://web.libera.chat/#NetBSD-code">#NetBSD-code</a> IRC channel on <a href="https://libera.chat/">libera</a> or get in touch with us via <a href="https://www.NetBSD.org/mailinglists/">mailing lists</a> to propose new projects!
</p>
<p>
Please note that unlike past years where Google Summer of Code was opened only to university students since this year if you are 18 or older you can be a GSoC contributor.
</p>
<p>
For more information about Google Summer of Code please give a look to the official <a href="https://summerofcode.withgoogle.com/">GSoC website</a>.
</p>
<p>
Looking forward to have a nice Google Summer of Code!
</p>https://blog.netbsd.org/tnf/entry/making_rockpro64_a_netbsd_serverMaking RockPro64 a NetBSD Servermatthew green2022-03-09T23:57:54+00:002022-03-09T23:57:54+00:00<p>The time has come to upgrade my SunBlade 2500s to something more power friendly and faster. I'd already removed one CPU and thus half the ram from two of these systems to reduce their power consumption, but it's still much higher than it could be.</p>
<p>After much searching, I've decided on <a href="https://wiki.pine64.org/wiki/ROCKPro64">Pine64's RockPro64</a> 4GiB ram model (technically, only 3.875GiB ram.) Pine64 make SBCs, laptops, phones, and various other mostly ARM gadgets, and the RockPro64 has the fastest CPU they ship (Rockchip RK3399), and can use a small "NAS Case", that is sufficient to house 2 HDDs and, at a stretch, upto 6 SSDs (cooling would become quite an issue at this point.)</p>
<p>In my SATA setup, I have 3 SSDs with a JMicron 585 card in the PCIe slot, two SSDs in the NAS case SSD region, and the third is in the HDD region with an adapter. I have used two SATA power splitters to convert the NAS case's 2 SATA power ports into 4, with the 4th one also powering a Noctua case fan. The cabling is not great with this, with enough SATA power cabling for 6 devices to lay. Probably a bespoke power cable to connect to the RockPro64 would make this nicer and probably improve cooling slightly, but I'm just using off-the-shelf components for now.</p>
<p>In the last year or so I've been working on making NetBSD/arm64 big-endian more featureful. In particular, I've added support for:
<ul><li>other-endian access disklabels, FFS file-systems in the NetBSD libsa</li>
<li>the "look 64 sectors later" for RAIDFrame partitions in MI efiboot (the x86 specific efiboot has more extensive support for this that should be ported over)</li>
<li>other-endian access to RAIDFrame labels in the kernel</li>
<li>updated the U-Boot package and featureset to include AHCI/SATA support, workaround some bugs and fix the newer U-Boot SPI loader location, and figured out how to default loading from either SATA or NVMe</li>
</ul></p>
<p>There are not too many special actions needed for this sort of setup compared to a normal NetBSD or Arm system. While I built my installations by hand, the standard NetBSD Arm images are suitable for this task. It's easiest to start from an SD card with the RockPro64 u-boot installed. There are two U-Boot images available, one for SD/eMMC, and one for SPI (there is an odd problem with the early SPI loader that requires a portion of the image to be different.) The pkgsrc package for sysutils/u-boot-rockpro64 version 2022.01 has these suggested methods for installing the U-Boot image (this package should be buildable on any supported pkgsrc platform).</p>
<p>To install U-Boot into the SD/eMMC card (can run on any system, the image must be written at 32KiB into the device):
<pre>
# dd if=/usr/pkg/share/u-boot/rockpro64/rksd_loader.img seek=64 of=/dev/rld0c
</pre>
<p>
To install U-Boot into the SPI flash (must be run on the host, and lives at the very start of the device:
<pre>
# dd if=/usr/pkg/share/u-boot/rockpro64/rkspi_loader.img bs=64k of=/dev/spiflash0
</pre>
</p>
<p>When booting from NVMe or SATA, one must drop to the U-Boot prompt and adjust the "boot_targets" value to put scsi* (SATA) or nvme* ahead of the mmc* and usb* options.</p>
<p>The original value for me:
<pre>
=> printenv boot_targets
boot_targets=mmc1 usb0 mmc0 nvme0 scsi0 pxe dhcp sf0
</pre>
</p>
<p>Which is then adjusted and saved with eg:
<pre>
=> setenv boot_targets nvme0 scsi0 mmc1 usb0 mmc0 pxe dhcp sf0
=> saveenv
Saving Environment to SPIFlash... Erasing SPI flash...Writing to SPI flash...done
OK
</pre></p>
<p>(In this list, mmc1 is the SD slot, mmc0 is the eMMC card, pxe and dhcp are netbooting, and sf0 attempts to load further U-Boot scripts from SPI.)</p>
<p>There are some minor issues with the RockPro64. It has no ability to use ECC memory. It only comes with 1 PCIe 4x slot, and Rockchip errata limited this to PCIe 1.x (though no NetBSD users encounted any issues with PCIe 2.x enabled, and this can be forced back on via a DTS patch.) It is possible to use a PCIe bridge (I have never done this, though I would like to try in the future) to enable more devices for booting, or to enable both a network and storage device.</p>
https://blog.netbsd.org/tnf/entry/public_netbsd_irc_channels_movedPublic NetBSD IRC chat channels moved to LiberaNia Alarie2021-05-30T18:23:29+00:002021-05-30T18:24:49+00:00<pHi everyone,</p>
<p>Due to the unfortunate situation regarding changes in administration on
freenode.net, and the resulting chaos, we have decided to move the public
NetBSD IRC channels from freenode to irc.libera.chat.</p>
<p>This includes:</p>
<ul>
<li><a href="https://web.libera.chat/#NetBSD">#NetBSD</a> - general discussion</li>
<li><a href="https://web.libera.chat/#netbsd-code">#netbsd-code</a> - development discussion</li>
<li><a href="https://web.libera.chat/#pkgsrc">#pkgsrc</a> - pkgsrc (primarily development) discussion</li>
</ul>
<p>You can find information on connecting to Libera at <a href="https://libera.chat">https://libera.chat/</a></p><p>Hi everyone,</p>
<p>Due to the unfortunate situation regarding changes in administration on
freenode.net, and the resulting chaos, we have decided to move the public
NetBSD IRC chat channels from freenode to irc.libera.chat.</p>
<p>This includes:</p>
<ul>
<li><a href="https://web.libera.chat/#NetBSD">#NetBSD</a> - general discussion</li>
<li><a href="https://web.libera.chat/#netbsd-code">#netbsd-code</a> - development discussion</li>
<li><a href="https://web.libera.chat/#pkgsrc">#pkgsrc</a> - pkgsrc (primarily development) discussion</li>
</ul>
<p>You can find information on connecting to Libera at <a href="https://libera.chat">https://libera.chat/</a></p>https://blog.netbsd.org/tnf/entry/allen_k_briggs_memorial_scholarshipAllen K. Briggs Memorial ScholarshipLeonardo Taccari2020-12-21T11:37:59+00:002020-12-21T11:37:59+00:00<p>
Allen Briggs was one of the earliest members of the NetBSD community,
pursuing his interest in macBSD, and moving to become a NetBSD
developer when the two projects merged. Allen was known for his
quiet and relaxed manner, and always brought a keen wisdom with
him; allied with his acute technical expertise, he was one of the
most valued members of the NetBSD community.
</p>
<p>
He was a revered member of the NetBSD core team, and keenly involved
in many aspects of its application; from working on ARM chips to
helping architect many projects, Allen was renowned for his expertise.
He was a distinguished engineer at Apple, and used his NetBSD
expertise there to bring products to market.
</p>
<p>
Allen lived in Blacksburg Virginia with his wife and twin boys and
was active with various community volunteer groups. His family
touched the families of many other NetBSD developers and those
friendships have endured beyond his passing.
</p>
<p>
We have received the following from Allen's family and decided to
share it with the NetBSD community. If you can, we would ask you
to consider contributing to his Memorial Scholarship.
</p>
<p>
<a href="https://www.ncssm.edu/donate/distance-education/allen-k-briggs-88-memorial-scholarship">https://www.ncssm.edu/donate/distance-education/allen-k-briggs-88-memorial-scholarship</a>
</p>
<p>
The Allen K. Briggs Memorial Scholarship is an endowment to provide
scholarships in perpetuity for summer programs at the North Carolina
School of Science & Math, which Allen considered to be a place that
fundamentally shaped him as a person. We would love to invite
Allen's friends and colleagues from the BSD community to donate to
this cause so that we can provide more scholarships to students
with financial need each year. We are approximately halfway to our
goal of $50K with aspirations to exceed that target and fund
additional scholarships.
</p>
<p>
Two quick notes on donating: <strong>Important!</strong> When donating, you must
select "Allen K. Briggs Memorial Scholarship" under designation
for the donation to be routed to the scholarship If you have the
option to use employer matching (i.e., donating to NCSSM through
an employer portal to secure a match from your employer), please
email the NCSSM Foundation's Director of Development, April Horton
(<code>april.horton@ncssm.edu</code>), after donating to let her know you want
your gift and employer match to go to the Allen K. Briggs Memorial
Scholarship Thanks in advance for your help. I'd be happy to answer
any questions you or any others have about this.
</p>
https://blog.netbsd.org/tnf/entry/default_window_manager_switched_toDefault window manager switched to CTWM in NetBSD-currentNia Alarie2020-09-28T08:33:28+00:002020-09-28T17:42:32+00:00<p>
For more than 20 years, NetBSD has shipped X11 with the "classic" default window manager of twm. However, it's been showing its age for a long time now.
</p>
<p>
In 2015, ctwm was imported, but after that no progress was made. ctwm is a fork of twm with some extra features - the primary advantages are that it's still incredibly lightweight, but highly configurable, and has support for virtual desktops, as well as a NetBSD-compatible license and ongoing development. Thanks to its configuration options, we can provide a default experience that's much more usable to people experienced with other operating systems.
</p><p>
For more than 20 years, NetBSD has shipped X11 with the "classic" default window manager of twm. However, it's been showing its age for a long time now.
</p>
<p>
In 2015, ctwm was imported, but after that no progress was made. ctwm is a fork of twm with some extra features - the primary advantages are that it's still incredibly lightweight, but highly configurable, and has support for virtual desktops, as well as a NetBSD-compatible license and ongoing development. Thanks to its configuration options, we can provide a default experience that's much more usable to people experienced with other operating systems.
</p>
<p>
Recently, I've been installing NetBSD with some people in real life and was inspired by their reactions to the default twm to improve the situation, so I played with ctwm, wrote a config, and used it myself for a week. It's now the default in NetBSD-current.
</p>
<a href="https://ftp.netbsd.org/pub/NetBSD/misc/nia/images/desktop.png"><img src="https://ftp.netbsd.org/pub/NetBSD/misc/nia/images/desktop.png" alt="" style="max-width: 750px;"></a>
<p>
We gain some nice features like an auto-generated application menu (that will fill up as packages are installed to /usr/pkg), and a range of useful keyboard shortcuts including volume controls - the default config should be fully usable without a mouse. It should also work at a range of screen resolutions. We can add HiDPI support after some larger bitmap fonts are imported - another advantage of ctwm is that we can support very slow and very fast hardware with one config.
</p>
<p>
If you're curious about ctwm, check out the <a href="https://www.ctwm.org/index.html">ctwm website</a>. It's also included in previous NetBSD releases, though not as the default window manager and not with this config.
</p>https://blog.netbsd.org/tnf/entry/gsoc_reports_benchmarking_netbsd_thirdGSoC Reports: Benchmarking NetBSD, third evaluation reportLeonardo Taccari2020-09-12T08:29:06+00:002020-09-12T08:29:06+00:00<p>This report was written by Apurva Nandan as part of Google Summer of Code 2020.</p>
<p>This blog post is in continuation of
<a href="//blog.NetBSD.org/tnf/entry/gsoc_reports_benchmarking_netbsd_first">GSoC Reports: Benchmarking NetBSD, first evaluation report</a>
and <a href="//blog.NetBSD.org/tnf/entry/gsoc_reports_benchmarking_netbsd_second">GSoC Reports: Benchmarking NetBSD, second evaluation report</a>
blogs, and describes my progress in the final phase of GSoC 2020 under
The NetBSD Foundation.</p>
<p>In the third phase, I upgraded to the latest stable version
<a href="https://www.phoronix-test-suite.com/">Phoronix Test Suite (PTS)</a> 9.8.0 in
pkgsrc-wip, resolved the TODOs and created patches for more
test-profiles to fix their installation and runtime errors on
NetBSD-current.</p><p>This report was written by Apurva Nandan as part of Google Summer of Code 2020.</p>
<h2>Introduction</h2>
<p>This blog post is in continuation of
<a href="//blog.NetBSD.org/tnf/entry/gsoc_reports_benchmarking_netbsd_first">GSoC Reports: Benchmarking NetBSD, first evaluation report</a>
and <a href="//blog.NetBSD.org/tnf/entry/gsoc_reports_benchmarking_netbsd_second">GSoC Reports: Benchmarking NetBSD, second evaluation report</a>
blogs, and describes my progress in the final phase of GSoC 2020 under
The NetBSD Foundation.</p>
<p>In the third phase, I upgraded to the latest stable version
<a href="https://www.phoronix-test-suite.com/">Phoronix Test Suite (PTS)</a> 9.8.0 in
pkgsrc-wip, resolved the TODOs and created patches for more
test-profiles to fix their installation and runtime errors on
NetBSD-current.</p>
<h2>Progress in the third phase of GSoC</h2>
<h3>wip/phoronix-test-suite TODO and update</h3>
<p>As a newer stable version of the Phoronix Test Suite was available in
upstream, I upgraded the Phoronix Test Suite from version 9.6.1 to
9.8.0 in pkgsrc-wip and is available as
<a href="https://pkgsrc.se/wip/phoronix-test-suite">wip/phoronix-test-suite</a>.
You can have a look at the
<a href="https://github.com/phoronix-test-suite/phoronix-test-suite/blob/master/ChangeLog">PTS Changelog</a>
to know about the improvements between these two versions.</p>
<p>To get the package ready for merge in pkgsrc upstream, I also resolved
the pkgsrc-wip TODOs.</p>
<h4>pkgsrc-wip commits:</h4>
<ul>
<li><a href="https://github.com/NetBSD/pkgsrc-wip/commit/77ef21385452fe098abb12f7772ac21f0aeb7f86">wip/phoronix-test-suite: update phoronix-test-suite to 9.8.0</a></li>
<li><a href="https://github.com/NetBSD/pkgsrc-wip/commit/7162cf86c3fced1702a4446d514b76640d7266f3">Resolved the pending TODOs</a></li>
</ul>
<p>If any new problems are encountered, please document them in
<code>wip/phoronix-test-suite/TODO</code> file and/or contact me.</p>
<h3>Testing of automated benchmarking framework</h3>
<p>I had been assigned a remote testing machine having Intel 6138 dual
processor, 40 cores, 80 threads, 192GB of RAM. I spent time reproducing
my automated framework i.e., Phoromatic-Anita Integration on the
machine. I was able to reproduce the integration framework working
without networking configuration, but the network bridge needs to be
setup on the remote machine and the integration script to be tested
with it. I shall continue this task in the post-GSoC period.</p>
<h4>Benchmarking Results</h4>
<p>I also performed benchmarking of NetBSD-9 amd64 native installation by
running 50 test-profiles on a remote machine assigned to me by mentors
and uploaded the benchmark results to
<a href="https://openbenchmarking.org/">OpenBenchmarking.org</a> at:</p>
<ul>
<li><a href="https://openbenchmarking.org/result/2008287-NE-3966268951">https://openbenchmarking.org/result/2008287-NE-3966268951</a></li>
</ul>
<h3>Test-profile debugging</h3>
<p>I then continued the task of maintaining/porting test-profiles and
fixed the following test-profiles:</p>
<h4>Timed FFmpeg Compilation</h4>
<p>This test times how long it takes to build FFmpeg.
This test is part of <code>Processor Test</code> category.</p>
<h5>Original Test-profile:</h5>
<p><a href="https://openbenchmarking.org/test/pts/build-ffmpeg">https://openbenchmarking.org/test/pts/build-ffmpeg</a></p>
<h5>Patched Test-profile:</h5>
<p><a href="https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/build-ffmpeg-1.0.1">https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/build-ffmpeg-1.0.1</a></p>
<h6>Commit:</h6>
<ul>
<li><a href="https://github.com/apurvanandan1997/pts-test-profiles-dev/commit/e9765185aabc01240cdb3671fb61f4c7d47e1b7e">Replace the make -> gmake for compatibility with NetBSD</a></li>
</ul>
<h4>Compile Bench</h4>
<p>Compilebench tries to age a filesystem by simulating some of the disk
IO common in creating, compiling, patching, stating and reading kernel
trees. It indirectly measures how well filesystems can maintain
directory locality as the disk fills up and directories age.
This test is part of <code>Disk Test</code> category.</p>
<h5>Original Test-profile:</h5>
<p><a href="https://openbenchmarking.org/test/pts/compilebench">https://openbenchmarking.org/test/pts/compilebench</a></p>
<h5>Patched Test-profile:</h5>
<p><a href="https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/compilebench-1.0.2">https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/compilebench-1.0.2</a></p>
<h6>Commit:</h6>
<ul>
<li><a href="https://github.com/apurvanandan1997/pts-test-profiles-dev/commit/528b1853a07c22d79d2d8483b3fe2aa951d3866f">Replaced python2 with NetBSD style naming python2.7</a></li>
</ul>
<h4>Timed MAFFT Alignment</h4>
<p>This test performs an alignment of 100 pyruvate decarboxylase sequences.
This test is part of <code>Processor Test</code> category.</p>
<h5>Original Test-profile:</h5>
<p><a href="https://openbenchmarking.org/test/pts/mafft">https://openbenchmarking.org/test/pts/mafft</a></p>
<h5>Patched Test-profile:</h5>
<p><a href="https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/mafft-1.5.0">https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/mafft-1.5.0</a></p>
<h6>Commits:</h6>
<ul>
<li><a href="https://github.com/apurvanandan1997/pts-test-profiles-dev/commit/2bb288c4d352693b42586e55caf68a847d5740f8">Replaced the make -> gmake for compatibility with NetBSD</a></li>
<li><a href="https://github.com/apurvanandan1997/pts-test-profiles-patches/commit/d31fee7256d165acfd22e23e04590ec322df3620">Patch for replacing /bin/bash interpreter with /usr/pkg/bin/bash</a></li>
</ul>
<h2>Future Plans</h2>
<p>This officially summarizes my GSoC project: Benchmark NetBSD, and my
end goal of the project that is to integrate Phoronix Test Suite with
NetBSD and Anita for automated benchmarking is complete and its
deployment on benchmark.NetBSD.org will be continued to be worked on
with the coordination of moderators and merging the wip of Phoronix
Test Suite 9.8.0 will be done by the pkgsrc maintainers in next days.</p>
<p>I want to thank my mentors and the NetBSD community without whose
constant support I wouldn't have achieved the goals.</p>https://blog.netbsd.org/tnf/entry/gsoc_reports_benchmarking_netbsd_secondGSoC Reports: Benchmarking NetBSD, second evaluation reportLeonardo Taccari2020-08-12T10:09:16+00:002020-08-12T10:09:16+00:00<p>This report was written by Apurva Nandan as part of Google Summer of Code
2020.</p>
<p>This blog post is in continuation of
<a href="//blog.NetBSD.org/tnf/entry/gsoc_reports_benchmarking_netbsd_first">GSoC Reports: Benchmarking NetBSD, first evaluation report</a>
blog and describes my progress in the second phase of GSoC 2020 under
The NetBSD Foundation.</p>
<p>In this phase, I worked on the automation of the regression suite made
using <a href="https://www.phoronix-test-suite.com">Phoronix Test Suite (PTS)</a>
and its integration with <a href="https://www.gson.org/netbsd/anita">Anita</a>.</p>
<p>The automation framework consists of two components Phoromatic server,
provided by Phoronix Test Suite in pkgsrc, and Anita, a Python tool for
automating NetBSD installation.</p>
<p>This report was written by Apurva Nandan as part of Google Summer of Code
2020.</p>
<p>This blog post is in continuation of
<a href="//blog.NetBSD.org/tnf/entry/gsoc_reports_benchmarking_netbsd_first">GSoC Reports: Benchmarking NetBSD, first evaluation report</a>
blog and describes my progress in the second phase of GSoC 2020 under
The NetBSD Foundation.</p>
<p>In this phase, I worked on the automation of the regression suite made
using <a href="https://www.phoronix-test-suite.com">Phoronix Test Suite (PTS)</a>
and its integration with <a href="https://www.gson.org/netbsd/anita">Anita</a>.</p>
<p>The automation framework consists of two components Phoromatic server,
provided by Phoronix Test Suite in pkgsrc, and Anita, a Python tool for
automating NetBSD installation.</p>
<h2>About Phoromatic</h2>
<p>Phoromatic is a remote management system for the Phoronix Test Suite,
which allows the automatic scheduling of tests, remote installation of
new tests, and the management of multiple test systems through a web
interface. Tests can be scheduled to run on a routine basis across
multiple test systems automatically. Phoromatic can also interface with
revision control systems to offer support for issuing new tests on a
context-basis, such as whenever a Git commit has been pushed. The test
results are then available from the web interface.</p>
<h3>Phoromatic client-server architecture</h3>
<p><img src="//www.NetBSD.org/~leot/blog-posts/imgs/pts_phoromatic-server-client-arch.png" alt="Phoromatic server-client architecture" width="800" /></p>
<p>The Phoromatic server relies upon a PHP/HHVM built-in web server
process and a PTS-hosted WebSocket server. The web server process
handles the web UI and the responsibilities of the Phoromatic server.</p>
<p>Phoromatic clients are testing machines installed with PTS that connect
to the Phoromatic web server through the HTTP port of the server.</p>
<h3>Phoromatic Setup</h3>
<p>To start the Phoromatic server, Phoromatic server HTTP port and web
server socket port needs to be set in
<code>~/.phoronix-test-suite/user-config.xml</code> as shown:</p>
<pre>
...
<Server>
<RemoteAccessPort>8640</RemoteAccessPort>
<Password></Password>
<WebSocketPort>8642</WebSocketPort>
<AdvertiseServiceZeroConf>TRUE</AdvertiseServiceZeroConf>
<AdvertiseServiceOpenBenchmarkRelay>TRUE</AdvertiseServiceOpenBenchmarkRelay>
<PhoromaticStorage>~/.phoronix-test-suite/phoromatic/</PhoromaticStorage>
</Server>
</pre>
<h3>Phoromatic Usage</h3>
<p>To start the Phoromatic web server for controlling local Phoronix Test Suite client systems:</p>
<p><code>$ phoronix-test-suite start-phoromatic-server</code></p>
<p>The Phoromatic web server will be hosted at <code>localhost:8640</code> and will require a local account creation on the server.</p>
<h4>Phoromatic Clients</h4>
<p>The Phoromatic client is used for connecting to a Phoromatic server to facilitate the automatic running of tests on that client.</p>
<p>Phoromatic clients can be created and connected to the server using the following command:</p>
<p><code>
$ phoronix-test-suite phoromatic.connect SERVER_IP:SERVER_HTTP_PORT/ACCOUNT_ID
</code></p>
<p>Phoromatic server interacts with the Phoromatic clients through the HTTP port specified in the <code>~/.phoronix-test-suite/user-config.xml</code>.</p>
<h4>Phoromatic Test-schedules</h4>
<p>A test schedule is used to facilitate automatically running a set of
test(s)/suite(s) on either a routine timed basis or whenever triggered
by an external script or process, e.g. Git/VCS commit, manually
triggered, etc.
Phoromatic provides an option for pre-install, pre-run, post-install
and post-run shell scripts that are executed on the Phoromatic
clients.
Test-schedules can be configured to run any tests on any specific
systems.</p>
<h2>About Anita</h2>
<p>Anita is a tool for automated testing of the NetBSD operating system.
Using Anita, we can download a NetBSD distribution and install it in a
virtual machine in a fully automated fashion. Anita is written in
Python and uses the pexpect module to “screen scrape” the sysinst
output over an emulated serial console and script the installation
procedure.</p>
<h3>Installation</h3>
<p>Anita can be installed on NetBSD, Linux and macOS systems using the following:</p>
<pre>
$ pip install pexpect
$ git clone https://github.com/gson1703/anita/
$ python setup.py install
</pre>
<h2>Phoromatic-Anita Integration</h2>
<p>I would like to describe the workflow here briefly:</p>
<ul>
<li>A test-schedule was created on the Phoromatic server meant to run
<code>pts/idle-1.2.0</code> test on the host machine that contains the
<a href="https://gist.github.com/apurvanandan1997/48b54402db1df3723920735f85fc7934">phoromatic-anita-integration.sh</a>
as a pre-run script.</li>
<li>The script performs the following:
<ul>
<li>Creates a mountable disk image with an executable script for
setting up Phoronix Test Suite and Phoromatic client creation on the
benchmarking VM systems.</li>
<li>Invokes Anita with the appropriate command-line options for
configurations and network setup and mounts the image to run the
configuration script on the VM.</li>
<li>Configuration script performs hostname change, DHCP setup, NFS
setup, <code>PKG_PATH</code> setup, PTS installation, its configuration and
connecting it to the Phoromatic server through a network bridge.</li>
</ul>
</li>
<li>Once the benchmarking VM systems get connected to the Phoromatic
server, Phoromatic server identifies the benchmarking VM systems with
their IP address, hostname and MAC address.</li>
<li>After the identification, Phoromatic initiates the pending tests on
VM (test-profiles are downloaded on the go in the VM and executed) and
maintains a history of the test result data.</li>
</ul>
<p>Few points to be noted:</p>
<ul>
<li>I have used a local PKG_PATH with a NFS server setup as PTS 9.6.1 is
available in wip and recompiling it would be a wastage of time. Later I
have planned to use the binary shard by Joyent:
<a href="https://pkgsrc.joyent.com/packages/NetBSD/9.99.69/amd64/All/">https://pkgsrc.joyent.com/packages/NetBSD/9.99.69/amd64/All/</a> once the
updated PTS gets upstreamed.</li>
<li>The host machine needs some one-time manual setup like installation
of QEMU, Anita, pexpect, term, cvs, etc., initial user registration on
Phoromatic server, Phoromatic port setup, network bridge setup. Apart
from this, the rest of the framework does not require user
supervision.</li>
</ul>
<h5>VM configuration script</h5>
<p>The following script is used as a pre-run script in the test-schedules for invoking Anita and setting up the VMs:</p>
<p><a href="https://gist.github.com/apurvanandan1997/48b54402db1df3723920735f85fc7934">https://gist.github.com/apurvanandan1997/48b54402db1df3723920735f85fc7934</a></p>
<h5>Networking Setup</h5>
<p>A bridged networking mode configuration of QEMU has been used in Anita
as multiple VMs will be able to accommodate with a single bridge
(created on the host machine, one-time setup) using
<a href="//man.NetBSD.org/dhcpcd.8">dhcpcd(8)</a>,
without complicated host forwarding setup (Phoromatic server requires
HTTP port forwarding).</p>
<p>In order to enable bridged networking for your QEMU guests, you must
first create and configure a bridge interface on your host.</p>
<pre>
# ifconfig virbr0 create
</pre>
<p>Next, you must specify the newly-created bridge interface in
<code>/etc/qemu/bridge.conf</code>:</p>
<pre>
$ sudo mkdir /etc/qemu
$ sudo touch /etc/qemu/bridge.conf && sudo chmod 644 /etc/qemu/bridge.conf
$ sudo sh -c "echo 'allow virbr0' >> /etc/qemu/bridge.conf"
</pre>
<p>Finally, in order for non-privileged processes to be able to invoke
qemu-bridge-helper, you must set the setuid bit on the utility:</p>
<p><code>
$ sudo chmod u+s /usr/local/libexec/qemu-bridge-helper
</code></p>
<p>For more details on the bridged mode networking setup in QEMU, please
refer to the following guides:</p>
<ul>
<li><a href="https://t.pagef.lt/basic-networking-with-qemu/">https://t.pagef.lt/basic-networking-with-qemu/</a></li>
<li><a href="https://www.NetBSD.org/docs/guide/en/chap-net-practice.html#chap-net-practice-bridge">https://www.NetBSD.org/docs/guide/en/chap-net-practice.html#chap-net-practice-bridge</a></li>
</ul>
<h5>Reproducing the framework</h5>
<p>To reproduce the framework, you need to have Phoronix Test Suite, QEMU,
Anita, pexpect, cvs, xterm, makefs installed on your host machine.</p>
<p>For example on NetBSD:</p>
<pre>
# pkg_add qemu
# pkg_add py37-anita
$ cd pkgsrc/wip/phoronix-test-suite
$ make install
</pre>
<p>The step-by-step process to get the framework after installing PTS,
including the one-time manual setup, can be summarized as follows: All
control and configuration of the Phoromatic Server is done via the
web-based interface when the Phoromatic Server is active.</p>
<ul>
<li>Configure the port of Phoromatic server as 8640 and web socket as
8642 as described above.</li>
<li>Start the Phoromatic server using the command stated above.
<img src="//www.NetBSD.org/~leot/blog-posts/imgs/pts_phoromatic-register.png" alt="Phoromatic login page image" width="800" /></li>
<li>Create your user account on the Phoromatic server using the web interface GUI.
<img src="//www.NetBSD.org/~leot/blog-posts/imgs/pts_phoromatic.png" alt="Phoromatic default page image" width="800" /></li>
<li>Disable client system approval for new system addition from the
settings menu in the web interface.</li>
<li>Connect the host machine as a Phoromatic client to the Phoromatic
server using the command stated above.</li>
<li>Create a test-schedule for the host machine with the
<a href="https://gist.github.com/apurvanandan1997/48b54402db1df3723920735f85fc7934">pre-run script</a>
as specified above and <code>pts/idle-1.2.0</code> as the test-profile.
<img src="//www.NetBSD.org/~leot/blog-posts/imgs/pts_create-sch.png" alt="Phoromatic create test-schedule image" width="800" /></li>
<li>Execute the test-schedule or assign it on a timed-schedule and watch it running!
<img src="//www.NetBSD.org/~leot/blog-posts/imgs/pts_host-machine-test-sch.png" alt="Phoromatic Anita integration test-scehdule" width="800" /></li>
<li>New VM systems with the latest NetBSD-current binaries and packages
will be created and identified by Phoromatic server automatically.</li>
<li>Once for all, we need to specify what benchmarking test-profiles need
to be run on the VM systems in the test-schedules section and it will
be taken care of by Phoromatic.
<img src="//www.NetBSD.org/~leot/blog-posts/imgs/pts_test-sch.png" alt="Phoromatic benchmarking system test-schedule image" width="800" /></li>
<li>The result history can also be viewed from Phoromatic web interface.</li>
</ul>
<p>You can have a look at the video to get a clearer picture of how to
setup the framework:</p>
<iframe width="800" height="450" src="https://www.youtube.com/embed/7OSaqDWIsxo" frameborder="0"></iframe>
<h2>Future Plans</h2>
<p>The regression suite is complete and final tasks of deploying it on
benchmark.NetBSD.org and upstreaming the wip of Phoronix Test Suite
will be done in the final phase of my GSoC project.
I want to thank my mentors for their constant support.</p>
https://blog.netbsd.org/tnf/entry/gsoc_reports_benchmarking_netbsd_firstGSoC Reports: Benchmarking NetBSD, first evaluation reportLeonardo Taccari2020-07-16T09:38:58+00:002020-07-17T16:00:11+00:00<p>This report was written by Apurva Nandan as part of Google Summer of Code 2020.</p>
<p>My GSoC project under NetBSD involves developing an automated
regression and performance test framework for NetBSD that offers
reproducible benchmarking results with detailed history and logs across
various hardware & architectures.</p>
<p>To achieve this performance testing framework, I am using the
<a href="https://www.phoronix-test-suite.com/">Phoronix Test Suite (PTS)</a>
which is an open-source, cross-platform automated testing/benchmarking
software for Linux, Windows and BSD environments. It allows the
creation of new tests using simple XML files and shell scripts and
integrates with revision control systems for per-commit regression
testing.</p><p>This report was written by Apurva Nandan as part of Google Summer of Code 2020.</p>
<p>My GSoC project under NetBSD involves developing an automated
regression and performance test framework for NetBSD that offers
reproducible benchmarking results with detailed history and logs across
various hardware & architectures.</p>
<p>To achieve this performance testing framework, I am using the
<a href="https://www.phoronix-test-suite.com/">Phoronix Test Suite (PTS)</a>
which is an open-source, cross-platform automated testing/benchmarking
software for Linux, Windows and BSD environments. It allows the
creation of new tests using simple XML files and shell scripts and
integrates with revision control systems for per-commit regression
testing.</p>
<h2>About PTS</h2>
<h3>PTS core</h3>
<p>PTS core is the engine of the Phoronix Test Suite and handles the following jobs:</p>
<ul>
<li>Regularly updating the test profiles, test suites, and repository index.</li>
<li>Downloading, compilation, installation and evaluation of test packages.</li>
<li>Test result management and uploading results and user-defined
test-profiles/suites to OpenBenchmarking.org</li>
</ul>
<h3>Test-profiles/Suites & OpenBenchmarking</h3>
<p>Test-profiles are light-weight XMLs and shell scripts that allow easy
installation of the benchmarking packages and their evaluation.
Test-profiles generally contains the following files:</p>
<ul>
<li><code>downloads.xml</code>: Stores the links of the benchmarking packages,
required additional packages, patches to be applied, etc. with their
hash sums.</li>
<li><code>install.sh</code>: Actual compilation, installation, and test
evaluation shell script. Also handles the task of applying patches.</li>
<li><code>results-definition.xml</code>: Meta-data to define the information for
test result data (e.g. result data units, LIB or HIB, etc.)</li>
<li><code>test-definition.xml</code>: Meta-data to specify test-profile details,
such as compatible OS, external dependencies, test arguments, etc.</li>
</ul>
<p>A simple test-profile
<a href="https://openbenchmarking.org/innhold/91db3ffff901d12dabd732bc568a44d02e5c6387">C-Ray-1.2.0</a>
can be seen to get an idea of the XMLs and shell scripts.</p>
<p>Test-suites are bundles of related test-profiles or more suites,
focusing on a subsystem or certain category of tests e.g. Disk Test
Suite, Kernel, CPU suite, etc.</p>
<p><a href="https://openbenchmarking.org/">OpenBenchmarking.org</a> serves as a
repository for storing test profiles, test suites, hence allowing
new/updated tests to be seamlessly obtained via PTS.
OpenBenchmarking.org also provides a platform to store the test result
data openly and do a comparison between any test results stored in the
OpenBenchmarking.org cloud.</p>
<h2>Usage</h2>
<h3>Test installation</h3>
<p>The command:</p>
<p><code>$ phoronix-test-suite install test-profile-name</code></p>
<p>Fetches the sources of the tests related to the the test-profile in
<code>~/.phoronix-test-suite/installed-tests</code>, applies patches and
carries out compilation of test sources for generating the executables
to run the tests.</p>
<h3>Test execution</h3>
<p>The command:</p>
<p><code>$ phoronix-test-suite run test-profile-name</code></p>
<p>Performs installation of the test (similar to <code>$ phoronix-test-suite
install test-profile-name</code>) followed by exectution the binary from
the compiled sources of the test in
<code>~/.phoronix-test-suite/installed-tests</code> and finally reports the
tests results/outcome to the user and provides an option to upload
results to OpenBenchmarking.org.</p>
<h2>Progress in the first phase of GSoC</h2>
<h3>pkgsrc-wip</h3>
<p>The first task performed by me was upgrading the Phoronix Test Suite
from version 8.8 to the latest stable version 9.6.1 in <code>pkgsrc-wip</code>
and is available as
<a href="https://pkgsrc.se/wip/phoronix-test-suite">wip/phoronix-test-suite</a>.
You can have a look at the
<a href="https://github.com/phoronix-test-suite/phoronix-test-suite/blob/master/ChangeLog">PTS Changelog</a>
to know about the improvements between these two versions.</p>
<h4>Major Commits:</h4>
<ul>
<li><a href="https://github.com/NetBSD/pkgsrc-wip/commit/0073e18af14a97e33542a15d5a2940ed0b353897">wip/phoronix-test-suite: update phoronix-test-suite to 9.6.1</a></li>
<li><a href="https://github.com/NetBSD/pkgsrc-wip/commit/959a9a0821fb155d11a2a68d2271233135375df1">Added required PHP dependencies & removed TODO file</a></li>
</ul>
<p>Currently, the PTS upgrade to 9.6.1 is subject to more modifications
and fixes before it gets finally merged to the <code>pkgsrc</code> upstream.
To test the Phoronix Test Suite 9.6.1 on NetBSD till then, you can
setup <code>pkgsrc-wip</code> on your system via:</p>
<pre>
$ cd /usr/pkgsrc
$ git clone git://wip.pkgsrc.org/pkgsrc-wip.git wip
</pre>
<p>To learn more about pkgsrc-wip please see
<a href="https://pkgsrc.org/wip/">The pkgsrc-wip project homepage</a>.</p>
<p>After setting up <code>pkgsrc-wip</code>, use the following command for
installation of PTS 9.6.1:</p>
<pre>
$ cd /usr/pkgsrc/wip/phoronix-test-suite
$ make install
</pre>
<p>If any new build/installation errors are encountered, please document
them in wip/phoronix-test-suite/TODO file and/or contact me.</p>
<h3>Testing & Debugging</h3>
<p>As we now know, having a look over OpenBenchmarking.org’s available
test-profiles section or using
<code>$ phoronix-test-suite list-available-tests</code>,
there are 309 test-profiles available on PTS for Linux, and only 166 of
309 tests have been ported to NetBSD (can be differentiated by a
thunder symbol on the test profile on the website). These 166
test-profiles can be fetch, build and executed on NetBSD using just a
single command <code>$ phoronix-test-suite run test-profile-name</code>. I am
ignoring the graphics ones in both Linux & NetBSD as GPU benchmarking
wasn’t required in the project.</p>
<p>Many of the test profiles available on NetBSD have installation, build,
or runtime errors i.e. they don’t work out of the box using the command
<code>$ phoronix-test-suite run test-profile-name</code>. I ran 90 of these
test profiles on a NetBSD-current x86_64 QEMU VM (took a lot of time
due to the heavy tests) and have reported the status in the sheet:
<a href="https://drive.google.com/file/d/1gW76JI7W-Jpeczns49k1bBIVr8bb6QuX/view?usp=sharing">NetBSD GSoC: Test Porting Progress Report</a></p>
<p>You can have a look at how a PTS test-profile under action looks like!</p>
<h4>Aircrack-NG test-profile demonstration</h4>
<p>Aircrack-ng is a tool for assessing WiFi/WLAN network security.</p>
<p>The PTS test-profile is available at:
<a href="https://openbenchmarking.org/test/pts/aircrack-ng">https://openbenchmarking.org/test/pts/aircrack-ng</a></p>
<p><img src="//www.NetBSD.org/~leot/blog-posts/imgs/pts_aircrack_ng_test.png" alt="Screenshot of aircrack-ng PTS test-profile results" width="800" /></p>
<p>For further information, complete results can be seen at
<a href="https://openbenchmarking.org/result/2007116-NI-AIRCRACKN51">https://openbenchmarking.org/result/2007116-NI-AIRCRACKN51</a></p>
<h4>AOBench test-profile demonstration</h4>
<p>AOBench is a lightweight ambient occlusion renderer, written in C.</p>
<p>The PTS test-profile is available at:
<a href="https://openbenchmarking.org/test/pts/aircrack-ng">https://openbenchmarking.org/test/pts/aircrack-ng</a></p>
<p><img src="//www.NetBSD.org/~leot/blog-posts/imgs/pts_aobench_test.png" alt="Screenshot of aobench PTS test-profile results" width="800" /></p>
<p>For further information, complete results can be seen at:
<a href="https://openbenchmarking.org/result/2007148-NI-AOBENCHTE94">https://openbenchmarking.org/result/2007148-NI-AOBENCHTE94</a></p>
<h4>Debugging and fixing non-working test-profiles</h4>
<p>After testing these test-profiles, I debugged the following build
errors in the following test-profiles:</p>
<ul>
<li>pts/t-test1-1.0.1: <code>memalign()</code> not available on NetBSD</li>
<li>pts/coremark-1.0.0: calling bmake instead of gmake</li>
<li>pts/apache-siege-1.0.4: Missing <code>signal.h</code> header</li>
<li>pts/mbw-1.0.0: <code>mempcpy()</code> not available on NetBSD</li>
</ul>
<p>Fixes to the above bugs can be found in the sheet or in the GitHub
repositories shared in the next paragraph.</p>
<p>The modified test-profiles have been pushed to
<a href="https://github.com/apurvanandan1997/pts-test-profiles-dev">pts-test-profiles-dev</a>
and the fix patches to
<a href="https://github.com/apurvanandan1997/pts-test-profiles-patches">pts-test-profiles-patches</a>.
These patches are automatically downloaded by the debugged
test-profiles and automatically applied before building of tests. The
steps to install the debugged test-profiles have been added in the
<code>README.md</code> of
<a href="https://github.com/apurvanandan1997/pts-test-profiles-dev">pts-test-profiles-dev</a>.</p>
<p>These patches have been added in the <code>downloads.xml</code> of the
modified test-profiles, and hence they get fetched during the test
installation. The <code>install.sh</code> script in these modified
test-profiles applies them on the benchmarking packages if the
operating system is detected as NetBSD, thereby removing the build
errors.</p>
<h4>coremark test-profile demonstration</h4>
<p>You can have a look at the patched coremark-1.0.0 test-profile under
execution:</p>
<p>The patched PTS test-profile is available at:
<a href="https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/coremark-1.0.0">https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/coremark-1.0.0</a></p>
<p>This is a test of EEMBC CoreMark processor benchmark.</p>
<p><img src="//www.NetBSD.org/~leot/blog-posts/imgs/pts_coremark_test.png" alt="Screenshot of patched coremark PTS test-profile results" width="800" /></p>
<p>For further information, complete results can be seen at:
<a href="https://openbenchmarking.org/result/2007148-NI-LOCALCORE64">https://openbenchmarking.org/result/2007148-NI-LOCALCORE64</a></p>
<h4>Porting more test-profiles to NetBSD</h4>
<p>Attempts were made to port new test-profiles from Linux to NetBSD, but
the major incompatibility issue was missing external dependencies in
NetBSD. Hence, this may reduce the number of test-profiles we can
actually port, but more sincere efforts will be made in this direction
in the next phase.</p>
<h2>Future Plans</h2>
<p>My next steps would involve porting the non-available tests to NetBSD
i.e. the non-graphics ones available on Linux but unavailable on
NetBSD, and fix the remaining test-profiles for NetBSD.</p>
<p>After gaining an availability of a decent number of test-profiles on
NetBSD, I will use Phoromatic, a remote tests management system for the
PTS (allows the automatic scheduling of tests, remote installation of
new tests, and the management of multiple test systems) to host the
live results of the automated benchmarking framework on
benchmark.NetBSD.org, thereby delivering a functional performance and
regression testing framework.</p>
https://blog.netbsd.org/tnf/entry/announcing_google_summer_of_code2Announcing Google Summer of Code 2020 projectsLeonardo Taccari2020-05-07T10:16:58+00:002020-05-07T10:18:46+00:00<p>
<a href='https://summerofcode.withgoogle.com/'><img alt="Google Summer of Code logo" src='//www.NetBSD.org/~leot/blog-posts/imgs/GSoC-icon-192.png' style="width: 80px; height: 80px; float: right;" /></a>
We are very happy to announce
<a href='https://summerofcode.withgoogle.com/organizations/5125029991284736/'>
The NetBSD Foundation Google Summer of Code 2020 projects</a>:
</p>
<ul>
<li>Apurva Nandan - <a href='https://summerofcode.withgoogle.com/projects/#4645452886048768'>Benchmark NetBSD</a></li>
<li>Jain Naman - <a href='https://summerofcode.withgoogle.com/projects/#4884895517638656'>Curses library automated testing</a></li>
<li>Nikita Gillmann - <a href='https://summerofcode.withgoogle.com/projects/#5101731152658432'>Make system(3) and popen(3) use posix_spawn(3) internally</a></li>
<li>Ayushi Sharma - <a href='https://summerofcode.withgoogle.com/projects/#6134980024991744'>Enhance the syzkaller support for NetBSD</a></li>
<li>Aditya Vardhan Padala - <a href='https://summerofcode.withgoogle.com/projects/#6449059977494528'>Rumpkernel Syscall Fuzzing</a></li>
<li>Nisarg Joshi - <a href='https://summerofcode.withgoogle.com/projects/#6486401496907776'>Fuzzing the network stack of NetBSD in a rumpkernel environment</a></li>
<li>Jason High - <a href='https://summerofcode.withgoogle.com/projects/#6728937511583744'>Extending the functionality of the netpgp suite</a></li>
</ul>
<p>
The community bonding period - where students get in touch with mentors and
community - started on May 4 and will go on until June 1.
The coding period will be June 1 to August 24.
</p>
<p>
Please welcome all our students and a big good luck to students and mentors!
</p>
<p>
A big thank you to <a href='https://www.google.com/'>Google</a> and The
NetBSD Foundation organization mentors and administrators!
</p>
<p>
Looking forward to having another nice Google Summer of Code!
</p>https://blog.netbsd.org/tnf/entry/porting_wine_to_amd64_on1Porting wine to amd64 on NetBSD, third evaluation reportLeonardo Taccari2019-08-21T19:11:41+00:002019-08-26T12:34:54+00:00<p>
This report was written by Naveen Narayanan as part of Google Summer of Code 2019.
</p>
<p>
This report encompasses the progress of the project during the
third coding period.
You can make sense of the overall progress of
the project by going through
the <a href="//blog.NetBSD.org/tnf/entry/porting_wine_to_amd64_on">first
evaluation report</a>
and <a href="//blog.NetBSD.org/tnf/entry/porting_wine_to_amd64_on2">second
evaluation report</a>.
</p>
<p>
Wine-4.4 (released on Mar 2019) is working fine on amd64
and i386. I have been able to use a script as a workaround for
the problem of setting <code>LD_LIBRARY_PATH</code>. My patch for
setting guard size to 0 and hence, precluding Wine from
segfaulting, that got upstreamed, can be
found <a href="https://source.winehq.org/git/wine.git/commit/de5392bfe7919c180d6650eec7b57b5386a4796f">here</a>.
I have updated the package to the latest development version of
Wine which is Wine-4.13 (released on Aug 2019). I have
added support to Wine pkgsrc packages to run tests using make
test, and at the time of writing, they are failing. I have also
noticed them fail on Linux non-pkgsrc environment and hence, will
require further investigation. Initially, they were disabled
owing to pkgsrc setting <code>FORTIFY_SOURCE</code> which is a macro that
provides support for detecting buffer overflows. In pkgsrc, the
<code>wip/wine*</code> packages honor <code>PKGSRC_USE_FORTIFY</code> variable passing
<code>_FORTIFY_SOURCE</code> macro accordingly. Programs compiled with
<code>FORTIFY_SOURCE</code> substitute wrappers for commonly used libc
functions that don't do bounds checking regularly, but could in some
cases. Wine unconditionally disables that via their configure
script because for some platforms that triggered false positives
in the past. However, in my experience, no false positive were
found.
</p><p>
This report was written by Naveen Narayanan as part of Google Summer of Code 2019.
</p>
<p>
This report encompasses the progress of the project during the
third coding period.
You can make sense of the overall progress of
the project by going through
the <a href="//blog.NetBSD.org/tnf/entry/porting_wine_to_amd64_on">first
evaluation report</a>
and <a href="//blog.NetBSD.org/tnf/entry/porting_wine_to_amd64_on2">second
evaluation report</a>.
</p>
<h2>WINE on amd64</h2>
<p>
Wine-4.4 (released on Mar 2019) is working fine on amd64
and i386. I have been able to use a script as a workaround for
the problem of setting <code>LD_LIBRARY_PATH</code>. My patch for
setting guard size to 0 and hence, precluding Wine from
segfaulting, that got upstreamed, can be
found <a href="https://source.winehq.org/git/wine.git/commit/de5392bfe7919c180d6650eec7b57b5386a4796f">here</a>.
I have updated the package to the latest development version of
Wine which is Wine-4.13 (released on Aug 2019). I have
added support to Wine pkgsrc packages to run tests using make
test, and at the time of writing, they are failing. I have also
noticed them fail on Linux non-pkgsrc environment and hence, will
require further investigation. Initially, they were disabled
owing to pkgsrc setting <code>FORTIFY_SOURCE</code> which is a macro that
provides support for detecting buffer overflows. In pkgsrc, the
<code>wip/wine*</code> packages honor <code>PKGSRC_USE_FORTIFY</code> variable passing
<code>_FORTIFY_SOURCE</code> macro accordingly. Programs compiled with
<code>FORTIFY_SOURCE</code> substitute wrappers for commonly used libc
functions that don't do bounds checking regularly, but could in some
cases. Wine unconditionally disables that via their configure
script because for some platforms that triggered false positives
in the past. However, in my experience, no false positive were
found.
</p>
<p>
Running tests on Wine-4.13 throws errors as you can see below:
<pre>
mode64# make test
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M adsldp.dll -p adsldp_test.exe.so sysinfo && touch sysinfo.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
002a:err:winediag:SECUR32_initNTLMSP ntlm_auth was not found or is outdated. Make sure that ntlm_auth >= 3.0.25 is in your path. Usually, you can find it in the winbind package of your distribution.
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f150,0x00000001,0x22f120) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EEA0 000000000002B028 000000000022EE9C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f0e0,0x00000001,0x22f0b0) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EE30 000000000002B318 000000000022EE2C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f0e0,0x00000001,0x22f0b0) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EE30 000000000002B318 000000000022EE2C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:adsldp:sysinfo_get_UserName 0000000000021670,000000000022F0C8: stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f980,0x00000001,0x22f950) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022F6D0 000000000002B318 000000000022F6CC): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f980,0x00000001,0x22f950) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022F6D0 000000000002B318 000000000022F6CC): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:adsldp:sysinfo_get_UserName 0000000000021670,000000000022FB48: stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so cred && touch cred.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
002e:fixme:cred:CredReadW unhandled type -1
002e:fixme:cred:CredReadW unhandled flags 0xdeadbeef
002e:err:cred:CredWriteW bad username L"winetest"
002e:err:cred:CredWriteW bad username (null)
002e:err:cred:CredWriteW bad username (null)
002e:fixme:cred:CredDeleteW unhandled type -1
002e:fixme:cred:CredDeleteW unhandled flags 0xdeadbeef
002e:fixme:cred:CredReadDomainCredentialsW (0x1b1d0, 0x0, 0x22fa50, 0x22f908) stub
002e:fixme:cred:CredReadDomainCredentialsW (0x1b1d0, 0x0, 0x22fa50, 0x22f908) stub
002e:fixme:cred:CredReadDomainCredentialsW (0x1b3c0, 0x0, 0x22fa50, 0x22f908) stub
cred.c:820: Tests skipped: CRED_TYPE_DOMAIN_VISIBLE_PASSWORD credentials are not supported or are disabled. Skipping
002e:fixme:cred:CredIsMarshaledCredentialW BinaryBlobCredential not checked
002e:fixme:cred:CredIsMarshaledCredentialW BinaryBlobCredential not checked
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt && touch crypt.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_lmhash && touch crypt_lmhash.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_md4 && touch crypt_md4.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_md5 && touch crypt_md5.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_sha && touch crypt_sha.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so eventlog && touch eventlog.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),(null)) stub
0046:fixme:advapi:OpenEventLogW (L"IDontExist",(null)) stub
0046:fixme:advapi:OpenEventLogW (L"IDontExist",L"deadbeef") stub
0046:fixme:advapi:OpenEventLogW Remote server not supported
0046:fixme:advapi:OpenEventLogW ((null),L"deadbeef") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW (L"",L"Application") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:GetEventLogInformation (0x0, 1, 0x0, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0x0, 0, 0x0, 0, 0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x0, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x0, 0, 0x22fb40) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 0, 0x22fb40) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 8, 0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x0) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x22fb40) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x0) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x22fb40) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:BackupEventLogW (0x0,(null)) stub
0046:fixme:advapi:BackupEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,(null)) stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:BackupEventLogW (0x0,L"backup2.evt") stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),(null)) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"idontexist.evt") stub
0046:fixme:advapi:OpenBackupEventLogW (L"IDontExist",(null)) stub
0046:fixme:advapi:OpenBackupEventLogW (L"IDontExist",L"idontexist.evt") stub
0046:fixme:advapi:OpenBackupEventLogW Remote server not supported
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
eventlog.c:546: Tests skipped: We don't have a backup eventlog to work with
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x0,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x22fa88,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x0,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000000,0x0,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000000,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000000,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000001,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000002,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x0000000d,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x0000000e,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000007,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa84) stub
eventlog.c:479: Tests skipped: No records in the 'Application' log
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:ClearEventLogW (0x0,(null)) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup2.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,(null)) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0020,0x0000,0x00000000,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000005,0x00000000,0x1b1e0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ClearEventLogW (0xcafe4242,(null)) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"Wine"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"Wine"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0004,0x0001,0x00000001,0x0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0004,0x0001,0x00000001,0x0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0001,0x00000002,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc1"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc1"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0010,0x0001,0x00000003,0x0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0010,0x0001,0x00000003,0x0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc20") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0001,0x0001,0x00000004,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc300"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc300"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0001,0x00000005,0x0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0002,0x0001,0x00000005,0x0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0000,0x0002,0x00000006,0x1b3d0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0000,0x0002,0x00000006,0x1b3d0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0010,0x0002,0x00000007,0x1b3d0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc1") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0008,0x0002,0x00000008,0x1b3d0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0008,0x0002,0x00000008,0x1b3d0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc20"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc20"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0002,0x00000009,0x1b3d0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc300") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0001,0x0002,0x0000000a,0x1b3d0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0001,0x0002,0x0000000a,0x1b3d0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:err:eventlog:ReportEventW L"First string"
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
eventlog.c:889: Tests skipped: No events were written to the eventlog
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "this name is too long", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x0) stub
0046:fixme:advapi:StartTraceA (0x0, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:ControlTraceA (cafe4242, "wine", 0x1b1e0, 1) stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so lsa && touch lsa.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
004a:fixme:advapi:LsaOpenPolicy ((null),0x22f960,0x000f0fff,0x22f930) stub
004a:fixme:security:GetWindowsAccountDomainSid (000000000022F690 0000000000019078 000000000022F68C): semi-stub
004a:fixme:advapi:LsaEnumerateAccountRights (0xcafe,0x22fa20,0x22f990,0x22f92c) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb10,0x000f0fff,0x22fae8) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb40,0x00000800,0x22fb00) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb40,0x00000800,0x22fb08) stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so registry && touch registry.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
registry.c:2850: Tests skipped: HKCR key merging not supported
registry.c:3146: Tests skipped: HKCR key merging not supported
004e:fixme:winspool:PerfOpen (null): stub
004e:fixme:winspool:PerfCollect L"Global", 0x22ead8, 0x22eabc, 0x22eac0: stub
004e:fixme:winspool:PerfClose stub
004e:fixme:winspool:PerfOpen (null): stub
004e:fixme:winspool:PerfCollect L"invalid counter name", 0x22ead8, 0x22eabc, 0x22eac0: stub
004e:fixme:winspool:PerfClose stub
registry.c:4032: Test failed: [ 9] expected 1168, got 2
registry.c:4032: Test failed: [10] expected 1814, got 2
*** Error code 2
Stop.
make[1]: stopped in /usr/pkgsrc/wip/wine64-dev/work/wine-4.13/wine64/dlls/advapi32/tests
*** Error code 1
Stop.
make: stopped in /usr/pkgsrc/wip/wine64-dev/work/wine-4.13/wine64
</pre>
</p>
<h3>Running programs on Wine</h3>
<p>
You can find obligatory screenshots of Wine-4.4/4.13 on amd64/i386
below:
</p>
<div style="font-size:80%; text-align:center">
<img alt="010 Editor (Professional Text/Hex Editor) on Wine-4.13 (amd64)" src="//www.NetBSD.org/~leot/blog-posts/imgs/wine_010editor.png" width="800" />
010 Editor (Professional Text/Hex Editor) on Wine-4.13 (amd64)</div>
<br />
<div style="font-size:80%; text-align:center">
<img alt="Notepad++ on Wine-4.4 (i386)" src="//www.NetBSD.org/~leot/blog-posts/imgs/wine_notepadpp.png" width="800" />
Notepad++ on Wine-4.4 (i386)</div>
<br />
<div style="font-size:80%; text-align:center">
<img alt="Pinball on Wine-4.13 (amd64)" src="//www.NetBSD.org/~leot/blog-posts/imgs/wine_pinball.png" width="800">
Pinball on Wine-4.13 (amd64)</div>
<h3>How to run Wine on NetBSD/amd64</h3>
<ul style="list-style-type:square;">
<li>Compile kernel with <strong>USER_LDT</strong> enabled.</li>
<li>Install kernel.</li>
<li>Clone wip repo.</li>
<li><code>cd /usr/pkgsrc/wip/wine64; make install</code> (for 4.4)</li> or
<li><code>cd /usr/pkgsrc/wip/wine64-dev; make install</code> (for 4.13)</li>
<li><code>wine notepad</code></li>
</ul>
<h3>Future Plans</h3>
<p>
Wine requires the kernel option
<a href="//mail-index.NetBSD.org/tech-kern/2017/02/05/msg021545.html">USER_LDT</a>
to be able to run 32-bit applications on amd64 - facilitated by
<strong>WoW64</strong>. Presently, this feature isn't enabled by
default on NetBSD and hence, the kernel has to be compiled with
<code>USER_LDT</code> enabled. I will work on getting the tests to pass and
finding a better approach to deal with <code>LD_LIBRARY_PATH</code> issue.
In addition to that, I shall work on incorporating a NetBSD VM to Wine
Test Bot infrastructure so we can preclude Wine from getting out of
shape on NetBSD in the future (work in progress).
</p>
<h3>Summary</h3>
<p>
Presently, Wine-4.4 and Wine-4.13 are working fine on amd64/i386. I
have been able to meet all the goals as per my original plan. I
would like to attribute the success to the support provided by my
mentors @leot, @maya and @christos. I would also like to thank my
mentor @maxv for solving the conflict between <code>SVS</code> and <code>USER_LDT</code>
kernel options. I intend to maintain the packages
<code>wine64</code>/<code>wine64-dev</code>/</code>wine32</code> for the foreseeable future. As stated
above, I shall try to set up a NetBSD VM as Wine Test Bot for
purpose of CI. Once again, thanks to Google for enabling me to do
what I love.
</p>
<h3>Source Code</h3>
<p>
<li><a href="https://source.winehq.org/git/wine.git/commit/de5392bfe7919c180d6650eec7b57b5386a4796f">Commit:
Wine</a></li>
<li><a href="https://wip.pkgsrc.org/cgi-bin/gitweb.cgi?p=pkgsrc-wip.git;a=search;h=311c305bf8afadbf45ffd9922a14a840e0b72a44;s=Naveen+Narayanan;st=author
">Commit:
pkgsrc WIP</a></li>
<li><a href="http://pkgsrc.se/wip/wine64">Package: Wine64</a></li>
<li><a href="http://pkgsrc.se/wip/wine64-dev">Package: Wine64-dev</a></li>
<li><a href="http://pkgsrc.se/wip/wine32">Package: Wine32</a></li>
</p>
https://blog.netbsd.org/tnf/entry/porting_wine_to_amd64_on2Porting wine to amd64 on NetBSD, second evaluation reportLeonardo Taccari2019-08-01T11:48:34+00:002019-08-01T11:48:34+00:00<p>
This report was written by Naveen Narayanan as part of Google Summer of Code 2019.
</p>
<p>
This report encompasses the progress of the project during the
second coding period.
</p>
<p>
As getting Wine to work with WoW64 support was of foremost
importance, my focus was on compat32 dependency packages without
which Wine's functionality would be limited and more importantly
untestable. Initially, being unaware of what to expect, I just
wanted Wine to run, at the earliest. So, with outmost support from
mentors, the consensus was to install libs from 32-bit packages to
<code>${PREFIX}/lib/32</code> and ignore everything else that came with the
respective packages.
</p><p>
This report was written by Naveen Narayanan as part of Google Summer of Code 2019.
</p>
<p>
This report encompasses the progress of the project during the
second coding period.
</p>
<h2>WINE on amd64</h2>
<p>
As getting Wine to work with WoW64 support was of foremost
importance, my focus was on compat32 dependency packages without
which Wine's functionality would be limited and more importantly
untestable. Initially, being unaware of what to expect, I just
wanted Wine to run, at the earliest. So, with outmost support from
mentors, the consensus was to install libs from 32-bit packages to
<code>${PREFIX}/lib/32</code> and ignore everything else that came with the
respective packages.
</p>
<p>
I had most of the compat32 packages ready after a couple of
days. And it was time we gave Wine a whirl. Well, the build was
successful. However, I had problems with 32-bit Xorg. The
applications which came packaged with Wine worked fine, but, other
Microsoft Windows applications like notepad++, Mario etc had a
hard time running. Additionally, I noticed that fontconfig went
wild and crashed spewing errors symptomatic of Wine (32-bit) not
playing nice with the fontconfig lib that came with 32-bit Xorg
package. On top of this, I found that build failed on certain
machines due to unavailability of headers. This made us reconsider
our decision to install 32-bit libs to <code>${PREFIX}/lib/32</code> and ignore
everything else which included headers and binaries.
</p>
<p>
Ultimately, we realized that it was time we found a proper
solution to the problem of where 32-bit packages should be
installed on NetBSD amd64 and then, we eventually settled on
<code>${PREFIX}/emul/netbsd32/</code>. It seemed logical, and I
got on with the task of adapting the respective Makefiles of
dependency packages to install
to <code>${PREFIX}/emul/netbsd32/</code>. Additionally, I packaged
fontconfig (32-bit) in high hopes that Wine would behave
appropriately with the former. Wine build was successful. However,
I noticed that Wine wasn't linking against the fontconfig libs
from my package but against the ones which came with 32-bit Xorg
package. Later, I realized, after consulting with mentors, that
pkgsrc doesn't search for pkgconfig files (*.pc)
in <code>${PREFIX}/emul/netbsd32/lib</code> by default. pkgsrc
sets
<a href="https://github.com/NetBSD/pkgsrc/blob/trunk/mk/tools/pkg-config.mk">
_PKG_CONFIG_LIBDIR</a> appropriately based
on <code>${LIBABISUFFIX}</code>. As you can see, it doesn't search
for .pc files in <code>${PREFIX}/emul/netbsd32/lib</code> and
hence, pkgconfig couldn't get the right flags for fontconfig
package. On the other hand, Wine configure script found fontconfig
libs provided by 32-bit Xorg package against which it linked
wine. A kludgy workaround was to use configure args for the
respective libs thereby, obviating configure from finding the
wrong libs. Finally, with the help of aforementioned kludgy
workarounds, we were able to build Wine and successfully run Mario
and Lua (32-bit binaries).
</p>
<img src="//www.NetBSD.org/~leot/blog-posts/imgs/wine_smbx.png" alt="Running smbx.exe and Lua installer in wine on NetBSD/amd64" height="1000" width="800" />
<br />
<img src="//www.NetBSD.org/~leot/blog-posts/imgs/wine_lua.png" alt="Running SciTE and Lua in wine on NetBSD/amd64" height="1000" width="800" />
<h3>How to run Wine on NetBSD -current</h3>
<ul style="list-style-type:square;">
<li> Build -current distribution. </li>
<li> Compile -current kernel with <strong>USER_LDT</strong> enabled and <strong>SVS</strong>
disabled. </li>
<li> Install kernel and distribution. </li>
<li> Clone wip repo. </li>
<li> <code>cd /usr/pkgsrc/wip/wine64; make install</code> </li>
<li> <code>export LD_LIBRARY_PATH=/usr/pkg/emul/netbsd32/lib; wine
mario.exe</code> </li>
</ul>
<h3>Future Plans</h3>
<p>
Wine requires the kernel option
<a href="//mail-index.NetBSD.org/tech-kern/2017/02/05/msg021545.html">USER_LDT</a>
to be able to run 32-bit applications on amd64 - facilitated by
<strong>WoW64</strong>. Presently, this feature isn't enabled by default on NetBSD
and hence, the kernel has to be compiled with USER_LDT enabled. This
also warrants the kernel option SVS to be disabled currently owing
to compatability issues. Work is being done to resolve the conflict
which would apparently allow USER_LDT to be enabled by default in
the GENERIC kernel.
</p>
<p>
<a href="https://github.com/NetBSD/pkgsrc/blob/trunk/mk/tools/pkg-config.mk">pkgsrc</a>
can be made to search for pkgconf config files in
<code>${PREFIX}/emul/netbsd32/lib</code> by setting
_PKG_CONFIG_LIBDIR to
<code>${BUILDLINK_DIR}${LIBABIPREFIX}/lib${LIBABISUFFIX}</code> as
per @maya's suggestion. It might take some more thought and time
before we finalize it.
</p>
<h3>Summary</h3>
<p>
Presently, Wine on amd64 is in test phase. It seems to work fine
with caveats like LD_LIBRARY_PATH which has to be set as 32-bit
Xorg libs don't have <code>${PREFIX}/emul/netbsd32/lib</code> in its rpath
section. The latter is due to us extracting 32-bit libs from
tarballs in lieu of building 32-bit Xorg on amd64. As previously
stated, pkgsrc doesn't search for pkgconfig files in
<code>${PREFIX}/emul/netbsd32/lib</code> which might have inadvertent effects
that I am unaware of as of now. I shall be working on these issues
during the final coding period. I would like to thank @leot, @maya
and @christos for saving me from shooting myself in the foot
many a time. I, admittedly, have had times when multiple
approaches, which all seemed right at that time, perplexed me. I
believe those are times when having a mentor counts, and I have
been lucky enough to have really good ones. Once again, thanks to
Google for this wonderful opportunity.
</p>https://blog.netbsd.org/tnf/entry/implementation_of_drm_ioctl_supportImplementation of DRM ioctl Support for NetBSD kernelChristos Zoulas2019-07-08T18:33:04+00:002019-07-08T18:33:04+00:00<p>
This report was prepared by Surya P as a part of Google Summer of Code 2019
</p>
<p>
Enabling support of DRM ioctls in linux emulation.
</p><p>
This report was prepared by Surya P as a part of Google Summer of Code 2019
</p>
<h2>
What is DRM ioctl ?
</h2>
<p>
Ioctls are input/output control system calls and DRM stands for direct rendering manager The DRM layer provides several services to graphics drivers, many of them driven by the application interfaces it provides through libdrm, the library that wraps most of the DRM ioctls. These include vblank event handling, memory management, output management, framebuffer management, command submission & fencing, suspend/resume support, and DMA services.
</p>
<h2>
Native DRM ioctl calls
</h2>
<p>
NetBSD was able to make native DRM ioctl calls with hardware rendering once xorg and proper mesa packages where installed. We used the glxinfo and glxgears applications to test this out.
</p>
<img alt="X desktop glxgears" src="//www.NetBSD.org/~christos/blog-posts/gsoc/2019/drm/image2.png" />
<h2>
DRM ioctl calls from emulation
</h2>
<p>
In order to make sure DRM ioctl calls where also made from the linux emulation layer of NetBSD . We used rpm and suse131 packages
In specific base,compat,X11,libdrm,libglx,libexpat packages where used.
To my surprise the applications kept segfaulting . I used glxgears and glxinfo rpm packages for this test .when I analyzed the segfault and traced the process , I was able to identify the cause of the segfault which was caused due to broken suse131 libdrm package which did not support nouveau based cards.
To further make user that the problem was with the suse packages , I downgraded to suse121 and as expected the glxinfo and glxgears rpm packages ran, but it was using software rendering instead of hardware rendering but nevertheless we were still able to see the DRM ioctl calls made by the emulation layer hence we added some print statements in the kernel source to identify the calls made.
</p>
<img alt="Bash shell showing ioctls" src="//www.NetBSD.org/~christos/blog-posts/gsoc/2019/drm/image1.png" />
<h2>
Summary
</h2>
<p>
Fixing the Suse131 package and enabling hardware rendering from emulation is of highest priority , I have also planned to port steam and its dependencies to NetBSD to incorporate some gaming on NetBSD! And finally conversion between 32bit DRM ioctl calls 64bit DRM ioctl calls will be implemented.
</p>
<p>
Last but not the least I would like to thank my mentor @christos , @maya , @leot for helping me out and guiding me throughout the process and Google for providing me with such a wonderful opportunity to work with NetBSD community.
</p>
https://blog.netbsd.org/tnf/entry/porting_netbsd_to_hummingboard_pulsePorting NetBSD to HummingBoard Pulse, Part 1jmcneill2019-06-30T13:05:27+00:002019-06-30T13:25:26+00:00<p>
<i>This report was written by Saurav Prakash as part of Google Summer of Code 2019.</i>
</p>
<p>
My venture into the first phase of The Google Summer of Code is nearing an end. The experience was enriching in every dimension, and the learning exposure I was subjected to was genuinely worthwhile. Here is a brief report on the work I have performed during this coding period.
</p><p>
<i>This report was written by Saurav Prakash as part of Google Summer of Code 2019.</i>
</p>
<p>
My venture into the first phase of The Google Summer of Code is nearing an end. The experience was enriching in every dimension, and the learning exposure I was subjected to was genuinely worthwhile. Here is a brief report on the work I have performed during this coding period.
</p>
<h2>About HummingBoard Pulse</h2>
<p>
<a href="https://www.solid-run.com/nxp-family/hummingboard-pulse/">HummingBoard Pulse</a> is an arm64 board developed by <a href="https://www.solid-run.com">SolidRun</a>, with iMX8 System-on-Chip, Dual/Quad core Cortex A53 processor. This project involves extending the GENERIC64 kernel to support iMX8 SoC.
</p>
<h2>Device Tree Files</h2>
<p>
We compare the compatible property of the root node in the device tree with the list of ARM platforms compiled into the kernel. We add the .dts file for <a href="https://github.com/SauravPrakash98/netbsd-src/blob/build_1/sys/arch/arm/dts/imx8mq-hummingboard-pulse.dts">imx8</a> which is compatible with the imx8mq.dtsi file adopted from the Linux mainline 5.1.4.
At this early stage, nodes for uart, iomux and clock only were created.
</p>
<h2>Board Platform Code</h2>
<p>
The platform code provides SoC specific <a href="https://github.com/SauravPrakash98/netbsd-src/blob/build_1/sys/arch/arm/nxp/imx8_platform.c">code</a> needed early at boot. The arm_platform structure for imx8 is initialised here. It contains function pointers like
.ap_attach_init_args,
.ap_device_register,
.ap_reset,
.ap_delay,
.ap_uart_freq,
.ap_mpstart
and a, platform_early_putchar_function so that we can send a character to UART.
</p>
<h2>Clock Driver</h2>
<p>
During the booting stage we only need to enable the uart clocks (IMX8MQ_CLK_UART*_ROOT) and its parents. This includes writing drivers for
fixed clocks (OSC_25M, OSC_27M, OSC_32K, and more),
fixed-factor clocks,
divider clocks,
mux clocks,
composite clocks (IMX8MQ_CLK_UART*),
gate clocks (IMX8MQ_CLK_UART*_ROOT),
sccg-pll clocks,
and frac-pll clocks.
</p>
<h2>UART Driver</h2>
<p>
The imx drivers are separated between "core driver" and "bus glue". <arch/arm/imx/imxuart.c> is a "core driver", and we need to write a fdt based <a href="https://github.com/SauravPrakash98/netbsd-src/blob/build_1/sys/arch/arm/nxp/imx8_uart.c">"bus glue"</a> for the custom bus, this non-fdt based imx code currently uses.
</p>
You can checkout the code from <a href="https://github.com/SauravPrakash98/netbsd-src/tree/build_1">here</a>.
<h2>What's Next</h2>
<p>
The sccg-pll, frac-pll, composite clocks aren't complete yet. So the focus in the next weeks would be to complete the unfinished work as required for the board to be booting till root. Finalising the UART driver code. Modifying the drivers written now, to suit the needs later. And then moving on further to write iomux driver, and then a USB driver for a storage device to boot from.
</p>
<p>
Lastly I would like to thank my mentors, @jmcneill, @martin @Cryo for being a constant support and non-flickering guidance throughout the process.
</p>https://blog.netbsd.org/tnf/entry/porting_wine_to_amd64_onPorting Wine to amd64 on NetBSD, first evaluation reportLeonardo Taccari2019-06-30T11:31:31+00:002019-06-30T11:31:31+00:00<p>
This report was written by Naveen Narayanan as part of Google Summer of Code 2019.
</p>
<p>
I have been working on porting Wine to amd64 on NetBSD as a GSoC
2019 project.
<a href="https://www.winehq.org/">Wine</a>
is a compatibility layer which allows running
Microsoft Windows applications on POSIX-complaint operating
systems. This report provides an overview of the progress of the
project during the first coding period.
</p><p>
This report was written by Naveen Narayanan as part of Google Summer of Code 2019.
</p>
<p>
I have been working on porting Wine to amd64 on NetBSD as a GSoC
2019 project.
<a href="https://www.winehq.org/">Wine</a>
is a compatibility layer which allows running
Microsoft Windows applications on POSIX-complaint operating
systems. This report provides an overview of the progress of the
project during the first coding period.
</p>
<h2>WINE on i386</h2>
<p>
Initially, when I started working on getting Wine-4.4 to build and
run on NetBSD i386 the primary issue that I faced was Wine
displaying black windows instead of UI, and this applied to any
graphical program I tried running with Wine.
</p>
<p>
I suspected it , as it is related to graphics, to be
an issue with the graphics driver or Xorg. Subsequently, I tried
building modular Xorg, and I tried running Wine on it only to
realize that Xorg being modular didn't affect it in the least.
After having tried a couple of configurations, I realized that trying
to hazard out every other probability is going to take an awful lot
of time that I didn't have. This motivated me to bisect the
repo using git, and find the first version of Wine which failed on NetBSD.
</p>
<p>
I started with the last version of Wine that worked on NetBSD which
is 1.9.18. After two days of git bisect, I had the culprit. It was a
commit that introduced a single clipboard manager thread per window
station in Wine. Later, I found that
<a href="//man.NetBSD.org/pthread_create.3">pthread_create(3)</a>
calls were borked
and didn't return anything. I tried to walk through the code to
know where in pthreads lib was the program getting stuck
at. Finally, I got a segfault.
</p>
<p>
I realized the address at which it segfaulted was identical to the
address at which Wine has been throwing an unhandled page
fault. I had a hunch that these two issues were some how correlated.
After learning more about memory mapping and /proc, I was sure black
windows was an effect of this unhandled page fault.
</p>
<p>
Eventually, I found that
<a href="//man.NetBSD.org/pthread_attr_setstack.3">pthread_attr_setstack(3)</a>
was setting the guard
size to 65536 bytes even though the man page said otherwise. And
Wine relied on it not being set. This resulted in out-of-bound
access which caused the unhandled page fault. After setting the
guard size to 0 using
<a href="//man.NetBSD.org/pthread_attr_setguardsize.3">pthread_attr_setguardsize(3)</a>,
Wine started
behaving fine.
</p>
<p>
I discussed about the patch with Wine devs, and they were happy to
upstream it as long as it didn't cause any inadvertent issues on
other platforms. Of course, who wouldn't want to play mario now?
</p>
<p>
<img alt="Screenshot of mario running under wine" src="//www.NetBSD.org/~leot/blog-posts/imgs/wine_mario.png" height="800" width="800" />
</p>
<h2>WINE on amd64</h2>
<p>
Compiling Wine with 32 bit support is a bit tricky on amd64.
I proceeded with the chroot approach as I wanted to know if Wine
worked as promised on NetBSD, and if it didn't, to what degree,
it required patching. So, as the first step, I compiled Wine on
amd64 and it ran fine. The subsequent step was to build a chroot
environment for i386. I corresponded with my mentors to learn
more about compat32_netbsd and successfully built an i386 chroot.
I had to compile Wine twice under i386 to have it inject the 32 bit
components to 64 bit Wine for WoW64 support. And to my amazement, it
worked!
</p>
<p>
<img alt="Screenshot of notepad running under wine32" src="//www.NetBSD.org/~leot/blog-posts/imgs/wine32.png" height="800" width="800" />
</p>
<h2>Summary</h2>
<p>
I think Wine-4.4 is in pretty good shape as of right now, but
packaging it is tricky especially since chroot isn't an option, as it
is privileged. I have been writing compat32 packages for dependencies
of Wine to have them crosscompiled on amd64. I shall be working on
getting Wine crosscompiled for 32 bit support on amd64 during the next
coding period.
On a different note and a very important one, I would like to thank my
mentors @christos, @leot, @maya, and @maxv for their valuable
suggestions. From where I see it, I wouldn't have reached this phase if
it weren't for them. I would also like to thank @kamil for his
feedback and input.
</p>https://blog.netbsd.org/tnf/entry/lldb_from_trunk_is_runningLLDB from trunk is running on NetBSD once again!Michał Górny2019-03-02T18:52:08+00:002019-03-02T18:52:08+00:00<p>Upstream describes LLDB as <cite>a next generation, high-performance debugger</cite>.
It is built on top of LLVM/Clang toolchain, and features great
integration with it. At the moment, it primarily supports debugging C,
C++ and ObjC code, and there is interest in extending it to more
languages.</p>
<p>Originally, LLDB was ported to NetBSD by Kamil Rytarowski. However,
multiple upstream changes and lack of continuous testing have resulted
in decline of support. So far we haven't been able to restore
the previous state.</p>
<p>In February, I have started working on LLDB, as contracted by the NetBSD
Foundation. LLDB used to work on NetBSD before but the support recently
regressed. Therefore, my four first goals as detailed in the <a class="reference external" href="http://blog.netbsd.org/tnf/entry/final_report_on_clang_lld#future-plans-lldb">previous
report</a> were:</p>
<blockquote>
<ol class="arabic simple">
<li><p>Restore tracing in LLDB for NetBSD (i386/amd64/aarch64) for
single-threaded applications.</p></li>
<li><p>Restore execution of LLDB regression tests, unless there is need
for a significant LLDB or kernel work, mark detected bugs as failing
or unsupported ones.</p></li>
<li><p>Enable execution of LLDB regression tests on the buildbot in order
to catch regressions.</p></li>
<li><p>Upstream NetBSD (i386/amd64) core(5) support. Develop LLDB
regression tests (and the testing framework enhancement)
as requested by upstream.</p></li>
</ol>
</blockquote>
<p>Of those tasks, I consider running regression tests on the buildbot
the highest priority. Bisecting regressions post-factum is hard due
to long build times, and having continuous integration working is going
to be very helpful to maintaining the code long-term.</p>
<p>In this report, I'd like to summarize what I achieved and what technical
difficulties I met.</p><p>Upstream describes LLDB as <cite>a next generation, high-performance debugger</cite>.
It is built on top of LLVM/Clang toolchain, and features great
integration with it. At the moment, it primarily supports debugging C,
C++ and ObjC code, and there is interest in extending it to more
languages.</p>
<p>Originally, LLDB was ported to NetBSD by Kamil Rytarowski. However,
multiple upstream changes and lack of continuous testing have resulted
in decline of support. So far we haven't been able to restore
the previous state.</p>
<p>In February, I have started working on LLDB, as contracted by the NetBSD
Foundation. My four first goals as detailed in the <a class="reference external" href="http://blog.netbsd.org/tnf/entry/final_report_on_clang_lld#future-plans-lldb">previous
report</a> were:</p>
<blockquote>
<ol class="arabic simple">
<li><p>Restore tracing in LLDB for NetBSD (i386/amd64/aarch64) for
single-threaded applications.</p></li>
<li><p>Restore execution of LLDB regression tests, unless there is need
for a significant LLDB or kernel work, mark detected bugs as failing
or unsupported ones.</p></li>
<li><p>Enable execution of LLDB regression tests on the buildbot in order
to catch regressions.</p></li>
<li><p>Upstream NetBSD (i386/amd64) core(5) support. Develop LLDB
regression tests (and the testing framework enhancement)
as requested by upstream.</p></li>
</ol>
</blockquote>
<p>Of those tasks, I consider running regression tests on the buildbot
the highest priority. Bisecting regressions post-factum is hard due
to long build times, and having continuous integration working is going
to be very helpful to maintaining the code long-term.</p>
<p>In this report, I'd like to summarize what I achieved and what technical
difficulties I met.</p>
<div class="section" id="the-kqueue-interoperability-issues">
<h2>The kqueue interoperability issues</h2>
<p>Given no specific clue as to why LLDB was no longer able to start
processes on NetBSD, I've decided to start by establishing the status of
the test suites. More specifically, I've started with a small subset of
LLDB test suite — unittests. In this section, I'd like to focus on two
important issues I had with them.</p>
<p>Firstly, one of the tests was hanging indefinitely. As I established,
the purpose of the test was to check whether the main loop
implementation correctly detects and reports when all the slaves
of a pty are disconnected (and therefore the reads on master would
fail). Through debugging, I've came to the conclusion that <span class="docutils literal">kevent()</span>
is not reporting this particular scenario.</p>
<p>I have built a simple test case (which is now <a class="reference external" href="http://cvsweb.netbsd.org/bsdweb.cgi/src/tests/kernel/kqueue/read/t_ttypty.c.diff?r1=1.2&r2=1.3&only_with_tag=MAIN">part of kqueue ATF
tests</a>) and confirmed it. Afterwards, I have attempted to establish
whether this behavior is correct. While <cite>kqueue(2)</cite> does not mention
ptys specifically, it states the following for pipes:</p>
<blockquote>
<dl>
<dt>Fifos, Pipes</dt>
<dd><p>Returns when there is data to read; data contains the
number of bytes available.</p>
<p>When the last writer disconnects, the filter will set
EV_EOF in flags. This may be cleared by passing in
EV_CLEAR, at which point the filter will resume
waiting for data to become available before returning.</p>
</dd>
</dl>
</blockquote>
<p>Furthermore, my test program indicated that FreeBSD exhibits
the described EV_EOF behavior. Therefore, I have decided to write
a kernel patch adding this functionality, submitted it to review
and eventually committed it after applying helpful suggestions from
Robert Elz (<a class="reference external" href="http://mail-index.netbsd.org/tech-kern/2019/02/07/msg024616.html">[PATCH v3] kern/tty_pty: Fix reporting EOF via kevent and
add a test case</a>).
I have also disabled the test case temporarily since the functionality
is non-critical to LLDB (<a class="reference external" href="https://github.com/llvm/llvm-project/commit/01486b22bb62a4020b39001df25442fa2b1d2228">r353545</a>).</p>
<p>Secondly, a few gdbserver-based tests were flaky — i.e. unpredictably
passed and failed every iteration. I've started debugging this with a test
whose purpose was to check verbose error messages support
in the protocol. To my surprise, it seemed as if gdbserver worked fine
as far as error message exchange was concerned. This packet was
followed by a termination request from client — and it seemed that
the server sometimes replies to it correctly, and sometimes terminates
just before receiving it.</p>
<p>While working on this particular issue, I've noticed a few deficiencies
in LLDB's error handling. In this case, this involved two major issues:</p>
<ol class="arabic simple">
<li><p>gdbserver ignored errors from main loop. As a result, if
<span class="docutils literal">kevent()</span> failed, it silently exited with a successful status.
I've fixed it to catch and report the error verbosely instead:
<a class="reference external" href="https://github.com/llvm/llvm-project/commit/2d874e53561c749fde6efa51650bb0c49d89851a">r354030</a>.</p></li>
<li><p>Main loop reported meaningless return value (<cite>-1</cite>) from <span class="docutils literal">kevent()</span>.
I've established that most likely all <span class="docutils literal">kevent()</span> implementation use
<cite>errno</cite> instead, and made the function return it:
<a class="reference external" href="https://github.com/llvm/llvm-project/commit/c23f82c026049d574eef73af3b85e63598d9b0a3">r354029</a>.</p></li>
</ol>
<p>After applying those two fixes, gdbserver clearly indicated the problem:
<span class="docutils literal">kevent()</span> returned due to <cite>EINTR</cite> (i.e. the process receiving
a signal). Lacking correct handling for this value, the main loop
implementation wrongly treated it as fatal error and terminated
the program. I've fixed this via implementing <cite>EINTR</cite> support for
<span class="docutils literal">kevent()</span> in <a class="reference external" href="https://github.com/llvm/llvm-project/commit/53eabaab3f59ca9b32a3775f1cb7bf51717fa354">r354122</a>.</p>
<p>This trivial fix not only resolved most of the flaky tests but also
turned out to be the root cause for LLDB being unable to start
processes. Therefore, at this point tracing for single-threaded
processes was restored on amd64. Testing on other platforms is pending.</p>
<p>Now, for the moral: working error reporting can save a lot of time.</p>
</div>
<div class="section" id="socket-issues">
<h2>Socket issues</h2>
<p>The next issue I hit while working on the unittests is rather curious,
and I have to admit I haven't managed to neither find the root cause
or build a good reproducer for it. Nevertheless, I seem to have caught
the gist of it and found a good workaround.</p>
<p>The test in question focuses on the high-level socket API in LLDB. It
is rather trivial — it binds a server in one thread, and tries to
connect to it from a second thread. So far, so good. Most of the time
the test works just fine. However, sometimes — especially early after
booting — it hangs forever.</p>
<p>I've debugged this thoroughly and came to the following conclusion:
the test binds to <span class="docutils literal">127.0.0.1</span> (i.e. purely IPv4) but tries to connect
to <span class="docutils literal">localhost</span>. The latter results in the client trying IPv6 first,
failing and then succeeding with IPv4. The connection is accepted,
the test case moves forward and terminates successfully.</p>
<p>Now, in the failing case, the IPv6 connection attempt succeeds, even
though there is no server bound to that port. As a result, the client
part is happily connected to a non-existing service, and the server part
hangs forever waiting for the connection to come.</p>
<p>I have attempted to reproduce this with an isolated test case,
reproducing the use of threads, binding to port zero, the IPv4/IPv6
mixup and I simply haven't been able to reproduce this. However,
curiously enough my test case actually <em>fixes</em> the problem. I mean, if I
start my test case before LLDB unit tests, they work fine afterwards
(until next reboot).</p>
<p>Being unable to make any further progress on this weird behavior, I've
decided to fix the test design instead — and make it connect to the same
address it binds to:
<a class="reference external" href="https://github.com/llvm/llvm-project/commit/f6e5594e811c70986ad8af3aa3f9540fcb7a1149">r353868</a>.</p>
</div>
<div class="section" id="getting-the-right-toolchain-for-live-testing">
<h2>Getting the right toolchain for live testing</h2>
<p>The largest problem so far was getting LLDB tests to interoperate with
NetBSD's clang driver correctly. On other systems, clang either
defaults to libstdc++, or has libc++ installed as part of the system
(FreeBSD, Darwin). The NetBSD driver wants to use libc++
but we do not have it installed by default.</p>
<p>While this could be solved via installing libc++ on the buildbot host,
I thought it would be better to establish a solution that would allow
LLDB to use just-built clang — similarly to how other LLVM projects
(such as OpenMP) do. This way, we would be testing the matching libc++
revision and users would be able to run the tests in a single checkout
out of the box.</p>
<p>Sadly, this is non-trivial. While it could be all hacked into
the driver itself, it does not really belong there. While it is
reasonable to link tests into the build tree, we wouldn't want regular
executables built by user to bind to it. This is why normally this is
handled via the test system. However, the tests in LLDB are
an accumulation of at least three different test systems, each one calling
the compiler separately.</p>
<p>In order to establish a baseline for this, I have created wrappers for
clang that added the necessary command-line options. The state-of-art
wrapper for clang looked like the following:</p>
<pre class="literal-block">#!/usr/bin/env bash
topdir=/home/mgorny/llvm-project/build-rel-master
cxxinc="-cxx-isystem $topdir/include/c++/v1"
lpath="-L $topdir/lib"
rpath="-Wl,-rpath,$topdir/lib"
pthread="-pthread"
libs="-lunwind"
# needed to handle 'clang -v' correctly
[ $# -eq 1 ] && [ "$1" = -v ] && exec $topdir/bin/clang-9-real "$@"
exec $topdir/bin/clang-9-real $cxxinc $lpath $rpath "$@" $pthread $libs</pre>
<p>The actual executable I renamed to <span class="docutils literal"><span class="pre">clang-9-real</span></span>, and this wrapper
replaced <span class="docutils literal">clang</span> and a similar one replaced <span class="docutils literal">clang++</span>. <span class="docutils literal"><span class="pre">clang-cl</span></span>
was linked to the real executable (as it wasn't
called in wrapper-relevant contexts), while <span class="docutils literal"><span class="pre">clang-9</span></span> was linked
to the wrapper.</p>
<p>After establishing a baseline of working tests, I've looked into
migrating the necessary bits one by one to the driver and/or LLDB test
system, removing the migrated parts and verifying whether tests pass
the same.</p>
<p>My proposal so far involves, appropriately:</p>
<ol class="arabic simple">
<li><p>Replacing <span class="docutils literal"><span class="pre">-cxx-isystem</span></span> with libc++ header search using path
relative the compiler executable:
<a class="reference external" href="https://reviews.llvm.org/D58592">D58592</a>.</p></li>
<li><p>Integrating <span class="docutils literal"><span class="pre">-L</span></span> and <span class="docutils literal"><span class="pre">-Wl,-rpath</span></span> with the LLDB test system:
<a class="reference external" href="https://reviews.llvm.org/D58630">D58630</a>.</p></li>
<li><p>Adding NetBSD to list of platforms needing <span class="docutils literal"><span class="pre">-pthread</span></span>:
<a class="reference external" href="https://github.com/llvm/llvm-project/commit/c10a8848734898970a216aa6d4fd453fe05ae606">r355274</a>.</p></li>
<li><p>The need for <span class="docutils literal"><span class="pre">-lunwind</span></span> is solved via switching the test failing
due to the lack of it to use libc++ instead of libstdc++:
<a class="reference external" href="https://github.com/llvm/llvm-project/commit/8085c1b3c1c93b5027b8dd611733d1e1ab68c4ab">r355273</a>.</p></li>
</ol>
<p>The reason for adjusting libc++ header search in the driver rather than
in LLDB tests is that the path is specific to building against libc++,
and the driver makes it convenient to adjust the path conditionally
to standard C++ library being used. In other words, it saves us from
hard-relying on the assumption that tests will be run against libc++
only.</p>
<p>I've went for integrating <span class="docutils literal"><span class="pre">-L</span></span> in the test system since we do not want
to link arbitrary programs to the libraries in LLVM's build directory.
Appending this path unconditionally should be otherwise harmless to
LLDB's tests, so that is the easier way to go.</p>
<p>Originally I wanted to avoid appending RPATHs. However, it seems that
the <span class="docutils literal">LD_LIBRARY_PATH</span> solution that works for Linux does not reliably
work on NetBSD with LLDB. Therefore, passing <span class="docutils literal"><span class="pre">-Wl,-rpath</span></span> along with
<span class="docutils literal"><span class="pre">-L</span></span> allowed me to solve the problem simpler.</p>
<p>Furthermore, those design solutions match other LLVM projects. I've
mentioned OpenMP before — so far we had to pass <span class="docutils literal"><span class="pre">-cxx-isystem</span></span> to its
tests explicitly but it passed <span class="docutils literal"><span class="pre">-L</span></span> for us. Those patches render
passing <span class="docutils literal"><span class="pre">-cxx-isystem</span></span> unnecessary, and therefore make LLDB follow
the suit of OpenMP.</p>
</div>
<div class="section" id="finishing-touches">
<h2>Finishing touches</h2>
<p>Having a reasonably working compiler and major regressions fixed, I have
focused on establishing a baseline for running tests. The goal is to
mark broken tests XFAIL or skip them. With all tests marked
appropriately, we would be able to start running tests on the buildbot
and catch regressions compared to this baseline. The current progress
on this can be see in <a class="reference external" href="https://reviews.llvm.org/D58527">D58527</a>.</p>
<p>Sadly, besides failing tests there is still a small number of flaky
or hanging tests which are non-trivial to detect. The upstream
maintainer, Pavel Labath is very helpful and I hope to be able to
finally get all the flaky tests either fixed or covered with his help.</p>
<p>Other fixes not worth a separate section include:</p>
<ul class="simple">
<li><p>fixing compiler warnings about empty format strings:
<a class="reference external" href="https://github.com/llvm/llvm-project/commit/42d9cd2d350df895f2e78de9a134a26526cc86cb">r354922</a>,</p></li>
<li><p>fixing two <span class="docutils literal">dlopen()</span> based test cases not to link <span class="docutils literal"><span class="pre">-ldl</span></span>
on NetBSD:
<a class="reference external" href="https://github.com/llvm/llvm-project/commit/65ebfaf0bee85a563b3f88da9ba1108cc3b032fd">r354617</a>,</p></li>
<li><p>finishing Kamil's patch for core file support: <a class="reference external" href="https://github.com/llvm/llvm-project/commit/4f134fb66019d948d1f68d62a92f2a01b22eb7ee">r354466</a>,
followup fix in <a class="reference external" href="https://github.com/llvm/llvm-project/commit/880b38d0bd854af8897f4fffd2bd5c3a19e3d271">r354483</a>,</p></li>
<li><p>removing dead code in main loop:
<a class="reference external" href="https://github.com/llvm/llvm-project/commit/257fcd9b17119455a02f965b95200a286435e942">r354050</a>,</p></li>
<li><p>fixing stand-alone builds after they've been switched to
LLVMConfig.cmake:
<a class="reference external" href="https://github.com/llvm/llvm-project/commit/e47f89cb2cba7f6441b851ac43a22d989ce5bb51">r353925</a>,</p></li>
<li><p>skipping lldb-mi tests when Python support (needed by lldb-mi) is disabled:
<a class="reference external" href="https://github.com/llvm/llvm-project/commit/8780771c51d6ced4998d24f71f75bc6c6e7d2acc">r353700</a>,</p></li>
<li><p>fixing incorrect initialization of <em>sigset_t</em> (not actually used right
now): <a class="reference external" href="https://github.com/llvm/llvm-project/commit/f048d448e0f58ff6b3e5d09cbdc91fc1c97afdc0">r353675</a>.</p></li>
</ul>
</div>
<div class="section" id="buildbot-updates">
<h2>Buildbot updates</h2>
<p>The last part worth mentioning is that the NetBSD LLVM buildbot has seen
some changes. Notably, zorg <a class="reference external" href="https://github.com/llvm/llvm-zorg/commit/c9b663d22ff703fb45e1bde658d1ae4378902b5c">r354820</a>
included:</p>
<ul class="simple">
<li><p>fixing the bot commit filtering to include all projects built,</p></li>
<li><p>renaming the bot to shorter <span class="docutils literal"><span class="pre">netbsd-amd64</span></span>,</p></li>
<li><p>and moving it to <span class="docutils literal">toolchain</span> category.</p></li>
</ul>
<p>One of the most useful functions of buildbot is that it associated every
successive build with new commits. If the build fails, it blames
the authors of those commits and reports the failure to them. However,
for this to work buildbot needs to be aware which projects are being
tested.</p>
<p>Our buildbot configuration has been initially based on one used for
LLDB, and it assumed LLVM, Clang and LLDB are the only projects built
and tested. Over time, we've added additional projects but we failed to
update the buildbot configs appropriately. Finally, with the help of
Jonas Hahnfeld, Pavel Labath and Galina Kistanova we've managed to
update the list and make the bot blame all projects correctly.</p>
<p>While at it, we were suggested to rename the bot. The previous name was
<span class="docutils literal"><span class="pre">lldb-amd64-ninja-netbsd8</span></span>, and others suggested that the developers
may ignore failures in other projects seeing <span class="docutils literal">lldb</span> there. Kamil
Rytarowski also pointed out that the version number confuses users to
believe that we're running separate bots for different versions.
The new name and category mean to clearly indicate that we're running
a single bot instance for multiple projects.</p>
</div>
<div class="section" id="quick-summary-and-future-plans">
<h2>Quick summary and future plans</h2>
<p>At this point, the most important regressions in LLDB have been fixed
and it is able to debug simple programs on amd64 once again. The test
suite patches are still waiting for review, and once they're approved I
still need to work on flaky tests before we can reliably enable that
on the buildbot. This is the first priority.</p>
<p>The next item on the TODO list is to take over and finish <a class="reference external" href="https://reviews.llvm.org/D32149">Kamil's patch
for core files with thread</a>. Most
notably, the patch requires writing tests, and verifying whether there
are no new bugs affecting it.</p>
<p>On a semi-related note, LLVM 8.0.0 will be released in a few days
and I will be probably working on updating src to the new version.
I will also try to convince Joerg to switch from unmaintained libcxxrt
to upstream libc++abi. Kamil also wanted to change libc++ include path
to match upstream (NetBSD is dropping <span class="docutils literal">/v1</span> suffix at the moment).</p>
<p>Once this is done, the next big step is to fix threading support.
Testing on non-amd64 arches is deferred until I gain access to some
hardware.</p>
</div>
<div class="section" id="this-work-is-sponsored-by-the-netbsd-foundation">
<h2>This work is sponsored by The NetBSD Foundation</h2>
<p>The NetBSD Foundation is a non-profit organization and welcomes any
donations to help us continue funding projects and services
to the open-source community. Please consider visiting the following URL
to chip in what you can:</p>
<p><a class="reference external" href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a></p>
https://blog.netbsd.org/tnf/entry/the_netbsd_foundation_participating_inThe NetBSD Foundation participating in Google Summer of Code 2019Leonardo Taccari2019-02-27T16:34:42+00:002019-02-27T16:34:42+00:00<p>
<a href='https://summerofcode.withgoogle.com/'><img alt="Google Summer of Code logo" src='//www.NetBSD.org/~leot/blog-posts/imgs/GSoC-icon-192.png' style="width: 80px; height: 80px; float: right;" /></a>
For the 4th year in a row and for the 13th time <a
href="https://summerofcode.withgoogle.com/organizations/5919342647050240/">The NetBSD Foundation</a>
will participate in
<a href="https://summerofcode.withgoogle.com/">Google Summer of Code 2019</a>!
</p>
<p>
If you are a student and would like to learn more about Google
Summer of Code please go to the
<a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a>
homepage.
</p>
<p>
You can find a list of projects in
<a href="https://wiki.NetBSD.org/projects/gsoc/">Google Summer of Code
project proposals</a> in the <a href="https://wiki.NetBSD.org/">wiki</a>.
</p>
<p>
Do not hesitate to get in touch with us via <a
href="ircs://chat.freenode.net:6697/#netbsd-code">#netbsd-code</a> IRC channel
on <a href="https://freenode.net/">Freenode</a> and via
<a href="https://www.NetBSD.org/mailinglists/">NetBSD mailing lists</a>!
</p>
<p>
Looking forward to have a great summer!
</p>
https://blog.netbsd.org/tnf/entry/announcing_google_summer_of_codeAnnouncing Google Summer of Code 2018 projectsLeonardo Taccari2018-05-01T10:15:52+00:002018-05-01T10:15:52+00:00<p>
<a href='https://summerofcode.withgoogle.com/'><img alt="Google Summer of Code logo" src='//www.NetBSD.org/~leot/blog-posts/imgs/GSoC-icon-192.png' style="width: 80px; height: 80px; float: right;" /></a>
We are very happy to announce
<a href='https://summerofcode.withgoogle.com/organizations/6559863631511552/'>
The NetBSD Foundation Google Summer of Code 2018 projects</a>:
</p>
<ul>
<li>Harsh Khatore - <a href='https://summerofcode.withgoogle.com/projects/#4535962579763200'>Modern cryptographic algorithms to netpgp, netpgpverify</a></li>
<li>Nizar Benshaqi - <a href='https://summerofcode.withgoogle.com/projects/#5611080471019520'>SQL Database for ATF tests results with online query and statistics page</a></li>
<li>Marwa Desouky - <a href='https://summerofcode.withgoogle.com/projects/#5844214449963008'>Tickless Kernel with high-resolution timers</a></li>
<li>Harry Pantazis - <a href='https://summerofcode.withgoogle.com/projects/#5908322608218112'>Kernel Undefined Behavior SANitizer</a></li>
<li>Does025 - <a href='https://summerofcode.withgoogle.com/projects/#6179441747689472'>Porting FreeBSD Atheros driver to NetBSD</a></li>
<li>Saad Mahmood - <a href='https://summerofcode.withgoogle.com/projects/#6292399387574272'>Machine-independent EFI bootloader</a></li>
<li>Yang Zheng - <a href='https://summerofcode.withgoogle.com/projects/#6417656001855488'>Integrate libFuzzer With the Basesystem</a></li>
<li>Keivan Motavalli - <a href='https://summerofcode.withgoogle.com/projects/#6485310259593216'>configuration files versioning in pkgsrc</a></li>
<li>R3x - <a href='https://summerofcode.withgoogle.com/projects/#6738600452947968'>Implementing Kernel Address Sanitizer (KASan) in the NetBSD kernel</a></li>
</ul><p>
<a href='https://summerofcode.withgoogle.com/'><img alt="Google Summer of Code logo" src='//www.NetBSD.org/~leot/blog-posts/imgs/GSoC-icon-192.png' style="width: 80px; height: 80px; float: right;" /></a>
We are very happy to announce
<a href='https://summerofcode.withgoogle.com/organizations/6559863631511552/'>
The NetBSD Foundation Google Summer of Code 2018 projects</a>:
</p>
<ul>
<li>Harsh Khatore - <a href='https://summerofcode.withgoogle.com/projects/#4535962579763200'>Modern cryptographic algorithms to netpgp, netpgpverify</a></li>
<li>Nizar Benshaqi - <a href='https://summerofcode.withgoogle.com/projects/#5611080471019520'>SQL Database for ATF tests results with online query and statistics page</a></li>
<li>Marwa Desouky - <a href='https://summerofcode.withgoogle.com/projects/#5844214449963008'>Tickless Kernel with high-resolution timers</a></li>
<li>Harry Pantazis - <a href='https://summerofcode.withgoogle.com/projects/#5908322608218112'>Kernel Undefined Behavior SANitizer</a></li>
<li>Does025 - <a href='https://summerofcode.withgoogle.com/projects/#6179441747689472'>Porting FreeBSD Atheros driver to NetBSD</a></li>
<li>Saad Mahmood - <a href='https://summerofcode.withgoogle.com/projects/#6292399387574272'>Machine-independent EFI bootloader</a></li>
<li>Yang Zheng - <a href='https://summerofcode.withgoogle.com/projects/#6417656001855488'>Integrate libFuzzer With the Basesystem</a></li>
<li>Keivan Motavalli - <a href='https://summerofcode.withgoogle.com/projects/#6485310259593216'>configuration files versioning in pkgsrc</a></li>
<li>R3x - <a href='https://summerofcode.withgoogle.com/projects/#6738600452947968'>Implementing Kernel Address Sanitizer (KASan) in the NetBSD kernel</a></li>
</ul>
<p>
We have started the community bonding period where students get in touch with
mentors and community and get more confident with documentation and the code
base.
The coding period will start from May 14 until August 6.
</p>
<p>
Please welcome all our students and a big good luck to students and mentors!
</p>
<p>
A big thank you also to <a href='https://www.google.com'>Google</a> and The
NetBSD Foundation Organization Administrators!
</p>
<p>
Looking forward to a nice Google Summer of Code!
</p>https://blog.netbsd.org/tnf/entry/google_summer_of_code_2017Google Summer of Code 2017Thomas Klausner2017-10-18T16:43:40+00:002017-10-18T16:43:40+00:00NetBSD participated in the 2017 edition of <a href="https://summerofcode.withgoogle.com/">Google of Summer of Code</a> with 3 students.
All of the students finished their projects successfully. The following links report about their activities:
<ul>
<li><a href="https://blog.netbsd.org/tnf/entry/gsoc_2017_reports_add_code">Leonardo Taccari: Add multi-packages support to pkgsrc</a>
<li><a href="http://coypu.sdf.org/2017-08-29-LFS">Maya Rashish: LFS cleanup</a>
<li><a href="https://bumblebeesky.tumblr.com/">Utkarsh Anand: Make Anita support multiple virtual machine systems</a>
</ul>
Congratulations to the students for finishing their projects successfully, and thanks to Google for sponsoring!
https://blog.netbsd.org/tnf/entry/announcing_netbsd_and_the_googleAnnouncing NetBSD and the Google Summer of Code Projects 2017Hubert Feyrer2017-05-05T19:16:02+00:002017-05-05T19:16:02+00:00We are very happy to announce that the selection process in this year's Summer of Code with its bargaining of slots and what student gets assigned to which project is over. As a result, the following students will take on their projects:
<p>
<ul>
<li> Leonardo Taccari will work add multi-packages support to pkgsrc.
<li> Maya Rashish will work on the LFS cleanup.
<li> Utkarsh Anand will make Anita support multiple virtual machine systems and more architectures within them to improve testing coverage.
</ul>
What follows now is a community bonding period until May 30th, followed by a coding period over the summer (it's Summer of Code, after all <img src="https://blog.netbsd.org/images/smileys/smile.gif" class="smiley" alt=":-)" title=":-)" />) until August 21st, evaluations, code submission and an announcement of the results on September 6th 2017.
<p>
Good luck to all our students and their mentors - we look forward to your work results, and welcome you to The NetBSD Project!
<p>
https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_buildsNetBSD fully reproducible buildsChristos Zoulas2017-02-20T17:01:18+00:002017-02-20T23:34:04+00:00Today (2017-02-20) NetBSD got our first reproducible build on the <a href=https://tests.reproducible-builds.org/netbsd/netbsd.html>debian build farm</a>. Here's a short description how we got here, what implementation choices we made, and what we had to fix.<h1>Introduction</h1>
<p>
I have been working on and off for almost a year trying to get
reproducible builds (the same source tree always builds an identical cdrom)
on NetBSD. I did not think at the time it would
take as long or be so difficult, so I did not keep a log of all
the changes I needed to make. I was also not the only one working
on this. Other NetBSD developers have been making improvements for
the past 6 years.
<p>I would like to acknowledge the NetBSD <a href="https://nxr.netbsd.org/xref/src/BUILDING">build
system</a> (aka <a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/build.sh">build.sh</a>)
which is a fully portable cross-build system.
This build system has given us a head-start in the reproducible
builds work.
<p>I would also like to acknowledge the work done by the
Debian folks who have provided a
<a href="https://tests.reproducible-builds.org/netbsd/netbsd.html">platform</a> to run, test and analyze
reproducible builds. Special mention to the
<a href="https://diffoscope.org/">diffoscope</a> tool that
gives an excellent overview of what's different between binary
files, by finding out what they are (and if they are containers
what they contain) and then running the appropriate
formatter and diff program to show what's different for each file.
<p>Finally other
developers who have started, motivated and did a lot of work getting
us here like Joerg Sonnenberger and Thomas Klausner for their work
on reproducible builds, and Todd Vierling and Luke Mewburn for their
work on <a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/build.sh">build.sh</a>.
<h1>Sources of difference</h1>
<p>
Here's is what we found that we needed to
fix, how we chose to fix it and why, and where are we now.
<p>
There are many reasons why two separate builds from the same sources
can be different. Here's an (incomplete) list:
<ol>
<li><h3>timestamps</h3>
Many things like to keep track of timestamps, specially archive
formats (<b>tar(1)</b>, <b>ar(1)</b>), filesystems etc. The way to handle each is
different, but the approach is to make them either produce files
with a 0 timestamp (where it does not matter like ar), or with a
specific timestamp when using 0 does not make sense
(it is not useful to the user).</li>
<li><h3>dates/times/authors etc. embedded in source files</h3>
Some programs like to report the date/time they were built, the
author, the system they were built on etc. This can be done either
by programmatically finding and creating source files containing
that information during build time, or by using standard macros such
as __DATE__, __TIME__ etc. Usually putting a constant time or
eliding the information (such as we do with kernels and bootblocks)
solves the problem.</li>
<li><h3>timezone sensitive code</h3>
Certain filesystem formats (iso 9660 etc.) don't store raw timestamps but
formatted times; to achieve this they convert from a timestamp
to localtime, so they are affected by the timezone.</li>
<li><h3>directory order/build order</h3>
The build order is not constant especially in the presence of
parallel builds; neither is directory scan order. If those
are used to create output files, the output files will need
to be sorted so they become consistent.</li>
<li><h3>non-sanitized data stored into files</h3>
Writing data structures into raw files can lead to problems.
Running the same program in different operating systems or
using ASLR makes those issues more obvious.</li>
<li><h3>symbolic links/paths</h3>
Having paths embedded into binaries (specially for debugging
information) can lead to binary differences. Propagation of the
logical path can prove problematic.</li>
<li><h3>general tool inconsistencies</h3>
<b>gcc(1)</b> profiling uses a PROFILE_HOOK macro on RISC targets that
utilizes the "current function" number to produce labels. Processing
order of functions is not guaranteed.
<b>gpt(8)</b> creation involves uuid generation; these are generally random.
block allocation on msdos filesystems had a random component.
<b>makefs(8)</b> uses timezones with timestamps
(<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/usr.sbin/makefs/cd9660/cd9660_conversion.c.diff?r1=1.4&r2=1.5">iso9660</a>), randomness for block selection
(<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/usr.sbin/makefs/ffs/buf.h.diff?r1=1.10&r2=1.11">msdos</a>),
stores stray pointers in superblock
(<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/usr.sbin/makefs/ffs/mkfs.c.diff?r1=1.34&r2=1.35">ffs</a>).</li>
<li><h3>toolchain</h3>
Every program that is used to generate other output needs to have
consistent results. In NetBSD this is done with
<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/build.sh">build.sh</a>, which
builds a set of tools from known sources before it can use those
tools to build the rest of the system). There is a large number of
<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/tools/">tools</a>.
There are also internal issues with the tools that make their output non
reproducible, such as nondeterministic symbol creation or capturing
parts of the environment in debugging information.</li>
<li><h3>build information / tunables / environment</h3>
There are many environment settings, or build variable settings that
can affect the build. This needs to be kept constant across builds
so we've changed the list of variables that are reported in
<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/etc/Makefile.params">
Makefile.params</a>:
<pre>
.if ${MKREPRO:Uno} != "yes"
RELEASEVARS+= BSDOBJDIR BSDSRCDIR BUILDID BUILDINFO BUILDSEED \
DESTDIR KERNARCHDIR KERNCONFDIR KERNOBJDIR KERNSRCDIR MAKE \
MAKEFLAGS NBUILDJOBS NETBSDSRCDIR OBJMACHINE OBJMACHINE_ARCH \
RELEASEDIR RELEASEMACHINEDIR TOOLDIR USR_OBJMACHINE X11SRCDIR
.endif
</pre>
</li>
<li><h3>making sure that the source tree has no local changes</h3></li>
</ol>
<h1>Variables controlling reproducible builds</h1>
<p>
Reproducible builds are controlled on NetBSD with two variables:
<a href="https://nxr.netbsd.org/s?n=25&start=0&q=MKREPRO&sort=relevancy&project=src"><b>MKREPRO</b></a> (which can be set to yes or no) and <a href="https://nxr.netbsd.org/search?q=MKREPRO_TIMESTAMP&project=src"><b>MKREPRO_TIMESTAMP</b></a>
which
is used to set the timestamp of the builds artifacts.
This is usually set to the
number of seconds from the epoch. The
<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/build.sh">build.sh</a>
<b>-P</b> flag handles
reproducible builds automatically: sets the <b>MKREPRO</b> variable to
yes, and then finds the latest source file timestamp in the tree
and sets <b>MKREPRO_TIMESTAMP</b> to that.
<h1>Handling timestamps</h1>
<p>
The first thing that we needed to understand was how
to deal with timestamps. Some of the timestamps are not very useful
(for example inside random ar archives) so we choose to 0 them out.
Others though become annoying if they are all 0. What does it mean
when you mount install media and all the dates on the files are
Jan 1, 1970?
<p>
We decided that a better timestamp would be the timestamp of the
most recently modified file in the source tree. Unfortunately this
was not easy to find on NetBSD, because we are still using CVS as
the source control system, and CVS does not have a good way to
provide that. For that we wrote a tool called
<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/usr.bin/cvslatest">cvslatest</a>, that
scans the CVS metadata files (CVS/Entries) and finds the latest
commit. This works well for freshly checked out trees (since CVS uses
the source timestamp when checking out), but not with updated trees
(because CVS uses the current time when updating files, so that
<b>make(1)</b> thinks they've been modified). To fix that, we've added a
new flag to the <b>cvs(1)</b> "update" command <b>-t</b>, that uses the source
checkout time.
<p>
The build system needs now to evaluate the tree for the latest
file running <b>cvslatest(1)</b> and find the latest timestamp in seconds
from the Epoch which is set in the <b>MKREPRO_TIMESTAMP</b> variable.
This is the same as <a href="https://reproducible-builds.org/specs/source-date-epoch/">
SOURCE_DATE_EPOCH</a>.
Various Makefiles are using this variable and <b>MKRERPO</b> to determine
how to produce consistent build artifacts.
<p>
For example many commands (<b>tar(1)</b>, <b>makefs(8)</b>, <b>gpt(8)</b>, ...) have been modified
to take a <b>--timestamp</b> or <b>-T</b> command line switch to generate
output files that use the given timestamp, instead of the current
time.
<p>
Other software (am-utils, acpica, bootblocks, kernel) used
__DATE__ or __TIME__, or
captured the user, machine, etc. from the environment and
had to be changed to a constant time, user, machine, etc.
<p>
<b>roff(7)</b> documents used the <b>td</b> macro to generate the date of formatting
in the document have been changed to conditionally use the macro based
on register <b>R</b>, for example as in
<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/share/doc/usd/19.memacros/intro.me.diff?r1=1.4&r2=1.5">intro.me</a>
and then the
<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/share/doc/usd/19.memacros/Makefile.diff?r1=1.4&r2=1.5">Makefile</a>
was changed to set that register for MKREPRO.
<h1> Handling Order</h1>
<p>
We don't control the build order of things and we also don't control
the directory order which can be filesystem dependent. The collation
order also is environment specific, and sorting needs to be stable
(we have not encountered that problem yet). Two different programs
caused us problems here:
<ul>
<li><b>file(1)</b> with the generation of the compiled
magic file using directory order (fixed by changing <b>file(1)</b>).</li>
<li><b>install-info(1)</b>, <b>texinfo(5)</b> files that have no specific order.
For that we developed another tool called
<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/usr.bin/sortinfo"><b>sortinfo(1)</b></a>
that sorts those files as a post-process step.</ul>
</ul>
<p>
Fortunately the filesystem builders and tar programs usually work with input
directories that appear to have a consistent order so far, so we did
not have to fix things there.
<h1>Permissions</h1>
<p>
NetBSD already keeps permissions for most things consistent in different
ways:
<ul>
<li>the build system uses <b>install(8)</b> and specifies ownership and mode.</li>
<li>the <b>mtree(8)</b> program creates build artifacts using consistent
ownership and permissions.</li>
</ul>
<p>
Nevertheless, the various architecture-specific distribution media installers
used <b>cp(1)</b>/<b>mkdir(1)</b> and needed to be corrected.
<h1>Toolchain</h1>
<p>
Most of the issues found had to do with capturing the environment in
debugging information. The two biggest issues were: <b>DW_AT_Producer</b>
and <b>DW_AT_comp_dir</b>:
<pre>
DW_AT_producer : (indirect string, offset: 0x80): GNU C99 5.4.0 \
-fno-canonical-system-headers -mtune=nocona \
-march=x86-64 -g -O2 -std=gnu99 -fPIE -fstack-protector \
-fdebug-prefix-map=$NETBSDSRCDIR=/usr/src \
-fdebug-prefix-map=$X11SRCDIR=/usr/xsrc \
-fdebug-regex-map=/usr/src/(.*)/obj.*=/usr/obj/\1 \
-fdebug-regex-map=/usr/src/(.*)/obj.*/(.*)=/usr/obj/\1/\2 \
--param ssp-buffer-size=1
</pre>
<p>
Here you see two changes we made for reproducible builds:
<ul>
<li>We chose to allow variable names (and have <b>gcc(1)</b> expand them)
for the source of the prefix map
because the source tree location can vary. Others have chosen to
skip <b>-fdebug-prefix-map</b> from the variables to be listed.</li>
<li>We added <b>-fdebug-regex-map</b> so that we could handle the
NetBSD specific objdir build functionality. Object directories can have
many flavors in NetBSD so it was difficult to use <b>-fdebug-prefix-map</b>
to capture that.
</ul>
<p>
<b>DW_AT_comp_dir</b> presented a different challenge. We got non-reproducibility when
building on paths where either the source or the object directories contained symbolic links.
Although <b>gcc(1)</b> does the right
thing handling logical paths (respects $PWD), we found that there were
problems both in the NetBSD <b>sh(1)</b> (fixed
<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/bin/sh/cd.c.diff?r1=1.46&r2=1.47">here</a>)
and in the NetBSD <b>make(1)</b> (fixed
<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/usr.bin/make/main.c.diff?r1=1.255&r2=1.256">here</a>).
Unfortunately we can't depend on the shell to obey the logical path so we
decided to go with:
<p>
<pre>
${MAKE} -C other/dir
</pre>
instead of:
<pre>
cd other/dir && ${MAKE}
</pre>
<p>
This works because <b>make(1)</b> is a tool (part of the toolchain we provide) whereas
<b>sh(1)</b> is not.
<p>
Another weird issue popped up on sparc64 where a single file in the whole
source tree does not build reproducibly. This file is asn1_krb5_asn1.c which
is generated in
<a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/crypto/external/bsd/heimdal/dist/lib/asn1/">here</a>.
The problem is that when profiling on RISC
machines gcc
uses the PROFILE_HOOK macro which in turn uses the "function number" to
generate labels. This number is assigned to each function in a source file
as it is being compiled. Unfortunately this number is not deterministic because
of optimization (a bug?), but fortunately turning optimization off fixes the
problem.
<h1>Status and future work</h1>
<p>
As of 2017-02-20 we have fully reproducible builds on amd64 and sparc64.
We are planning to work on the following areas:
<ul>
<li>Vary more parameters on the system build (filesystem types, build OS's)</li>
<li>Verify that cross building is reproducible</li>
<li>Verify that unprivileged builds work</li>
<li>Test on all the platforms</li>
</ul>
https://blog.netbsd.org/tnf/entry/firefox_51_on_sparc64_weFirefox 51 on sparc64 - we did not hit the wall yetmartin2017-02-08T15:39:07+00:002017-02-08T15:39:07+00:00Keeping a current firefox working is a tough task. All NetBSD architectures are "tier 3" from the Mozilla foundations point of view.<p>
Onodera-san (who does most of the pkgsrc work for mozilla based pkgs - and others) does a great job.<p>
And on "strange" architectures like sparc64 it is even worse...<p><h1>Background</h1>
I am using a sparc64 Sun Blade 2500 (silver) as a desktop machine - for my pretty light "desktop" needs. Besides the usual developer tools (editors, compilers, subversion, hg, git) and admin stuff (all text based) I need mpg123 and mserv for music queues, Gimp for image manipulation and of course Firefox.<p>
Recently I updated all my installed pkgs to pkgsrc-current and as usual the new Firefox version failed to build.<p>
Fortunately the issues were minor, as they all had been handled upstream for Firefox 52 already, all I needed to do was back-porting a few fixes. This made the pkg build, but after a few minutes of test browsing, it crashed. Not surprisingly this was reproducible, any web site trying to play audio triggered it. A bit surprising though: the same happened on an amd64 machine I tried next. After a bit digging the bug was easy to fix, and upstream already took the fix and committed it to the libcubeb repository.<p>
<h1>Success!</h1>
So I am now happily editing this post using Firefox 51 on the Blade 2500.<p>
<a href="http://www.netbsd.org/~martin/ff51_screen.png"><img src="http://www.netbsd.org/~martin/ff51_screen.png" width="80%" /></a><p>
I saw one crash in two days of browsing, but unfortunately could not (yet) reproduce it (I have gdb attached now). There will be future pkg updates certainly.<p>
<h1>Future Obstacles</h1>
You may have read elsewhere that Firefox will start to require a working Rust compiler to build.<p>
This is a bit unfortunate, as Rust (while academically interesting) is right now not a very good implementation language if you care about portability. The only available compiler requires a working LLVM back end, which we are still debugging. Our auto-builds produce sparc sets with LLVM, but the result is not fully working (due to what we believe being code gen bugs in LLVM). It seems we need to fix this soon (which would be good anyway, independent of the Rust issue). Besides the back end, only very recently traces of sparc64 support popped up in Rust. However, we still have a few firefox versions time to get it all going. I am optimistic.<p>
Another upcoming change is that Cairo (currently used as 2D graphics back end, at least on sparc64) will be phased out and Skia will be the only supported software rendering target. Unfortunately Skia does (as of now) not support any big endian machine at all. I am looking for help getting Skia to work on big endian hardware in general, and sparc64 in particular.<p>
<h1>Alternatives</h1>
Just in case, I tested a few other browsers and (so far) they all failed:
<ul>
<li>NetSurf<br>Nice, small, has a few "tweaks" and does not yet support JavaScript good enough for many sites</li>
<li>Midori</br>They call it "lightweight" but it is based on WebKit, which alone is a few times more heavy than all of Firefox. It crashes immediately at startup on sparc64 (I am investigating, but with low priority - actually I had to replace the hard disk in my machine to make enough room for the debug object files for WebKit - it takes ~20GB)</li>
</ul><p>
So, while it is a bit of a struggle to keep a modern browser working on my favorite odd-ball architecture, it seems we will get at least to the Firefox 52 ESR release, and that should give us enough time to get Rust working and hopefully continue with Firefox.<p>https://blog.netbsd.org/tnf/entry/netbsd_org_outage_2017_01NetBSD.org outage 2017-01-16Taylor R Campbell2017-01-16T17:34:51+00:002017-01-16T20:56:13+00:00<p>NetBSD.org DNS is down because our secondary stopped serving our zone and the network of our primary went offline.</p>
<p><i>[Update, 2017-01-16 20:48 UTC: NetBSD.org is back up now.]</i></p><p>
<i>[Update, 2017-01-16 20:48 UTC: All issues have been resolved and
NetBSD.org should be functioning normally now.]</i>
</p>
<p>
As some of you may have noticed, the netbsd.org DNS records are broken
right now.
Two independent incidents caused this to happen:
</p>
<ol>
<li>
UC Berkeley stopped providing secondary DNS service for netbsd.org
at adns1.berkeley.edu and adns2.berkeley.edu on a few hours
notice, before we had time to update the NS delegations in the
ORG. zone.
<i>[Update, 2017-01-16 20:48 UTC: The Berkeley network
administrators have graciously restored service until we can
update our NS delegations. Thanks, Berkeley—for doing this
on a US holiday, no less!]</i>
</li>
<li>
The network hosting our primary nameserver ns.netbsd.de went offline
at around 14:00 UTC for reasons unknown.
<i>[Update, 2017-01-16 20:48 UTC: The network and ns.netbsd.de are
back up.]</i>
</li>
</ol>
<p>
We have sent a request to our registrar to update the delegations to
make ns.netbsd.org primary and ns.netbsd.de secondary.
We asked them to expedite the process, but owing to the weekend, when
the humans who process these requests are off work, it may take
another day for that to take effect, and another day after that for
the cached TTLs on the old NS delegations to Berkeley's nameservers
to expire.
</p>
<p>
As a provisional stop gap, if you need access to the netbsd.org zone
and you can prime your DNS cache with manually entered records, you
can prime it with the following records to restore service at your
site:
</p>
<blockquote>
<pre>
netbsd.org. 86400 IN NS ns.netbsd.org.
netbsd.org. 86400 IN NS ns.netbsd.de.
ns.netbsd.org. 86400 IN A 199.233.217.200
ns.netbsd.org. 86400 IN AAAA 2001:470:a085:999::53
</pre>
</blockquote>
<p>
(Presumably if you are reading this you have some DNS records in the
netbsd.org zone cached, but they will eventually expire and you may
need some others.)
</p>
https://blog.netbsd.org/tnf/entry/more_disk_s_funMore disk(s) funmartin2016-07-07T11:55:45+00:002016-07-08T08:23:05+00:00<p>When I got my Sun T1000 machine, it came with a ~80 GB hard disk - good enough for a NetBSD installation, but a bit challenged when you want to use logical domains. Time to expand disk space, or maybe make it faster? But these 1U server machines do not offer a lot of room for extensions, and it is sometimes tricky to get hold of the official extension options nowadays.</p>
<p>So I had fun with disks and modern replacements again...</p><p>When I got my Sun T1000 machine, it came with a ~80 GB hard disk - good enough for a NetBSD installation, but a bit challenged when you want to use logical domains. Time to expand disk space, or maybe make it faster? But these 1U server machines do not offer a lot of room for extensions, and it is sometimes tricky to get hold of the official extension options nowadays.</p>
<p>So I had fun with disks and modern replacements again... (after the <a href="http://blog.netbsd.org/tnf/entry/what_to_do_when_you">scsi2sd adventures</a>).</p>
<p>The T1000 were offered in single 3.5" disk configurations (this is what I got) or with two 2.5" disks. The mainboard has two sata (sas?) connectors. But the disk tray in my machine is not usable for mounting two 2.5" HDs.<p>
<p>But: the machine has a spare PCIe slot:<br/>
<a href="http://www.netbsd.org/~martin/t1k-pcie-back.jpg"><img width="100%" src="http://www.netbsd.org/~martin/t1k-pcie-back.jpg" alt="PCIe slot on the back of a T1000"/></a><br/>
and we support NVMe, so why not put a M.2 module in an adapter card in there? <a href="http://mail-index.netbsd.org/port-sparc64/2016/06/26/msg002537.html">I tried and failed</a>. I learned that the T1000's PCIe x4 slot is PCIe version 1, but modern NVMe modules do not fancy working in anything older than PCIe version 3.</p>
<p>Too bad, but whatever. I switched things around, my amd64 desktop machine got a faster "SSD" (i.e. the NVMe), another machine got the SSD freed from there and I ended up with a spare 128 GB sata SSD.</p>
<p>Now the T1000 had a sata connector free, but no way to place the disk and no power connector for it. The mainboard had a legacy 5/12V jack and the single disk used a strange power/sata connector:<br/>
<center><a href="http://www.netbsd.org/~martin/t1k-sata-pwr-connector.png"><img width="50%" src="http://www.netbsd.org/~martin/t1k-sata-pwr-connector.png" alt="SATA and power connector for the single disk in my T1000"/></a></center><br/>
but of course that could be replaced easily. I cut a Y-sata power split and one legacy 5/12V power connector from an old (broken) PSU and soldered a dual sata power connector cable that connected to the jack on the mainboard.</p>
<p>I found a small cavity in the at the front of the machine in the sheet separating PSU and disk from the fans and mainboard. Big enough to push a sata power connector and a sata connector through! So I just glued the SSD at the bottom of the chasis in front of the fans (it should be low enough to not harm airflow significantly):</br>
<a href="http://www.netbsd.org/~martin/t1k-with-sdd.jpg"><img width="100%" src="http://www.netbsd.org/~martin/t1k-with-sdd.jpg" alt="SSD on bottom of chasis"/></a><br/></p>
<p>It was a bit tricky to fit the custom power cabling and the standard sata cable alongside the disk, and I hope the disk will not get too hot to unsolder the connectors ;-)</br>
<a href="http://www.netbsd.org/~martin/t1k-custom-pwr-cabling.jpg"><img width="100%" src="http://www.netbsd.org/~martin/t1k-custom-pwr-cabling.jpg" alt="Custom power cabling for the SSD"/></a><br/>
Hardware service technicians would hate this, as usually the T1000 offers really easy and fast hard disk replacement, but I suppose no one but me will ever deal with hardware failure on this particular machine.</p>
<p>Now thats it, fast base system on SSD and plenty of space for other LDOMs data on the second disk:
<pre>
mpt0 at pci3 dev 2 function 0: Symbios Logic SAS1064 (rev. 0x02)
mpt0: interrupting at ivec 7c0
mpt0: Phy 0: Link Status Unknown
mpt0: Phy 0: Link Status Unknown
scsibus0 at mpt0: 63 targets, 8 luns per target
scsibus0: waiting 2 seconds for devices to settle...
sd0 at scsibus0 target 0 lun 0: <ATA, Samsung SSD 840, 9B0Q> disk fixed
sd0: 111 GB, 114474 cyl, 16 head, 127 sec, 512 bytes/sect x 234441648 sectors
sd0: tagged queueing
sd1 at scsibus0 target 1 lun 0: <ATA, ST2000DM001-9YN1, CC4H> disk fixed
sd1: 1863 GB, 1907730 cyl, 16 head, 127 sec, 512 bytes/sect x 3907029168 sectors
sd1: tagged queueing
</pre>
</p>
<p>Unfortunately the machine is louder than hell, and even though the rack is in our basement, the sound level is still irritating elsewhere in the house (massive stone walls, old style german construction). My wife officially hates the T1000.</p>
<p>Now that the hardware is ready, it is time to get it fully supported and import the LDOM management/utilities. I need to find time and help Palle!</p>https://blog.netbsd.org/tnf/entry/what_to_do_when_youWhat to do when you run out of (ancient) 50 pin SCSI disks?martin2016-05-27T17:42:10+00:002016-05-27T17:44:01+00:00<p>Unable to buy new 50 pin SCSI disks, and not willing to spend huge amounts of money on fast SCSI disks then slowed down by 50pin adapters, I looked for alternative solutions for the root disks of my mac68k and alpha machines.</p><p>
Recently on a few mailing lists a discussion about creating a simple turbo-channel USB adapter came up. I have an old Alpha machine with turbo channel and no on-board USB, so considered ordering one - but wanted to know upfront whether that alpha machine still worked.
<p/>
<p>
I started with updating the installation on the alpha to -current, and ran out of disk space!
</p>
<p>
No big deal, it still had the original ~1GB disk (RZ26) in it, and a full NetBSD install nowadays needs slightly more.
I have plenty of unused SCSI disks lying around, but when looking for 50pin ones, I found I have none left.
</p>
<p>
Another machine I was looking at for a different root disk solution was an old mac68k (Quadra 640 AV). I had replaced the disk it came with by an 8GB one, but it draws too much power for the internal PSU, so I had to move it to an external SCSI enclosure, which is inconvenient.
</p>
<p>
Additionally both disks made quite some noise and probably would not last a lot longer. So I started looking for "cheap", silent and low-power alternatives.
</p>
<p>
I found the <a href="http://www.codesrc.com/mediawiki/index.php?title=SCSI2SD">SCSI2SD</a> project, which looked promising and had a firmware update earlier this year. I ordered a few boards from <a href="https://www.itead.cc/scsi2sd.html">idead studio</a> and started testing.
</p>
<p>
The solution is mostly plug & play, a configuration utility allows updating the firmware and configuring the devices.
</p>
<p>
I did several bonnie++ runs to get an idea of the speed achievable. All machines are not fast, and disk access has always been slow. To get an overall idea, I also tested on a faster alpha (still with slow scsi disks). Then I moved both test machines over to "root on sd card" and tested again. The results look like this:
</p>
<table>
<tr>
<th rowspan=3>Test</th>
<th colspan=6>Sequential Output</th>
<th colspan=4>Sequential Input</th>
<th colspan=2>Random Seeks</th>
<th colspan=6>Sequential Create</th>
<th colspan=6>Random Create</th>
</tr>
<tr>
<th colspan=2>Per Chr</th>
<th colspan=2>Block</th>
<th colspan=2>Rewrite</th>
<th colspan=2>Per Chr</th>
<th colspan=2>Block</th>
<th rowspan=2>K/s</th>
<th rowspan=2>CPU</th>
<th colspan=2>Create</th>
<th colspan=2>Read</th>
<th colspan=2>Delete</th>
<th colspan=2>Create</th>
<th colspan=2>Read</th>
<th colspan=2>Delete</th>
</tr>
<tr>
<th>K/s</th>
<th>CPU</th>
<th>K/s</th>
<th>CPU</th>
<th>K/s</th>
<th>CPU</th>
<th>K/s</th>
<th>CPU</th>
<th>K/s</th>
<th>CPU</th>
<th>/s</th>
<th>CPU</th>
<th>/s</th>
<th>CPU</th>
<th>/s</th>
<th>CPU</th>
<th>/s</th>
<th>CPU</th>
<th>/s</th>
<th>CPU</th>
<th>/s</th>
<th>CPU</th>
</tr>
<tr>
<td>Ref: DS20</td>
<td>3436</td>
<td>35</td>
<td>3460</td>
<td>6</td>
<td>2935</td>
<td>5</td>
<td>9565</td>
<td>94</td>
<td>19494</td>
<td>285.7</td>
<td>16</td>
<td>3</td>
<td>878</td>
<td>81</td>
<td>31793</td>
<td>99</td>
<td>2939</td>
<td>54</td>
<td>940</td>
<td>84</td>
<td>1351</td>
<td>96</td>
<td>203</td>
<td>10</td>
</tr>
<tr>
<td>Alpha HD</td>
<td>906</td>
<td>92</td>
<td>2146</td>
<td>41</td>
<td>980</td>
<td>24</td>
<td>947</td>
<td>93</td>
<td>2178</td>
<td>23</td>
<td>78.5</td>
<td>14</td>
<td>36</td>
<td>95</td>
<td>926</td>
<td>72</td>
<td>290</td>
<td>90</td>
<td>37</td>
<td>96</td>
<td>39</td>
<td>97</td>
<td>25</td>
<td>34</td>
</tr>
<tr>
<td>Alpha SD</td>
<td>548</td>
<td>77</td>
<td>897</td>
<td>19</td>
<td>450</td>
<td>9</td>
<td>490</td>
<td>97</td>
<td>1140</td>
<td>9</td>
<td>24.9</td>
<td>4</td>
<td>85</td>
<td>89</td>
<td>1245</td>
<td>75</td>
<td>437</td>
<td>84</td>
<td>90</td>
<td>93</td>
<td>119</td>
<td>96</td>
<td>29</td>
<td>16</td>
</tr>
<tr>
<td>Mac HD</td>
<td>98</td>
<td>95</td>
<td>737</td>
<td>89</td>
<td>373</td>
<td>91</td>
<td>94</td>
<td>96</td>
<td>647</td>
<td>89</td>
<td>14.1</td>
<td>47</td>
<td>22</td>
<td>93</td>
<td>502</td>
<td>75</td>
<td>122</td>
<td>81</td>
<td>23</td>
<td>94</td>
<td>31</td>
<td>97</td>
<td>14</td>
<td>43</td>
</tr>
<tr>
<td>Mac SD</td>
<td>95</td>
<td>95</td>
<td>646</td>
<td>89</td>
<td>339</td>
<td>92</td>
<td>93</td>
<td>96</td>
<td>636</td>
<td>79</td>
<td>14.5</td>
<td>75</td>
<td>22</td>
<td>90</td>
<td>473</td>
<td>75</td>
<td>124</td>
<td>85</td>
<td>23</td>
<td>92</td>
<td>32</td>
<td>97</td>
<td>13</td>
<td>46</td>
</tr>
</table>
<p>
So even for machines with very slow hard disk access, there is a (relatively small?) performance hit with the SCSI2SD solution. Whether it is too bad for your application, depends on various factors - all of the affected machines are very slow overall anyway.
</p>
<p>
Just a few days ago a new hardware revision of the SCSI2SD cards has been announced which is supposed to improve performance in the future (when the firmware has caught up), so it may be worth to wait (or get the newer hardware) if performance is important.
</p>
<p>
I have a VAX machine that could go with this solution as well, but results are expected to be similar (or worse) than for the slow alpha, as my vax is pretty fast (for a vax) and has a decent SCSI subsystem.
</p>
<p>For reference, below are the full dmesg from the fast reference and both test machines after switching to SD root, and the sniplet for the original hard disk</p>
<h1>AlphaServer DS20</h1>
<p>"Fast" reference machine<br/>
file system: FFSv2, log</p>
<code>
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,<br/>
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016<br/>
The NetBSD Foundation, Inc. All rights reserved.<br/>
Copyright (c) 1982, 1986, 1989, 1991, 1993<br/>
The Regents of the University of California. All rights reserved.<br/>
<br/>
NetBSD 7.99.29 (GENERIC-$Revision: 1.368 $) #7: Mon May 23 20:44:04 CEST 2016<br/>
martin@martins.aprisoft.de:/ssd/src/sys/arch/alpha/compile/GENERIC<br/>
AlphaServer DS20 500 MHz, s/n AY94910150<br/>
8192 byte page size, 1 processor.<br/>
total memory = 1024 MB<br/>
(2712 KB reserved for PROM, 1021 MB used by NetBSD)<br/>
avail memory = 994 MB<br/>
timecounter: Timecounters tick every 0.976 msec<br/>
Kernelized RAIDframe activated<br/>
mainbus0 (root)<br/>
cpu0 at mainbus0: ID 0 (primary), 21264-4<br/>
cpu0: Architecture extensions: 0x303<PAT,MVI,FIX,BWX><br/>
tsc0 at mainbus0: 21272 Core Logic Chipset, Cchip rev 0<br/>
tsc0: 8 Dchips, 2 memory buses of 32 bytes<br/>
tsc0: arrays present: 512MB, 512MB, 0MB, 0MB, Dchip 0 rev 1<br/>
tsp0 at tsc0<br/>
pci0 at tsp0 bus 0<br/>
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok<br/>
sio0 at pci0 dev 5 function 0: vendor 1080 product c693 (rev. 0x00)<br/>
cypide0 at pci0 dev 5 function 1: Cypress 82C693 IDE Controller (rev. 0x00)<br/>
cypide0: bus-master DMA support present, but unused (registers at unsafe address 0x10000)<br/>
cypide0: primary channel wired to compatibility mode<br/>
cypide0: primary channel interrupting at isa irq 14<br/>
atabus0 at cypide0 channel 0<br/>
cypide1 at pci0 dev 5 function 2: Cypress 82C693 IDE Controller (rev. 0x00)<br/>
cypide1: hardware does not support DMA<br/>
cypide1: primary channel wired to compatibility mode<br/>
cypide1: secondary channel interrupting at isa irq 15<br/>
atabus1 at cypide1 channel 0<br/>
ohci0 at pci0 dev 5 function 3: vendor 1080 product c693 (rev. 0x00)<br/>
ohci0: interrupting at isa irq 10<br/>
ohci0: OHCI version 1.0, legacy support<br/>
usb0 at ohci0: USB revision 1.0<br/>
isa0 at sio0<br/>
lpt0 at isa0 port 0x3bc-0x3bf irq 7<br/>
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo<br/>
com0: console<br/>
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo<br/>
pckbc0 at isa0 port 0x60-0x64<br/>
attimer0 at isa0 port 0x40-0x43<br/>
pcppi0 at isa0 port 0x61<br/>
midi0 at pcppi0: PC speaker<br/>
spkr0 at pcppi0<br/>
isabeep0 at pcppi0<br/>
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2<br/>
mcclock0 at isa0 port 0x70-0x71: mc146818 compatible time-of-day clock<br/>
attimer0: attached to pcppi0<br/>
tsp1 at tsc0<br/>
pci1 at tsp1 bus 0<br/>
pci1: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok<br/>
tlp0 at pci1 dev 7 function 0: Macronix MX98713 Ethernet, pass 0.0<br/>
tlp0: interrupting at dec 6600 irq 47<br/>
tlp0: Ethernet address 00:40:05:50:ee:9b<br/>
tlp0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX<br/>
isp0 at pci1 dev 8 function 0: QLogic 1020 Fast Wide SCSI HBA<br/>
isp0: interrupting at dec 6600 irq 43<br/>
isp1 at pci1 dev 9 function 0: QLogic 1020 Fast Wide SCSI HBA<br/>
isp1: interrupting at dec 6600 irq 39<br/>
tsciic0 at tsc0<br/>
iic0 at tsciic0: I2C bus<br/>
timecounter: Timecounter "clockinterrupt" frequency 1024 Hz quality 0<br/>
timecounter: Timecounter "PCC" frequency 499882560 Hz quality 1000<br/>
scsibus0 at isp0: 16 targets, 8 luns per target<br/>
scsibus1 at isp1: 16 targets, 8 luns per target<br/>
scsibus0: waiting 2 seconds for devices to settle...<br/>
scsibus1: waiting 2 seconds for devices to settle...<br/>
uhub0 at usb0: vendor 1080 OHCI root hub, class 9/0, rev 1.00/1.00, addr 1<br/>
uhub0: 2 ports with 2 removable, self powered<br/>
sd0 at scsibus0 target 0 lun 0: <COMPAQ, BB00911CA0, 3B05> disk fixed<br/>
sd0: 8678 MB, 5273 cyl, 20 head, 168 sec, 512 bytes/sect x 17773524 sectors<br/>
sd0: sync (50.00ns offset 8), 16-bit (40.000MB/s) transfers, tagged queueing<br/>
sd1 at scsibus0 target 1 lun 0: <COMPAQ, BB00912301, B016> disk fixed<br/>
sd1: 8678 MB, 5273 cyl, 20 head, 168 sec, 512 bytes/sect x 17773524 sectors<br/>
sd1: sync (50.00ns offset 8), 16-bit (40.000MB/s) transfers, tagged queueing<br/>
sd2 at scsibus0 target 2 lun 0: <COMPAQ, BB00911CA0, 3B05> disk fixed<br/>
sd2: 8678 MB, 5273 cyl, 20 head, 168 sec, 512 bytes/sect x 17773524 sectors<br/>
sd2: sync (50.00ns offset 8), 16-bit (40.000MB/s) transfers, tagged queueing<br/>
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec<br/>
sd3 at scsibus0 target 3 lun 0: <COMPAQ, BB00912301, B016> disk fixed<br/>
sd3: 8678 MB, 5273 cyl, 20 head, 168 sec, 512 bytes/sect x 17773524 sectors<br/>
sd3: sync (50.00ns offset 8), 16-bit (40.000MB/s) transfers, tagged queueing<br/>
cd0 at scsibus1 target 5 lun 0: <DEC, RRD47 (C) DEC, 1206> cdrom removable<br/>
raid0: RAID Level 1<br/>
raid0: Components: /dev/sd0a /dev/sd1a<br/>
raid0: Total Sectors: 17773440 (8678 MB)<br/>
raid1: RAID Level 1<br/>
raid1: Components: /dev/sd2a /dev/sd3a<br/>
raid1: Total Sectors: 17773440 (8678 MB)<br/>
root on raid0a dumps on raid0b<br/>
root file system type: ffs<br/>
kern.module.path=/stand/alpha/7.99.29/modules<br/>
</code>
<h1>Alpha DEC 3000</h1>
<h2>with SCSI2SD</h2>
<code>
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,<br/>
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016<br/>
The NetBSD Foundation, Inc. All rights reserved.<br/>
Copyright (c) 1982, 1986, 1989, 1991, 1993<br/>
The Regents of the University of California. All rights reserved.<br/>
<br/>
NetBSD 7.99.29 (GENERIC-$Revision: 1.368 $) #3: Thu May 5 09:17:11 CEST 2016<br/>
martin@martins.aprisoft.de:/ssd/src/sys/arch/alpha/compile/GENERIC<br/>
DEC 3000 - M300, 150MHz, s/n <br/>
8192 byte page size, 1 processor.<br/>
total memory = 98304 KB<br/>
(2048 KB reserved for PROM, 96256 KB used by NetBSD)<br/>
avail memory = 82352 KB<br/>
timecounter: Timecounters tick every 0.976 msec<br/>
Kernelized RAIDframe activated<br/>
mainbus0 (root)<br/>
cpu0 at mainbus0: ID 0 (primary), 21064-1<br/>
tcasic0 at mainbus0<br/>
tc0 at tcasic0: 12.5 MHz clock<br/>
sfb0 at tc0 slot 6 offset 0x2000000: 1280x1024, 8bpp<br/>
wsdisplay1 at sfb0 kbdmux 1<br/>
wsmux1: connecting to wsdisplay1<br/>
ioasic0 at tc0 slot 5 offset 0x0: slow mode<br/>
le0 at ioasic0 offset 0xc0000: address 08:00:2b:3c:93:27<br/>
le0: 32 receive buffers, 8 transmit buffers<br/>
zsc0 at ioasic0 offset 0x100000<br/>
vsms0 at zsc0 channel 0<br/>
wsmouse0 at vsms0 mux 0<br/>
zstty0 at zsc0 channel 1 (console i/o)<br/>
zsc1 at ioasic0 offset 0x180000<br/>
lkkbd0 at zsc1 channel 0<br/>
wskbd0 at lkkbd0 mux 1<br/>
wskbd0: connecting to wsdisplay1<br/>
zsc1: channel 1 not configured<br/>
mcclock0 at ioasic0 offset 0x200000: mc146818 compatible time-of-day clock<br/>
bba0 at ioasic0 offset 0x240000<br/>
audio0 at bba0: full duplex, playback, capture, mmap<br/>
tcds0 at tc0 slot 4 offset 0x0: TurboChannel Dual SCSI (baseboard)<br/>
asc0 at tcds0 chip 0: NCR53C94, 25MHz, SCSI ID 0<br/>
scsibus0 at asc0: 8 targets, 8 luns per target<br/>
timecounter: Timecounter "clockinterrupt" frequency 1024 Hz quality 0<br/>
timecounter: Timecounter "PCC" frequency 150006528 Hz quality 1000<br/>
lkkbd0: no keyboard<br/>
scsibus0: waiting 2 seconds for devices to settle...<br/>
sd0 at scsibus0 target 1 lun 0: <codesrc, SCSI2SD, 1.0> disk fixed<br/>
sd0: 3712 MB, 473 cyl, 255 head, 63 sec, 512 bytes/sect x 7603200 sectors<br/>
sd0: async, 8-bit transfers<br/>
root on sd0a dumps on sd0b<br/>
root file system type: ffs<br/>
kern.module.path=/stand/alpha/7.99.29/modules<br/>
</code>
<h2>with original RZ26</h2>
<code>
sd0 at scsibus0 target 3 lun 0: <DEC, RZ26 (C) DEC, 392A> disk fixed<br/>
sd0: 1001 MB, 2570 cyl, 14 head, 57 sec, 512 bytes/sect x 2050860 sectors<br/>
sd0: sync (200.00ns offset 15), 8-bit (5.000MB/s) transfers, tagged queueing<br/>
</code>
<h1>Mac Quadra 640AV</h1>
<p>File system: FFSv1, log</p>
<h2>with SCSI2SD</h2>
<code>
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,<br/>
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016<br/>
The NetBSD Foundation, Inc. All rights reserved.<br/>
Copyright (c) 1982, 1986, 1989, 1991, 1993<br/>
The Regents of the University of California. All rights reserved.<br/>
<br/>
NetBSD 7.99.29 (MAC-BETH) #18: Tue May 10 05:40:16 CEST 2016<br/>
martin@night-owl.duskware.de:/usr/src/sys/arch/mac68k/compile/MAC-BETH<br/>
Apple Macintosh Centris 660AV (68040)<br/>
cpu: delay factor 800<br/>
fpu: mc68040<br/>
total memory = 45056 KB<br/>
avail memory = 40248 KB<br/>
timecounter: Timecounters tick every 16.666 msec<br/>
mrg: 'Quadra AV ROMs' ROM glue, tracing off, debug off, silent traps<br/>
mainbus0 (root)<br/>
obio0 at mainbus0<br/>
esp0 at obio0 addr 0: address 0x2da000: NCR53C96, 25MHz, SCSI ID 7<br/>
scsibus0 at esp0: 8 targets, 8 luns per target<br/>
adb0 at obio0<br/>
intvid0 at obio0 @ 50100600: CIVIC video subsystem<br/>
intvid0: 640 x 480, 65536 color<br/>
macfb0 at intvid0<br/>
wsdisplay0 at macfb0 (kbdmux ignored)<br/>
mc0 at obio0: address 08:00:07:46:74:0a<br/>
zsc0 at obio0 chip type 0 <br/>
zsc0 channel 0: d_speed 9600 DCD clk 0 CTS clk 0<br/>
zstty0 at zsc0 channel 0 (console i/o)<br/>
zsc0 channel 1: d_speed 9600 DCD clk 0 CTS clk 0<br/>
zstty1 at zsc0 channel 1<br/>
nubus0 at mainbus0<br/>
timecounter: Timecounter "clockinterrupt" frequency 60 Hz quality 0<br/>
timecounter: Timecounter "VIA1 T2" frequency 783360 Hz quality 100<br/>
scsibus0: waiting 2 seconds for devices to settle...<br/>
adb0 (direct, Cuda): 2 targets<br/>
aed0 at adb0 addr 0: ADB Event device<br/>
akbd0 at adb0 addr 2: keyboard II (ISO layout)<br/>
wskbd0 at akbd0 (mux ignored)<br/>
ams0 at adb0 addr 3: 1-button, 100 dpi mouse<br/>
wsmouse0 at ams0 (mux ignored)<br/>
sd0 at scsibus0 target 0 lun 0: <codesrc, SCSI2SD, 4.6> disk fixed<br/>
sd0: 15104 MB, 1925 cyl, 255 head, 63 sec, 512 bytes/sect x 30934016 sectors<br/>
sd0: async, 8-bit transfers<br/>
cd0 at scsibus0 target 3 lun 0: <SONY, CD-ROM CDU-8003A, 1.9a> cdrom removable<br/>
cd0: async, 8-bit transfers<br/>
boot device: sd0<br/>
root on sd0a dumps on sd0b<br/>
root file system type: ffs<br/>
kern.module.path=/stand/mac68k/7.99.29/modules<br/>
</code>
<h2>with hard disk</h2>
<code>
sd0 at scsibus0 target 0 lun 0: <SEAGATE, ST39236LW, 0004> disk fixed<br/>
sd0: 8761 MB, 14384 cyl, 3 head, 415 sec, 512 bytes/sect x 17942584 sectors<br/>
sd0: async, 8-bit transfers, tagged queueing<br/>
</code>
https://blog.netbsd.org/tnf/entry/talks_about_blacklistdtalks about blacklistdMatthew Sporleder2016-04-04T20:21:58+00:002016-04-04T20:21:58+00:00Watch a video by Christos Zoulas (with good audio!) talking about blacklistd
<br />
<a href="https://www.youtube.com/watch?v=fuuf8G28mjs">blacklistd by Christos Zoulas</a>
https://blog.netbsd.org/tnf/entry/happy_23rd_birthday_srcHappy 23rd Birthday, src!William J. Coldwell2016-03-21T12:54:14+00:002016-03-21T12:55:31+00:00And so it began...<br>
<br>
revision 1.1<br>
date: 1993-03-21 10:45:37 +0100; author: cgd; state: Exp;<br>
branches: 1.1.1;<br>
<br>
Initial revision<br>
<br>
and we continue this legacy.<br>
<br>
https://blog.netbsd.org/tnf/entry/hands_on_experience_with_edgerouterHands on experience with EdgeRouter ERLite-3martin2015-05-07T12:18:08+00:002021-04-23T16:29:36+00:00My EdgeRouter ERLite-3 just has been delivered. Setup was easy (the NetBSD version of "plug & play"), and I really like this hardware.<p>
Of course first testing showed up first errors - so this will be an interesting experience! <p>After spending last year on improving ARM test coverage (and then fixing bugs when found), which greatly improved ARM test results, I now turned to MIPS. I used to have some SGI O2 machines, but for some (yet unclear) reason, none of them powers up any more. I also had an alchemy base MeshCube, but that broke too.<p>
Some time ago, Ingenics donated a CI20 board to me and I am in the process of getting that testable. Michael Lorenz did most of the needed <a href="http://blog.netbsd.org/tnf/entry/ci20_reaches_userland">work</a> already, but there are issues with the network driver, which make automatic test runs hard.<p>
When a few days ago cavium/octeon support was added to NetBSD-current, I just had to buy an <a href="https://www.ubnt.com/edgemax/edgerouter-lite/">ERLite-3</a> device:<p>
<a href="https://cdn.NetBSD.org/pub/NetBSD/misc/jdc/wiki/IMG_0583.JPG">
<image width="100%" src="https://cdn.NetBSD.org/pub/NetBSD/misc/jdc/wiki/IMG_0583.jpeg">
</a>
<p>
This is probably the cheapest MIPS64 hardware currently available. It is in-store at many customer electronics shops all over the world.
<p>
The ERLite-3 uses a big-endian dual-core octeon chipset, has external serial console (via a cisco console cable) and three gigE ethernet ports. It also features an internal (built in) USB hard disk. It can be configured to boot the NetBSD kernel from that usb disk, or load it via tftp from another server. But booting from internal "disk" plus multiple gigE ports makes it an ideal device for a NetBSD based home or small office firewall, DNS cache, DHCP server setup.
<p>
From the NetBSD point of view this device is as plug&play as you can get. Connect the serial console cable, reboot it, press Ctrl-C during early boot and get to the u-boot prompt. Prepare a user land and kernel, like:
<pre>
build.sh -m evbmips64-eb sets
build.sh -m evbmips64-eb kernel=ERLITE
</pre>
The kernel build will produce a "netbsd" and "netbsd.elf32" file. Extract the created sets in the NFS root directory, cd to the dev directory there and run "sh MAKEDEV all". Put "netbsd" into the NFS root directory. Copy "netbsd.elf32" to the tftp server directory (I renamed mine to "erlite.elf32") and then go back to the edge router console.
<pre>
dhcp
tftp $loadaddr erlite.elf32
bootoctlinux
</pre>
and your kernel should boot. (Never mind the funny name of the boot command).
<p>
The obligatory dmesg:
<pre>
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
NetBSD 7.99.15 (ERLITE) #1: Wed May 6 21:40:41 CEST 2015
martin@night-owl.duskware.de:/usr/src/sys/arch/evbmips/compile/ERLITE
total memory = 512 MB
avail memory = 490 MB
timecounter: Timecounters tick every 10.000 msec
mainbus0 (root)
cpu0 at mainbus0: 500.00MHz (hz cycles = 5000000, delay divisor = 500)
cpu0: Cavium CN50xx (0xd0601) Rev. 1 with software emulated floating point
cpu0: 64 TLB entries, 512TB (49-bit) VAs, 512TB (49-bit) PAs, 256MB max page size
cpu0: 32KB/128B 4-way set-associative L1 instruction cache
cpu0: 16KB/128B 64-way set-associative write-back coherent L1 data cache
iobus0 at mainbus0
iobus0: initializing POW
iobus0: initializing FPA
com0 at iobus0: address=0x0001180000000800: ns16650, no ERS, working fifo
com0: console
com at iobus0: address=0x0001180000000c00 not configured
octeon_rnm0 at iobus0: address=0x0001180040000000
octeon_rnm0: random number generator enabled: 1hz
octeon_twsi at iobus0: address=0x0001180000001000 not configured
octeon_mpi at iobus0: address=0x0001070000001000 not configured
octeon_gmx0 at iobus0: address=0x0001180008000000
cnmac0 at octeon_gmx0: address=0x0001180008000000: RGMII
cnmac0: Ethernet address 04:18:d6:f0:4b:e4
atphy0 at cnmac0 phy 7: Atheros AR8035 10/100/1000 PHY, rev. 2
atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX-FDX, 1000baseT-FDX, auto
cnmac1 at octeon_gmx0: address=0x0001180008000000: RGMII
cnmac1: Ethernet address 04:18:d6:f0:4b:e5
atphy1 at cnmac1 phy 6: Atheros AR8035 10/100/1000 PHY, rev. 2
atphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX-FDX, 1000baseT-FDX, auto
cnmac2 at octeon_gmx0: address=0x0001180008000000: RGMII
cnmac2: Ethernet address 04:18:d6:f0:4b:e6
atphy2 at cnmac2 phy 5: Atheros AR8035 10/100/1000 PHY, rev. 2
atphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX-FDX, 1000baseT-FDX, auto
dwctwo0 at iobus0: address=0x0001180068000000
usb0 at dwctwo0: USB revision 2.0
bootbus0 at mainbus0
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
timecounter: Timecounter "mips3_cp0_counter" frequency 500000000 Hz quality 100
uhub0 at usb0: vendor 0000 DWC2 root hub, class 9/0, rev 2.00/1.00, addr 1
uhub0: 1 port with 1 removable, self powered
umass0 at uhub0 port 1 configuration 1 interface 0
umass0: vendor 13fe USB DISK 2.0, rev 2.00/1.00, addr 2
umass0: using SCSI over Bulk-Only
scsibus0 at umass0: 2 targets, 1 lun per target
sd0 at scsibus0 target 0 lun 0: <, USB DISK 2.0, PMAP> disk removable
sd0: 3824 MB, 959 cyl, 255 head, 32 sec, 512 bytes/sect x 7831552 sectors
boot device: <unknown>
root device: cnmac0
dump device:
file system (default generic):
root on cnmac0
mountroot: trying nfs...
nfs_boot: trying DHCP/BOOTP
cnmac0: link state UP (was UNKNOWN)
cnmac0: link state DOWN (was UP)
cnmac0: link state UP (was DOWN)
nfs_boot: DHCP next-server: 192.168.150.188
nfs_boot: my_domain=duskware.de
nfs_boot: my_addr=192.168.150.192
nfs_boot: my_mask=255.255.254.0
nfs_boot: gateway=192.168.151.1
root on 192.168.150.188:/hosts/erlite
root time: 0x554ad8bc
root file system type: nfs
kern.module.path=/stand/evbmips/7.99.15/modules
WARNING: no TOD clock present
WARNING: using filesystem time
WARNING: CHECK AND RESET THE DATE!
init path (default /sbin/init):
init: copying out path `/sbin/init' 11
pid 1(init): ABI set to N32 (e_flags=0x20000027)
</pre>
<p>
The port to octeon (thanks IIJ!) is new, but already quite complete. The second cpu core is not yet started, but I hope this will be fixed very soon.
No MIPS has been part of the automatic test runs recently, so there is some fallout, as expected. On first try I got several core files, among others from pkg_info, pkg_admin and gdb. The latter makes analyzing the crashes a bit more challenging, but give me a few days ;-}
<p>https://blog.netbsd.org/tnf/entry/an_internet_ready_os_fromAn Internet-Ready OS From Scratch in a Week — Rump Kernels on Bare MetalAntti Kantee2014-08-08T00:14:54+00:002014-08-08T00:14:54+00:00<p>
The most time-consuming part of operating system development is obtaining
enough drivers to enable the OS to run real
applications which interact with the real world. NetBSD's <a
href="http://rumpkernel.org/">rump kernels</a> allow reducing
that time to almost zero, for example for developing special-purpose operating
systems for the cloud and embedded IoT devices. This article describes
an experiment in creating an OS by using a rump kernel for drivers.
It attempts to avoid going into full detail on the principles
of rump kernels,
which are available for interested readers from
<a href="http://rumpkernel.org">rumpkernel.org</a>.
</p>
<p>
The most time-consuming part of operating system development is obtaining
enough drivers to enable the OS to run real
applications which interact with the real world. NetBSD's <a
href="http://rumpkernel.org/">rump kernels</a> allow reducing
that time to almost zero, for example for developing special-purpose operating
systems for the cloud and embedded IoT devices. This article describes
an experiment in creating an OS by using a rump kernel for drivers.
It attempts to avoid going into full detail on the principles
of rump kernels,
which are available for interested readers from
<a href="http://rumpkernel.org">rumpkernel.org</a>. We start by defining
the terms in the title:
</p>
<p>
<ul>
<li><i>OS</i>: operating system, i.e. the overhead that enables applications to run</li>
<li><i>internet-ready</i>: supports POSIX applications and talks TCP/IP</li>
<li><i>a week</i>: 7 days, in this case the period between Wednesday night
last week and Wednesday night this week</li>
<li><i>from scratch</i>: began by writing the assembly instructions for the kernel entry point</li>
<li><i>rump kernel</i>: partial kernel consisting of unmodified NetBSD kernel
drivers</li>
<li><i>bare metal</i>: what you get from the BIOS/firmware</li>
</ul>
</p>
<p>
Why would anyone want to write a new OS? If you look at our definition
of "OS", you notice that you want to keep the OS as small as possible.
Sometimes you might not care, e.g. in case of a desktop PC, but other
times when hardware resources are limited or you have high enough security
concerns, you actually might care. For example, NetBSD itself is not able to
run on systems without a MMU, but the OS described in this article does
not use virtual memory at all, and yet it can run most of the same
applications as NetBSD can. Another example: if you want to finetune
the OS to suit your application, it's easier to tune a simple OS than a
very complicated general purpose OS. The motivation for this work came
in fact from someone who was looking to provision applications as services
on top of VMWare, but found that no existing solution supported the
system interfaces his applications needed without dragging an entire
classic OS along for the ride.
</p>
<p>
Let's move on to discussing what an OS needs to support for it to be able
to host for example a web server written for a regular OS such as
Linux or the BSDs. The list gets quite long. You need a file system
where the web server reads the served pages from, you need a TCP/IP
stack to communicate with the clients, and you need a network interface
driver to be able to send and receive packets. Furthermore, you need the
often overlooked, yet very surprisingly complicated system call handlers.
For example, opening a socket is not really very complicated to handle.
Neither is reading and writing data. However, when you start piling
things like <tt>fcntl(O_NONBLOCK)</tt> and <tt>poll()</tt> on top,
things get trickier. By a rough estimate, if you run an httpd on
NetBSD, approximately 100k lines of code from kernel are used just to
service the requests that the httpd makes. If you do the math (and
<tt>bc</tt> did), there are 86400 seconds in a week. The OS we are
discussing is able to run an off-the-shelf httpd, but definitely I did not
write >1 line of code per second 24/7 during the past week.
</p>
<h3>Smoke and Mirrors, CGI Edition</h3>
<p>
The key to happiness is not to write 100k lines of code from scratch, nor
to port it from another OS, as both are time-consuming and error-prone
techniques, and error-proneness leads to even more consumption of time.
Rump kernels come into the picture as the key to happiness and provide
the necessary drivers.
</p>
<p>
As the old saying goes: "<u>rump</u> kernels do
not an OS make", and we need the rest of the bits that make up the OS
side of the software stack from somewhere. These bits need to make it
seem like the drivers in a rump kernel are running inside the NetBSD
kernel, hence "smoke and mirrors". What is surprising is how little code
needs to exist between the drivers and the hardware, just some hundreds
of lines of code. More specifically, in the bare metal scenario we need
support for:
</p>
<ul>
<li>low level machine dependent code</li>
<li>thread support and a scheduler</li>
<li>rump kernel hypercall layer</li>
<li>additionally: bundling the application into a bootable image</li>
</ul>
<p>
The figure below illustrates the rump kernel software stack.
The arrows correspond to the above list (in reverse order).
We go over the list starting from the top of the list (bottom
of the figure).
</p>
<a href="http://ftp.netbsd.org/pub/NetBSD/misc/pooka/images/baremetal-rumpstack.png"><img src="http://ftp.netbsd.org/pub/NetBSD/misc/pooka/images/baremetal-rumpstack.png" alt="rump kernel software stack" width="700"/></a>
<p>
<b>Low level machine dependent
code</b> is what the OS uses to get the CPU and devices to talking terms with
the rest of OS. Before we can do anything useful, we need to bootstrap.
Bootstrapping x86-32 is less work than one would expect, which incidentally
is also why the OS runs only in 32bit mode (adding 64bit support would not
likely be many hours of work — and patches are welcome). Thanks to the <a
href="https://www.gnu.org/software/grub/manual/multiboot/multiboot.html"</a>Multiboot
specification</a>, the bootstrap code is more or less just a question
of setting the stack pointer and jumping to C code. In C code
we need to parse the amount of physical memory available and initialize
the console. Since NetBSD device drivers mainly use interrupts, we also need
interrupt support for the drivers to function correctly. On x86,
interrupt support means setting up the CPU's interrupt descriptor
tables and programming the interrupt controller. Since rump kernels
do not support interrupts, in addition we need a small interrupt
stub that transfers the interrupt request to a thread context which
calls the rump kernel. In total,
the machine dependent code is only a few hundred lines. The <a
href="http://wiki.osdev.org/">OSDev.org wiki</a> contains a lot of
information which was useful when hammering the hardware into shape.
The other source of x86 hardware knowledge was x86 support in NetBSD.
</p>
<p>
<b>Threads and scheduling</b> might sound intimidating, but they are
not. First, rump kernels can run on top of any kinds
of threads you throw at them, so we can just use the ones which are
the simplest to implement: cooperative threads. Note, simple does
not mean poorly performing threads, and in fact the predictability of
cooperative threads, at least in my opinion, makes them
more likely to perform better than preemptive threading in
cases where you are honing an OS for a single application.
Second, I already had access to an implementation
which served as the basis: Justin Cormack's work on
<a href="http://wiki.rumpkernel.org/Platforms%3A-Userspace-fibers">userspace
fibers</a>, which in turn has its roots in Xen MiniOS we use for
running <a href="http://wiki.rumpkernel.org/Platforms%3A-Xen-DomU">rump
kernel on the Xen hypervisor</a>, could be re-purposed as the threads+scheduler
implementation, with the context switch code kindly borrowed from MiniOS.
</p>
<p>
The <b>rump kernel hypercall</b> interface is what rump kernels themselves
run on. While the implementation is platform-specific, our baremetal OS
shares a large portion of its qualities with the Xen platform that was
already supported. Therefore, most of the Xen implementation applied
more or less directly. One notable exception to the similarities is
that Xen paravirtualized devices are not available on bare metal and
therefore we access all I/O devices via the PCI bus.
</p>
<p>
All we need now is the
<b>application, a.k.a. "userspace"</b>. Support for application interfaces
(POSIX syscalls, libc, etc.) readily exists for rump kernels, so we
just use what is already available. The only remaining issue is building
the bundle that we bootstrap. For that, we can repurpose Ian Jackson's
<a href="https://github.com/rumpkernel/rumprun-xen/tree/master/app-tools">app-tools</a> which were originally written for the rump kernel Xen
platform. Using app-tools, we could build a bootable image containing
thttpd simply by running the app-tools wrappers for <tt>./configure</tt>
and <tt>make</tt>. The image below illustrates part of the build output,
along with booting the image in QEMU and testing that the httpd really
works. The use of QEMU, i.e. software-emulated bare metal, is due to
convenience reasons.
</p>
<a href="http://ftp.netbsd.org/pub/NetBSD/misc/pooka/images/baremetal-thttpd.png"><img src="http://ftp.netbsd.org/pub/NetBSD/misc/pooka/images/baremetal-thttpd.png" alt="thttpd booting on bare metal" width="700"/></a>
<h3>Conclusions</h3>
<p>
You probably noticed that whole thing is just bolting a lot of working
components together while writing minimal amounts of necessary glue.
That is exactly the point: never write or port or hack what you can reuse
without modification. Code reusability has always been the strength of
NetBSD and rump kernels add another dimension to that quality.
</p>
<p>
The source code for the OS discussed in this post is
available under a 2-clause BSD license from <a href="http://repo.rumpkernel.org/
rumpuser-baremetal">repo.rumpkernel.org/rumpuser-baremetal</a>.
</p>