USBNET: A story of networking and threads that won't stop pulling

Once upon a time a developer wrote a temperature sensor driver for one system and decided to port it to another, similar, system. For days and then weeks, the developer waited and waited for the other, similar, system to arrive.

One day, joy of joys, the other, similar, system arrived. A National Day of Celebration commenced, and the porting effort began. Over hours, perhaps even as many as five hours of time, the sensors were finally able to tell you whether they were hot or not.

This other, similar, system suddenly was purposeless and was pondering what else life could provide when the Remote Server task popped up and said "Hey, first media file is free", and sadly, this other, similar, system tried its first media file, and then purchased many, many more media files, and suddenly this other, similar, system was suddenly hooked.

Unfortunately, this other, similar, system had a problem talking on the network without falling asleep on the job, so the developer says "let's try a USB network instead!", and initially this seemed like a good idea. Many bits were transferred over USB, but soon whispers of a lurking monster, known to developers, experience or otherwise, as KASSERT, were heard and soon found common.

The developer attempted other USB network as the other, similar, system was destined to be flown many thousands of miles away soon, but the only other option was similarly plagued by the KASSERT monster. The developer reached into his Bag Of Holding and pulled out a magical weapon known capable of slaying the KASSERT monster. The mighty blade of MPSAFE was free again!

After much planning and many failed attacks, the developer was able to slay the KASSERT monster and all the bits wanting to be transferred were able to be.

For a day and for a night there were celebrations. Much food and ale was consumed until finally everyone was asleep, and the music and lights were finally turned off. In the morning a great clean up was needed and while the developer was cleaning off the shiny, happy and working USB network the original USB network was accidentally reconnected. Oh no, the KASSERT monster has returned! Lock your doors and hide your children.

The developer quickly pulled out MPSAFE again, and once again the KASSERT monster was slain, though the face of this monster was different to the previous monster. The developer then searched and searched for surely they were more KASSERT monsters to be found. Indeed, many many others were found, though they retreated to safety after two more of their number were slain by the mighty MPSAFE.

The developer called upon his friends Shared and Code and with them forged a new weapon against the KASSERT monster, using the mighty MPSAFE in ways unheard of before. After much research and planning, and with the help of some friends, the USBNET was born. All the angels and all the birds of the world were said to sing all at once at this moment, for the USBNET would bring happiness to both the Code Deletionist and the Code Sharers, bound to war against each other from time immemorial.

With this new USBNET the developer was able to blaze a trail across the landscape, searching out each KASSERT monster lurking in every USB network corner. All told, fourteen faces of KASSERT monster were found and the developer and his friends have slain seven of these faces, with the remaining seven under attack, life looks grim for them.

The other, similar, system is safe now. Turns out that MPSAFE also has cleared up the sleeping problem using the cousins, NET and FDT in a tight, dual-blade combination.

Let the world rejoice, for soon the KASSERT monster will be no more!

--mrg @ 2019-08-11

tl;dr:

i fixed many bugs across several USB ethernet adapters and got sick of fixing the same bug across multiple drivers so made common code for them to use instead. the original 4 drivers fixed were axen(4), axe(4), cdce(4), and ure(4). the current list of fixed drivers, at time of writing, includes smsc(4), udav(4) and urndis(4). all drivers except umb(4) are ported but either not tested or not yet working with the usbnet(9) framework.

update 2019-09-02:

all 13 known working drivers converted and will be present in netbsd 9.

[0 comments]

 

Postfix 2.8.6 imported into NetBSD-current

Postfix 2.8.6 was imported into NetBSD-current last Friday. The changes since Postfix 2.8.5 are:

• The Postfix SMTP daemon sent "bare" newline characters instead of <CR><LF> when a header_checks REJECT pattern matched multi-line header. This bug was introduced with Postfix 1.1.

• The Postfix SMTP daemon sent "bare" newline characters instead of <CR><LF> when an smtpd_proxy_filter returned a multi-line response. This bug was introduced with Postfix 2.1.

• For compatibility with future EAI (email address internationalization) implementations, the Postfix MIME processor no longer enforces the strict_mime_encoding_domain check on unknown message subtypes such as message/global*. This check is disabled by default.

• The Postfix master daemon could report a panic error ("master_spawn: at process limit") after the process limit for some service was reduced with "postfix reload". This bug existed in all Postfix versions.

[0 comments]

 

Postfix 2.8.5 imported into NetBSD-current

Postfix 2.8.5 was imported into NetBSD-current today. The changes since Postfix 2.8.4 are:

• The Postfix Milter client logged a "milter miltername: malformed reply" error when a Milter sent an SMTP response without enhanced status code (i.e. "XXX Text" instead of "XXX X.X.X Text").
• The Postfix Milter client sent a random {client_connections} macro value when the remote SMTP client was not subject to any smtpd_client_* limit. As a workaround, it now sends a zero value instead.

[0 comments]

 

Postfix 2.8.4 imported into NetBSD-current

Postfix 2.8.4 was imported into NetBSD-current today. The changes since Postfix 2.8.2 are:

• Performance: a high load of DSN success notification requests could slow down the queue manager. Solution: make the trace client asynchronous, just like the bounce and defer clients.
• The local(8) delivery agent ignored table lookup errors in mailbox_command_maps, mailbox_transport_maps, fallback_transport_maps and (while bouncing mail to alias) alias owner lookup.
• Workaround: dbl.spamhaus.org rejects lookups with "No IP queries" even if the name has an alphanumerical prefix. We play safe, and skip both RHSBL and RHSWL queries for names ending in a numerical suffix.
• The "sendmail -t" command reported "protocol error" instead of "file too large", "no space left on device" etc.
• The Postfix Milter client reported a temporary error instead of "file too large" in three cases.

[0 comments]

 

Postfix 2.8.2 imported into NetBSD-current

Postfix 2.8.2 was imported into NetBSD-current today. The changes since Postfix 2.8.1 are:

• Bugfix: postscreen DNSBL scoring error. When a client disconnected and then reconnected before all DNSBL results for the earlier session arrived, DNSBL results for the earlier session would be added to the score for the later session. This is very unlikely to have affected any legitimate mail.
• Workaround: the SMTP client did not support mail to [ipv6:ipv6addr].

[0 comments]

 

Postfix 2.8.1 imported into NetBSD-current

Postfix 2.8.1 was imported into NetBSD-current today. The changes since Postfix 2.7.* are:

• The postscreen daemon (a zombie blocker in front of Postfix) is now included with the stable release. postscreen now supports TLS and can log the rejected sender, recipient and helo information. See the POSTSCREEN_README file for recommended usage scenarios.
• Support for DNS whitelisting (permit_rhswl_client), and for pattern matching to filter the responses from DNS white/blacklist servers (e.g., reject_rhsbl_client zen.spamhaus.org=127.0.0.[1..10]).
• Improved message tracking across SMTP-based content filters; the after-filter SMTP server can log the before-filter queue ID (the XCLIENT protocol was extended).
• Support for 'footers' that are appended to SMTP server "reject" responses. See "smtpd_reject_footer" in the postconf(5) manpage.

[2 comments]

 

Postfix 2.7.2 imported into NetBSD-current

Postfix 2.7.2 was imported into NetBSD-current today. The changes since Postfix 2.7.1 are:

• Postfix no longer automatically appends the system default CA (certificate authority) certificates, when it reads the CA certificates specified with {smtp, lmtp, smtpd}_tls_CAfile or with {smtp, lmtp, smtpd}_tls_CApath. This prevents third-party certificates from getting mail relay permission with the permit_tls_all_clientcerts feature. Unfortunately, this change may cause compatibility problems with configurations that rely on certificate verification for other purposes. To get the old behavior, specify "tls_append_default_CA = yes".
• A prior fix for compatibility with Postfix < 2.3 was incomplete. When pipe-to-command delivery fails with a signal, mail is now correctly deferred, instead of being returned to sender.
• Poor smtpd_proxy_filter TCP performance over loopback (127.0.0.1) connections was fixed by adapting the output buffer size to the MTU.
• The SMTP server no longer applies the reject_rhsbl_helo feature to non-domain forms such as network addresses. This would cause false positives with dbl.spamhaus.org.
• The Postfix SMTP server failed to deliver a "421" response and hang up the connection after Milter error. Instead, the server delivered a "503 Access denied" response and left the connection open, due to some Postfix 1.1 workaround for RFC 2821.
• The milter_header_checks parser failed to enable any of the actions that have no effect on message delivery (warn, replace, prepend, ignore, dunno, and ok).

[0 comments]

 

Postfix 2.7.1 imported into NetBSD-current

Postfix 2.7.1 was imported into NetBSD-current today. The major changes since Postfix 2.6.6 are:

• Improved before-queue content filter performance. With smtpd_proxy_options = speed_adjust, the Postfix SMTP server receives the entire message before it connects to a before-queue content filter. Typically, this allows Postfix to handle the same mail load with fewer content filter processes.
• Improved address verification performance. The verify database is now persistent by default, and it is automatically cleaned periodically. Under overload conditions, the Postfix SMTP server no longer waits up to 6 seconds for an address probe to complete.
• Support for reputation management based on the local SMTP client IP address. This is typically implemented with FILTER transportname: actions in access maps or header/body checks, and mail delivery transports in master.cf with unique smtp_bind_address values.

[0 comments]

 

Postfix 2.6.6 imported into NetBSD-current

Postfix 2.6.6 was imported into NetBSD-current today. The following bugs have been fixed since version 2.6.5:

• postmulti -p command did not skip disabled instances.
• In the multi_instance_wrapper parameter, the expansion of $command_directory and $daemon_directory was broken.
• The address_verify_poll_count parameter value was not made stress-dependent by default. This defeated the purpose of making other settings stress-dependent by default with Postfix 2.6.
• Milter applications would hang up after receiving an unexpected SMFIC_HEADER (mail header) command. This problem happened with Milters that (legitimately) do not send replies for SMFIC_RCPT (recipient address) or SMFIC_DATA (start of message) commands.
• Core dump while an printing error message for a malformed %<letter> sequence in LDAP, MySQL or PostgreSQL lookup table configuration.
• Mail with zero recipients was forever stuck in the queue. This happened when postsuper -r was run after all the recipients of a message were delivered (or bounced), but before the message was deleted from the queue.
• With hostnames such as 1-2-3-4, the valid_hostname() fuction did not recognize the - as a non-numeric character, causing a legitimate name to be rejected as invalid.
• The VRFY command did not accept a mailbox address inside <>.

[0 comments]

 

openresolv imported into NetBSD

openresolv has been imported into NetBSD, which allows more than one daemon to update /etc/resolv.conf sanely and configure local nameservers for enhanced DNS, especially if running on a VPN. dhcpcd already uses resolvconf when available and dhclient in NetBSD has been patched to use it.

This is important for NetBSD, as many packages support resolvconf, but only when /sbin/resolvconf exists. This meant that a lot of packages that supported resolvconf, failed to work with any resolvconf implementation from pkgsrc.

PPP users who maintain their own scripts are encouraged to try it out :)

[1 comment]

 

Summer of Code results: A tool to dump and restore pf(4) state

This summer I mentored Arnaud Degroote's Summer of Code project 'A tool to dump/restore the pf state table'.

[Read More] [0 comments]

 

Postfix 2.6.5 imported into NetBSD-current

Postfix 2.6.5, the latest stable version of the popular mail transport agent, was imported into NetBSD-current today. The following bugs have been fixed since version 2.6.2:

• The Postfix Milter client got out of step with a Milter application after the application sent a "quarantine" request at end-of-message time. The Milter application would still be in the end-of-message state, while Postfix would already be working on the next SMTP event, typically, QUIT or MAIL FROM. In the latter case, Milter responses for the previously-received email message would be applied towards the next MAIL FROM transaction. This problem was diagnosed with help from Alban Deniz.
• The Postfix SMTP server would abort with an "unexpected lookup table" error when an SMTPD policy server was mis-configured in a particular way.

[0 comments]