June 07, 2016 posted by Kamil Rytarowski
The pkgsrc team has prepared the 50th release of their package management system, with the 2016Q1 version. It's infrequent event, as the 100th release will be held after 50 quarters.
The NetBSD team has prepared series of interviews with the authors. The next one is with Jonathan Perkin, a developer in the Joyent team.
Hi Jonathan, please introduce yourself.
Hello! Thirty-something, married with 4 kids. Obviously this means life
is usually pretty busy! I work as a Software Engineer for Joyent, where
we provide SmartOS zones (also known as "containers" these days) running
pkgsrc. This means I am in the privileged position of getting paid to
work full-time on what for many years has been my hobby.
First of all, congratulations on the 50th release of pkgsrc! How do you feel about this anniversary?
I've been involved in pkgsrc since 2001, which was a few years before we
started the quarterly releases. Back then and during the early 2000s
there was a significant amount of work going into the pkgsrc
infrastructure to give it all the amazing features that we have today, but
that often meant the development branch had some rough edges while those
features were still being integrated across the tree.
The quarterly releases gave users confidence in building from a stable
release without unexpected breakages, and also helped developers to
schedule any large changes at the appropriate time.
At Joyent we make heavy use of the quarterly releases, producing new
SmartOS images for each branch, so for example our 16.1.x images are based
on pkgsrc-2016Q1, and so on.
Reaching the 50th release makes me feel old! It also makes me feel proud
that we've come a long way, yet still have people who want to be involved
and continue to develop both the infrastructure and packages.
I'd also like to highlight the fantastic work of the pkgsrc releng team,
who work to ensure the releases are kept up-to-date until the next one is
released. They do a great job and we benefit a lot from their work.
What are the main benefits of the pkgsrc system?
For me the big one is portability. This is what sets it apart from all
other package managers (and, increasingly, software in general), not just
because it runs on so many platforms but because it is such a core part of
the infrastructure and has been constantly developed and refined over the
years. We are now up to 23 distinct platforms, not counting different
distributions, and adding support for new ones is relatively easy thanks
to the huge amount of work which has gone into the infrastructure.
The other main benefit for me is the buildlink infrastructure and various
quality checks we have. As someone who distributes binary packages to end
users, it is imperative that those packages work as expected on the target
environment and don't have any embarrassing bugs. The buildlink system
ensures (amongst other things) that dependencies are correct and avoids
many issues around build host pollution. We then have a number of QA
scripts which analyse the generated package and ensure that the contents
are accurate, RPATHs are correct, etc. It's not perfect and there are
more tests we could write, but these catch a lot of mistakes that would
otherwise go undetected until a user submits a bug report.
Others for me are unprivileged support, signed packages, multi-version
support, pbulk, and probably a lot of other things I've forgotten and take
Where and how do you use pkgsrc?
As mentioned above I work on pkgsrc for SmartOS. We are probably one of
the biggest users of pkgsrc in the world, shipping over a million package
downloads per year and rising to our users, not including those
distributed as part of our images or delivered from mirrors. This is
where I spend the majority of my time working on pkgsrc, and it is all
performed remotely on a number of zones running in the Joyent Public
Cloud. The packages we build are designed to run not just on SmartOS but
across all illumos distributions, and so I also have an OmniOS virtual
machine locally where I test new releases before announcing them.
As an OS X user, I also use pkgsrc on my MacBook. This is generally where
I perform any final tests before committing changes to pkgsrc so that I'm
reasonably confident they are correct, but I also install a bunch of
different packages from pkgsrc (mutt, ffmpeg, nodejs, jekyll, pstree etc)
for my day-to-day work. I also have a number of Mac build servers in my
loft and at the Joyent offices in San Francisco where I produce the binary
OS X packages we offer which are starting to become popular among users
looking for an alternative to Homebrew or MacPorts.
Finally, I have a few Linux machines also running in the Joyent Public
Cloud which I have configured for continuous bulk builds of pkgsrc trunk.
These help me to test any infrastructure changes I'm working on to ensure
that they are portable and correct.
On all of these machines I have written infrastructure to perform builds
inside chroots, ensuring a consistent environment and allowing me to work
on multiple things simultaneously. They all have various tools installed
(git, pkgvi, pkglint, etc) to aid my personal development workflow. We
then make heavy use of GitHub and Jenkins to manage automatic builds when
pushing to various branches.
What are the pkgsrc projects you are currently working on?
One of my priorities over the past year has been on performance. We build
a lot of packages (over 40,000 per branch, and we support up to 4 branches
simultaneously), and when the latest OpenSSL vulnerability hits it's
critical to get a fix out to users as quickly as possible. We're now at
the stage where, with a couple of patches, we can complete a full bulk
build in under 3 hours. There is still a lot of room for improvement
though, so recently I've been looking at slibtool (a libtool replacement
written in C) and supporting dash (a minimal POSIX shell which is faster
There are also a few features we've developed at Joyent that I continue to
maintain, such as our multiarch work (which combines 32-bit and 64-bit
builds into a single package), additional multi-version support for MySQL
and Percona, SMF support, and a bunch of other patches which aren't yet
ready to be integrated.
I'm also very keen on getting new users into pkgsrc and turning them into
developers, so a lot of my time has been spent on making pkgsrc more
accessible, whether that's via our pkgbuild image (which gives users a
ready-made pkgsrc development environment) or the developer guides I've
written, or maintaining our https://pkgsrc.joyent.com/ website. There's
lots more to do in this area though to ensure users of all abilities can
Most of my day-to-day work though is general bug fixing and updating
packages, performing the quarterly release builds, and maintaining our
If you analyze the current state of pkgsrc, which improvements and changes do you wish for the future?
More users and developers! I am one of only a small handful of people who
are paid to work on pkgsrc, the vast majority of the work is done by our
amazing volunteer community. By its very nature pkgsrc requires constant
effort updating existing packages and adding new ones. This is something
that will never change and if anything the demand is accelerating, so we
need to ensure that we continue to train up and add new developers if we
are to keep up.
We need more documentation, more HOWTO guides, simpler infrastructure,
easier patch submission, faster and less onerous on-boarding of
developers, more bulk builds, more development machines. Plenty to be
getting on with!
Some technical changes I'd like to see are better upgrade support, launchd
support, integration of a working alternative pkg backend e.g. IPS, bmake
IPC (so we don't need to recompute the same variables over and over), and
Do you have any practical tips to share with the pkgsrc users?
Separate your build and install environments, so e.g. build in chroots or
in a VM then deploy the built packages to your target. Trying to update a
live install is the source of many problems, and there are few things more
frustrating than having your development environment be messed up by an
upgrade which fails part-way through.
For brand new users, document your experience and tell us what works and
what sucks. Many of us have been using pkgsrc for many many years, and
have lost your unique ability to identify problems, inconsistencies, and
If you run into problems, connect to Freenode IRC #pkgsrc, and we'll try
to help you out. Hang out there even if you aren't having problems!
Finally, if you like pkgsrc, tell your friends, write blog posts, post to
Hacker News etc. It's amazing to me how unknown pkgsrc is despite being
around for so long, and how many people love it when they discover it.
More users leads to more developers, which leads to improved pkgsrc, which
leads to more users, which...
What's the best way to start contributing to pkgsrc and what needs to be done?
Pick something that interests you and just start working on it. The great
thing about pkgsrc is that there are open tasks for any ability, from
documentation fixes all the way through adding packages to rewriting large
parts of the infrastructure.
When you have something to contribute, don't worry about whether it's
perfect or how you are to deliver it. Just make it available and let us
know via PR, pull request, or just mail, and we can take it from there.
Do you plan to participate in the upcoming pkgsrcCon 2016 in Kraków (1-3 July)?
I am hoping to. If so I usually give a talk on what we've been working on
at Joyent over the past year, and will probably do the same.