NetBSD Bloghttps://blog.netbsd.org/tnf/feed/entries/atom2024-03-17T12:20:22+00:00Apache Roller (incubating)https://blog.netbsd.org/tnf/entry/gsoc_2018_reports_configuration_files3GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 4: configuration deployment, pkgtools and future improvementsThomas Klausner2018-08-26T12:24:59+00:002018-08-26T12:24:59+00:00<p>
As introduced in <a href=" http://blog.netbsd.org/tnf/entry/gsoc_2018_reports_configuration_files1">GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 2: remote repositories (git and CVS)</a> pkgsrc supports using some version control system repositories to search for site-specific configuration for packages being installed, and deploys it on the system.
</p>
<h2>Configuration deployment: intro to VCSCONFPULL</h2>
<p>
Support for configuration tracking is in scripts, pkginstall scripts, that get built into binary packages and are run by <code>pkg_add</code> upon installation. The idea behind the proposal suggested that users of the new feature should be able to store revisions of their installed configuration files, and of package-provided default, both in local or remote repositories. With this capability in place, it doesn't take much to make the scripts "pull" configuration from a VCS repository at installation time.
</p>
<p>
That's what setting <code>VCSCONFPULL=yes</code> in <code>pkg_install.conf</code> after having enabled <code>VCSTRACK_CONF</code> does:
You are free to use official, third party prebuilt packages that have no customization in them, enable these options, and point pkgsrc to a private conf repository. If it contains custom configuration for the software you are installing, an attempt will be made to use it and install it on your system. If it fails, pkginstall will fall back to using the defaults that come inside the package. RC scripts are always deployed from the binary package, if existing and <code>PKG_RCD_SCRIPTS=yes</code> in pkg_install.conf or the environment.
</p>
<p>This will be part of packages, not a separate solution like configuration management tools. It doesn't support running scripts on the target system to customize the installation, it doesn't come with its domain-specific language, it won't run as a daemon or require remote logins to work. It's quite limited in scope, but you can define a <code>ROLE</code> for your system in pkg_install.conf or in the environment, and pkgsrc will look for configuration you or your organization crafted for such a role (e.g., public, standalone webserver vs reverse proxy or node in a database cluster)
</p>
<p>Configuration files should be put in branches named according to a specific convention, their paths relative to the repository is implied to be their absolute path, prepending a "/", on the target system.</p>
<p>Branch names define the package name they contain configuration for, the role, the version of the package the configuration is made for and a range of compatible software releases that should work with such configuration options, a best match is attemped inside of compatible ranges to the nearest software release, and roles can be undefined (both on the branch, to make it apt for any system, or on the target to ignore roles specified in branch names where they don't matter). More on that later, now some practical examples using git:
</p>
<p>Set up a remote repository on some machine, if you are not using a public facing service or private repositories in third party services:</p>
<pre>
$ id
uid=1001(vcs) gid=1001(vcs) groups=1001(vcs)
$ mkdir confpullexample.git
$ cd confpullexample.git/
$ git --bare init
Initialized empty Git repository in /usr/home/vcs/confpullexample.git/
</pre>
<p>Check for correct settings on the target machine: (a <code>REMOTEVCS</code> URI is required, the <code>VCSDIR</code> is NOT used)</p>
<pre>
pkghost# cat /usr/pkg/etc/pkg_install.conf
VCSTRACK_CONF=yes
VCSCONFPULL=yes
REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/confpullexample.git
VCS=git
</pre>
<p>Do no reuse repositories that track existing configuration files from packages, set a new one here. It will be structured differently, with branches that can specialize between them and yet contain configuration files common to each one (from master/trunk/...). Some technicalities are needed before making practical examples. I will now cheat and cite direcly the comment in <code>mk/pkginstall/versioning</code>:</p>
<p>
The remote configuration repository should contain branches named
according to the following convention:
<code>
category_pkgName_pkgVersion_compatRangeStart_compatRangeEnd_systemRole
</code>
an optional field may exist that explicitates part of the system hostname
<code>
category_pkgName_pkgVersion_compatRangeStart_compatRangeEnd_systemRole_hostname
</code>.
</p>
<p>
the branch should contain needed configuration files. Their path relative
to the repository is then prepended with a "/" and files force copied
to the system and chmod 0600 executed on them.
Permission handling and removal upon package uninstallation are not supported.
</p>
<p>
The branch to be used, among the available ones, is chosen this way:
branches named according to the convention that provide configuration
for <code>category/packageName</code> are filtered from the VCS output;
then, all branches whose ranges are compatible with the version of the
package being installed are selected. The upper bound of the range is
excluded as a compatible release if using sequence based identifiers.
If system role is set through the <code>ROLE</code> environment variable,
and it's different from "any",
and branches exists whose role is different from "any", then their
role gets compared with the one defined on the system or in pkg_add config.
The last part of the branch name is optional and, if present, is compared
character by character with the system hostname,
finally selecting the branches that best match it.
As an example, a branch named <code>mail_postfix_3.3.0_3.0.0_3.3.20_mailrelay_ams</code>
will match with system hostname <code>amsterdam09</code>.
A system with its <code>ROLE</code> unspecified or set to <code>ANY</code> will select branches
independently of the role they are created for, scoring and using the one
with the best matching optional hostname and/or nearest to the target release
as explained below:
</p>
</p>
The checks now further refine the candidates: if a branch pkgVersion exactly
corresponds with the version of the package being installed,
that branch gets selected, otherwise the procedure uses the one
which is closest to the package version being installed.
Non-numerical values in package versions are accounted for
when checking for an exact match, and are otherwise ignored.
Only integer versions and dot-separated sequence based identifiers are
understood when checking for compatible software ranges and for the closest
branch, if no branch exactly matches with the package version being installed.
Dates are handled provided they follow the ISO 8601 scheme: YYYY-MM-DD, YYYYMMDD
</p>
<p>
Let's suppose that an hypothetical team uses a common ssh configuration, on all systems, to disable root and passwordless logins, and enable logins from users in a specific group.</p>
<p>the main/master branch of the repository will then contain one custom object,
<code>
etc/ssh/sshd_config
</code>
that will get included when branching from there (specific removals are always possible).<p>
<pre>
$ whoami
devuser
$ file $HOME/.ssh/id_rsa
/home/devuser/.ssh/id_rsa: PEM RSA private key
$ pwd
/home/devuser/sim
$ git clone ssh://vcs@192.168.100.112/usr/home/vcs/confpullexample.git .
Cloning into '.'...
The authenticity of host '192.168.100.112 (192.168.100.112)' can't be established.
ECDSA key fingerprint is SHA256:RMkiZlYqNIKlgDQUvhFBXLUpW2qcLd1nuEi4NaROkLg.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.100.112' (ECDSA) to the list of known hosts.
warning: You appear to have cloned an empty repository.
$ mkdir -p etc/ssh
$ cat /etc/ssh/sshd_config > etc/ssh/sshd_config
</pre>
<p>
customize it as needed, then
</p>
<pre>
$ echo "PermitEmptyPasswords no" >> $HOME/sim/etc/ssh/sshd_config
$ echo "PermitRootLogin no" >> $HOME/sim/etc/ssh/sshd_config
$ echo "AllowGroup ops" >> $HOME/sim/etc/ssh/sshd_config
</pre>
<p>and load it in the repo:<p>
<pre>
$ git add etc/ssh/sshd_config
$ git commit -m "add common sshd_config to master"
[master (root-commit) ec91ffc] add common sshd_config to master
1 file changed, 134 insertions(+)
create mode 100644 etc/ssh/sshd_config
$ git push
Counting objects: 5, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (5/5), 1.89 KiB | 966.00 KiB/s, done.
Total 5 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/confpullexample.git
* [new branch] master -> master
</pre>
<p>With respect to this example, two different nginx configuration sets will exist, one for reverse proxies (role <code>reverseproxy</code>) and one for standalone webservers (role <code>webserver</code>). A standalone webserver will also install a default database configuration file, which will be customized for clustered postgresql instances (roles <code>dbcluster-master</code> and <code>dbcluster-replicas</code>).</p>
<p>Furthermore, nodes part of the same cluster will have hostnames the likes of <code>toontowndc-node03</code> so that branches ending with <code>_toontowndc-node</code> will match and have configuration files with the addresses of nodes in the same cluster deployed.</p>
<p>The branch <code>net_net-snmp_5.7.3_5.2_6_any</code> will work on all systems you want to monitor via snmpd independently of their defined role, provided the version of net-snmp being installed is >= 5.2 and < 6</p>
<p>In order to keep the tutorial short, I will only show how to deploy configuration for postgresql.</p>
<p>I'll start with a generic configuration file, for the role <code>webserver</code>;
the branch <code>databases_postgresql10_10.4nb1_10.0_11_default</code> may be created to contain default configuration from the package (the role isn't any, so it will not be accidentally selected on systems that define a role), then branch it into the specialized roles webserver, dbcluster-master, dbcluster-deplicas.
</p>
<p>A master-node address specific to each cluster site will be included in <code>recovery.conf</code> by specializing the branch ...<code>dbcluster-replicas</code> to branches specific for earch location, e.g., <code>dbcluster-replicas_toontowndc</code> that will be deployed on db nodes in the "Toontown" site.
</p>
<p>NOTE that files from the branch will get copied unconditionally on the system, replacing existing files or files coming with the binary package, even if they are not installed to location/handled by the <code>+FILES</code> script</p>
<p>This means that it's possible to write configuration files to <code>${pgsql_home}/data</code>, by default <code>/usr/pkg/pgsql/data/</code>, where the rc.d script usually sets postgresql to look into.</p>
<pre>
$ pwd
/home/devuser
$ mkdir -p sim/usr/pkg/pgsql/data
$ cd sim
$ ls -a
. .. .git etc usr
$ git checkout -b databases_postgresql10_10.4nb1_10.0_11_default
Switched to a new branch 'databases_postgresql10_10.4nb1_10.0_11_default'
$ ls
etc usr
$ cp /home/devuser/postgresql.conf.sample usr/pkg/pgsql/data/postgresql.conf
$ cp /home/devuser/pg_hba.conf.sample usr/pkg/pgsql/data/pg_hba.conf
$ vi usr/pkg/pgsql/data/pg_hba.conf
$ tail -n 23 usr/pkg/pgsql/data/pg_hba.conf
# Put your actual configuration here
# ----------------------------------
#
# If you want to allow non-local connections, you need to add more
# "host" records. In that case you will also need to make PostgreSQL
# listen on a non-local interface via the listen_addresses
# configuration parameter, or via the -i or -h command line switches.
@authcomment@
# TYPE DATABASE USER ADDRESS METHOD
@remove-line-for-nolocal@# "local" is for Unix domain socket connections only
@remove-line-for-nolocal@local all all ident
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
@remove-line-for-nolocal@local replication all ident
#host replication all 127.0.0.1/32 md5
#host replication all ::1/128 md5
$ echo "log_destination = 'syslog'" >> usr/pkg/pgsql/data/postgresql.conf
$ echo "syslog_ident = 'postgres'" >> usr/pkg/pgsql/data/postgresql.conf
$ git add usr/pkg/pgsql/*
$ git commit -m "import common config for not to be deployed default role"
[databases_postgresql10_10.4nb1_10.0_11_default 1ce080d] import common config for not to be deployed default e
2 files changed, 749 insertions(+)
create mode 100644 usr/pkg/pgsql/data/pg_hba.conf
create mode 100644 usr/pkg/pgsql/data/postgresql.conf
$ git push --set-upstream origin databases_postgresql10_10.4nb1_10.0_11_default
Counting objects: 8, done.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (8/8), 9.10 KiB | 4.55 MiB/s, done.
Total 8 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/confpullexample.git
* [new branch] databases_postgresql10_10.4nb1_10.0_11_default -> databases_postgresql10_10.4nb1_10.0_11_default
Branch 'databases_postgresql10_10.4nb1_10.0_11_default' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_default'
</pre>
<p>the branch with role webserver will get deployed, and doesn't differ in configuration from the default branch:</p>
<pre>
$ git checkout -b databases_postgresql10-server_10.4nb1_10.0_11_webserver
Switched to a new branch 'databases_postgresql10-server_10.4nb1_10.0_11_webserver'
$ git push --set-upstream origin databases_postgresql10_10.4nb1_10.0_11_webserver
Total 0 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/confpullexample.git
* [new branch] databases_postgresql10_10.4nb1_10.0_11_webserver -> databases_postgresql10_10.4nb1_10.0_11_webserver
Branch 'databases_postgresql10_10.4nb1_10.0_11_webserver' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_webserver'
</pre>
<p>now change it for the role dbcluster-master:</p>
<pre>
$ git checkout -b databases_postgresql10_10.4nb1_10.0_11_dbcluster-master
M usr/pkg/pgsql/data/pg_hba.conf
M usr/pkg/pgsql/data/postgresql.conf
Switched to a new branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-master'
$ vi usr/pkg/pgsql/data/pg_hba.conf
$ tail -n 23 usr/pkg/pgsql/data/pg_hba.conf
# Put your actual configuration here
# ----------------------------------
#
# If you want to allow non-local connections, you need to add more
# "host" records. In that case you will also need to make PostgreSQL
# listen on a non-local interface via the listen_addresses
# configuration parameter, or via the -i or -h command line switches.
@authcomment@
# TYPE DATABASE USER ADDRESS METHOD
@remove-line-for-nolocal@# "local" is for Unix domain socket connections only
@remove-line-for-nolocal@local all all ident
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
@remove-line-for-nolocal@local replication all ident
host replication all 127.0.0.1/32 trust
host replication all ::1/128 trust
$ cat << EOF >> usr/pkg/pgsql/data/postgresql.conf
> listen_addresses = '*'
> wal_level = hot_standby
> max_wal_senders = 10
> hot_standby = on
> archive_mode = on
> archive_command = 'cp %p /usr/pkg/pgsql/archive/%f'
> EOF
</pre>
<p>No automation exists to run mkdir other than creating a placeholder file. you will have to chown the archive dir!
File permissions are not yet handled. This may change in the future, by reusing the +DIRS script with new input for example, if deemed necessary by the community.</p>
<pre>
$ mkdir -p usr/pkg/pgsql/archive/
$ echo "remember to chown this dir to the user pgsql runs as" > usr/pkg/pgsql/archive/placeholder
$ git add usr/pkg/pgsql/archive/placeholder
$ git commit -m "create dbcluster-master role for postgresql10"
[databases_postgresql10_10.4nb1_10.0_11_dbcluster-master 98fe514] create dbcluster-master role for postgresql0
3 files changed, 9 insertions(+), 2 deletions(-)
create mode 100644 usr/pkg/pgsql/archive/placeholder
$ git push -u origin databases_postgresql10_10.4nb1_10.0_11_dbcluster-master
Counting objects: 10, done.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (10/10), 827 bytes | 827.00 KiB/s, done.
Total 10 (delta 2), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/confpullexample.git
* [new branch] databases_postgresql10_10.4nb1_10.0_11_dbcluster-master -> databases_postgresql10_10.4nb1_10.0_11_dbcluster-master
Branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-master' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-master'
</pre>
<p>create a generic node/replica configuration from the default config files and add the recovery.conf file...</p>
<pre>
$ git checkout databases_postgresql10_10.4nb1_10.0_11_default
Switched to branch 'databases_postgresql10_10.4nb1_10.0_11_default'
Your branch is up to date with 'origin/databases_postgresql10_10.4nb1_10.0_11_default'.
$ mkdir -p usr/pkg/pgsql/archive/
$ echo "remember to chown this dir to the user pgsql runs as" > usr/pkg/pgsql/archive/placeholder
$ cp /home/devuser/recovery.conf.sample usr/pkg/pgsql/data/recovery.conf
$ echo "standby_mode = 'on'" >> usr/pkg/pgsql/data/recovery.conf
$ echo "restore_command = 'cp /usr/pkg/pgsql/archive/%f \"%p%\"'" >> usr/
$ git checkout -b databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas
A usr/pkg/pgsql/archive/placeholder
A usr/pkg/pgsql/data/recovery.conf
Switched to a new branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas'
$ git add usr/pkg/pgsql/archive/placeholder
$ git add usr/pkg/pgsql/data/recovery.conf
$ git add usr/pkg/pgsql/data/postgresql.conf
$ git add usr/pkg/pgsql/data/pg_hba.conf
$ git commit -m "create generalized dbcluster-replica role schema"
[databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas 9260a88] create generalized dbcluster-replica role schema
2 files changed, 161 insertions(+)
create mode 100644 usr/pkg/pgsql/archive/placeholder
create mode 100644 usr/pkg/pgsql/data/recovery.conf
$ git push -u origin databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas
Counting objects: 9, done.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (9/9), 2.71 KiB | 2.71 MiB/s, done.
Total 9 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/confpullexample.git
* [new branch] databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas -> databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas
Branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas'.
</pre>
<p>Now specialize the schema for the cluster at @ToonTown by specifying part of the hostname the machines will have in that cluster:</p>
<pre>
$ git checkout -b databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node
Switched to a new branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node'
$ echo "primary_conninfo = 'host=10.0.10.200 port=5432 user=replicauser application_name=toons'" >> usr/pkg/pgsql/data/recovery.conf
$ git add usr/pkg/pgsql/data/recovery.conf
$ git commit -m "create specialized branch for dbcluster at ToonTown dc"
[databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node 90d9731] create specialized branch for dbcluster at ToonDown dc
1 file changed, 1 insertion(+)
$ git push -u origin databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node
Counting objects: 7, done.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (7/7), 604 bytes | 604.00 KiB/s, done.
Total 7 (delta 2), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/confpullexample.git
* [new branch] databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node -> databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node
Branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node'.
</pre>
<p>Let's try to deploy it on a node!</p>
<pre>
pkghost# whoami
root
pkghost# hostname
pkghost
pkghost# cat /usr/pkg/etc/pkg_install.conf
VCSTRACK_CONF=yes
VCSCONFPULL=yes
REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/confpullexample.git
VCS=git
pkghost# echo "ROLE=dbcluster-replicas" >> /usr/pkg/etc/pkg_install.conf
pkghost# hostname toontowndc-node05
toontowndc-node05# #show use with pkg_add and a local package, for once
toontowndc-node05# /usr/pkg/sbin/pkg_add ./postgresql10-10.3.tgz
Trying to deploy configuration from ssh://vcs@192.168.100.112/usr/home/vcs/confpullexample.git via git
About to use remote branch databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node
Cloning into 'confpullexample'...
remote: Enumerating objects: 45, done.
remote: Counting objects: 100% (45/45), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 45 (delta 5), reused 0 (delta 0)
Receiving objects: 100% (45/45), 15.23 KiB | 12.00 KiB/s, done.
Resolving deltas: 100% (5/5), done.
/tmp/pkgsrcdeploy-17103/work/etc/ssh/sshd_config -> /etc/ssh/sshd_config
/tmp/pkgsrcdeploy-17103/work/usr/pkg/pgsql/archive/placeholder -> /usr/pkg/pgsql/archive/placeholder
/tmp/pkgsrcdeploy-17103/work/usr/pkg/pgsql/data/pg_hba.conf -> /usr/pkg/pgsql/data/pg_hba.conf
/tmp/pkgsrcdeploy-17103/work/usr/pkg/pgsql/data/postgresql.conf -> /usr/pkg/pgsql/data/postgresql.conf
/tmp/pkgsrcdeploy-17103/work/usr/pkg/pgsql/data/recovery.conf -> /usr/pkg/pgsql/data/recovery.conf
toontowndc-node05# tail /etc/ssh/sshd_config
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# PermitTTY no
# ForceCommand cvs server
PermitEmptyPasswords no
PermitRootLogin no
AllowGroup ops
toontowndc-node05# tail /usr/pkg/pgsql/data/recovery.conf
#---------------------------------------------------------------------------
# HOT STANDBY PARAMETERS
#---------------------------------------------------------------------------
#
# Hot Standby related parameters are listed in postgresql.conf
#
#---------------------------------------------------------------------------
standby_mode = 'on'
restore_command = 'cp /usr/pkg/pgsql/archive/%f "%p%"'
primary_conninfo = 'host=10.0.10.200 port=5432 user=replicauser application_name=toons'
toontowndc-node05# ls /tmp/pkgsrcdeploy-17103
ls: /tmp/pkgsrcdeploy-17103: No such file or directory
</pre>
<p>The same way goes if you should choose to set <code>VCS=hg</code> or <code>VCS=svn</code> in <code>pkg_install.conf</code>. All are supported when pulling configuration, you should adapt to managing branches the way these other VCSs expect you to work with them. It might be in scope for this tutorial, but it's getting a bit long!
Also remember to set an appropriate URI in REMOTEVCS for these other Version Control Systems.</p>
<p>
CVS is not supported: tags (branch names) cannot contain dots with CVS, and this breaks the chosen naming scheme for many software applications. It could be replaced with another special meaning character when using this vcs, but this workaround would make for possible confusion in branch naming. Furthermore, as with svn, there is no way to list remote branches before getting a local copy of the repository, with the added headcache of having to parse its log to extract branch names. I considered it not to be worth the hassle, but I'm open to adding PULL mode support for cvs if required by any user.</p>
<p>
One word about subversion: I expect you to follow conventions and keep branches at /branches/ in your repository!</p>
<h2>pkgconftrack store: storing changes when not upgrading packages</h2>
<p>
Even when merging changes in configuration files is not attempted because of <code>VCSAUTOMERGE=no</code> or unset, pkgsrc will keep track of installed configuration files by committing them to the configuration repository, as a user-modified file under <code>user/$filePath</code> (see mk/pkginstall/files).
No VCS will commit files that haven't changed since the last revision (I hope!).
</p>
<p>But what if you made changes to a configuration file and you want to store it in the configuration repository as a manually edited file, neither waiting for its package to be updated nor forcing a reinstallation? You could interact with your vcs of choice direcly, or just use the new script in pkgsrc at <code>pkgtools/pkgconftrack</code>
</p>
<pre>
pkgconf# cd pkgtools/pkgconftrack/
pkgconf# make install
===> Skipping vulnerability checks.
[...]
===> Installing binary package of pkgconftrack-1.0
pkgconf# pkgconftrack
prefix: /usr/pkg, VCSDIR: /var/confrepo, VCS: svn, REMOTEVCS: no
: unknown action
Usage: pkgconftrack [-p PREFIX] [-m commit message] store packagename [... packagenames]
</pre>
<p>
<code>pkgconftrack</code> will search for pkgsrc VCS configuration on your shell environment or in <code>pkg_install.conf</code>, reading it via <code>pkg_admin config-var $VARNAME</code>.</p>
<p>pkg_admin, in order to work with the correct configuration file, and look in the right package database to check if a package exists and its list of configuration files, is called relative to the prefix.
By default, if unspecified, <code>/usr/pkg</code> is assumed and <code>/usr/pkg/sbin/pkg_admin</code> is executed.
</p>
<p>You can work in packages of a different prefix by calling <code>pkgconftrack -p /path/to/other/prefix</code> followed by other options (you are free to chose a custom commit message with <code>-m "my commit message"</code>) and the action to be performed.
</p>
<p>
As of now, pkgconftrack only support one action: <code>store</code> followed by one or more packages you wish to store configuration files from, into the configuration repository.
</p>
<p>When storing configuration for more than one package, a unique commit is made (if using a VCS other than the old RCS, which doesn't support multi file commits and atomic transactions). This opens the way to storing config files for a service made working by a combination of software packages when these are in a known-good status, to be accessed and restored checking out one commit id.</p>
<p>So, let's say one last change was needed to have mail working:
</p>
<pre>
pkgconf# diff /usr/pkg/etc/spamd.conf /usr/pkg/share/examples/spamd/spamd.conf
7a8
> #
33c34
< :method=https:\
---
> :method=http:\
pkgconf# pkgconftrack -m "5 august 2018: mail service: blacklist only accessible via https" store spamd postfix dovecot opendkim
prefix: /usr/pkg, VCSDIR: /var/confrepo, VCS: svn, REMOTEVCS: no
Storing configuration files for opendkim
A opendkim.conf
Package not found: dovecot in the pkgdb for /usr/pkg
Package not found: postfix in the pkgdb for /usr/pkg
Storing configuration files for spamd
Adding usr/pkg/etc/opendkim.conf
Transmitting file data .done
Committing transaction...
Committed revision 7.
pkgconf#
</pre>
<p>Yeah, I haven't really installed a mail server just to test pkgconftrack, so neither dovecot nor postfix are installed on the system!</p>
<h2>future improvements</h2>
<p>
Well, it all begins with the new features being tested by end users, more bugs being found, changes made as requested.
Then, once things get more stable, the versioning script could be reimplemented as part of pkgtasks, and files.subr changed there to interact with the new functions.
</p>
<p>
More VCSs could be handled, there could be more automation in switching repositories and adding remote sources, but all this is maybe best kept in a separate tool such as pkgconftrack. Speaking of which, well, it stores installed config files for one or more packages at once, but it does not restore them yet!
RCS needs to be supported, being the default Version Control System, but it has no concept of an atomic transaction involving more than one file, so there would be nothing the user could reference to when asking the script to restore configuration. The script could track, by checking the log for all files, for identical commit message and restore each revision of each file having the same commit message, but what if some user mistakingly reused the same commit message as part of different transactions? Should each custom commit message be prepended with a timestamp?</p>
<p>All this would still lead to differences in referencing to a transaction, or the simulation thereof, and in listing available changes the user can select for restore.</p>
<p>pkgconftrack could also help in interactively reviewing conflicting automerge results, some scripts already do it, and in restoring the last user-installed file from the repository in case of breakage. This should really consist in a cp from vcsdir/user/path/to/file /path/to/installed/file, maybe preceded by a checkout, but there is good marging for making the tool more useful.</p>
<p>And yes, configuraction deployment/<code>VCSCONFPULL</code> does not handle permissions or the creation of empty directories, executable files: I think this would require making the way users interact with the configuration repository more complex, more akin to a configuration management and monitoring software, and widen the chances for mistakes when working across branches. Any good idea of how to implement these missing bits while still keeping things simple for users?
</p>
<p>I'd really like to see the code tried out! it's now at <a href="https://github.com/kmotavalli/pkgsrc/tree/gsoc">Github</p>.
https://blog.netbsd.org/tnf/entry/gsoc_2018_reports_configuration_files2GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 3: remote repositories (SVN and Mercurial)Thomas Klausner2018-08-26T12:12:53+00:002018-08-26T12:12:53+00:00<p>
In <a href=" http://blog.netbsd.org/tnf/entry/gsoc_2018_reports_configuration_files1">GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 2: remote repositories (git and CVS)</a> structural changes to the configuration tracking feature of pkgsrc got introduced, along with a short walkthrough in using remote Git and CVS repositories.
</p>
<p>This third post will introduce SVN and Mercurial support, leaving configuration deployment for the <a href=" http://blog.netbsd.org/tnf/entry/gsoc_2018_reports_configuration_files3">final</a> entry. Stay tuned!
<h2>SVN and remotes</h2>
<p>
Subversion, when used locally, needs to be set this way, as you can expect:
</p>
<pre>
pkghost# cat /usr/pkg/etc/pkg_install.conf
VCSTRACK_CONF=yes
VCS=svn
VCSDIR=/var/svnconfdir
VCSAUTOMERGE=yes
</pre>
<p>
A local repository will be created at VCSDIR/localsvn, files will get extracted and checkedin at <code>VCSDIR/defaults</code>, <code>VCSDIR/automerged</code>, <code>VCSDIR/user</code>
</p>
<pre>
pkghost# make
[...]
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Installing for spamd-20060330nb5
=> Generating pre-install file lists
=> Creating installation directories
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/man/spamd.conf.5 /ro5
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd.8 /root/8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-set8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb.8 /roo8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd.8 8
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-c
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd /rootc
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb /ron
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd c
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/etc/spamd.conf /root/d
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for spamd-20060330nb5
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb5.tgz
===> Building binary package for spamd-20060330nb5
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb5.tgz
===> Installing binary package of spamd-20060330nb5
REGISTER /var/svnconfdir/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb5: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf
Conf commit: pkgsrc: add spamd-20060330nb5
===========================================================================
The following files should be created for spamd-20060330nb5:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $
Don't forget to add the spamd ports to /etc/services:
spamd 8025/tcp # spamd(8)
spamd-cfg 8026/tcp # spamd(8) configuration
===========================================================================
pkghost# ls -lah /var/svnconfdir/localsvn/
total 3.2K
drwxr-xr-x 6 pkgvcsconf pkgvcsconf 512B Aug 4 07:22 .
drwx------ 6 pkgvcsconf pkgvcsconf 512B Aug 4 07:22 ..
-rw-r--r-- 1 pkgvcsconf pkgvcsconf 246B Aug 4 07:22 README.txt
drwxr-xr-x 2 pkgvcsconf pkgvcsconf 512B Aug 4 07:22 conf
drwxr-sr-x 6 pkgvcsconf pkgvcsconf 512B Aug 4 07:22 db
-r--r--r-- 1 pkgvcsconf pkgvcsconf 2B Aug 4 07:22 format
drwxr-xr-x 2 pkgvcsconf pkgvcsconf 512B Aug 4 07:22 hooks
drwxr-xr-x 2 pkgvcsconf pkgvcsconf 512B Aug 4 07:22 locks
pkghost# vi /usr/local/etc/spamd.conf
whitelist:\
:white:\
:method=file:\
:file=/var/mail/whitelist.txt:
/usr/pkg/etc/spamd.conf: 86 lines, 2767 characters
.
pkghost# make replace
[...]
</pre>
<p>
and so on, you can come up with what follows.
Let's test a remote repository, starting with preparations on the server:
</p>
<pre>
$ hostname
vers
$ cd /usr/home/vcs
$ svnadmin create svnremote
</pre>
<p>That's it, no branches or releases are used, so you don't need to create any further structure on the repository, the <code>+VERSIONING</code> script will take care of the remaining bits (you should migrate data from a local repository to the remote one, if you desire so, before running package installations).
</p>
<p>
Use a similarly structured <code>REMOTEVCS</code> URI: <code>svn+ssh://vcs@192.168.100.112/usr/home/vcs/svnremote</code></p>
<p>(as with CVS, you can also run it as a standalone network server without ssh transport and authentication)</p>
<pre>
pkghost# cat /usr/pkg/etc/pkg_install.conf
VCSTRACK_CONF=yes
VCS=svn
VCSDIR=/var/svnwrkremote
VCSAUTOMERGE=yes
REMOTEVCS=svn+ssh://vcs@192.168.100.112/usr/home/vcs/svnremote
pkghost# make
[...]
pkghost# make install
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Installing for spamd-20060330nb6
=> Generating pre-install file lists
=> Creating installation directories
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/man/spamd.conf.5 /ro5
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd.8 /root/8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-set8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb.8 /roo8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd.8 8
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-c
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd /rootc
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb /ron
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd c
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/etc/spamd.conf /root/d
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for spamd-20060330nb6
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb6.tgz
===> Building binary package for spamd-20060330nb6
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb6.tgz
===> Installing binary package of spamd-20060330nb6
svn: E155007: '/var/svnwrkremote/defaults' is not a working copy
Committing transaction...
Committed revision 1.
Checked out revision 1.
Committing transaction...
Committed revision 2.
Checked out revision 2.
Committing transaction...
Committed revision 3.
Checked out revision 3.
A usr
A pkg
A etc
A spamd.conf
REGISTER /var/svnwrkremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb6: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf
Adding usr
Adding usr/pkg
Adding usr/pkg/etc
Adding usr/pkg/etc/spamd.conf
Transmitting file data .done
Committing transaction...
Committed revision 4.
Conf commit: pkgsrc: add spamd-20060330nb6
===========================================================================
The following files should be created for spamd-20060330nb6:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $
Don't forget to add the spamd ports to /etc/services:
spamd 8025/tcp # spamd(8)
spamd-cfg 8026/tcp # spamd(8) configuration
===========================================================================
pkghost#
</pre>
<p>since I didn't show it with CVS, let's try again some simulated file edits:
</p>
<pre>
pkghost# tail -n 5 /usr/pkg/etc/spamd.conf
#
#whitelist:\
# :white:\
# :method=file:\
# :file=/var/mail/whitelist.txt:
pkghost# vi /usr/pkg/etc/spamd.conf; tail -n 5 /usr/pkg/etc/spamd.conf
/usr/pkg/etc/spamd.conf: 86 lines, 2767 characters
.
#
whitelist:\
:white:\
:method=file:\
:file=/var/mail/whitelist.txt:
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Replacing for spamd-20060330nb6
===> Updating using binary package of spamd-20060330nb6
/usr/bin/env /usr/pkg/sbin/pkg_add -K /var/db/pkg -U -D /root/pkgsrc/mail/spamd/work/.packages/spamd-2006033z
===========================================================================
The following users are no longer being used by spamd-20060330nb6,
and they can be removed if no other software is using them:
_spamd
===========================================================================
===========================================================================
The following groups are no longer being used by spamd-20060330nb6,
and they can be removed if no other software is using them:
_spamd
===========================================================================
===========================================================================
The following files are no longer being used by spamd-20060330nb6,
and they can be removed if no other packages are using them:
/usr/pkg/etc/spamd.conf
===========================================================================
REGISTER /var/svcwrkremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb6: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb6: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
Saving the currently user-installed revision to /var/svnwrkremote/user//usr/pkg/etc/spamd.conf
A usr
A pkg
A etc
A spamd.conf
Adding usr
Adding usr/pkg
Adding usr/pkg/etc
Adding usr/pkg/etc/spamd.conf
Transmitting file data .done
Committing transaction...
Committed revision 5.
Conf commit: pkgsrc: backup user conf before attempting merge for spamd-20060330nb6
A defaults/usr/pkg/etc/spamd.conf
Export complete.
Merged with no conflicts. installing it to /usr/pkg/etc/spamd.conf!
A usr
A pkg
A etc
A spamd.conf
A defaults/usr/pkg/etc/spamd.conf
Export complete.
Revert from the last revision of /var/svcwrkremote/user//usr/pkg/etc/spamd.conf if needed
Adding usr
Adding usr/pkg
Adding usr/pkg/etc
Adding usr/pkg/etc/spamd.conf
Transmitting file data .done
Committing transaction...
Committed revision 6.
Conf commit: pkgsrc: add spamd-20060330nb6
===========================================================================
The following files should be created for spamd-20060330nb6:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $
Don't forget to add the spamd ports to /etc/services:
spamd 8025/tcp # spamd(8)
spamd-cfg 8026/tcp # spamd(8) configuration
===========================================================================
pkghost# cd /var/svnwrkremote/
pkghost# ls
automerged automergedfiles defaults user
pkghost# cd automerged
pkghost# svn update
Updating '.':
At revision 6.
pkghost# svn log
------------------------------------------------------------------------
r6 | vcs | 2018-08-04 13:22:29 +0000 (Sat, 04 Aug 2018) | 1 line
pkgsrc: add spamd-20060330nb6
------------------------------------------------------------------------
r1 | vcs | 2018-08-04 13:19:29 +0000 (Sat, 04 Aug 2018) | 1 line
initial import
------------------------------------------------------------------------
pkghost# cd $HOME/pkgsrc/mail/spamd
pkghost# grep REVISION Makefile
PKGREVISION= 6
pkghost# sed -i "s/PKGREVISION=.*6/PKGREVISION=7/g" Makefile
pkghost# grep REVISION Makefile
PKGREVISION=7
pkghost# make clean; make extract; sed -i "s/http/https/g" work/spamd-20060330/etc/spamd.conf
pkghost# make update
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Deinstalling for spamd-20060330nb7
Running /usr/pkg/sbin/pkg_delete -K /var/db/pkg -r spamd-20060330nb6
===========================================================================
[...]
===> Installing for spamd-20060330nb7
=> Generating pre-install file lists
=> Creating installation directories
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/man/spamd.conf.5 /ro5
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd.8 /root/8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-set8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb.8 /roo8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd.8 8
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-c
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd /rootc
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb /ron
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd c
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/etc/spamd.conf /root/d
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for spamd-20060330nb7
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb7.tgz
===> Building binary package for spamd-20060330nb7
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb7.tgz
===> Installing binary package of spamd-20060330nb7
REGISTER /var/svcwrkremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb7: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb7: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
A automerged/usr/pkg/etc/spamd.conf
Export complete.
Saving the currently installed revision to /var/svcwrkremote/automerged//usr/pkg/etc/spamd.conf
Sending usr/pkg/etc/spamd.conf
Transmitting file data .done
Committing transaction...
Committed revision 7.
Conf commit: pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb7
A defaults/usr/pkg/etc/spamd.conf
Export complete.
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!
--- /usr/pkg/etc/spamd.conf 2018-08-04 09:01:09.536149671 +0000
+++ /var/svcwrkremote/defaults//usr/pkg/etc/spamd.conf.automerge 2018-08-04 09:10:52.131955252 +0000
@@ -15,7 +15,7 @@
# may be applied to each blacklist.
#
# As of November 2004, a place to search for black lists is
-# http://spamlinks.net/filter-bl.htm
+# https://spamlinks.net/filter-bl.htm
#
# Some of the URLs below point to www.openbsd.org locations. Those
# files are likely to be mirrored to other OpenBSD www mirrors located
@@ -25,45 +25,45 @@
all:\
:spews1:china:korea:
-# Mirrored from http://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2
+# Mirrored from https://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2
spamhaus:\
:black:\
:msg="SPAM. Your address %A is in the Spamhaus Block List\n\
- See http://www.spamhaus.org/sbl and\
- http://www.abuse.net/sbl.phtml?IP=%A for more details":\
- :method=http:\
+ See https://www.spamhaus.org/sbl and\
+ https://www.abuse.net/sbl.phtml?IP=%A for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/SBL.cidr.gz:
-# Mirrored from http://www.spews.org/spews_list_level1.txt
+# Mirrored from https://www.spews.org/spews_list_level1.txt
spews1:\
:black:\
:msg="SPAM. Your address %A is in the spews level 1 database\n\
- See http://www.spews.org/ask.cgi?x=%A for more details":\
- :method=http:\
+ See https://www.spews.org/ask.cgi?x=%A for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/spews_list_level1.txt.gz:
-# Mirrored from http://www.spews.org/spews_list_level2.txt
+# Mirrored from https://www.spews.org/spews_list_level2.txt
spews2:\
:black:\
:msg="SPAM. Your address %A is in the spews level 2 database\n\
- See http://www.spews.org/ask.cgi?x=%A for more details":\
- :method=http:\
+ See https://www.spews.org/ask.cgi?x=%A for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/spews_list_level2.txt.gz:
-# Mirrored from http://www.okean.com/chinacidr.txt
+# Mirrored from https://www.okean.com/chinacidr.txt
china:\
:black:\
:msg="SPAM. Your address %A appears to be from China\n\
- See http://www.okean.com/asianspamblocks.html for more details":\
- :method=http:\
+ See https://www.okean.com/asianspamblocks.html for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/chinacidr.txt.gz:
-# Mirrored from http://www.okean.com/koreacidr.txt
+# Mirrored from https://www.okean.com/koreacidr.txt
korea:\
:black:\
:msg="SPAM. Your address %A appears to be from Korea\n\
- See http://www.okean.com/asianspamblocks.html for more details":\
- :method=http:\
+ See https://www.okean.com/asianspamblocks.html for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/koreacidr.txt.gz:
#relaydb-black:\
Revert from the penultimate revision of /var/svcwrkremote/automerged//usr/pkg/etc/spamd.conf if needed
Sending usr/pkg/etc/spamd.conf
Transmitting file data .done
Committing transaction...
Committed revision 8.
Sending usr/pkg/etc/spamd.conf
Transmitting file data .done
Committing transaction...
Committed revision 9.
Conf commit: pkgsrc: add spamd-20060330nb7
===========================================================================
The following files should be created for spamd-20060330nb7:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
[...]
===> Cleaning for spamd-20060330nb7
</pre>
<h2>HG and remotes</h2>
<p>
Yes, mercurial.
Mercurial, like git, uses a local directory (namely <code>.hg</code>) located under the working directory <code>VCSDIR</code>, it is also used to store configuration and URIs to remote repositories.
to initialize and test it (as said, no repository and vcs migrations are supported by pkgsrc itself, you should take care of migrations yourself if you want to), just set <code>pkg_install.conf</code> to use a local mercurial repo and install a package:
</p>
<pre>
pkghost# rm /usr/pkg/etc/spamd.conf
pkghost# /usr/pkg/sbin/pkg_delete spamd
pkghost# vi /usr/pkg/etc/pkg_install.conf; cat /usr/pkg/etc/pkg_install.conf
/usr/pkg/etc/pkg_install.conf: 4 lines, 66 characters
.
VCSTRACK_CONF=yes
VCS=hg
VCSDIR=/var/hglocaldir
VCSAUTOMERGE=yes
pkghost# make
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
=> Checksum SHA1 OK for spamd-20060330.tar.gz
[...]
</pre>
<p>To save some space and time, let's just say that it works as expected:</p>
<pre>
[...]
Conf commit: pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb10
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!
--- /usr/pkg/etc/spamd.conf 2018-08-04 09:50:00.738099463 +0000
+++ /var/hglocaldir/defaults//usr/pkg/etc/spamd.conf.automerge 2018-08-04 09:52:18.467604445 +0000
@@ -15,7 +15,7 @@
# may be applied to each blacklist.
#
# As of November 2004, a place to search for black lists is
-# http://spamlinks.net/filter-bl.htm
+# https://spamlinks.net/filter-bl.htm
#
[...]
</pre>
<p>One nice thing about mercurial is the simplicity enabling one to clone a local repository to a remote server.
The script, when using mercurial, tries exacly that, this should succeed if the remote repository is empty.</p>
<pre>
if ${TEST} "$_REMOTE" != "no" -a "$_REMOTE" != "NO"; then
execute "hg clone . \"$_REMOTE\""
execute "hg --repository \"$_VCSDIR\" push \"$_REMOTE\""
execute "hg --repository \"$_VCSDIR\" pull \"$_REMOTE\""
fi
</pre>
<p>let's init an empty repository on the server and let pkgsrc clone the existing files over there!</p>
<pre>
$ cd
$ hostname
vers
$ pwd
/home/vcs
$ hg init pkgconftest
$ ls pkgconftest/
$ ls pkgconftest/.hg/
00changelog.i requires store
$
</pre>
<p>URIs for <code>REMOTEVCS</code> take the following format, should you choose to use ssh instead of <code>hg server</code>, <code>http</code> or other access methods that you'll find documented on official Mercurial resources, as with svn, git and cvs:
</p>
<p>
<code>ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest</code>
</p>
<pre>
pkghost# vi /usr/pkg/etc/pkg_install.conf; cat /usr/pkg/etc/pkg_install.conf
/usr/pkg/etc/pkg_install.conf: 5 lines, 127 characters
.
VCSTRACK_CONF=yes
VCS=hg
VCSDIR=/var/hglocaldir
VCSAUTOMERGE=yes
REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
pkghost# make replace
[...]
===> Updating using binary package of spamd-20060330nb10
/usr/bin/env /usr/pkg/sbin/pkg_add -K /var/db/pkg -U -D /root/pkgsrc/mail/spamd/work/.packages/spamd-2006033z
===========================================================================
The following users are no longer being used by spamd-20060330nb10,
and they can be removed if no other software is using them:
_spamd
===========================================================================
===========================================================================
The following groups are no longer being used by spamd-20060330nb10,
and they can be removed if no other software is using them:
_spamd
===========================================================================
===========================================================================
The following files are no longer being used by spamd-20060330nb10,
and they can be removed if no other packages are using them:
/usr/pkg/etc/spamd.conf
===========================================================================
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 5 changesets with 6 changes to 3 files
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
no changes found
pulling from ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
no changes found
defaults/usr/pkg/etc/spamd.conf already tracked!
REGISTER /var/hglocaldir/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb10: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb10: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
Saving the currently installed revision to /var/hglocaldir/automerged//usr/pkg/etc/spamd.conf
automerged/usr/pkg/etc/spamd.conf already tracked!
Failed to commit conf: backup preexisting conf before attempting merge for spamd-20060330nb10
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
no changes found
hg: failed to push changes to the remote repository ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!
Revert from the penultimate revision of /var/hglocaldir/automerged//usr/pkg/etc/spamd.conf if needed
Failed to commit conf: add spamd-20060330nb10
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
no changes found
hg: failed to push changes to the remote repository ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
===========================================================================
The following files should be created for spamd-20060330nb10:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
[...]</pre>
<p>Yes, hg exits in error when there are no changes to be pushed. Everything is working fine.</p>
<p>Let's simulate some more package-provided changes in defaults</p>
<pre>
pkghost# make clean; sed -i "s/PKGREVISION=10/PKGREVISION=11/g" Makefile; make extract
pkghost# sed -i "s/http/https/g" work/spamd-20060330/etc/spamd.conf
pkghost# vi work/spamd-20060330/etc/spamd.conf; head work/spamd-20060330/etc/spamd.conf
work/spamd-20060330/etc/spamd.conf: 87 lines, 2812 characters
.
# $OpenBSD: spamd.conf,v 1.17 2006/02/01 20:22:43 dhartmei Exp $
#
# spamd config file, read by spamd-setup(8) for spamd(8)
#
# See spamd.conf(5)
#THIS IS A NEW COMMENT!
#
# Configures whitelists and blacklists for spamd
#
# Strings follow getcap(3) convention escapes, other than you
pkghost# make update
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Deinstalling for spamd-20060330nb11
Running /usr/pkg/sbin/pkg_delete -K /var/db/pkg -r spamd-20060330nb10
===========================================================================
The following users are no longer being used by spamd-20060330nb10,
and they can be removed if no other software is using them:
[...]
===> Installing for spamd-20060330nb11
[...]
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for spamd-20060330nb11
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb11.tgz
===> Building binary package for spamd-20060330nb11
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb11.tgz
===> Installing binary package of spamd-20060330nb11
abort: repository usr/home/vcs/pkgconftest already exists!
abort: could not create remote repo!
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
no changes found
pulling from ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
no changes found
defaults/usr/pkg/etc/spamd.conf already tracked!
REGISTER /var/hglocaldir/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb11: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb11: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
Saving the currently installed revision to /var/hglocaldir/automerged//usr/pkg/etc/spamd.conf
automerged/usr/pkg/etc/spamd.conf already tracked!
Conf commit: pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb11
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!
--- /usr/pkg/etc/spamd.conf 2018-08-04 10:43:37.554182167 +0000
+++ /var/hglocaldir/defaults//usr/pkg/etc/spamd.conf.automerge 2018-08-04 10:50:23.582806647 +0000
@@ -3,6 +3,7 @@
# spamd config file, read by spamd-setup(8) for spamd(8)
#
# See spamd.conf(5)
+#THIS IS A NEW COMMENT!
#
# Configures whitelists and blacklists for spamd
#
Revert from the penultimate revision of /var/hglocaldir/automerged//usr/pkg/etc/spamd.conf if needed
Conf commit: pkgsrc: add spamd-20060330nb11
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 2 changes to 2 files
===========================================================================
The following files should be created for spamd-20060330nb11:
[...]
</pre>
<p>
Also see <a href=" http://blog.netbsd.org/tnf/entry/gsoc_2018_reports_configuration_files3">GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 4: configuration deployment, pkgtools and possible improvements</a>
</p>
https://blog.netbsd.org/tnf/entry/gsoc_2018_reports_configuration_files1GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 2: remote repositories (git and CVS)Thomas Klausner2018-08-26T12:10:39+00:002018-08-26T12:10:39+00:00<p>
Prepared by Keivan Motavalli as part of GSoC 2018.
</p>
<p>
In <a href="https://blog.netbsd.org/tnf/entry/gsoc_2018_reports_configuration_files">Configuration files versioning in pkgsrc, Part 1</a> basic pkgsrc support for tracking package configuration changes was introduced, as were the features that attempt to automatically merge installed configuration files with new defaults, all shown using RCS as the backing Version Control System and passing options by setting environment variables.
</p>
<p>
Some of them have changed (conf tracking now needs to be explicitly enabled), support is now also in <code>pkg_install.conf</code> and aside from RCS, pkgsrc now supports CVS, Git, mercurial and SVN, both locally and connecting to remote resources.
</p>
<p>
Furthermore, pkgsrc is now able to deploy configuration from packages being installed from a remote, site-specific vcs repository.
</p>
<p>User modified files are always tracked even if automerge functionality is not enabled, and a new tool, pkgconftrack(1), exists to manually store user changes made outside of package upgrade time.
</p>
<p>
Version Control software is executed as the same user running <code>pkg_add</code> or <code>make install</code>, unless the user is "root". In this case, a separate, unprivileged user, <code>pkgvcsconf</code>, gets created with its own home directory and a working login shell (but no password). The home directory is not strictly necessary, it exists to facilitate migrations betweens repositories and vcs changes; it also serves to store keys used to access remote repositories.</p>
<p>
Files, objects and other data in the working directory are only accessible by UID 0 and pkgvcsconf, but be aware, if you want to login to remote repositories via ssh, keys will now need to be placed at pkgvcsconf own <code>$HOME/.ssh/</code> directory, and ssh-agent, if you need to use it, also should have its socket, as defined by SSH_AUTH_SOCK, accessible by <code>pkgvcsconf:pkgvcsconf</code>.
</p>
<h2>New pkgsrc options</h2>
<p>
NOVCS, which was used to disable configuration tracking via Version Control Systems, doesn't exist anymore as an option or environment variable.
In its place, <code>VCSTRACK_CONF</code> now needs to be set truthfully in order to enable configuration files version tracking.
This can be done temporarily via environment variables before using pkgsrc or installing binary packages, such as by exporting </p>
<pre>export VCSTRACK_CONF=yes</pre> <p> on the tty, or permanently in <code>pkg_install.conf</code> located under the <code>PREFIX</code> in use.</p>
<p>
After bootstrapping a pkgsrc branch that supports the new features, check new options by calling, if you are using <code>/usr/pkg</code> as the PREFIX:
</p>
<pre>man -M /usr/pkg/man/ pkg_install.conf</pre>
<p>
Remember to edit <code>/usr/pkg/etc/pkg_install.conf</code> in order to set <code>VCSTRACK_CONF=yes</code>.
Also try out <code>VCSAUTOMERGE=yes</code> to attempt merging installed configuration files with new defaults, as described in the <a href="https://blog.netbsd.org/tnf/entry/gsoc_2018_reports_configuration_files">first blog post</a>. Installed configuration files are always backed up before an attempt is made to merge changes, and if conflicts are reported no changes get automatically installed in place of the existing configuration files, so that the user can manually review them.
<p>
Both <code>VCSTRACK_CONF</code> and <code>VCSAUTOMERGE</code> variables, if set on the environment, take precedence over values defined in <code>pkg_install.conf</code> in order to allow for an easy override at runtime. All other vcs-related options defined in <code>pkg_install.conf</code> replace those defined as environment variables.
</p>
<p>
These are:
</p>
<p>
<code>VCS</code>, sets the backend used to track configuration files.
If unset, rcs gets called. pkgsrc now also supports Git, SVN, HG and CVS. More on that will get introduced later in the post.
</p>
<p>
<code>VCSDIR</code>, the path used as a working directory by pkginstall scripts run at package installation by pkg_add and looked up by <code>pkgconftrack(1)</code>. Files are copied there before being tracked in a VCS or merged, this directory also stores a local configuration repository or objects and metadata associated with remote repositories, according to the solution in use.
By default, <code>/var/confrepo</code> is assumed as the default working directory and initialized.
</p>
<p>
<code>REMOTEVCS</code>, an URI describing how to access a remote repository, according to the format required by the chosen VCS. Remote repositories are disabled if this variable is unset, or if it is set to "no".
</p>
<p>
<code>VCSCONFPULL</code>, set it to "yes" to have pkgscr attempt to deploy configuration files from a remote repository upon package installation, instead of using package provided defaults. It now works with git, svn and mercurial.
</p>
<p>
<code>ROLE</code>, the role defined for the system, used when deploying configuration from a remote, site repository for packages being installed.
</p>
<p>
More details on system roles and configuration deployment are down in this post.
</p>
<h2>rcs vs network-capable systems</h2>
<p>
RCS, as stated, is the default backend because of its simplicity and widespread presence.
When packages are being compiled from source, system rcs is used if present, otherwise it gets installed automatically by pkgsrc as a TOOL when the first package gets built, being an <code>mk/pkginstall/bsd.pkginstall.mk</code> tool dependency.
</p>
<p>
Please note that binaries defined as pkgsrc TOOLs will get their path, as on the build system, hardcoded in binary packages install scripts.
This shouldn't be a problem for NetBSD binary package users since RCS is part of the base system installation and paths will match. Users who choose binary packages on other platforms and got <code>pkgtools/pkg_install</code> (e.g., <code>pkg_add</code>) by running pkgsrc <code>bootstrap/bootstrap</code> script should also be ok, since packages get built as part of the bootstrap procedure, making <code>bsd.pkginstall.mk</code> tool dependency on RCS tick.
Also note that even when using other systems, automerge functionality depends on <code>merge</code>, part of Revision Control System.
</p>
<p>
Other software, such as git, svn, cvs and hg is not installed or called in pkginstall scripts as TOOL dependencies: they get searched in the PATH defined on the environment, and must be manually installed by the user via pkgsrc itself or other means before setting <code>VCS=</code> to the desired system.
</p>
<p>Each will get briefly introduced, with an example setting up and using remote repositories.
</p>
<p>
pkgsrc will try to import existing and new files into the chosen VCS when it gets switched, and a previously undefined <code>REMOTEVCS</code> URI may be added to an existing repository, already existing files may get imported and pushed to the repository.
It all depends on the workflow of the chosen system, but users SHOULD NOT expect pkgsrc to handle changes and the addition of remote repositories for them. At best there will be inconsistencies and the loss of revisions tracked in the old solution: the handling of repository switches is out of scope for this project.
</p>
<p>
Users who decide to switch to a different Version Control System should manually convert their local repository and start clean. Users who want to switch from local to using a remote repository should also, when setting it up, take care in populating it with local data and then start over, preferably by removing the path specified by <code>VCSDIR</code> or by setting it to a new path.
pkgsrc will try its best to ensure new package installations don't fail because of changes made without taking the aforementioned steps, but no consistency with information stored in the old system should then be expected.
</p>
<h2>Some words on REMOTEVCS</h2>
<p>
<code>REMOTEVCS</code>, if defined and different from "no", points to the URI of a remote repository in a way understandable by the chosen version control system.
It may include the access method, the user name, the remote FQDN or IP address, and the path to the repository.
if a password gets set via <code>userName:password@hostname</code>, care should be taken to only set <code>REMOTEVCS</code> as an environment variable, leaving out login information from <code>pkg_install.conf</code>.
</p>
<p>
The preferred way to access remote repositories is to use ssh with asymmetrical keys. These should be placed under <code>$HOME/.ssh/</code> for the user pkgsrc runs under, or under <code>pkgvcsconf</code> home directory if you are installing packages as root. If keys are password protected or present in non-standard search paths, <code>ssh-agent</code> should be started and the appropriate key unlocked via <code>ssh-add</code> before package installations or upgrades are run.
</p>
<p>If running ssh-agent, <code>SSH_AUTH_SOCK</code> and <code>SSH_AGENT_PID</code> variables must be appropriately exported on the environment the package installations will take place in; <code>CVS_RSH</code> should be set to the path of the <code>ssh</code> executable and exported on the environment when using CVS to access a remote repository over ssh.
</p>
<h2>Git and remotes</h2>
<p>
git, pretty straightforwardly, will store objects, metadata and configuration at <code>VCSDIR/.git</code>, while VCSDIR is defined as the working-tree files get checked out to.
</p>
<p>The configuration under VCSDIR/.git includes the definition of the remote repository, which the <code>+VERSIONING</code> script will set once in the <code>PREPARE</code> phase; the script can also switch the remote to a new one, when using git. Data consistency should be ensured by actions taken beforehand by the user.</p>
<p>Using git instead of rcs is simply done by setting <code>VCS=git</code> in <code>pkg_install.conf</code></p>.
<p>pkgsrc will inizialize the repository.</p>
<p>Please do this before installing packages that come with configuration files, then remove the old VCSDIR (e.g., by running <code>rm -fr /var/confrepo</code>) and let pkgsrc take care of the rest, or set VCSDIR to a new path.
</p>
<p>If you wish to migrate an existing repository to git, seek documentation for scripts such as <code>rcs-fast-export</code> and <code>git fast-import</code>, but initialize it under a new working directory. Running git on top of an older, different repository without migrating data first will allow you to store new files, but automerge may break in strange ways when it tries to extract the first revision of a configuration file from the repository, if it is only tracked in the previous version control system.
</p>
<p>
Now, some steps needed to setup a remote git repository are also common to cvs, svn and mercurial:
I will run a server at <code>192.168.100.112</code> with ssh access enabled for the user <code>vcs</code>.
An ssh key has been generated under <code>/root/.ssh</code> and added to vcs@192.168.100.112 <code>authorized_keys</code> list for the pkgsrc host to automatically login as vcs on the remote instance.
</p>
<p>In order to create a remote repository, login as <code>vcs</code> on the server and run</p>
<pre>
$ pwd
/usr/home/vcs
$ mkdir gitconftrack.git
$ cd gitconftrack.git/
$ git --bare init
Initialized empty Git repository in /usr/home/vcs/gitconftrack.git/
$ cat /usr/home/vcs/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABA[...]
$ /etc/rc.d/sshd status
sshd is running as pid 1343.
</pre>
<p>now, on the client, set the URI as <code>ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git</code> (full path here! hg and svn use paths relative to the user's home directory</p>
<pre>
pkghost# echo "VCSTRACK_CONF=yes" >> /usr/pkg/etc/pkg_install.conf
pkghost# echo "VCS=git" >> /usr/pkg/etc/pkg_install.conf
pkghost# echo "VCSDIR=/var/gitandremote" >> /usr/pkg/etc/pkg_install.conf
pkghost# echo "VCSAUTOMERGE=yes" >> /usr/pkg/etc/pkg_install.conf
pkghost# echo "REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git
pkghost# cat /usr/pkg/etc/pkg_install.conf
VCSTRACK_CONF=yes
VCS=git
VCSDIR=/var/gitandremote
VCSAUTOMERGE=yes
REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git
pkghost#
</pre>
<p>
and try to install <code>spamd</code> as with the examples in the first post (it comes with a configuration file and is written in shell scripts that don't need to be compiled or many dependencies installed when testing things out)
<pre>
pkghost# cd pkgsrc/mail/spamd
pkghost# make
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
=> Fetching spamd-20060330.tar.gz
[...]
pkghost# make install
pkghost# make install
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Installing binary package of spamd-20060330nb2
fatal: Not a git repository: '/var/gitandremote/.git'
Initialized empty Git repository in /var/gitandremote/.git/
fatal: Couldn't find remote ref master
fatal: The remote end hung up unexpectedly
REGISTER /var/gitandremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb2: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf
[master (root-commit) f84f8ee] pkgsrc: add spamd-20060330nb2
1 file changed, 86 insertions(+)
create mode 100644 defaults/usr/pkg/etc/spamd.conf
Conf commit: pkgsrc: add spamd-20060330nb2
Counting objects: 7, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (7/7), 1.34 KiB | 688.00 KiB/s, done.
Total 7 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git
* [new branch] master -> master
===========================================================================
The following files should be created for spamd-20060330nb2:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $
Don't forget to add the spamd ports to /etc/services:
spamd 8025/tcp # spamd(8)
spamd-cfg 8026/tcp # spamd(8) configuration
===========================================================================
pkghost#
</pre>
<p>Let's see what's at the <code>VCSDIR</code> now...</p>
<pre>
pkghost# cd /var/gitandremote/
pkghost# ls
.git automerged defaults user
pkghost# find defaults/ -type f -print
defaults/usr/pkg/etc/spamd.conf
pkghost# git log
commit f84f8ee278ec53f55d652e5c6bf52554216cfd49 (HEAD -> master, origin/master)
Author: kmotavalli <my@email.tld>
Date: Fri Aug 3 15:00:26 2018 +0000
pkgsrc: add spamd-20060330nb2
pkghost# git remote -v
origin ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git (fetch)
origin ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git (push)
</pre>
<p>on the server:</p>
<pre>
$ hostname
vers
$ pwd
/usr/home/vcs/gitconftrack.git
$ git log
commit f84f8ee278ec53f55d652e5c6bf52554216cfd49 (HEAD -> master)
Author: kmotavalli <my@email.tld>
Date: Fri Aug 3 15:00:26 2018 +0000
pkgsrc: add spamd-20060330nb2
$
</pre>
<p>All fine, let's simulate automerge:</p>
<pre>
pkghost# vi /usr/pkg/etc/spamd.conf
[...]
# Whitelists are done like this, and must be added to "all" after each
# blacklist from which you want the addresses in the whitelist removed.
#
#whitelist:\
# :white:\
# :method=file:\
# :file=/var/mail/whitelist.txt:
whitelist:\
:white:\
:method=file:\
:file=/var/mail/whitelist.txt:
:x
/usr/pkg/etc/spamd.conf: 86 lines, 2767 characters
pkghost# cd $HOME/pkgsrc/mail/spamd; make replace
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Replacing for spamd-20060330nb2
===> Updating using binary package of spamd-20060330nb2
/usr/bin/env /usr/pkg/sbin/pkg_add -K /var/db/pkg -U -D /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb2.tgz
===========================================================================
The following users are no longer being used by spamd-20060330nb2,
and they can be removed if no other software is using them:
_spamd
===========================================================================
===========================================================================
The following groups are no longer being used by spamd-20060330nb2,
and they can be removed if no other software is using them:
_spamd
===========================================================================
===========================================================================
The following files are no longer being used by spamd-20060330nb2,
and they can be removed if no other packages are using them:
/usr/pkg/etc/spamd.conf
===========================================================================
Everything up-to-date
From ssh://192.168.100.112/usr/home/vcs/gitconftrack
* branch master -> FETCH_HEAD
Already up to date.
REGISTER /var/gitandremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb2: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb2: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
Saving the currently user-installed revision to /var/gitandremote/user//usr/pkg/etc/spamd.conf
[master eacdef9] pkgsrc: backup user conf before attempting merge for spamd-20060330nb2
1 file changed, 86 insertions(+)
create mode 100644 user/usr/pkg/etc/spamd.conf
Conf commit: pkgsrc: backup user conf before attempting merge for spamd-20060330nb2
Counting objects: 7, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (7/7), 1.44 KiB | 1.44 MiB/s, done.
Total 7 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git
f84f8ee..eacdef9 master -> master
Merged with no conflicts. installing it to /usr/pkg/etc/spamd.conf!
Revert from the last revision of /var/gitandremote/user//usr/pkg/etc/spamd.conf if needed
[master 4f1d514] pkgsrc: add spamd-20060330nb2
1 file changed, 86 insertions(+)
create mode 100644 automerged/usr/pkg/etc/spamd.conf
Conf commit: pkgsrc: add spamd-20060330nb2
Counting objects: 2, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 280 bytes | 280.00 KiB/s, done.
Total 2 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git
eacdef9..4f1d514 master -> master
===========================================================================
The following files should be created for spamd-20060330nb2:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $
Don't forget to add the spamd ports to /etc/services:
spamd 8025/tcp # spamd(8)
spamd-cfg 8026/tcp # spamd(8) configuration
===========================================================================
</pre>
<p>A new file backing up the installed configuration, as edited by the user, has been created in the working directory at <code>/var/gitandremote/user/usr/pkg/etc/spamd.conf</code> and pushed to the remote repository.
</p>
<p>You can check it out and rever from it in case something breaks:</p>
<pre>
pkghost# cd /var/gitandremote/user/
pkghost# ls
usr
pkghost# git checkout usr/pkg/etc/spamd.conf
pkghost# tail /var/gitandremote/user/usr/pkg/etc/spamd.conf
# :method=exec:\
# :file=/usr/local/bin/relaydb -4lw:
# Whitelists are done like this, and must be added to "all" after each
# blacklist from which you want the addresses in the whitelist removed.
#
whitelist:\
:white:\
:method=file:\
:file=/var/mail/whitelist.txt:
pkghost#
</pre>
<p>running git log on the server, we can see two new commits: one pushed when backing up the installed configuration file, one after the automerge succeded and the package got installed.</p>
<pre>
$ hostname
vers
$ pwd
/usr/home/vcs/gitconftrack.git
$ git log
commit 4f1d514159860906154eeaaa24a10ec572a4d062 (HEAD -> master)
Author: kmotavalli <user@email.tld>
Date: Fri Aug 3 15:12:45 2018 +0000
pkgsrc: add spamd-20060330nb2
commit eacdef9f52b49e3ce6f544008dc7c75dc89cbd67
Author: kmotavalli <user@email.tld>
Date: Fri Aug 3 15:12:43 2018 +0000
pkgsrc: backup user conf before attempting merge for spamd-20060330nb2
commit f84f8ee278ec53f55d652e5c6bf52554216cfd49
Author: kmotavalli <user@email.tld>
Date: Fri Aug 3 15:00:26 2018 +0000
pkgsrc: add spamd-20060330nb2
$
</pre>
<p>Furthermore, I'll simulate the addition of a new comment to the package provided configuration file, then the change of a variable, which should generate an automerge conflict that needs manual user review. (Note: you don't need to change configuration files in pkgsrc package working directory! This is only done to simulate a change coming with a package upgrade!</p>
<pre>
pkghost# cd $HOME/pkgsrc/mail/spamd; make clean
===> Cleaning for spamd-20060330nb2
pkghost# make fetch
pkghost# sed -i 's/PKGREVISION=.*2/PKGREVISION=3/g' Makefile
pkghost# make extract
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
=> Checksum SHA1 OK for spamd-20060330.tar.gz
=> Checksum RMD160 OK for spamd-20060330.tar.gz
=> Checksum SHA512 OK for spamd-20060330.tar.gz
===> Installing dependencies for spamd-20060330nb3
=> Build dependency cwrappers>=20150314: found cwrappers-20180325
===> Overriding tools for spamd-20060330nb3
===> Extracting for spamd-20060330nb3
pkghost# head work/spamd-20060330/etc/spamd.conf
# spamd config file, read by spamd-setup(8) for spamd(8)
#
# See spamd.conf(5)
#
# Configures whitelists and blacklists for spamd
#
# Strings follow getcap(3) convention escapes, other than you
# can have a bare colon (:) inside a quoted string and it
# will deal with it. See spamd-setup(8) for more details.
pkghost# sed -i 's/method=http/method=https/g' work/spamd-20060330/etc/spamd.conf
.
pkghost# make update
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Deinstalling for spamd-20060330nb4
Running /usr/pkg/sbin/pkg_delete -K /var/db/pkg -r spamd-20060330nb3
===========================================================================
The following users are no longer being used by spamd-20060330nb3,
and they can be removed if no other software is using them:
_spamd
===========================================================================
===========================================================================
The following groups are no longer being used by spamd-20060330nb3,
and they can be removed if no other software is using them:
_spamd
===========================================================================
===========================================================================
The following files are no longer being used by spamd-20060330nb3,
and they can be removed if no other packages are using them:
/usr/pkg/etc/spamd.conf
===========================================================================
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Patching for spamd-20060330nb4
=> Applying pkgsrc patches for spamd-20060330nb4
=> Fixing configuration paths.
===> Creating toolchain wrappers for spamd-20060330nb4
===> Building for spamd-20060330nb4
[...]
=> Creating /root/pkgsrc/mail/spamd/work/.rc.d/pfspamd
===> Installing for spamd-20060330nb4
=> Generating pre-install file lists
=> Creating installation directories
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/man/spamd.conf.5 /ro5
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd.8 /root/8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-set8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb.8 /roo8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd.8 8
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-c
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd /rootc
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb /ron
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd c
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/etc/spamd.conf /root/d
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for spamd-20060330nb4
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb4.tgz
===> Building binary package for spamd-20060330nb4
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb4.tgz
===> Installing binary package of spamd-20060330nb4
Everything up-to-date
From ssh://192.168.100.112/usr/home/vcs/gitconftrack
* branch master -> FETCH_HEAD
Already up to date.
REGISTER /var/gitretest/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb4: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb4: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
Saving the currently installed revision to /var/gitretest/automerged//usr/pkg/etc/spamd.conf
[master 8970767] pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb4
1 file changed, 17 insertions(+), 17 deletions(-)
Conf commit: pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb4
Counting objects: 7, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (7/7), 599 bytes | 599.00 KiB/s, done.
Total 7 (delta 1), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git
8dd7368..8970767 master -> master
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!
--- /usr/pkg/etc/spamd.conf 2018-08-04 06:13:39.017744154 +0000
+++ /var/gitretest/defaults//usr/pkg/etc/spamd.conf.automerge 2018-08-04 06:15:09.799758773 +0000
@@ -15,7 +15,7 @@
# may be applied to each blacklist.
#
# As of November 2004, a place to search for black lists is
-# http://spamlinks.net/filter-bl.htm
+# https://spamlinks.net/filter-bl.htm
#
# Some of the URLs below point to www.openbsd.org locations. Those
# files are likely to be mirrored to other OpenBSD www mirrors located
@@ -25,45 +25,45 @@
all:\
:spews1:china:korea:
-# Mirrored from http://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2
+# Mirrored from https://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2
spamhaus:\
:black:\
:msg="SPAM. Your address %A is in the Spamhaus Block List\n\
- See http://www.spamhaus.org/sbl and\
- http://www.abuse.net/sbl.phtml?IP=%A for more details":\
- :method=http:\
+ See https://www.spamhaus.org/sbl and\
+ https://www.abuse.net/sbl.phtml?IP=%A for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/SBL.cidr.gz:
-# Mirrored from http://www.spews.org/spews_list_level1.txt
+# Mirrored from https://www.spews.org/spews_list_level1.txt
spews1:\
:black:\
:msg="SPAM. Your address %A is in the spews level 1 database\n\
- See http://www.spews.org/ask.cgi?x=%A for more details":\
- :method=http:\
+ See https://www.spews.org/ask.cgi?x=%A for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/spews_list_level1.txt.gz:
-# Mirrored from http://www.spews.org/spews_list_level2.txt
+# Mirrored from https://www.spews.org/spews_list_level2.txt
spews2:\
:black:\
:msg="SPAM. Your address %A is in the spews level 2 database\n\
- See http://www.spews.org/ask.cgi?x=%A for more details":\
- :method=http:\
+ See https://www.spews.org/ask.cgi?x=%A for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/spews_list_level2.txt.gz:
-# Mirrored from http://www.okean.com/chinacidr.txt
+# Mirrored from https://www.okean.com/chinacidr.txt
china:\
:black:\
:msg="SPAM. Your address %A appears to be from China\n\
- See http://www.okean.com/asianspamblocks.html for more details":\
- :method=http:\
+ See https://www.okean.com/asianspamblocks.html for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/chinacidr.txt.gz:
-# Mirrored from http://www.okean.com/koreacidr.txt
+# Mirrored from https://www.okean.com/koreacidr.txt
korea:\
:black:\
:msg="SPAM. Your address %A appears to be from Korea\n\
- See http://www.okean.com/asianspamblocks.html for more details":\
- :method=http:\
+ See https://www.okean.com/asianspamblocks.html for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/koreacidr.txt.gz:
#relaydb-black:\
Revert from the penultimate revision of /var/gitretest/automerged//usr/pkg/etc/spamd.conf if needed
[master 1eef6d8] pkgsrc: add spamd-20060330nb4
1 file changed, 17 insertions(+), 17 deletions(-)
Conf commit: pkgsrc: add spamd-20060330nb4
Counting objects: 7, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (7/7), 563 bytes | 563.00 KiB/s, done.
Total 7 (delta 1), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git
8970767..1eef6d8 master -> master
===========================================================================
The following files should be created for spamd-20060330nb4:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $
Don't forget to add the spamd ports to /etc/services:
spamd 8025/tcp # spamd(8)
spamd-cfg 8026/tcp # spamd(8) configuration
===========================================================================
===> Cleaning for spamd-20060330nb4
[...]
pkghost# cat /var/gitandremote/automergedfiles
/usr/pkg/etc/spamd.conf
pkghost#
</pre>
<p>Should conflicts arise, these will be highlited as per rcs 3-way <code>merge</code> syntax in the file <code>/var/gitandremote/defaults/usr/pkg/etc/spamd.conf.automerge</code> and no action taken.
</p>
<p>In that case, manually review the conflict opening spamd.conf.automerge in your favorite text editor, then manually copy it in place of the existing installed configuration file for spamd at <code>/usr/pkg/etc/spamd.conf</code>
</p>
<h2>CVS and remotes</h2>
<p>
CVS is used by having a <code>CVSROOT</code> directory at <code>VCSDIR/CVSROOT</code>, with <code>VCSDIR</code> as the working directory.
The same precautions treated illustrating Git usage also apply when chosing to use CVS (and svn, mercurial).
To use CVS, simply set <code>VCS=cvs</code> in <code>PREFIX/etc/pkg_install.conf</code> and change the <code>VCSDIR</code> to a new path.
</p>
<p>
pkgsrc will try to import existing files in CVS if the <code>VCSDIR</code> is not empty, but past files revision will not get migrated from whatever version control system was in use, nor will files present in the old repository but not checked out in the working directory. This will likely make automerge malfunction, so it's up to the user to migrate preeexisting repositories to CVS or to chose a new, empty path as the VCS working directory, if automerge has never been used up to this change (<code>VCSAUTOMERGE</code> not set to <code>yes</code> or VCSDIR/automergedfiles not existing/empty).
</p>
<p>See CVS documentation for a description of how to form remote URIs (that will get passed to cvs with the -d flag)</p>
<p>
In this example, I will use <code>:ext:vcs@192.168.100.112:/usr/home/vcs/cvsconftrack</code> as the REMOTEVCS uri
</p>
<pre>
pkghost# cat /usr/pkg/etc/pkg_install.conf
VCSTRACK_CONF=yes
VCS=cvs
VCSDIR=/var/cvsandremote
VCSAUTOMERGE=yes
REMOTEVCS=:ext:vcs@192.168.100.112:/usr/home/vcs/cvsconftrack
pkghost# echo "export CVS_RSH=ssh" >> $HOME/.profile
pkghost# export CVS_RSH=ssh
pkghost#
</pre>
<p>
and initialize a remote cvs repository on the server at 192.168.100.112:
</p>
<pre>
$ hostname
vers
$ pwd
/usr/home/vcs
$ cvs -d/usr/home/vcs/cvsconftrack init
</pre>
<p>
manually migrate existing data from other systems to the new repository if needed, then try installing some package:
<pre>
pkghost# pwd
/root/pkgsrc/mail/spamd
pkghost# make install
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Installing binary package of spamd-20060330nb5
cvs rlog: cannot find module `defaults' - ignored
cvs checkout: Updating automerged
cvs checkout: Updating defaults
cvs checkout: Updating user
cvs update: Updating automerged
cvs update: Updating defaults
cvs update: Updating user
? usr/pkg
Directory /usr/home/vcs/cvsconftrack/defaults/usr added to the repository
? pkg/etc
Directory /usr/home/vcs/cvsconftrack/defaults/usr/pkg added to the repository
? etc/spamd.conf
Directory /usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc added to the repository
cvs add: scheduling file `spamd.conf' for addition
cvs add: use 'cvs commit' to add this file permanently
REGISTER /var/cvsandremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb5: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf
RCS file: /usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/spamd.conf,v
done
Checking in defaults/usr/pkg/etc/spamd.conf;
/usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/spamd.conf,v <-- spamd.conf
initial revision: 1.1
done
Conf commit: pkgsrc: add spamd-20060330nb5
===========================================================================
The following files should be created for spamd-20060330nb5:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $
Don't forget to add the spamd ports to /etc/services:
spamd 8025/tcp # spamd(8)
spamd-cfg 8026/tcp # spamd(8) configuration
===========================================================================
pkghost#
</pre>
<p>On a successive run, the remote repository will get used without files being imported and checked out:
</p>
<pre>
pkghost# cd /root/pkgsrc/pkgtools/pkgin/
pkghost# make install
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Installing for pkgin-0.9.4nb8
=> Generating pre-install file lists
=> Creating installation directories
[...]
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for pkgin-0.9.4nb8
=> Creating binary package /root/pkgsrc/pkgtools/pkgin/work/.packages/pkgin-0.9.4nb8.tgz
===> Building binary package for pkgin-0.9.4nb8
=> Creating binary package /root/pkgsrc/packages/All/pkgin-0.9.4nb8.tgz
===> Installing binary package of pkgin-0.9.4nb8
cvs rlog: Logging defaults
cvs rlog: Logging defaults/usr
cvs rlog: Logging defaults/usr/pkg
cvs rlog: Logging defaults/usr/pkg/etc
cvs update: Updating automerged
cvs update: Updating defaults
cvs update: Updating defaults/usr
cvs update: Updating defaults/usr/pkg
cvs update: Updating defaults/usr/pkg/etc
cvs update: Updating user
cvs [add aborted]: there is a version in usr already
cvs [add aborted]: there is a version in pkg already
cvs [add aborted]: there is a version in etc already
? pkgin/repositories.conf
Directory /usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/pkgin added to the repository
cvs add: scheduling file `repositories.conf' for addition
cvs add: use 'cvs commit' to add this file permanently
REGISTER /var/cvsandremote/defaults//usr/pkg/etc/pkgin/repositories.conf
pkgin-0.9.4nb8: copying /usr/pkg/share/examples/pkgin/repositories.conf.example to /usr/pkg/etc/pkgin/repositories.conf
RCS file: /usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/pkgin/repositories.conf,v
done
Checking in defaults/usr/pkg/etc/pkgin/repositories.conf;
/usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/pkgin/repositories.conf,v <-- repositories.conf
initial revision: 1.1
done
Conf commit: pkgsrc: add pkgin-0.9.4nb8
===========================================================================
$NetBSD: MESSAGE,v 1.3 2010/06/10 08:05:00 is Exp $
First steps before using pkgin.
. Modify /usr/pkg/etc/pkgin/repositories.conf to suit your platform
. Initialize the database :
# pkgin update
===========================================================================
pkghost#
</pre>
<p>
Yes, I know, it's a bit too chatty. Once the script gets used (and tested) some more cvs should be invoked in a more quiet way.
</p>
<p>See the <a href=" http://blog.netbsd.org/tnf/entry/gsoc_2018_reports_configuration_files2">next post</a> for a walkthrough in using SVN and Mercurial!
</p>
https://blog.netbsd.org/tnf/entry/gsoc_2018_reports_configuration_filesGSoC 2018 Reports: Configuration files versioning in pkgsrc, Part 1Leonardo Taccari2018-07-20T08:39:53+00:002018-07-20T08:39:53+00:00<p>
Starting with this post I will describe how, as part of the Google Summer of Code 2018,
support for configuration files versioning is shaping up in pkgsrc.
</p><p>
Prepared by Keivan Motavalli as part of GSoC 2018.
</p>
<p>
Packages may install code (both machine executable code and
interpreted programs), documentation and manual pages, source
headers, shared libraries and other resources such as graphic
elements, sounds, fonts, document templates, translations and
configuration files, or a combination of them.
</p>
<p>
Configuration files are usually the mean through which the behaviour
of software without a user interface is specified. This covers
parts of the operating systems, network daemons and programs in
general that don't come with an interactive graphical or textual
interface as the principal mean for setting options.
</p>
<p>
System wide configuration for operating system software tends to
be kept under <code>/etc</code>, while configuration for software installed via
pkgsrc ends up under <code>LOCALBASE/etc</code>
(e.g., <code>/usr/pkg/etc</code>).
</p>
<p>
Software packaged as part of pkgsrc provides example configuration
files, if any, which usually get extracted to
<code>LOCALBASE/share/examples/PKGBASE/</code>.
</p>
<p>
After a package has been extracted pre-pending the
<code>PREFIX(/LOCALBASE?)</code>
to relative file paths as listed in the <code>PLIST</code> file, metadata entries
(such as <code>+BUILD_INFO</code>, <code>+DESC</code>, etc) get extracted to
<code>PKG_DBDIR/PKGNAME-PKGVERSION</code> (creating files under
<code>/usr/pkg/pkgdb/tor-0.3.2.10</code>, as an example).
</p>
<p>
Some shell script also get extracted there, such as <code>+INSTALL</code> and
<code>+DEINSTALL</code>. These incorporate further snippets that get copied
out to distinct files after <code>pkg_add</code> executes the <code>+INSTALL</code> script
with <code>UNPACK</code> as argument.
</p>
<p>
Two main frameworks exist taking care of installation and deinstallation
operations: <code>pkgtasks</code>, still experimental, is structured as a library
of POSIX-compliant shell scripts implementing functions that get
included from <code>LOCALBASE/share/pkgtasks-1</code> and called by the
<code>+INSTALL</code> and <code>+DEINSTALL</code> scripts upon execution.
</p>
<p>
Currently pkgsrc defaults to using the <code>pkginstall</code>
framework, which as mentioned copies out from the main file separate,
monolithic scripts handling the creation and removal of directories
on the system outside the <code>PKGBASE</code>, user accounts, shells, the setup
of fonts... Among these and other duties, <code>+FILES ADD</code>, as
called by <code>+INSTALL</code>, copies with correct permissions files from the
<code>PKGBASE</code> to the system, if required by parts of the package such as init
scripts and configuration files.
</p>
<p>
Files to be copied are added as comments to the script at package
build time, here's an example:
</p>
<pre>
# FILE: /etc/rc.d/tor cr share/examples/rc.d/tor 0755
# FILE: etc/tor/torrc c share/examples/tor/torrc.sample 0644
</pre>
<p>
"c" indicates that <code>LOCALBASE/share/examples/rc.d/tor</code>
is to be copied in place to <code>/etc/rc.d/tor</code> with permissions 755,
"r" that it is to be handled as an rc.d script.
</p>
<p>
<code>LOCALBASE/share/examples/tor/torrc.sample</code>, the example file coming
with default configuration options for the tor network daemon, is
to be copied to <code>LOCALBASE/etc/tor/torrc</code>.
</p>
<p>
As of today, this only happens if the package has never been
installed before and said configuration file doesn't already exist
on the system, this to avoid overwriting explicit option changes
made by the user (or site administrator) when upgrading or reinstalling
packages.
</p>
<p>
Let's see where how it's done...
actions are defined under case switches:
</p>
<pre>
case $ACTION in
ADD)
${SED} -n "/^\# FILE: /{s/^\# FILE: //;p;}" ${SELF} | ${SORT} -u |
while read file f_flags f_eg f_mode f_user f_group; do
…
case "$f_flags:$_PKG_CONFIG:$_PKG_RCD_SCRIPTS" in
*f*:*:*|[!r]:yes:*|[!r][!r]:yes:*|[!r][!r][!r]:yes:*|*r*:yes:yes)
if ${TEST} -f "$file"; then
${ECHO} "${PKGNAME}: $file already exists"
elif ${TEST} -f "$f_eg" -o -c "$f_eg"; then
${ECHO} "${PKGNAME}: copying $f_eg to $file"
${CP} $f_eg $file
[...]
[...]
</pre>
<p>
Programs and commands are called using variables set in the script
and replaced with platform specific paths at build time, using the
<code>FILES_SUBST</code> facility (see <code>mk/pkginstall/bsd.pkginstall.mk</code>) and
platform tools definitions under <code>mk/tools</code>.
</p>
<p>
In order to also store revisions of example configuration files in
a version control system, <code>+FILES</code> needs to be modified to always
store revisions in a VCS, and to attempt merging changes non
interactively when a configuration file is already installed on
the system.
</p>
<p>
In order to avoid breakage, installed configuration is backed up
first in the VCS, separating user-modified files from files that
have been already automatically merged in the past, in order to
allow the administrator to easily restore the last manually edited
file in case of breakage.
</p>
<p>
Branches are deliberately not used, since not everyone may wish to
get familiar with version control systems technicalities when
attempting to make a broken system work again.
</p>
<p>
Here's what the modified pkginstall <code>+FILES</code> script does when installing spamd:
</p>
<pre>
case "$f_flags:$_PKG_CONFIG:$_PKG_RCD_SCRIPTS" in
*f*:*:*|[!r]:yes:*|[!r][!r]:yes:*|[!r][!r][!r]:yes:*|*r*:yes:yes)
if ${TEST} "$_PKG_RCD_SCRIPTS" = "no" -a ! -n "$NOVCS"; then
</pre>
<p>
VCS functionality only applies to configuration files, not to rc.d
scripts, and only if the environment variable <code>$NOVCS</code>
is unset. Set it to any value - yes will work :) - to disable the
handling of configuration file revisions.
</p>
<p>
A small note: these options could, in the future, be parsed by
<code>pkg_add</code> from some configuration file and passed calling
setenv before executing <code>+INSTALL</code>, without the need to
pass them as arguments and thus minimizing code path changes.
</p>
<p>
<code>$VCSDIR</code> is used to set a working directory for VCS
functionality different from the default one, <code>VARBASE/confrepo</code>.
</p>
<p>
<code>VCSDIR/automergedfiles</code>
is a textual list made by the absolute paths of installed configuration
files already automatically merged in the past during package
upgrades.
</p>
<p>
Manually remove entries from the list when you make manual
configuration changes after a package has been automatically merged!
</p>
<p>
And don't worry: automatic merging is disabled by default, set
<code>$VCSAUTOMERGE</code> to enable it.
</p>
<p>
When a configuration file already exists on the system, if it is
absent from <code>VCSDIR/automergedfiles</code>, it is assumed to be user
edited and copied to
</p>
<p>
<code>VCSDIR/user/path/to/installed/file</code> is a working file
REGISTERed (added and committed) to the version control system.
</p>
<p>
Check it out and restore it from there in case of breakage!
</p>
<p>
If the file is about to get automatically merged, and the operation
already succeeded in the past, then you can find automatically
merged revisions of installed configuration files under
<code>VCSDIR/automerged/path/to/installed/file</code>
checkout the required revision!
</p>
<p>
A new script, <code>+VERSIONING</code>, handles operations such as
<code>PREPARE</code> (checks that a vcs repository is initialized),
<code>REGISTER</code> (adds a configuration file from the working directory to the repo),
<code>COMMIT</code> (commit multiple <code>REGISTER</code> actions after all configuration
has been handled by the <code>+FILES</code> script, for VCSs that support atomic
transactions), <code>CHECKOUT</code> (checks out the last revision of a file to
the working directory) and <code>CHECKOUT-FIRST</code> (checks out the first
revision of a file).
</p>
<p>
The version control system to be used as a backend can be set
through <code>$VCS</code>. It default to RCS, the Revision Control System, which
works only locally and doesn't support atomic transactions.
</p>
<p>
It will get setup as a tool when bootstrapping pkgsrc on platforms
that don't already come with it.
</p>
<p>
Other backends such as CVS are supported and more will come; these,
being used at the explicit request of the administrator, need to
be already installed and placed in a directory part of <code>$PATH</code>.
</p>
<p>
Let's see what happens with <code>rcs</code> when <code>NOVCS</code> is unset, installing
spamd (for the first time).
</p>
<pre>
cd pkgsrc/mail/spamd
# bmake
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
> Fetching spamd-20060330.tar.gz
[...]
bmake install
===> Installing binary package of spamd-20060330nb2
spamd-20060330nb2: Creating group ``_spamd''
spamd-20060330nb2: Creating user ``_spamd''
useradd: Warning: home directory `/var/chroot/spamd' doesn't exist, and -m was not specified
rcs: /var/confrepo/defaults//usr/pkg/etc/RCS/spamd.conf,v: No such file or directory
/var/confrepo/defaults//usr/pkg/etc/spamd.conf,v <-- /var/confrepo/defaults//usr/pkg/etc/spamd.conf
initial revision: 1.1
done
REGISTER /var/confrepo/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb2: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf
===========================================================================
The following files should be created for spamd-20060330nb2:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $
Don't forget to add the spamd ports to /etc/services:
spamd 8025/tcp # spamd(8)
spamd-cfg 8026/tcp # spamd(8) configuration
===========================================================================
</pre>
<p>
<code>/usr/pkg/etc/spamd.conf</code> didn't already exists, so in the end,
as usual, the example/default configuration
<code>/usr/pkg/share/examples/spamd/spamd.conf</code> gets copied to
<code>PKG_SYSCONFDIR/spamd.conf</code>.
</p>
<p>
The modified <code>+FILES</code> script also copied the example
file under the VCS working directory at
<code>/var/confrepo/default/share/examples/spamd/spamd.conf</code>
it then REGISTEREd this (initial) revision of the default configuration with RCS.
</p>
<p>
When installing an updated (ouch!) spamd package, the installed
configuration at <code>/usr/pkg/etc/spamd.conf</code> won't get touched, but a
new revision of <code>share/examples/spamd/spamd.conf</code> will get stored
using the revision control system.
</p>
<p>
For VCSs that support them, remote repositories can also be used via <code>$REMOTEVCS</code>.
From the <code>+VERSIONING</code> comment:
</p>
<pre>
REMOTEVCS, if set, must contain a string that the chosen VCS understands as
an URI to a remote repository, including login credentials if not specified
through other means. This is non standard across different backends, and
additional environment variables and cryptographic material
may need to be provided.
</pre>
<p>
So, if using CVS accessing a remote repository over ssh, one should setup keys on the systems,
then set and export
</p>
<pre>
VCS=cvs
CVS_RSH=/usr/bin/ssh
REMOTEVCS=user@hostname:/path/to/existing/repo
</pre>
<p>
Remember to initialize (e.g., <code>mkdir -p /path/to/repo; cd /path/to/repo;
cvs init</code>) the repository on the remote system before attempting to
install new packages.
</p>
<p>
Let's try to make a configuration change to spamd.conf and reinstall it:
</p>
<p>
I will enable whitelists uncommenting
</p>
<pre>
#whitelist:\
# :white:\
# :method=file:\
# :file=/var/mail/whitelist.txt:
</pre>
<p>
...and enable automerge:
</p>
<pre>
export VCSAUTOMERGE=yes
bmake install
[...]
merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!
</pre>
<p>
No conflicts get reported, diff shows no output since the installed
file is already identical to the automerged one, which is installed
again and contains the whitelisting options uncommented:
</p>
<pre>
more /usr/pkg/etc/spamd.conf
# Whitelists are done like this, and must be added to "all" after each
# blacklist from which you want the addresses in the whitelist removed.
#
whitelist:\
:white:\
:method=file:\
:file=/var/mail/whitelist.txt:
</pre>
<p>
Let's simulate instead the addition of a new configuration option
in a new package revision: this shouldn't generate conflicts!
</p>
<pre>
bmake extract
===> Extracting for spamd-20060330nb2
vi work/spamd-20060330/etc/spamd.conf
# spamd config file, read by spamd-setup(8) for spamd(8)
#
# See spamd.conf(5)
# this is a new comment!
#
</pre>
<p>
save, run <code>bmake; bmake install</code>:
</p>
<pre>
===> Installing binary package of spamd-20060330nb2
RCS file: /var/confrepo/defaults//usr/pkg/etc/spamd.conf,v
done
/var/confrepo/defaults//usr/pkg/etc/spamd.conf,v <-- /var/confrepo/defaults//usr/pkg/etc/spamd.conf
new revision: 1.9; previous revision: 1.8
done
REGISTER /var/confrepo/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb2: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb2: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
saving the currently installed revision to /var/confrepo/automerged//usr/pkg/etc/spamd.conf
RCS file: /var/confrepo/automerged//usr/pkg/etc/spamd.conf,v
done
/var/confrepo/automerged//usr/pkg/etc/spamd.conf,v <-- /var/confrepo/automerged//usr/pkg/etc/spamd.conf
file is unchanged; reverting to previous revision 1.1
done
/var/confrepo/defaults//usr/pkg/etc/spamd.conf,v --> /var/confrepo/defaults//usr/pkg/etc/spamd.conf
revision 1.1
done
merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!
--- /usr/pkg/etc/spamd.conf 2018-07-09 22:21:47.310545283 +0200
+++ /var/confrepo/defaults//usr/pkg/etc/spamd.conf.automerge 2018-07-09 22:29:16.597901636 +0200
@@ -5,6 +5,7 @@
# See spamd.conf(5)
#
# Configures whitelists and blacklists for spamd
+# this is a new comment!
#
# Strings follow getcap(3) convention escapes, other than you
# can have a bare colon (:) inside a quoted string and it
revert from the last revision of /var/confrepo/automerged//usr/pkg/etc/spamd.conf if needed
===========================================================================
The following files should be created for spamd-20060330nb2:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $
Don't forget to add the spamd ports to /etc/services:
spamd 8025/tcp # spamd(8)
spamd-cfg 8026/tcp # spamd(8) configuration
===========================================================================
</pre>
<pre>
more /usr/pkg/etc/spamd.conf
[...]
# See spamd.conf(5)
#
# Configures whitelists and blacklists for spamd
# this is a new comment!
#
# Strings follow getcap(3) convention escapes, other than you
[...]
# Whitelists are done like this, and must be added to "all" after each
# blacklist from which you want the addresses in the whitelist removed.
#
whitelist:\
:white:\
:method=file:\
:file=/var/mail/whitelist.txt:
</pre>
<p>
We're set for now. In case of conflicts merging, the user is
notified, the installed configuration file is not replaced and the
conflict can be manually resolved by opening the file (as an example,
<code>/var/confrepo/defaults/usr/pkg/etc/spamd.conf.automerge</code>)
in a text editor.
</p>
https://blog.netbsd.org/tnf/entry/report_from_pkgsrccon_2018Report from pkgsrcCon 2018Leonardo Taccari2018-07-14T08:55:45+00:002023-03-07T22:32:37+00:00<p>
On July 7th and 8th there was
<a href="https://www.pkgsrc.org/pkgsrcCon/2018/">pkgsrcCon 2018</a>
in <a href="https://en.wikipedia.org/wiki/Berlin">Berlin</a>,
<a href="https://en.wikipedia.org/wiki/Germany">Germany</a>. It was my first
<a href="https://www.pkgsrc.org/pkgsrcCon/">pkgsrcCon</a>
and it was really really nice... So, let's share a report about it,
what we have done, the talk presented and everything else!
</p><p>
On July 7th and 8th there was
<a href="https://www.pkgsrc.org/pkgsrcCon/2018/">pkgsrcCon 2018</a>
in <a href="https://en.wikipedia.org/wiki/Berlin">Berlin</a>,
<a href="https://en.wikipedia.org/wiki/Germany">Germany</a>. It was my first
<a href="https://www.pkgsrc.org/pkgsrcCon/">pkgsrcCon</a>
and it was really really nice... So, let's share a report about it,
what we have done, the talk presented and everything else!
</p>
<h2>Friday (06/07): Social Event</h2>
<p>
I arrived by plane at
<a href="https://en.wikipedia.org/wiki/Berlin_Tegel_Airport">Berlin Tegel Airport</a>
in the middle of the afternoon. TXL buses were pretty full but after waiting
for 3 of them, I was finally in the direction for
<a href="https://en.wikipedia.org/wiki/Berlin_Hauptbahnhof">Berlin Hauptbahnhof</a>
(nice thing about the buses is that after many are getting
too full they start to arrive minute after minute!)
and then took the S7 for <a
href="https://en.wikipedia.org/wiki/Berlin_Jannowitzbr%C3%BCcke_station">Berlin
Jannowitzbrücke station</a>, just a couple of minutes on foot to
<a href="http://republik-berlin.de/">republik-berlin</a> (for the Friday
social event).
</p>
<p>
On 18:00 we met in
<a href="http://republik-berlin.de/">republik-berlin</a> for the social event.
We had good burgers there and <del>one^W</del><del>two^W</del>some beers
together!
</p>
<p>
The place were a bit noisy for the Belgium vs Brazil World Cup match, but we
still had nice discussions together (and also without losing a lot of
people cheering on! :))
</p>
<p>
There was also a table tennis table and <code>spz</code>, <code>maya</code>,
<code>youri</code> and myself played (I'm a terrible table tennis player
but it was very funny to play the wild west without any rules! :)).
</p>
<h2>Saturday (07/07): Talks session</h2>
<style type="text/css">
video {
width: 100% !important;
height: auto !important;
}
</style>
<h3>Meet & Greet -- Pierre Pronchery (<code>khorben</code>), Thomas Merkel (<code>tm</code>)</h3>
<p>
Pierre and Thomas welcomed us (aliens! :)) in
<a href='https://c-base.org/'>c-base</a>. c-base is a space station under
Berlin (or probably one of the oldest hackerspace, at least old enough that the
word "hackerspace" even didn't existed!).
</p>
<p>
<a href="https://pkgsrc.org/pkgsrcCon/2018/slides/pkgsrcCon-2018-01-khorben-intro.pdf">Slides
(PDF)</a> are available!
</p>
<p>
<video width="854" height="480" poster="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-01-khorben-intro.jpg" controls="controls" preload="none">
<source type="video/mp4" src="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-01-khorben-intro.mp4" />
</video>
</p>
<h3>Keynote: Beautiful Open Source -- Hugo Teso</h3>
<p>
Hugo talked about his experience as an open source developer and focused in
particular how important is the user interface.
</p>
<p>
He discussed that examinating some projects he worked on: Inguma, Bokken,
Iaitö and <a href="https://github.com/radareorg/cutter">Cutter</a>
extracting patterns about his experience.
</p>
<p>
<a href="https://pkgsrc.org/pkgsrcCon/2018/slides/pkgsrcCon-2018-02-hugo-keynote.pdf">Slides
(PDF)</a> are available!
</p>
<p>
<video width="854" height="480" poster="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-02-hugo-keynote.jpg" controls="controls" preload="none">
<source type="video/mp4" src="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-02-hugo-keynote.mp4" />
</video>
</p>
<h3>The state of desktops in pkgsrc -- Youri Mouton (<code>youri</code>)</h3>
<p>
Youri discussed about the state of desktop environments (DE) in pkgsrc starting
with <a href="https://xfce.org/">xfce</a>,
<a href="https://mate-desktop.org/">MATE</a>,
<a href="https://lxde.org/">LXDE</a>,
<a href="https://www.kde.org/">KDE</a> and
<a href="https://www.defora.org/">Defora</a>.
</p>
<p>
He then discussed about the WIP desktop environments:
<a href="https://github.com/linuxmint/Cinnamon">Cinnamon</a>,
<a href="https://lxqt.org/">LXQT</a>,
<a href="https://www.gnome.org/gnome-3/">Gnome 3</a> and
<a href="http://cdesktopenv.sourceforge.net/">CDE</a>, hardware support and
login managers.
</p>
<p>
Especially for the WIP desktop environments help is more than welcomed so if
you're interested in any of that, would like to help (that's also a great way to
start involved in pkgsrc!) please get in touch with <code>youri</code> and/or
give a look at the <code>wip/*/TODO</code> files in <code>pkgsrc-wip</code>!
</p>
<p>
<video width="854" height="480" poster="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-03-youri-desktops.jpg" controls="controls" preload="none">
<source type="video/mp4" src="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-03-youri-desktops.mp4" />
</video>
</p>
<h3>NetBSD & Mercurial: One year later -- Jörg Sonnenberger (<code>joerg</code>)</h3>
<p>
Jörg started discussing about <a href="https://git-scm.com/">Git</a> (citing
<a href="https://gregoryszorc.com/blog/2017/12/11/high-level-problems-with-git-and-how-to-fix-them/">
High-level Problems with Git and How to Fix Them - Gregory Szorc</a>) and then
discussed on why using <a href="https://www.mercurial-scm.org/">Mercurial</a>.
</p>
<p>
Then he announced the latest changes: <code>hgmaster.NetBSD.org</code> and
<code>anonhg.NetBSD.org</code> that permits to experiment with Mercurial and
<code>source-changes-hg@</code> and <code>pkgsrc-changes-hg@</code> mailing
lists.
</p>
<p>
The talk ended describing missing/TODO steps.
</p>
<p>
<a href="https://www.NetBSD.org/~joerg/pkgsrccon/">Slides (HTML)</a> are available!
</p>
<p>
<video width="854" height="480" poster="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-04-joerg-hg.jpg" controls="controls" preload="none">
<source type="video/mp4" src="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-04-joerg-hg.mp4" />
</video>
</p>
<h3>Maintaining qmail in 2018 -- Amitai Schleier (<code>schmonz</code>)</h3>
<p>
Amitai shared his long experience in maintaining
<a href="https://cr.yp.to/qmail.html">qmail</a>.
</p>
<p>
A lot of lesson learned in doing that were shared and it was also funny to see
that at a certain point from MAINTAINER he was more and more involved doing that
and ending up writing patches and tools for qmail.
</p>
<p>
<a href="https://schmonz.com/2018/07/07/pkgsrccon-2018-maintaining-qmail-in-2018/slides/">Slides (HTML)</a>
are available!
</p>
<p>
<video width="854" height="480" poster="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-05-schmonz-qmail.jpg" controls="controls" preload="none">
<source type="video/mp4" src="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-05-schmonz-qmail.mp4" />
</video>
</p>
<h3>A beginner's introduction to GCC -- Maya Rashish (<code>maya</code>)</h3>
<p>
Maya discussed about GCC. First she talked about an overview of the toolchain
(in general) and the corresponding GCC projects, how to pass flags to each of
them and how to stop the compilation process for each of them.
</p>
<p>
Then she talked about the black magic that happens in preprocessor, for example, what
a program does an <code>#include <math.h></code> and why
<code>__NetBSD__</code> is defined.
</p>
<p>
We then saw that with <code>-save-temps</code> is possible to save all
intermediary results and how this is very helpful to debug possible problems.
</p>
<p>
Compiler, assembler and linker were then discussed. We have also seen
<code>specfiles</code>, <code>readelf</code> and other GCC internals.
</p>
<p>
<a href="https://www.NetBSD.org/gallery/presentations/maya/2018-pkgsrccon-gcc/index.html">Slides (HTML)</a>
are available!
</p>
<p>
<video width="854" height="480" poster="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-06-maya-gcc.jpg" controls="controls" preload="none">
<source type="video/mp4" src="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-06-maya-gcc.mp4" />
</video>
</p>
<h3>Handling the workflow of pkgsrc-security -- Leonardo Taccari (<code>leot</code>)</h3>
<p>
I discussed about the workflow of the pkgsrc Security Team (<code>pkgsrc-security</code>).
</p>
<p>
I gave a brief introduction to <a href="https://www.nongnu.org/nmh/">nmh
(new MH)</a> message handling system.
</p>
<p>
Then talked about the mission, tasks and workflow of the <code>pkgsrc-security</code>.
</p>
<p>
For the last part of the talk, I tried to put everything together
and showed how to try to automate some part of the <code>pkgsrc-security</code>
with <code>nmh</code> and some shell scripting.
</p>
<p>
<a href="https://www.NetBSD.org/gallery/presentations/leot/pkgsrccon2018/pkgsrc-security-workflow.pdf">Slides (PDF)</a>
are available!
</p>
<p>
<video width="854" height="480" poster="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-07-leot-security.jpg" controls="controls" preload="none">
<source type="video/mp4" src="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-07-leot-security.mp4" />
</video>
</p>
<h3>Preaching for releng-pkgsrc -- Benny Siegert (<code>bsiegert</code>)</h3>
<p>
Benny discussed about pkgsrc Releng team (<code>releng-pkgsrc</code>).
</p>
<p>
The talk started discussing about the pkgsrc Quarterly Releases.
Since 2003Q4, every quarter a new pkgsrc release is released. Stable releases are
the basis for binary packages. Security, build and bug fixes get applied over
the liftime of the release via pullups, until the next quarterly release.
The release procedure and freeze period were also discussed.
</p>
<p>
Then we examined the life of a pullup. Benny first introduced what a
pullup is, the rules for requesting them and a practical example of how to file
a good pullup request.
Under the hood parts of releng were also discussed, for example how tickets are
handled with <code>req</code>, help script to ease the pullup, etc..
</p>
<p>
The talk concluded with the importance of releng-pkgsrc and also a call for
volunteers to join releng-pkgsrc! (despite they're really doing a great work,
at the moment there is a shortage of members in releng-pkgsrc, so, if you are
interested and would like to join them please get in touch with them!)
</p>
<p>
<video width="854" height="480" poster="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-08-benny-releng.jpg" controls="controls" preload="none">
<source type="video/mp4" src="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-08-benny-releng.mp4" />
</video>
</p>
<h3>Something old, something new, something borrowed -- [anonymous]</h3>
<p>
[anonymous] discussed about the state of
<a href="https://wiki.NetBSD.org/ports/macppc/">NetBSD/macppc</a> port.
</p>
<p>
Lot of improvements and news happened (a particular kudos to
<code>macallan</code> for doing an amazing work on the <code>macppc</code> port!)!
<code>HEAD-llvm</code> builds for <code>macppc</code> were added;
<a href="//man.NetBSD.org/macppc/awacs.4">awacs(4)</a>
Bluetooth support, IPsec support, Veriexec support are all enabled
by default now. </p>
<p>
radeonfb(4) and XCOFF boot loader had several improvements and now DVI is
supported on the G4 Mac Mini.
</p>
<p>
The other big news in the <code>macppc</code> land is the G5 support that will
probably be interesting also for possible pkgsrc bulk builds.
</p>
<p>
[anonymous] also discussed about some current problems (and workarounds!), bulk builds
takes time, no modern browser with JavaScript support is easily available right
now but also how using macppc port helped to spot several bugs.
</p>
<p>
Then he discussed about <a href="https://upspin.io/">Upspin</a> (please also
give a look to the corresponding package in
<code>wip/go-upspin</code>!)
</p>
<h3>Magit -- Christoph Badura (<code>bad</code>)</h3>
<p>
Christoph talk was a live introduction to <a href="https://magit.vc/">Magit</a>, a
<a href="https://git-scm.com/">Git</a> interface for <a
href="https://www.gnu.org/software/emacs/">Emacs</a>.
</p>
<p>
The talk started quoting <a href="https://mickens.seas.harvard.edu/">James
Mickens</a> <a href="https://vimeo.com/146524997#t=22m26s">It Was Never Going to Work, So
Let's Have Some Tea</a> talk presented at
<a href="https://www.usenix.org/conference/lisa15">USENIX LISA15</a> when James
Mickens talked about an high level picture of how Git works.
</p>
<p>
We then saw how to clone a repository inside Magit, how to navigate the
commits, how to create a new branch, edit a file and look at unstaged changes,
stage just some hunks of a change and commit them and how to rebase them
(everything is just one or two keystrokes far!).
</p>
<p>
<video width="854" height="480" poster="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-10-bad-magit.jpg" controls="controls" preload="none">
<source type="video/mp4" src="https://video.wiedi.hk/pkgsrcCon-2018/pkgsrcCon-2018-10-bad-magit.mp4" />
</video>
</p>
<h3>Post conf dinner</h3>
<p>
After the talks we had some burgers and beers together at
<a href="https://www.facebook.com/spudbencerberlin">Spud Bencer</a>.
</p>
<p>
We formed several groups to go there from c-base and I was actually in the
group that went there on foot so it was also a nice chance to sightsee Berlin
(thanks to <code>khorben</code> for being a very nice guide! :)).
</p>
<h2>Sunday (08/07): Hacking session</h2>
<h3>An introduction to Forth -- Valery Ushakov (<code>uwe</code>)</h3>
<p>
On Sunday morning Valery talked about
<a href="https://en.wikipedia.org/wiki/Forth_(programming_language)">Forth</a>
from the ground up.
</p>
<p>
We saw how to implement a Forth interpreter step by step and discussed
threaded code.
</p>
<p>
Unfortunately the talk was not recorded... However, if
you are curious I suggest taking a look to
<a href="https://bitbucket.org/nbuwe/forth">nbuwe/forth BitBucket repository</a>.
<code>internals.txt</code> file also contains a lot of interesting resources
about Forth.
</p>
<p>
<a href="https://twitter.com/YouriMouton/status/1015919065318285313/">
<img alt="Learning about Forth from uwe !@netbsd #pkgsrcCon" src='//www.NetBSD.org/~leot/blog-posts/imgs/DhlE_9JWsAEbl8C_small.jpg' />
</a>
</p>
<h3>Hacking session</h3>
<p>
After Valery talk there was the hacking session where we hacked on pkgsrc,
discussed together, etc..
</p>
<p>
Late in the afternoon some of us visited
<a href="http://www.computerspielemuseum.de/">Computerspielemuseum</a>.
</p>
<p>
More than 50 years of computer games were covered there and it was fun to also
play to several historical and also more recent video games.
</p>
<p>
We then met again for a dinner together in
<a href="https://en.wikipedia.org/wiki/Potsdamer_Platz">Potsdamer Platz</a>.
</p>
<p>
<img alt="Group photograph of the pkgsrcCon 2018 kindly taken by Gilberto Taccari" src='//www.NetBSD.org/~leot/blog-posts/imgs/pkgsrccon2018-groupphoto.jpg' />
</p>
<h2>Conclusion</h2>
<p>
<a href="https://www.pkgsrc.org/pkgsrcCon/2018/">pkgsrcCon 2018</a> was
really really great!
</p>
<p>
First of all I would like to thank all the pkgsrcCon organizers:
<code>khorben</code> and <code>tm</code>. It was very well organized and
everything went well, thank you Pierre and Thomas!
</p>
<p>
A big thank you also to
<code>wiedi</code>, just after few hours all the recordings of the talk were
shared and that's really impressive!
</p>
<p>
Thanks also to <code>youri</code> and
<a href="https://twitter.com/gilbertotcc">Gilberto</a> for photographs.
</p>
<p>
Last, but not least, thanks to
<a href="https://www.NetBSD.org/foundation/">The NetBSD Foundation</a> for
supporting three developers to attend the conference.
<a href="https://c-base.org/">c-base</a> for kindly providing a very nice location
for the pkgsrcCon.
Our sponsors: <a href="https://www.defora.net/">Defora
Networks</a> for sponsoring the t-shirts and badges for the conference and
<a href="https://www.skylime.net/">SkyLime</a> for sponsoring the catering on
Saturday.
</p>
<p>
Thank you!
</p>
https://blog.netbsd.org/tnf/entry/gsoc_2017_reports_add_codeGSoC 2017 Reports: Add <code>SUBPACKAGES</code> support to pkgsrc, part 1Leonardo Taccari2017-08-31T17:06:53+00:002017-08-31T17:38:48+00:00<p>
In this blog post series I will discuss about <code>SUBPACKAGES</code> work
done during
<a href="https://summerofcode.withgoogle.com/">Google Summer of Code 2017</a>.
</p>
<p>
In this first part I'll briefly introduce what are <code>SUBPACKAGES</code>,
why and when can be useful and finally we'll give a quick look to a trivial
<a href="https://www.pkgsrc.org/">pkgsrc</a> package that uses them. At the end
we'll also dive a bit on parts of the pkgsrc infrastructure that needed to be
adjusted for implementing that.
</p><p>
In this blog post series I will discuss about <code>SUBPACKAGES</code> work
done during
<a href="https://summerofcode.withgoogle.com/">Google Summer of Code 2017</a>.
</p>
<p>
In this first part I'll briefly introduce what are <code>SUBPACKAGES</code>,
why and when can be useful and finally we'll give a quick look to a trivial
<a href="https://www.pkgsrc.org/">pkgsrc</a> package that uses them. At the end
we'll also dive a bit on parts of the pkgsrc infrastructure that needed to be
adjusted for implementing that.
</p>
<h2>Introduction</h2>
<p>
<code>SUBPACKAGES</code> (on some package systems they are known as
<em>multi-packages</em>, but this term for pkgsrc is already used by packages
that can be built against several versions (e.g. Python, PHP, Ruby packages))
consist in generating multiple binary packages from a single pkgsrc package.
For example, from a pkgsrc package - <code>local/frobnitzem</code> - we will
see how to generate three separate binary packages:
<code>frobnitzem-foo</code>, <code>frobnitzem-bar</code> and
<code>frobnitzem-baz</code>.
</p>
<p>
This can be useful to separate several components of binary
packages (and avoid to run the <code>extract</code> and
<code>configure</code> phase two times!),
for <a href="https://blog.netbsd.org/tnf/entry/gsoc_2016_reports_split_debug">debugpkgs</a>
(so that all <code>*.debug</code> files containing debug symbols are contained
in a separate <code>-debugpkg</code> package that can be installed only when it
is needed), etc..
</p>
<h2>A simple <code>SUBPACKAGES</code> package: <code>frobnitzem</code>!</h2>
<p>
To understand how <code>SUBPACKAGES</code> works and can be useful let's start
to see an example of it in practice: <code>frobnitzem</code>.
</p>
<p>
<code>frobnitzem</code> is a trivial package that just install three scripts in
<code>${PREFIX}/bin</code>, let's see it:
</p>
<pre>
% cd pkgsrc/local/frobnitzem
% tree
.
|-- DESCR
|-- Makefile
|-- PLIST
`-- files
`-- frobnitzem
|-- frobnitzem-bar
|-- frobnitzem-baz
`-- frobnitzem-foo
2 directories, 6 files
% find . -type f | xargs tail -n +1
==> ./Makefile <==
# $NetBSD$
DISTNAME= frobnitzem-0
CATEGORIES= local
MASTER_SITES= # empty
DISTFILES= # empty
MAINTAINER= leot%NetBSD.org@localhost
HOMEPAGE= http://netbsd.org/~leot/gsoc2017-diary/
COMMENT= Simple subpackages example
LICENSE= public-domain
FILESDIR= ${.CURDIR}/../../local/frobnitzem/files
WRKSRC= ${WRKDIR}/frobnitzem
NO_BUILD= yes
do-extract:
${CP} -r ${FILESDIR}/frobnitzem ${WRKDIR}
do-install:
${INSTALL_SCRIPT_DIR} ${DESTDIR}${PREFIX}/bin
${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-foo ${DESTDIR}${PREFIX}/bin
${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-bar ${DESTDIR}${PREFIX}/bin
${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-baz ${DESTDIR}${PREFIX}/bin
.include "../../mk/bsd.pkg.mk"
==> ./files/frobnitzem/frobnitzem-bar <==
#!/bin/sh
echo "bar"
==> ./files/frobnitzem/frobnitzem-baz <==
#!/bin/sh
echo "baz"
==> ./files/frobnitzem/frobnitzem-foo <==
#!/bin/sh
echo "foo"
==> ./PLIST <==
@comment $NetBSD$
bin/frobnitzem-bar
bin/frobnitzem-baz
bin/frobnitzem-foo
==> ./DESCR <==
frobnitzem, collection of foo, bar, baz scripts.
(Or, a bit more seriously this is just a very simple package to
test subpackages support!)
</pre>
<p>
Nothing fancy, just three simple scripts, <code>frobnitzem-{bar,baz,foo}</code>
that will respectively print to the standard output <code>bar</code>,
<code>baz</code> and <code>foo</code>.
Let's build and install the <code>frobnitzem</code> package:
</p>
<pre>
% make install
===> Installing dependencies for frobnitzem-0
===> Overriding tools for frobnitzem-0
===> Extracting for frobnitzem-0
[...]
===> Installing for frobnitzem-0
[...]
===> Installing binary package of frobnitzem-0
[...]
</pre>
<p>
And now let's try scripts installed as part of the <code>frobnitzem</code>
package:
</p>
<pre>
% foreach w (bar baz foo)
... frobnitzem-$w
... end
bar
baz
foo
</pre>
<p>
Okay, as we expected. Despite <code>frobnitzem-{foo,bar,baz}</code> don't do
anything particularly useful we can split the <code>frobnitzem-0</code>
package in three separate subpackages: <code>frobnitzem-foo-0</code>,
<code>frobnitzem-bar-0</code> and
<code>frobnitzem-baz-0</code> (they provides different functionalities and can
also coexist if they're in separated binary packages).
</p>
<p>
To do that we need to slighty adjust <code>Makefile</code>, split the
<code>PLIST</code> in <code>PLIST.{foo,bar,baz}</code> (one for each separate
subpackage), split the <code>DESCR</code> in <code>DESCR.{foo,bar,baz}</code>.
So, at the end in <code>local/frobnitzem</code> we'll have:
</p>
<pre>
% tree
.
|-- DESCR.bar
|-- DESCR.baz
|-- DESCR.foo
|-- Makefile
|-- PLIST.bar
|-- PLIST.baz
|-- PLIST.foo
`-- files
`-- frobnitzem
|-- frobnitzem-bar
|-- frobnitzem-baz
`-- frobnitzem-foo
2 directories, 10 files
</pre>
<h3>Splitting <code>DESCR</code> and <code>PLIST</code></h3>
<p>
<code>DESCR</code> and <code>PLIST</code> splits are straightforward. We just
provide a separate <code>DESCR.<spkg></code> for each subpackage,
e.g. for the <code>foo</code> subpackage:
</p>
<pre>
% cat DESCR.foo
frobnitzem, collection of foo, bar, baz scripts.
(Or, a bit more seriously this is just a very simple package to
test subpackages support!)
This package provide the foo functionalities.
</pre>
<p>
Similarly, regarding <code>PLIST</code>s, we just provide a separate
<code>PLIST.<spkg></code> for each subpackage, e.g. for the
<code>foo</code> subpackage:
</p>
<pre>
% cat PLIST.foo
@comment $NetBSD$
bin/frobnitzem-foo
</pre>
<h3><code>Makefile</code> changes</h3>
<p>
In Makefile we'll need to list all <code>SUBPACKAGES</code> hence we'll add the
following line as first paragraph:
</p>
<pre>
SUBPACKAGES= foo bar baz
</pre>
<p>
We'll then need to define a <code>PKGNAME.<spkg></code> for each
subpackages:
</p>
<pre>
PKGNAME.foo= frobnitzem-foo-0
PKGNAME.bar= frobnitzem-bar-0
PKGNAME.baz= frobnitzem-baz-0
</pre>
<p>
...and similarly <code>COMMENT</code> variable should be defined for each
subpackage via <code>COMMENT.<spkg></code>:
</p>
<pre>
COMMENT.foo= Simple subpackages example (foo)
COMMENT.bar= Simple subpackages example (bar)
COMMENT.baz= Simple subpackages example (baz)
</pre>
<p>
To recap here how we have adjusted Makefile, all the other lines rest unchanged:
</p>
<pre>
% sed '/LICENSE/q' < Makefile
# $NetBSD$
SUBPACKAGES= foo bar baz
DISTNAME= frobnitzem-0
PKGNAME.foo= frobnitzem-foo-0
PKGNAME.bar= frobnitzem-bar-0
PKGNAME.baz= frobnitzem-baz-0
MASTER_SITES= # empty
DISTFILES= # empty
CATEGORIES= local
MAINTAINER= leot%NetBSD.org@localhost
HOMEPAGE= http://netbsd.org/~leot/gsoc2017-diary/
COMMENT.foo= Simple subpackages example (foo)
COMMENT.bar= Simple subpackages example (bar)
COMMENT.baz= Simple subpackages example (baz)
LICENSE= public-domain
</pre>
<p>
Finally we can install <del>it^W</del>them! The usual <code>make
install</code> will generate three binary packages
(<code>frobnitzem-foo-0.tgz</code>, <code>frobnitzem-bar-0.tgz</code>,
<code>frobnitzem-baz-0.tgz</code>) and install all of them:
</p>
<pre>
% make install
===> Installing dependencies for frobnitzem-0
[...]
===> Overriding tools for frobnitzem-0
===> Extracting for frobnitzem-0
[...]
===> Installing for frobnitzem-0
[...]
=> Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-foo-0.tgz
=> Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-bar-0.tgz
=> Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-baz-0.tgz
[...]
===> Installing binary package of frobnitzem-foo-0
===> Installing binary package of frobnitzem-bar-0
===> Installing binary package of frobnitzem-baz-0
[...]
</pre>
<p>
Now we can try them and use
<a href="//man.NetBSD.org/pkg_info.1">pkg_info(1)</a>
to get some information about them:
</p>
<pre>
% frobnitzem-foo
foo
% pkg_info -Fe /usr/pkg/bin/frobnitzem-foo
frobnitzem-foo-0
% pkg_info frobnitzem-baz
Information for frobnitzem-baz-0:
Comment:
Simple subpackages example (baz)
Description:
frobnitzem, collection of foo, bar, baz scripts.
(Or, a bit more seriously this is just a very simple package to
test subpackages support!)
This package provide the baz functionalities.
Homepage:
http://netbsd.org/~leot/gsoc2017-diary/
% pkg_info -L frobnitzem-bar
Information for frobnitzem-bar-0:
Files:
/usr/pkg/bin/frobnitzem-bar
</pre>
<p>
So we can see that <code>make install</code> actually installed three different
binary packages.
</p>
<p>
To deinstall all <code>SUBPACKAGES</code> we can run <code>make deinstall</code>
in the <code>local/frobnitzem</code> directory (that will remove all
subpackages) or we can just manually invoke
<a href="//man.NetBSD.org/pkg_delete.1">pkg_delete(1)</a>.
</p>
<h2>An high-level look at how SUBPACKAGES support is implemented</h2>
<p>
Most of the changes needed are in <code>mk/pkgformat/pkg/</code> hierarchy
(previously known as <code>mk/flavour</code> and then renamed and generalized to
other package formats during
<a href="http://addpackageforma.sourceforge.net/">Anton Panev's Google Summer of Code 2011</a>).
</p>
<p>
The code in <code>mk/pkgformat/${PKG_FORMAT}/</code> handle the interaction of
pkgsrc with the particular <code>${PKG_FORMAT}</code>, e.g. for <code>pkg</code>
populate meta-data files used by
<a href="//man.NetBSD.org/pkg_create.1">pkg_create(1)</a>,
install/delete packages via
<a href="//man.NetBSD.org/pkg_add.1">pkg_add(1)</a>,
and
<a href="//man.NetBSD.org/pkg_delete.1">pkg_delete(1)</a>,
etc.
</p>
<p>
For more information
<a href="http://cvsweb.netbsd.org/bsdweb.cgi/~checkout~/pkgsrc/mk/pkgformat/README?rev=1.3"><code>mk/pkgformat/README</code></a>
is a good introduction to <code>pkgformat</code> hierarchy.
</p>
<p>
Most of the changes done respect the following template:
</p>
<pre>
.if !empty(SUBPACKAGES)
. for _spkg_ in ${SUBPACKAGES}
[... code that handles SUBPACKAGES case ...]
. endfor
.else # !SUBPACKAGES
[... existing (and usually completely unmodified) code ...]
.endif # SUBPACKAGES
</pre>
<p>
In particular, in <code>mk/pkgformat/pkg/</code> targets were adjusted to
create/install/deinstall/etc. all subpackages.
</p>
<p>
Apart <code>mk/pkgformat</code> other changes were needed in
<code>mk/install/install.mk</code> in order to adjust the <code>install</code>
phase for <code>SUBPACKAGES</code>.
</p>
<p>
Regarding <code>PLIST.<spkg></code> handling <code>mk/plist/plist.mk</code>
needed some adjustments to honor each PLIST per-subpackage.
</p>
<p>
<code>mk/bsd.pkg.mk</code> needed to be adjusted too in order to honor several
per-subpackage variables (the <code>*.<spkg></code> ones) and
per-subpackage <code>DESCR.<spkg></code>.
</p>
<h2>Conclusion</h2>
<p>
In this first part of this blog post series we have seen what are
<code>SUBPACKAGES</code>, when and why they can be useful.
</p>
<p>
We have then seen a practical example of them taking a very trivial package
and learned how to "subpackage-ify" it.
</p>
<p>
Then we have described - from an high-level perspective - the changes needed to
the pkgsrc infrastructure for the <code>SUBPACKAGES</code> features that we have
used. If you are more interested in them please give a look to the
<a href="https://github.com/iamleot/pkgsrc/tree/debugpkg">pkgsrc debugpkg
branch</a> that contains all work done described in this blog post.
</p>
<p>
In the next part we will see how to handle <code>*DEPENDS</code> and
<code>buildlink3</code> inclusion for subpackages.
</p>
<p>
I would like to thanks <a href="https://www.google.com/">Google</a> for
organizing <a href="https://summerofcode.withgoogle.com/">Google Summer of
Code</a>, the entire <a href='https://www.netbsd.org/foundation/'>The NetBSD
Foundation</a> and in particular my mentors Taylor R. Campbell, William J.
Coldwell and Thomas Klausner for providing precious guidance during these three
months. A special thank you also to Jörg Sonnenberger who provided very
useful suggestions. Thank you!
</p>https://blog.netbsd.org/tnf/entry/gsoc_2016_reports_split_debug2GSoC 2016 Reports: Split debug symbols for pkgsrc builds, part 2Leonardo Taccari2016-09-14T20:21:19+00:002016-09-14T20:21:19+00:00<p>
<a href='https://summerofcode.withgoogle.com/'>Google Summer of Code</a> 2016 is
now over. We have polished the code and documentation and
submitted the final term evaluation on 23rd of August 2016. The mentors
evaluated us in the following week.
</p>
<p>
If you are impatient (and this time impatience is definitely a virtue!) please
take a look to all
<a href='https://www.netbsd.org/foundation/'>The NetBSD Foundation</a>
<a href='https://summerofcode.withgoogle.com/organizations/6246531984261120/'>
GSoC 2016 projects' code submissions</a>!
</p><p>
<a href='https://summerofcode.withgoogle.com/'>Google Summer of Code</a> 2016 is
now over. We have polished the code and documentation and
submitted the final term evaluation on 23rd of August 2016. The mentors
evaluated us in the following week.
</p>
<p>
If you are impatient (and this time impatience is definitely a virtue!) please
take a look to all
<a href='https://www.netbsd.org/foundation/'>The NetBSD Foundation</a>
<a href='https://summerofcode.withgoogle.com/organizations/6246531984261120/'>
GSoC 2016 projects' code submissions</a>!
</p>
<h2>Introduction</h2>
<p>
In <a href='https://blog.netbsd.org/tnf/entry/gsoc_2016_reports_split_debug'>part 1</a>
we have learned what happens under the hood when we split debugging symbols: how
debugging information is stored/stripped off, the relevant ELF sections involved,
various tools to dump that information, etc..
</p>
<p>
In particular, we have studied what happens on NetBSD when we set the
<code>MKDEBUG*</code> flags.
</p>
<p>
With this background we can finally try to implement this functionality in the
<a href='https://www.pkgsrc.org'>pkgsrc</a> infrastructure. In this blog post we
will first give a practical look at <code>PKG_DEBUGDATA</code> and then we
will analyze the code that I have written in the first mid-term of the GSoC to
accomplish that.
</p>
<h2><code>PKG_DEBUGDATA</code> in action</h2>
<p>
Before digging in the implementation of debugdata functionality let's see what
it produces!
</p>
<p>
In <a href='https://blog.netbsd.org/tnf/entry/gsoc_2016_reports_split_debug'>part 1</a>
we took as example a very simple program that just prints out the lyrics of
Ten Green Bottles song. After packaging it as
<a href='https://github.com/iamleot/pkgsrc/tree/debugpkg/local/green-bottles'>
<code>local/green-bottles</code></a>
let's just build it without any special pkgsrc system variables set:
</p>
<pre>
$ cd pkgsrc/local/green-bottles
$ make install
[...]
$ green-bottles
ten green bottles hanging on the wall
[...]
$ pkg_info -L green-bottles
Information for green-bottles-0:
Files:
/usr/pkg/bin/green-bottles
$ gdb `which green-bottles`
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
[...]
Reading symbols from /usr/pkg/bin/green-bottles...(no debugging symbols found)...done.
(gdb) quit
</pre>
<p>
That's expected because we did not set any special <code>CFLAGS</code> nor
<code>INSTALL_UNSTRIPPED</code>.
</p>
<p>
Before setting <code>PKG_DEBUGDATA</code> to "yes" let's inspect its
documentation via the <code>help</code> target:
</p>
<pre>
$ make help topic=PKG_DEBUGDATA
===> mk/bsd.debugdata.mk (keywords: INSTALL_UNSTRIPPED PKG_DEBUGLEVEL PKG_DEBUGDATA debug):
# This Makefile fragment is included by bsd.pkg.mk and implements the
# logic needed to strip debug symbols off the program/libraries.
#
# User-settable variables:
#
# PKG_DEBUGDATA
# If "yes", install stripped debug symbols for all programs and shared
# libraries. Please notice that if it is defined INSTALL_UNSTRIPPED will
# be also defined internally.
#
# PKG_DEBUGLEVEL
# Used to control the granularity of the debug information. Can be
# "small", "default", "detailed".
# TODO: the name of this variable should be changed because it can be
# TODO: easily confused with PKG_DEBUG_LEVEL!
#
# See also:
# INSTALL_UNSTRIPPED
#
</pre>
<p>
Now that we have finally an idea of what <code>PKG_DEBUGDATA</code> does, let's
set it and reinstall the green-bottles package:
</p>
<pre>
$ make clean deinstall
[...]
$ PKG_DEBUGDATA=yes make install
[...]
===> Installing for green-bottles-0
[...]
=> Stripping debug symbols
=> Installing debug data
[...]
=> Checking for missing debug data in green-bottles-0
[...]
</pre>
<p>
Apart from the various phases we can see three extra messages that are now printed
out: stripping of debug symbols, installing of debug data and a check for
missing debug data.
</p>
<p>
If we inspect the files installed we can now also see
<code>green-bottles.debug</code>:
</p>
<pre>
$ pkg_info -L green-bottles
Information for green-bottles-0:
Files:
/usr/pkg/bin/green-bottles.debug
/usr/pkg/bin/green-bottles
</pre>
<p>
Indeed if we now try to run it through
<a href="//man.NetBSD.org/gdb.1">gdb(1)</a>:
</p>
<pre>
$ gdb `which green-bottles`
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
[...]
Reading symbols from /usr/pkg/bin/green-bottles...Reading symbols from /usr/pkg/bin/green-bottles.debug...done.
done.
(gdb) b main
Breakpoint 1 at 0xad0: file green-bottles.c, line 29.
(gdb) run
Starting program: /usr/pkg/bin/green-bottles
Breakpoint 1, main () at green-bottles.c:29
29 {
(gdb)
[...]
</pre>
<p>
<code>green-bottles</code> was installed as usual without requiring any changes
from <code>MAINTAINER</code>s, in fact the various <code>.debug</code> files
are generated, installed and added to the <code>PLIST</code> dynamically.
</p>
<h2>A look at <code>mk/bsd.debugdata.mk</code> and <code>mk/check/check-debugdata.mk</code> implementation</h2>
<p>
All the functionalities that implement stripping of the debug data from
installed programs and libraries are implemented entirely in
<a
href='https://github.com/iamleot/pkgsrc/blob/f1f18ea7ee6e8c8f7a1d56c0c9cbab401fdf2bbc/mk/bsd.debugdata.mk'>
<code>bsd.debugdata.mk</code></a>.
</p>
<p>
<a href='https://github.com/iamleot/pkgsrc/blob/1c6eec9ea0a9455baabe6964a58148b2442864fd/mk/check/check-debugdata.mk'>
<code>check/check-debugdata.mk</code></a>
checks that all programs and libraries have the debug
data correctly stripped into their corresponding <code>*.debug</code> files.
</p>
<h3><code>mk/bsd.debugdata.mk</code></h3>
<p>
<a href='https://github.com/iamleot/pkgsrc/blob/f1f18ea7ee6e8c8f7a1d56c0c9cbab401fdf2bbc/mk/bsd.debugdata.mk#L1'>The
first 23 lines of <code>bsd.debugdata.mk</code></a> contain a comment that is also
accessible via the <a
href='https://www.netbsd.org/docs/pkgsrc/devfaq.html#devfaq.documentation'>help</a>
target. In particular it describes all the user-settable variables:
</p>
<ul>
<li>
<code>PKG_DEBUGDATA</code>: if "yes" it turns on debugdata functionality to
generate the <code>*.debug</code> files and then install them as part of the
package.
</li>
<li>
<code>PKG_DEBUGLEVEL</code>: useful to control the granularity of the debug
information, i.e. the <code>-g</code> level flag passed to the compiler. It can
be "small", "default" and "detailed".
</li>
</ul>
<p>
Then <code>_VARGROUPS</code> and <code>_USER_VARS.<vargroup></code>
are set accordingly. These are used by the <code>show-all</code> target (if you
are more curious regarding that please just try <code>make show-all</code> and/or
give a look to
<a href='http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/mk/misc/show.mk?rev=1.12'>
mk/misc/show.mk</a>).
</p>
<p>
After various definitions of some used variables (should be self-explainable)
<a
href='https://github.com/iamleot/pkgsrc/blob/f1f18ea7ee6e8c8f7a1d56c0c9cbab401fdf2bbc/mk/bsd.debugdata.mk#L54'>
<code>_FIND_DEBUGGABLEDATA_ERE</code></a> is defined as an Extended Regular
Expression that is used to match potential files that can be stripped.
<code>_FIND_DEBUGGABLEDATA_ERE</code> is then used to limit the file list
generated via
<a
href='https://github.com/iamleot/pkgsrc/blob/f1f18ea7ee6e8c8f7a1d56c0c9cbab401fdf2bbc/mk/bsd.debugdata.mk#L56'>
<code>_FIND_DEBUGGABLEDATA_FILELIST_CMD</code></a>
</p>
<p>
Lines <a
href='https://github.com/iamleot/pkgsrc/blob/f1f18ea7ee6e8c8f7a1d56c0c9cbab401fdf2bbc/mk/bsd.debugdata.mk#L63'>
63-86</a> pass the appropriate debug flags and debug level to the compiler.
</p>
<p>
Then <code>_PLIST_DEBUGDATA</code> is added to the <code>PLIST_SRC_DFLT</code>
variable. <code>PLIST_SRC_DFLT</code> contains all the default
<code>PLIST</code>s that
usually resides in <code>pkgsrc/category/PLIST*</code> (e.g. <code>PLIST</code>,
<code>PLIST.common</code>, <code>PLIST.NetBSD</code>, etc.). If you are more
interested in how it works, please give a look to
<a href='http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/mk/plist/plist.mk?rev=1.49'>
mk/plist/plist.mk</a>.
In this way all the <code>*.debug</code> files listed in
<code>_PLIST_DEBUGDATA</code> are dynamically appended to the package
<code>PLIST</code>.
</p>
<p>
In lines
<a href='https://github.com/iamleot/pkgsrc/blob/f1f18ea7ee6e8c8f7a1d56c0c9cbab401fdf2bbc/mk/bsd.debugdata.mk#L90'>
90-111</a> <code>generate-strip-debugdata</code> target is defined. It
basically just does the operations we have explored in
<a href='https://blog.netbsd.org/tnf/entry/gsoc_2016_reports_split_debug'>part 1</a>
with little adjustments for pkgsrc:
</p>
<ul>
<li>
Check if the current file is strippable via
<a
href="//man.NetBSD.org/objdump.1">objdump(1)</a>
</li>
<li>
If the current file is strippable <code>objcopy
--only-keep-debug program|library program.debug|library.debug</code> is invoked
in order to populate the <code>.debug</code> file.
</li>
<li>
If the current file is strippable <code>objcopy
--strip-debug -p -R .gnu_debuglink
--add-gnu-debuglink=program.debug|library.debug program|library</code> is
invoked to delete the debug information from the current file (program or
library) and to add the
<code>.gnu_debuglink</code> ELF section to it so that the debugger knows where
to pick the corresponding <code>.debug</code> file.
</li>
<li>
Append the <code>.debug</code> path and filename to the
<code>_PLIST_DEBUGDATA</code> file.
</li>
</ul>
<p>
To accomplish that <code>RUN</code> is used to run all shell commands without
printing them. Please note that when <code>RUN</code> is used all shell commands
should be terminated with a semicolon. For more information regarding
<code>RUN</code> please take a look at <code>make help topic=RUN</code>.
</p>
<p>
In lines
<a href='https://github.com/iamleot/pkgsrc/blob/f1f18ea7ee6e8c8f7a1d56c0c9cbab401fdf2bbc/mk/bsd.debugdata.mk#L113'>
113-122</a> <code>install-strip-debugdata</code> is defined. It just installs all
<code>*.debug</code> files via <code>INSTALL_DATA</code>.
<code>install-strip-debugdata</code> target will then be invoked after the
<code>post-install</code> phase.
</p>
<h3><code>mk/check/check-debugdata.mk</code></h3>
<p>
First
<a href='https://github.com/iamleot/pkgsrc/blob/1c6eec9ea0a9455baabe6964a58148b2442864fd/mk/check/check-debugdata.mk#L1'>
26 lines of check-debugdata.mk</a> contain a comment for the pkgsrc online
documentation. It describes the following user-settable variables:
</p>
<ul>
<li>
<code>CHECK_DEBUGDATA</code>: if "yes" it enables debugdata checks (by default
most checks are usually enabled only if PKG_DEVELOPER is "yes")
</li>
</ul>
<p>
...and the following package-settable variables:
</p>
<ul>
<li>
<code>CHECK_DEBUGDATA_SKIP</code>: list of shell patterns that should be
excluded from the check.
</li>
<li>
<code>CHECK_DEBUGDATA_SUPPORTED</code>: whether the check should be enabled for
the package or not.
</li>
</ul>
<p>
Like what it was done for
<a href='https://github.com/iamleot/pkgsrc/blob/f1f18ea7ee6e8c8f7a1d56c0c9cbab401fdf2bbc/mk/bsd.debugdata.mk'>
bsd.debugdata.mk</a>, <code>_VARGROUPS</code> and
<code>_{USER,PKG}_VARS.check-debugdata</code> are defined accordingly.
</p>
<p>
Similarly to what was done for <code>_FIND_DEBUGGABLEDATA_ERE</code> and
<code>_FIND_DEBUGGABLEDATA_FILELIST_CMD</code> in
<a href='https://github.com/iamleot/pkgsrc/blob/f1f18ea7ee6e8c8f7a1d56c0c9cbab401fdf2bbc/mk/bsd.debugdata.mk#L54'>
bsd.debugdata.mk</a>,
<code>_CHECK_DEBUGDATA_ERE</code> and
<code>_CHECK_DEBUGDATA_FILELIST_CMD</code>
<a href='https://github.com/iamleot/pkgsrc/blob/1c6eec9ea0a9455baabe6964a58148b2442864fd/mk/check/check-debugdata.mk#L41'>
are defined</a>.
</p>
<p>
In lines
<a
href='https://github.com/iamleot/pkgsrc/blob/1c6eec9ea0a9455baabe6964a58148b2442864fd/mk/check/check-debugdata.mk#L54'>
54-103</a> <code>_check-debugdata</code> target is defined.
<code>_check-debugdata</code> performs the following checks, respectively:
</p>
<ul>
<li>
Check that the program/library is readable.
</li>
<li>
If the program/library is a file format recognizable by
<a href="//man.NetBSD.org/objdump.1">objdump(1)</a>
check if it has a <code>.gnu_debuglink</code> ELF section.
</li>
<li>
Check that the respective <code>.debug</code> file of the program/library is
readable.
</li>
<li>
Print a warning message if the <code>.debug</code> file does not contain a
<code>.debug_info</code> ELF section.
</li>
</ul>
<h3>Other minor changes in <code>mk/*</code></h3>
<p>
In order to instruct pkgsrc to pick up <code>bsd.debugdata.mk</code> and
<code>checks/check-debugdata.mk</code>, nothing intrusive though:
</p>
<ul>
<li>
Add
<a href='https://github.com/iamleot/pkgsrc/commit/5d5f39ae260856257021755dc6b7b5a3aa925a91'>
<code>.include "bsd.debugdata.mk"</code></a> in <code>bsd.pkg.mk</code>.
</li>
<li>
Add
<a href='https://github.com/iamleot/pkgsrc/commit/df25620f1b912e8f7a5b466690477be7583982e0'>
<code>TOOLS_PLATFORM.objcopy</code></a>
and
<a href='https://github.com/iamleot/pkgsrc/commit/d3d12b4d558718ea8434b4143664d6783bc18b7a'>
<code>TOOLS_PLATFORM.objdump</code></a>
to <code>mk/tools/tools.NetBSD.mk</code>
(other <code>OPSYS</code>s will also needed to be adjusted similarly)
</li>
<li>
Add
<a href='https://github.com/iamleot/pkgsrc/commit/b16737b00e455b3bf016602eb40f2b3297b4e7d7'>
<code>CHECK_DEBUGDATA_SUPPORTED</code></a> to
<code>check/bsd.check-vars.mk</code>.
</li>
<li>
Add
<a href='https://github.com/iamleot/pkgsrc/commit/dc8894c54079c1a652b422f8353c9ba995bc6547'>
<code>.include "check-debugdata.mk"</code></a> in
<code>check/bsd.check.mk</code>.
</li>
</ul>
<h2>Conclusion</h2>
<p>
In this blog post we have learned what <code>PKG_DEBUGDATA</code> does via a
practical example.
</p>
<p>
Then we have examinated how <code>bsd.debugdata.mk</code> and
<code>checks/check-debugdata.mk</code> are implemented.
</p>
<p>
<code>PKG_DEBUGDATA</code> is completely agnostic if a package uses
GNU configure, cmake, etc., and it is expected to work without any changes.
I have only tested it with simple packages also containing libraries and the
ones that uses libtool but I have still not tested it in the wild and some minor
adjustements can be probably needed.
</p>
<p>
Thanks again to <a href="https://www.google.com/">Google</a> for
organizing
<a href='https://summerofcode.withgoogle.com/'>Google Summer of Code</a>,
<a href='https://www.netbsd.org/foundation/'>The NetBSD Foundation</a> and in
particular my mentors David Maxwell, Jöerg Sonnenberger,
Taylor R. Campbell, Thomas Klausner and William J. Coldwell.
</p>
<h2>References</h2>
<ul>
<li>
<a href='https://www.netbsd.org/docs/pkgsrc/'>The pkgsrc guide, Alistair Crooks, Hubert Feyrer, The pkgsrc Developers</a>
</li>
<li>
<a href='http://cvsweb.netbsd.org/bsdweb.cgi/src/usr.bin/make/PSD.doc/tutorial.ms?rev=1.12'>PMake -- A Tutorial, Adam de Boor</a>
(available under <code>/usr/share/doc/ref1/make/</code> in PS and text format)
</li>
<li>
<a href="//man.NetBSD.org/make.1">make(1)</a>
</li>
<li>
<a href="//man.NetBSD.org/objdump.1">objdump(1)</a>
</li>
<li>
<a href="//man.NetBSD.org/objcopy.1">objcopy(1)</a>
</li>
<li>
<a href="//man.NetBSD.org/objcopy.1">gdb(1)</a>
</li>
<li>
<a href='http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/mk/misc/show.mk?rev=1.12'>pkgsrc/mk/misc/show.mk, 1.12</a>
</li>
<li>
<a href='http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/mk/plist/plist.mk?rev=1.49'>pkgsrc/mk/plist/plist.mk, 1.49</a>
</li>
<li>
<a href='http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/mk/bsd.pkg.mk?rev=1.2021'>pkgsrc/mk/bsd.pkg.mk, 1.2021</a>
</li>
<li>
<a href='http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/mk/check/bsd.check-vars.mk?rev=1.8'>pkgsrc/mk/check/bsd.check-vars.mk 1.8</a>
</li>
<li>
<a href='http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/mk/check/bsd.check.mk?rev=1.8'>pkgsrc/mk/check/bsd.check.mk 1.8</a>
</li>
</ul>
https://blog.netbsd.org/tnf/entry/gsoc_2016_reports_split_debugGSoC 2016 Reports: Split debug symbols for pkgsrc builds, part 1Leonardo Taccari2016-06-22T20:17:02+00:002016-06-22T21:08:44+00:00<p>
For the 10th time <a href='https://www.netbsd.org/foundation/'>The NetBSD
Foundation</a> was <a
href='https://summerofcode.withgoogle.com/organizations/6246531984261120/#projects'>selected</a> for the
<a href='https://summerofcode.withgoogle.com/'>GSoC</a> 2016!
</p>
<p>
Now that we're near the first mid-term evaluation and have written the code
during these weeks it's also the right time to start writing some reports
regarding our projects in this series of blog posts.
</p>
<p>
For the 10th time <a href='https://www.netbsd.org/foundation/'>The NetBSD
Foundation</a> was <a
href='https://summerofcode.withgoogle.com/organizations/6246531984261120/#projects'>selected</a> for the
<a href='https://summerofcode.withgoogle.com/'>GSoC</a> 2016!
</p>
<p>
Now that we're near the first mid-term evaluation and have written the code
during these weeks it's also the right time to start writing some reports
regarding our projects in this series of blog posts.
</p>
<h2>About <em>Split debug symbols for pkgsrc builds</em> GSoC project</h2>
<p>
As part of <em>Split debug symbols for pkgsrc builds</em> GSoC project I'm
working to provide support for pkgsrc packages for splitted packages that just
contain debug symbols for their correspondent package (e.g. for the
<code>foo-0.1.2.tgz</code> package there will be a corresponding
<code>foo-0.1.2-debugpkg.tgz</code> package that just contains stripped debug
symbols of all the former binaries and libraries installed by
<code>foo-0.1.2</code>).
</p>
<p>
If you're more curious and you would like to know more information about it
please take a look to the
<a href='https://NetBSD.org/~leot/misc/gsoc2016/gsoc-debugpkg.pdf'>proposal</a>.
</p>
<h2>Introduction</h2>
<p>
In this blog post we will learn how debug information are stored and stripped
off from the programs and/or libraries. We will first write a simple program and
a <code>Makefile</code> to analyze what <code>MKDEBUG*</code> flags in <a
href='https://www.NetBSD.org/'>NetBSD</a> do. Then we will take a look more in
depth to how everything is implemented in the various <code>src/share/*.mk</code>
files and at the end we will give a look to related works already implemented in
<code>RPM</code> and <code>dpkg</code>.
</p>
<p>
A pretty long list of references is also provided for the most curiouses
readers!
</p>
<h2>A quick introduction to ELF and how debug information are stored/stripped off</h2>
<p>
In order to become familiar with ELF format a good starting point are
<a href='https://en.wikipedia.org/wiki/Object_file'>Object file</a> and
<a href='https://en.wikipedia.org/wiki/Executable_and_Linkable_Format'>
Executable and Linkable Format</a> pages from <a
href='https://www.wikipedia.org/'>Wikipedia, the free
encyclopedia</a>.
</p>
<p>
Trying to describe ELF format is not easy in short terms so, it is strongly
suggested to read the nice article series written by Eric Youngdale for
<a href='https://www.linuxjournal.com/'>Linux Journal</a>:
<a href='https://www.linuxjournal.com/article/1059'>The ELF Object File Format: Introduction</a>
and
<a href='https://www.linuxjournal.com/article/1060'>The ELF Object File Format by Dissection</a>.
Please note that these two resources should be enough to completely understand
this blog post!
</p>
<p>
After reading the above resources we have just learned that every programs
and libraries in <a href="https://www.NetBSD.org/">NetBSD</a> (and several
other Unix-like operating systems) uses the ELF format. There are four types
of ELF object files:
</p>
<ul>
<li>
executable
</li>
<li>
relocatable
</li>
<li>
shared
</li>
<li>
core
</li>
</ul>
<p>
For more information regarding them please give a look to
<a href="//man.NetBSD.org/elf.5">elf(5)</a>.
</p>
<p>
We are interested to understand what happens when we compile the
programs/libraries with debugging options (basically the <code>-g</code> option).
</p>
<p>
NetBSD already supports everything out of the box and so we can quickly start
looking at it just writing a simple Makefile and a program that will print the
lyrics of the famous <em>Ten Green Bottles</em> song! To avoid all the hassle of
providing (multiple times!) the right flags to the compiler and manually
invoke the right tool we can just write a very simple <code>Makefile</code> that will do
everything for us:
</p>
<pre>
$ cat green-bottles/Makefile
# $NetBSD$
NOMAN= # defined
PROG= green-bottles
.include <bsd.prog.mk>
</pre>
<p>
Now that we have the <code>Makefile</code> we can start writing the <code>green-bottles</code> <code>PROG</code>ram
(<em>please note that all the green bottles accidentally fall were properly
recycled during the writing of this article</em>):
</p>
<pre>
$ cat green-bottles/green-bottles.c
#include <stdio.h>
void
sing_green_bottles(int n)
{
const char *numbers[] = { "no more", "one", "two", "three", "four", "five",
"six", "seven", "eight", "nine", "ten" };
if ((1 <= n) && (n <= 10)) {
printf("%s green bottle%s hanging on the wall\n",
numbers[n], n > 1 ? "s" : "");
printf("%s green bottle%s hanging on the wall\n",
numbers[n], n > 1 ? "s" : "");
printf("and if %s green bottle should accidentally fall,\n",
n > 2 ? "one" : "that");
printf("there'll be %s green bottles hanging on the wall.\n",
numbers[n - 1]);
}
return;
}
/*
* Sing the famous `Ten Green Bottles' song.
*/
int
main(void)
{
int i;
for (i = 10; i > 0; i--) {
sing_green_bottles(i);
}
return 0;
}
</pre>
<p>
OK! Now everything is ready and if we just invoke <a
href="//man.NetBSD.org/make.1">make(1)</a> we'll build the
program. However, we would like to inspect what's happening behind the scenes,
so we'll look at each steps. Please note that right now it is not important that
you'll understand everything because we'll look at what <a
href="//man.NetBSD.org/make.1">make(1)</a> magic do in more
details later.
</p>
<p>
First, we compile the C program to generate the <em>relocatable</em> object file, i.e.
<code>green-bottles.o</code>:
</p>
<pre>
$ cd green-bottles/
$ make green-bottles.o
# compile green-bottles/green-bottles.o
gcc -O2 -fPIE -std=gnu99 -Werror -c green-bottles.c
ctfconvert -g -L VERSION green-bottles.o
</pre>
<p>
Let's see what <a href="//man.NetBSD.org/file.1">file(1)</a> says regarding it:
</p>
<pre>
$ file green-bottles.o
green-bottles.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
</pre>
<p>
In order to get more information we can use <a
href="//man.NetBSD.org/readelf.1">readelf(1)</a> tool provided by the
<a href='https://www.gnu.org/software/binutils/'>binutils (GNU binary utilities)</a>,
e.g. via <code>readelf -h</code> (the <code>-h</code> option is used
to just print the file headers, if you would like to get more information you
can use the <code>-a</code> option instead):
</p>
<pre>
$ readelf -h green-bottles.o
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: REL (Relocatable file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x0
Start of program headers: 0 (bytes into file)
Start of section headers: 2816 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 0 (bytes)
Number of program headers: 0
Size of section headers: 64 (bytes)
Number of section headers: 17
Section header string table index: 13
</pre>
<p>
We can see the 17 sections always via <code>readelf</code> (<code>-S</code> option).
Now let's recompile it but via the debugging options turned on:
</p>
<pre>
$ make green-bottles.o MKDEBUG=yes
# compile green-bottles/green-bottles.o
gcc -O2 -fPIE -g -std=gnu99 -Werror -c green-bottles.c
ctfconvert -g -L VERSION -g green-bottles.o
</pre>
<p>
If we are careful we can see that unlike the previous make incantation now the
<code>-g</code> option is passed to the compiler... Let's see if we can inspect that via
<code>readelf</code>:
</p>
<pre>
$ readelf -h green-bottles.o
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: REL (Relocatable file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x0
Start of program headers: 0 (bytes into file)
Start of section headers: 6424 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 0 (bytes)
Number of program headers: 0
Size of section headers: 64 (bytes)
Number of section headers: 29
Section header string table index: 25
</pre>
<p>
We can note several differences compared to the previous relocatable file
compiled without <code>MKDEBUG</code>:
</p>
<ul>
<li>
Start of section headers (previously 2816, now 6424)
</li>
<li>
Number of section headers (previously 17, now 29)
</li>
<li>
Section header string table index (previously 13, now 25)
</li>
</ul>
<p>
If we compare the sections between the two relocatable files (tips: using:
<code>readelf -WS green-bottles.o | sed -nEe 's/^ \[ *([0-9]+)\] ([^ ]*) .*/\2/p'</code>
is a possible way to do it) we can observe the following new ELF sections:
</p>
<ul>
<li>
<code>.debug_info</code>: contains main DWARF DIEs (Debugging Information Entry)
</li>
<li>
<code>.debug_abbrev</code>: contains abbreviations used in <code>.debug_info</code> section
</li>
<li>
<code>.debug_loc</code>: contains location expressions
</li>
<li>
<code>.debug_aranges</code>: contains a table for lookup by addresses of program entities
(i.e. <em>data objects</em>, <em>types</em>, <em>functions</em>)
</li>
<li>
<code>.debug_ranges</code>: contains address ranges referenced by DIEs
</li>
<li>
<code>.debug_line</code>: contains line number program
</li>
<li>
<code>.debug_str</code>: contains all strings referenced by <code>.debug_info</code>
</li>
<li>
other <code>.rela.debug_*</code>
</li>
</ul>
<p>
It's time to finally build the program:
</p>
<pre>
$ make green-bottles
rm -f .gdbinit
touch .gdbinit
# link green-bottles/green-bottles
gcc -pie -shared-libgcc -o green-bottles green-bottles.o -Wl,-rpath-link,/lib -L=/lib
ctfmerge -t -g -L VERSION -o green-bottles green-bottles.o
</pre>
<p>
We can observe:
</p>
<pre>
$ readelf -h green-bottles
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Shared object file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x730
Start of program headers: 64 (bytes into file)
Start of section headers: 6448 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 7
Size of section headers: 64 (bytes)
Number of section headers: 31
Section header string table index: 27
</pre>
<p>
...and for its counterpart compiled via <code>MKDEBUG=yes</code>:
</p>
<pre>
$ readelf -h green-bottles
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Shared object file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x730
Start of program headers: 64 (bytes into file)
Start of section headers: 8304 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 7
Size of section headers: 64 (bytes)
Number of section headers: 38
Section header string table index: 34
</pre>
<p>
Not so surprisingly the number of the 7 extra sections are exactly the
<code>.debug_*</code> ones!
</p>
<p>
Now that it's clear the difference between the program compiled with/without
<code>-g</code> option let's see what happen when the debug symbols are stripped off the
program:
</p>
<pre>
$ make green-bottles.debug MKDEBUG=yes
# create green-bottles/green-bottles.debug
( objcopy --only-keep-debug green-bottles green-bottles.debug && objcopy --strip-debug -p -R .gnu_debuglink --add-gnu-debuglink=green-bottles.debug green-bottles ) || (rm -f green-bottles.debug; false)
</pre>
<p>
We can try to describe what happened with an image:
</p>
<p>
<img src="https://www.netbsd.org/~leot/gsoc2016-diary/blog-posts/imgs/green-bottles.png" alt="green-bottles and green-bottles.debug ELF sections" />
</p>
<p>
The first <a href="//man.NetBSD.org/objcopy.1">objcopy(1)</a>
incantation generate the <code>green-bottles.debug</code>
file.
The second <a href="//man.NetBSD.org/objcopy.1">objcopy(1)</a>
incantation strip the debug symbols off
<code>green-bottles</code> (now that they're stored in
<code>green-bottles.debug</code> they are no more needed) and add the
<code>.gnu_debuglink</code> ELF section to it.
</p>
<p>
Let's quickly look them via <a href="//man.NetBSD.org/file.1">file(1)</a>:
</p>
<pre>
$ file green-bottles green-bottles.debug
green-bottles: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /usr/libexec/ld.elf_so, for NetBSD 7.99.29, not stripped
green-bottles.debug: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter *empty*, for NetBSD 7.99.29, not stripped
</pre>
<p>
Using <code>readelf</code> we can note that now <code>green-bottles</code> has 32 sections and
<code>green-bottles.debug</code> has 38 sections. <code>green-bottles</code> has one extra section that
was added by the <a href="//man.NetBSD.org/objcopy.1">objcopy(1)</a> incantation, let's see it:
</p>
<pre>
$ readelf -x '.gnu_debuglink' green-bottles
Hex dump of section '.gnu_debuglink':
0x00000000 67726565 6e2d626f 74746c65 732e6465 green-bottles.de
0x00000010 62756700 90b06f1c bug...o.
</pre>
<p>
The <code>.gnu_debuglink</code> section contain the <a
href='http://netbsd.gw.com/cgi-bin/man-cgi?basename+1'>basename(3)</a> of the <code>.debug</code> file and
its CRC32. The <code>.gnu_debuglink</code> section is used to properly pick the correct
<code>.debug</code> file from the <code>DEBUGDIR</code> directory (we'll see how it will work later when
we will invoke the <a href='https://www.gnu.org/software/gdb/'>GNU debugger</a>).
</p>
<p>
Regarding the sections in the <code>.debug</code> file all of them are preserved but several
have no data, we can check that by invoking:
</p>
<pre>
$ readelf `seq -f '-x %g' 0 37` green-bottles.debug
$ readelf `seq -f '-x %g' 0 31` green-bottles
</pre>
<p>
...and comparing their respective output.
</p>
<p>
Now that everything should be clearer we can just try to invoke it through
<a href="//man.NetBSD.org/gdb.1">gdb(1)</a>
and see what happens:
</p>
<pre>
$ gdb ./green-bottles
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64--netbsd".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./green-bottles...Reading symbols from /tmp/green-bottles/green-bottles.debug...done.
done.
(gdb) b main
Breakpoint 1 at 0xac0: file green-bottles.c, line 29.
(gdb) b sing_green_bottles
Breakpoint 2 at 0x940: file green-bottles.c, line 5.
(gdb) run
Starting program: /tmp/green-bottles/green-bottles
Breakpoint 1, main () at green-bottles.c:29
29 {
(gdb) n
32 for (i = 10; i > 0; i--) {
(gdb) n
33 sing_green_bottles(i);
(gdb) print i
$1 = 10
(gdb) cont
Continuing.
Breakpoint 2, sing_green_bottles (n=10) at green-bottles.c:5
5 {
(gdb) bt
#0 sing_green_bottles (n=10) at green-bottles.c:5
#1 0x00000000b7802ad7 in main () at green-bottles.c:33
[... we can now looks and debug it as we wish! ...]
</pre>
<p>
So we can see that the <code>green-bottles.debug</code> file is loaded from the same
directory where <code>green-bottles</code> program was present (in our case
<code>/tmp/green-bottles/</code> but if a corresponding file <code>.debug</code> is not
found gdb look for it in the <code>DEBUGDIR</code>, i.e. <code>/usr/libdata/debug/</code>;
e.g. for <code>/usr/bin/yes</code> it will look for debug symbols in
<code>/usr/libdata/debug//usr/bin/yes.debug</code>).
This is the same for all other programs and libraries.
</p>
<h2>A look to what MKDEBUG and MKDEBUGLIB do</h2>
<p>
NetBSD already provides <code>MKDEBUG</code> and <code>MKDEBUGLIB</code> <a
href="//man.NetBSD.org/mk.conf.5">mk.conf(5)</a> variables to achieve the
separation of the debug symbols. They respectively split symbols from programs
and libraries.
</p>
<p>
The implementation to do that is in
<a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.prog.mk'>src/share/mk/bsd.prog.mk</a> (for programs) and
<a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.lib.mk'>src/share/mk/bsd.lib.mk</a> (for libraries).
Several global variables used are
defined in <a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.own.mk'>src/share/mk/bsd.own.mk</a>.
</p>
<h3>bsd.prog.mk</h3>
<p>
In <a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.prog.mk?r=1.299#59'>bsd.prog.mk:58</a>
if <code>MKDEBUG</code> is defined and not "no" [sic] the <code>-g</code> flag is
added to <code>CFLAGS</code>.
</p>
<p>
In <a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.prog.mk?r=1.299#310'>bsd.prog.mk:310</a>
the internal <code>__progdebuginstall</code> make target is defined to
install the <code>.debug</code> file for the respective program. It is then called from
<a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.prog.mk?r=1.299#589'>bsd.prog.mk:589</a>
and <a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.prog.mk?r=1.299#604'>bsd.prog.mk:604</a>
(respectively for <code>MKUPDATE == "no"</code> and <code>MKUPDATE != "no"</code>, please note
the dependency operators <code>!</code> vs <code>:</code> for the two cases).
</p>
<p>
In <a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.prog.mk?r=1.299#437'>bsd.prog.mk:437</a>
<code>_PROGDEBUG.${_P}</code> is defined as <code>${PROGNAME.${_P}}.debug</code>,
inside a for loop. <code>${_P}</code> is just an element of the <code>${PROGS}</code> and <code>${PROGS_CXX}</code>
lists.
E.g.: for <a href='https://nxr.netbsd.org/xref/src/bin/echo/Makefile'>src/bin/echo</a>
<code>echo</code> is the <code>PROG</code> value. <a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.prog.mk'>bsd.prog.mk</a>
turns single-program <code>PROG</code> and <code>PROG_CXX</code> variable into the multi-word <code>PROGS</code> and
<code>PROGS_CXX</code> variables.
</p>
<p>
In <a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.prog.mk?r=1.299#545'>bsd.prog.mk:545</a>
there is the most important part. After checking if
<code>_PROGDEBUG.${_P}</code> is defined a <code>${_PROGDEBUG.${_P}}</code> target is defined and
<code>${OBJCOPY}</code> is invoked two times. In the first incantation the
<code>${_PROGDEBUG.${_P}}</code> file (containing the strip debug symbols) is generated for
<code>${_P}</code>. The second incantation is needed to get rid of (now no more needed) debug
symbols from <code>${_P}</code> and <code>--add-gnu-debuglink</code> add a <code>.gnu_debuglink</code> section to
<code>${_P}</code> containing the filename of the <code>${_PROGDEBUG.${_P}}</code>; e.g. for <code>echo</code> it
will be <code>echo.debug</code> (plus the CRC32 of <code>echo.debug</code> - padded as needed).
Regarding other options used by <code>${OBJCOPY}</code> we should note the <code>-p</code> option needed
to preserve dates and <code>-R</code> is added in order to be sure to update the
<code>.gnu_debuglink</code> section.
</p>
<p>
For a gentler introduction and to understand why these steps are needed please
read <a
href='http://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html'>(gdb.info)Separate Debug Files</a>
(you can just use <a href="//man.NetBSD.org/info.1">info(1)</a>, i.e. <code>info
'(gdb.info)Separate Debug Files'</code>).
</p>
<h3>bsd.lib.mk</h3>
<p>
The logic and <a href="//man.NetBSD.org/objcopy.1">objcopy(1)</a>
incantation are similar to the ones used in
<a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.prog.mk'>bsd.prog.mk</a>.
The most interesting part is in <a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.lib.mk?r=1.367#622'>bsd.lib.mk:622</a>.
Apart the <code>*.debug</code>
files if <code>MKDEBUGLIB</code> is defined and not "no" [sic] also <code>*_g.a</code> archives are
created for the respective libraries archives (although they are stored
directly in the several <code>lib/</code> directories not in <code>/usr/libdata/debug/</code>).
</p>
<h3>bsd.own.mk</h3>
<p>
In <a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.own.mk'>bsd.own.mk</a>
various <code>DEBUG*</code> variables are defined:
</p>
<ul>
<li>
<code>DEBUGDIR</code>: where <code>*.debug</code> files are stored. Please notice that this is also
the place where debugging symbols are looked (for more information please
give a look to <a href="//man.NetBSD.org/objcopy.1">objcopy(1)</a>)
</li>
<li>
<code>DEBUGGRP</code>: the <code>-g</code> option passed to
<a href="//man.NetBSD.org/install.1">install(1)</a>
for installing debug symbols
</li>
<li>
<code>DEBUGOWN</code>: the <code>-o</code> option passed to
<a href="//man.NetBSD.org/install.1">install(1)</a>
for installing debug symbols
</li>
<li>
<code>DEBUGMODE</code>: the <code>-m</code> option passed to
<a href="//man.NetBSD.org/install.1">install(1)</a>
for installing debug symbols
</li>
</ul>
<h2>Related works</h2>
<h3>dpkg</h3>
<p>
The <a href='https://www.debian.org/doc/manuals/developers-reference/'>Debian Developer's Reference</a>
written by the Developer's Reference Team
has a <a href='https://www.debian.org/doc/manuals/developers-reference/ch06.en.html#bpp-dbg'>
Best practices for debug packages (section 6.7.9)</a>.
The logic used is more or less the same of the one used by <code>src/share/mk</code> in
NetBSD and described above.
</p>
<p>
After a quick inspection of
<a href='https://sources.debian.net/data/main/d/debhelper/9.20160403/dh_strip'>dh_strip</a>
(part of <code>debhelper</code> package) some interesting ideas to look further are:
</p>
<ul>
<li>
the <a href="//man.NetBSD.org/file.1">file(1)</a> logic used in <code>testfile()</code> subroutine
</li>
<li>
handling of non-C/C++ programming languages: <a href='https://ocaml.org/'>OCaml</a> native code shared
libraries (<code>*.cmxs</code>) and <a href='https://nodejs.org/'>nodejs</a> binaries (<code>*.node</code>)
</li>
</ul>
<h3>RPM</h3>
<p>
The <a href='https://fedoraproject.org/wiki/Fedora_Project_Wiki'>
Fedora Project Wiki</a> contains some interesting tips, in particular
regarding most common issues that happens in stripping debugging symbols in
the
<a href='https://fedoraproject.org/wiki/Packaging:Debuginfo'>
Packaging:Debuginfo</a> page.
Some of the logic is handled in
<a href='http://rpm.org/gitweb?p=rpm.git;a=blob;f=scripts/find-debuginfo.sh'>
find-debuginfo.sh</a>.
</p>
<p>
Another interesting resource is the
<a href='https://fedoraproject.org/wiki/Releases/FeatureBuildId'>
Releases/FeatureBuildId page</a>.
The page discusses what Red Hat have done regarding using the <code>.note.gnu.build-id</code>
section and why have done them.
</p>
<p>
(Yet another) interesting idea adopted by Fedora developers is the
<a href='https://fedoraproject.org/wiki/Features/MiniDebugInfo'>
Features/MiniDebugInfo</a>.
More information regarding <code>MiniDebugInfo</code> are also present in
<a href='https://sourceware.org/gdb/onlinedocs/gdb/MiniDebugInfo.html'>(gdb.info)MiniDebugInfo</a>.
Please note that this is not completely related to
stripping debugging symbols (indeed the <code>MiniDebugInfo</code> is directly stored in
program/library!) but can be considered in order to provide better <code>.core</code> (both
in the pkgsrc and NetBSD cases).
</p>
<p>
Mark J. Wielaard presented in
<a href='https://fosdem.org/2016/'>FOSDEM 2016</a> a talk that summarizes many of the
thematics discussed in this diary. Abstract, video recording and more resources
are available in the FOSDEM website correspective event page:
<a href='https://fosdem.org/2016/schedule/event/where_are_your_symbols_debuginfo_and_sources/'>
Where are your symbols, debuginfo and sources?</a>.
Apart his talk a very interesting reading is his
<a href='https://gnu.wildebeest.org/blog/mjw/2016/02/02/where-are-your-symbols-debuginfo-and-sources/'>
blog post regarding the talk</a>.
In the blog post there are a lot of interesting information, all worth to be
taken in consideration also for the pkgsrc case.
</p>
<h2>Conclusion</h2>
<p>
In this blog post we have learned what's happening when we use
<code>MKDEBUG*</code>
<a href="//man.NetBSD.org/mk.conf.5">mk.conf(5)</a> variables
and how everything works.
</p>
<p>
We have also gave a quick look to other related works, in particular
<code>RPM</code> and <code>dpkg</code> package managers.
</p>
<p>
If you are curious on what I'm doing right now and you would like to also look
at the code you can give a look to the
<a href='https://github.com/iamleot/pkgsrc'>git pkgsrc repository</a> repository
fork in the
<a href='https://github.com/iamleot/pkgsrc/tree/debugpkg'>debugpkg</a> branch.
</p>
<p>
Apart the several references discussed above if you would like to learn more
about several aspects that wasn't discussed there...
<a href='http://dwarfstd.org/doc/Debugging%20using%20DWARF-2012.pdf'>Introduction to the DWARF Debugging Format</a>
written by Michael Eager
is a good starting point for DWARF (debugging data format); you can also use
<code>objdump -g</code> to show these information in the <code>*.debug</code>
files.
Regarding <a href='https://www.gnu.org/software/gdb/'>GDB</a> a
gentle introduction to it is
<a href='http://dirac.org/linux/gdb/'>Using GNU's GDB Debugger</a>
by Peter Jay Salzman.
</p>
<p>
I would like to thanks <a href="https://www.google.com/">Google</a> for
organizing <a href='https://summerofcode.withgoogle.com/'>Google Summer of Code</a>
and <a href='https://www.netbsd.org/foundation/'>The NetBSD Foundation</a>,
without them I would not be able to work on this project!
</p>
<p>
A particular and big thank you goes to my mentors David Maxwell, Jöerg
Sonnenberger, Taylor R. Campbell, Thomas Klausner and William J. Coldwell for
the invaluable help, guidance and feedbacks they're providing!
</p>
<h2>References</h2>
<ul>
<li>
<a href='https://www.linuxjournal.com/article/1059'>The ELF Object File Format: Introduction, Eric Youngdale</a>
</li>
<li>
<a href='https://www.linuxjournal.com/article/1060'>The ELF Object File Format by Dissection, Eric Youngdale</a>
</li>
<li>
<a href='http://refspecs.linuxbase.org/elf/elf.pdf'>Tool Interface Standard (TIS) Executable and Linking Format (ELF) Specification Version 1.2, TIS Committee</a>
</li>
<li>
<a href='http://dwarfstd.org/doc/DWARF4.pdf'>DWARF Debugging Information Format Version 4, DWARF Debugging Information Format Committee</a>
</li>
<li>
<a href='http://dwarfstd.org/doc/Debugging%20using%20DWARF-2012.pdf'>Introduction to the DWARF Debugging Format, Michael Eager</a>
</li>
<li>
<a href='http://sourceware.org/gdb/download/onlinedocs/gdb/index.html'>Debugging with GDB Tenth Edition for GDB Version 7.10.1, Free Software Foundation, Inc.</a>
</li>
<li>
<a href='http://dirac.org/linux/gdb/'>Using GNU's GDB Debugger, Peter Jay Salzman</a>
</li>
<li>
<a href='https://www.debian.org/doc/manuals/developers-reference/'>Debian Developer's Reference, Developer's Reference Team</a>
</li>
<li>
<a href='https://gnu.wildebeest.org/blog/mjw/2016/02/02/where-are-your-symbols-debuginfo-and-sources/'>
Where are your Symbols, Debuginfo and Sources?, Mark J. Wielaard</a>
</li>
<li>
<a href="//man.NetBSD.org/make.1">make(1)</a>
</li>
<li>
<a href="//man.NetBSD.org/elf.5">elf(5)</a>
</li>
<li>
<a href="//man.NetBSD.org/file.1">file(1)</a>
</li>
<li>
<a href="//man.NetBSD.org/gcc.1">gcc(1)</a>
</li>
<li>
<a href="//man.NetBSD.org/readelf.1">readelf(1)</a>
</li>
<li>
<a href="//man.NetBSD.org/objdump.1">objdump(1)</a>
</li>
<li>
<a href="//man.NetBSD.org/objcopy.1">objcompy(1)</a>
</li>
<li>
<a href="//man.NetBSD.org/gdb.1">gdb(1)</a>
</li>
<li>
<a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.README?r=1.353'>
src/share/bsd.README, 1.353
</a>
</li>
<li>
<a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.prog.mk?r=1.299'>
src/share/bsd.prog.mk, 1.299
</a>
</li>
<li>
<a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.lib.mk?r=1.367'>
src/share/bsd.lib.mk, 1.367
</a>
</li>
<li>
<a href='https://nxr.netbsd.org/xref/src/share/mk/bsd.own.mk?r=1.927'>
src/share/bsd.own.mk, 1.927
</a>
</li>
</ul>https://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interview_withpkgsrc 50th release interviews - Joerg SonnenbergerKamil Rytarowski2016-05-31T05:24:40+00:002016-05-31T13:23:25+00:00<p>The pkgsrc team has prepared the 50th release of their package management system, with the 2016Q1 version. It's infrequent event, as the 100th release will be held after 50 quarters.</p>
<p>The NetBSD team has prepared series of interviews with the authors. We started with Joerg Sonnenberger, a developer well known for his LLVM on NetBSD contribution.</p><p>The pkgsrc team has prepared the 50th release of their package management system, with the 2016Q1 version. It's infrequent event, as the 100th release will be held after 50 quarters.</p>
<p>The NetBSD team has prepared series of interviews with the authors. We started with Joerg Sonnenberger, a developer well known for his LLVM on NetBSD contribution.</p>
<p><b>Hi Joerg, please introduce yourself.</b></p>
<p>Hello Kamil,</p>
<p>I'm a software developer with a background in mathematics. Since I am doing Open Source development primarily for fun, my interests are changing over time and cover many different topics.</p></b>
<p><b>First of all, congratulations on the 50th release of pkgsrc! How do you feel about this anniversary?</b></p>
<p>The pkgsrc quarterly branches were one of the reasons for me to start looking at pkgsrc -- a bit more than a decade ago. As such, my 40th branch has passed recently as well, so it is a bit personal as well. I'm looking back proudly to what we have achieved, but also a bit sad to some of the faces and names that disappeared from the community over that time.</p>
<p><b>What are the main benefits of the pkgsrc system?</b></p>
<p>For me, the biggest advantage of pkgsrc is to provide me with the freedom to have a consistent environment across different platforms. Having to deal with BSD, Solaris and Linux flavours, it is very important to be able to sit down and get a consistent set of tools, consistent patches for programs to depend on etc.</p>
<p><b>Where and how do you use pkgsrc?</b></p>
<p>I'm using it on my own NetBSD and SmartOS systems as well as for providing software images for my customers at work.</p>
<p><b>What are the pkgsrc projects you are currently working on?</b></p>
<p>After that time focusing on bug hunting for the clang builds, I've recently returned to working on pbulk again. There are a number of areas where the distributed build can be improved. My main goal at the moment is to reduce the required coupling of master and slave. Sharing the distfiles is more challenging than it seems on first sight. NFS or null mounts are one option, but often, different packages sharing the same tarball are built in parallel. On an update, they will try to fetch the same file concurrently and often fail badly. The receipt I ended up using for the NetBSD 7.0 bulk builds is a low-memory virtual machine with nginx distribution the distfiles to all build slaves via HTTP(S). If a new file is obtained from elsewhere, it will be uploaded back to my local mirror via PUT. This has been quite reliable so far, but I still have to document the various steps involved for others. The new part will be better handling of the binary packages themselve. Using something like PUT again makes it easier to provide signatures without exposing the key material to all builds.</p>
<p><b>If you analyze the current state of pkgsrc, which improvements and changes do you wish for the future?</b></p>
<p>The biggest new term wish is that Leonardo's Summer of Code project will be successful and we finally get a proper infrastructure for building multiple packages in one step. The removal of non-DESTDIR support was kind of a show stopper for doing this sanely and that one is finally done. It didn't even take 10 years to reach that point From an infrastructural point of view, I want to tackle the shell fragment library topic again soon and also work on improving the build performance. The cwrapper support has been pending for a while, someone (else) has to sit down and fix the USE_CROSSBASE handling. I think that's the only real fallout we have left in the tree for making it the default, with all the performance boosts it can provide. More resources for builds is a general wish, but that also often requires porting to new hardware and the like.</p>
<p><b>Do you have any practical tips to share with the pkgsrc users?</b></p>
<p>The only practical tip is to aggressively use virtual machines or chroots for building whenever you can't just use binary packages. I.e. if you can't use pre-made binary packages, make your own.</p>
<p><b>What's the best way to start contributing to pkgsrc and what needs to be done?</b></p>
<p>Adopt a package, keep it updated and care for it and its environment. Everything tends to start from that. While pkglint is a handy tool for improving your own contributions, especially in pkgsrc-wip, it often helps even more to chat with a regular contributor whenever more challenging problems happen. Unlike many Linux packaging system, we try very hard to make simple things easy and compact. Many problems have different solutions and I sometimes look back to my own changes from years ago and find better solutions now. It is hard to put this kind of experience into documentation. None had the resources for training Watson yet.</p>
<p><b>Do you plan to participate in the upcoming pkgsrcCon 2016 in Kraków (1-3 July)?</b></p>
<p>I plan to come, I just don't know if I have time to prepare a presentation for it</p>
<p>Joerg</p>https://blog.netbsd.org/tnf/entry/pkgsrc_2016q1_the_fiftieth_pkgsrcpkgsrc-2016Q1 - the Fiftieth pkgsrc ReleaseAlistair Crooks2016-05-09T20:34:57+00:002016-05-09T20:34:57+00:00A perspective, looking back 19 years over pkgsrc, to celebrate the 50th quarterly pkgsrc release.<p>I was asked to prepare some stats on pkgsrc's growth, as part of the
celebrations of <a href=https://www.pkgsrc.org/>pkgsrc-2016Q1</a>, pkgsrc's fiftieth quarterly release.</p>
<p>This <a href=http://mail-index.netbsd.org/netbsd-announce/2004/02/02/0000.html>mail from 2004</a>
says:</p>
<blockquote>By my calculations, at the end of December 2003, there were 4310
packages in the NetBSD Packages Collection, up from 4170 the previous
month, a rise of only 140. We also tagged a new branch for pkgsrc,
which is being actively maintained - the branch is called
"pkgsrc-2003Q4". As the name implies, we will be branching pkgsrc on
a regular basis from now on, and maintaining the branch.</blockquote>
<p>We had previously cut pkgsrc releases to coincide with NetBSD
releases, and for some other events, but pkgsrc-2003Q4 can be seen
as the first branch where pkgsrc was really its own entity. It can be
checked out with the "pkgsrc-2003Q4" tag, and browsed at:</p>
<ul>
<li> <a href=http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/?only_with_tag=pkgsrc-2003Q4>the cvsweb URL</a></li>
<li> <a href=https://github.com/jsonn/pkgsrc.git>the git mirror</a></li>
</ul>
<p>We keep a complete log of changes we have made to pkgsrc since its inception.
At the lowest level, pkgsrc-2003Q4 had 3059 pkgsrc entries, whilst 2016Q1
had 14311 pkgsrc entries. Due to different versions of some languages, like
python, perl, ruby and php, the number of packages is a much higher number.
For instance, 2003Q4 had 4310 packages, and 2016Q1 had 17154.
We have also provided a histogram of
pkgsrc activity from its inception until now.</p>
<img src=https://www.netbsd.org/~agc/pkgsrc-perspective-2016Q1.png alt=pkgsrc-adds/updates/deletes height=638 width=522></img>
<p>To see how pkgsrc has spread across platforms, 2003Q4 supported 7 platforms,
2016Q1 supports 23 platforms.</p>
<p>In addition, pkgsrc has now grown its own conference. The first pkgsrccon was
in Vienna in 2004 -- see <a href=https://www.pkgsrc.org/pkgsrcCon/2004/>the original 2004
pkgsrccon information</a> and this
year's pkgsrccon is in Krakow -- see <a href=https://www.pkgsrc.org/pkgsrcCon/2016/>the 2016 pkgsrccon information</a>
We also produced a map of pkgsrccon venues for those interested
in the history and geography of pkgsrccon.</p>
<img src=https://www.netbsd.org/~agc/pkgsrccon.png alt=pkgsrccon-sites height=506 width=504>
<p>
Alistair Crooks<br>
for the pkgsrc team<br>
Sun May 8 11:26:15 PDT 2016</p>https://blog.netbsd.org/tnf/entry/pkgsrc_wip_migrating_to_netbsdpkgsrc-wip migrating to NetBSD.org, gitThomas Klausner2015-08-31T19:40:27+00:002015-08-31T20:17:40+00:00If everything goes as planned, the pkgsrc-wip CVS repository will be converted to git and hosted on NetBSD.org by end of September.
<p>
In July we cleaned up the repository so it can be converted easily; since then we've been working on the infrastructure and details of the conversion.
The main tasks are now finished. We have set up a server for it which hosts a <a href="http://wip.pkgsrc.org/cgi-bin/gitweb.cgi?p=pkgsrc-wip;a=summary">preliminary git conversion (on wip.pkgsrc.org)</a> of the CVS repository, created a <a href="https://netbsd.org/mailinglists/#pkgsrc-wip-changes">mailing list</a> for the commit messages, <a href="http://mail-index.netbsd.org/pkgsrc-wip-changes/">pkgsrc-wip-changes</a>, and prepared a <a href="https://wiki.netbsd.org/users/wiz/authorslist/">list of authors</a> for the conversion.
<p>
We've also provided a conversion of pkgsrc-wip based on data from July so that it can be tested on (nearly) live data. If you are interested in beta-testing the setup, send a suggestion for a username and an SSH public key to <a href="mailto:wiz@NetBSD.org">me</a>. Details on how to test are on the <a href="https://wiki.netbsd.org/users/wiz/wip-users/">NetBSD wiki</a> but will probably change some more over time.
<p>
We still need help for the conversion: if you are or were a wip contributor, please let <a href="mailto:wiz@NetBSD.org">me</a> know by <b>September 15</b> what name and email to use for the conversion from CVS to git. This conversion will not be done again, so after that date, the commit data will be final.
https://blog.netbsd.org/tnf/entry/pkgsrc_for_illumospkgsrc for Illumosimil2011-04-18T07:21:45+00:002011-07-31T10:03:59+00:00The Illumos project is "a community maintained derivative of the OpenSolaris ON source, including open source replacements for closed bits, and additional changes" (from <a href="http://www.illumos.org/">http://www.illumos.org/</a>).
A couple of month ago, the Illumos community launched "The Illumos pkgsrc project", thus communicating for the first time on pkgsrc being officially supported by a SunOS derivative.
<p>
<p>
The Illumos project is "a community maintained derivative of the OpenSolaris ON source, including open source replacements for closed bits, and additional changes" (from <a href="http://www.illumos.org/">http://www.illumos.org/</a>).
A couple of month ago, the Illumos community launched "The Illumos pkgsrc project", thus communicating for the first time on pkgsrc being officially supported by a SunOS derivative.
</p>
<p>
First thing done was a bootstrap wrapper written by Mads Worsøe Duun, <a href="https://sites.google.com/site/theillumospkgsrcproject/ipp-download">ipp</a>. This wrapper permits to easily prepare a pkgsrc environment on an OpenIndiana/Illumos/SunOS 5.11 platforms.
</p>
<p>
At some point i got involved into the project initiated by Mads, and we quickly got a build zone from OpenIndiana in order to bring some reality to the project. After a month of work, we finally got about 6000 packages built for this platform, and eventually were given a hosting zone for those.
</p>
<p>
Also, Mads just released a <code>pkgadd(1)</code> package which includes a pre-configured <code>pkgin</code> binary so those packages are immediately available for use.
As of today, it is possible to install more than 6000 pkgsrc binary packages on a SunOS 5.11 based platform, as easily as in NetBSD, DragonFly BSD or MINIX.
</p>
<p>
Many thanks to everyone who provided patches and workarounds in order to make this possible. Special thanks to the OpenIndiana crowd for providing us the build and hosting zones.
</p>
<p>
<ul>
<li><a href="http://www.illumos.org/projects/worsoe">The Illumos pkgsrc project</a></li>
<li><a href="http://pkgsrc-repo.uk.openindiana.org/reports/last/report.html">pkgsrc-2011Q1 bulk build report</a></li>
<li><a href="http://pkgsrc-repo.uk.openindiana.org/2011Q1/">pkgsrc-2011Q1 binary packages repository</a></li>
<li><a href="https://sites.google.com/site/theillumospkgsrcproject/ipp-download">pkgadd(1) startup package and ipp downloads</a></li>
<li><a href="http://www.illumos.org/projects/worsoe/wiki">Work-in-progress Wiki</a></li>
</ul>
</p>https://blog.netbsd.org/tnf/entry/pkgin_0_4_0pkgin 0.4.0imil2011-02-14T13:53:11+00:002011-02-14T14:57:50+00:00<p>After a year of PR's, feedbacks and various fixes, pkgin 0.4.0 is now available and its package, <code>pkgtools/pkgin</code> is up-to-date.</p><p>After a year of PR's, feedbacks and various fixes, pkgin 0.4.0 is now available and its package, <code>pkgtools/pkgin</code>, is up-to-date.</p>
<p>For the record, pkgin is a binary package manager aimed at simplifying manipulation of 3rd party software available as <code>pkgsrc</code> binaries. Its usage mimics similar tools like Debian's <code>apt</code> or RedHat's <code>yum</code>. While pkgin does not pretend to be as polished as those well known tools, it is now used in production in various companies (including mine), where clients cannot be told to "wait for the end of the build" ;)</p>
<p>0.4.0 brings some features that have been requested during the past year such as <i>download-only</i>, package reinstallation, <i>chroot</i> (in order to install packages to a chrooted environment), and bandwith calculation; thanks to Baptiste Daroussin for those two.</p>
<p>pkgin 0.4.0 also has now native support for MINIX 3 thanks to Gautam B.T. from MINIX, and better SunOS 5.1[0-1] support.</p>
<p>As of today, i've built and tested pkgin on the following platforms:
<ul>
<li>NetBSD 4.0</li>
<li>NetBSD 5.{0,1}</li>
<li>NetBSD current</li>
<li>DragonFly BSD 2.0 to 2.8</li>
<li>DragonFly BSD current</li>
<li>Solaris 10/SunOS 5.10</li>
<li>Opensolaris/SunOS 5.11</li>
<li>Debian GNU/Linux 5 and 6</li>
<li>Mac OS X 10.{5,6}</li>
<li>MINIX 3.1.8</li>
</ul>
</p>
<p>Last but not least, pkgin no longer depends on SQLite package, as the <a href="http://www.sqlite.org/amalgamation.html">Amalgamation</a> source file is now part of pkgin's tree. That means 0.4.0 binary installation will be as simple as pkg_add http://your.favorite.repository/All/pkgin-0.4.0.tgz</p>
<p>As always, testing and feedback is more than appreciated</p>https://blog.netbsd.org/tnf/entry/pkgsrccon_2010_more_than_pkgsrcpkgsrcCon 2010 - More Than pkgsrcMarc Balmer2010-06-15T12:35:14+00:002010-06-15T12:35:14+00:00pkgsrcCon 2010, the technical conference for people working on pkgsrc, took place may 28 - 30 at the Departement Informatik, University of Basel, Switzerland. Packed with interesting talks and in-depth discussions about the matter, the confence was a great success.
<p>
pkgsrcCon 2010, the technical conference for people working on pkgsrc, took place may 28 - 30 at the Departement Informatik, University of Basel, Switzerland. Packed with interesting talks and in-depth discussions about the matter, the confence was a great success.
<p>
This year's conference brought a new aspect to pkgsrcCon: The discussion of packing third party software in general. Not only the usual suspects attended this years conference, but also ports maintainers from OpenBSD and FreeBSD. From the OpenBSD team Jasper Lievisse Adriaanse and Landry Breuil joined, Erwin Lansing and Beat Gätzi represented FreeBSD. Jasper and Landry had registered for pkgsrcCon quite a while ago, Jasper even gave a presentation on how the OpenBSD ports system and team works and what the differences to pkgsrc are.
<p>
While attending a FreeBSD ports session during BSDCan in Ottawa and being an OpenBSD ports veteran of many years myself, I realized that all three teams have similar goals, but try to solve them in very different ways and all face more or less the same problems (e.g. patches not being accepted by upstream). So I asked Erwin Lansing, a FreeBSD portmeister, if he would like to join pkgsrcCon to start a discussion among all three teams. There is always something you can steal from the other groups and be it only the diffs to make a piece of software run on BSD.
<p>
During the conference, I moderated a panel session to figure out the current state, goals, and problems of the three projects, which was a highly interesting and lively discussion. In fact people so much liked the idea of inter-project discussions that this will continue somehow. One idea was a BSD third party software session during EuroBSDCon or to hold a pkgsrcCon right after an OpenBSD ports hackathon.
<p>
Of course pkgsrc specific topics were on the agenda as well and we got some deep insight and status updates on our favourite packaging system. NetBSD developer Michael van Elst presented even one more packaging system, OpenPKG.
<p>
All in all we spent three interesting days in Basel and we have to thank Vera Hardmeier for the excellent organisation, Petra Zeidler for putting together an interesting conference program and, last, but not least, Prof. Dr. Christian Tschudin for letting us use the rooms at his institute.
https://blog.netbsd.org/tnf/entry/the_pkgsrc_2010q1_releaseThe pkgsrc-2010Q1 ReleaseMatthias Scheler2010-04-20T19:50:33+00:002010-04-20T19:50:33+00:00<p>
The pkgsrc developers are happy to announce the new pkgsrc-2010Q1 release, which has support for even more packages than previous releases. Some major packages have also been updated in this release.
</p>
<p>
At the same time, the pkgsrc-2009Q4 release has been deprecated, and continuing engineering starts on the pkgsrc-2010Q1 release.
</p>
<p>
Some highlights of the new pkgsrc-2010Q1 release are:
<ul>
<li>we have almost finished the transition to DESTDIR installation, where a staging directory is used to make a binary package, which is then managed by the pkg_install tools</li>
<li>gnome has been updated to version 2.28.1, kde to 4.3.5</li>
<li>we have started changing packages to default to KDE4 instead of KDE3. For now, the old packages are still available as *-kde3 e.g. <code>amarok</code> is the KDE4 package, and <code>amarok-kde3</code> is the KDE3 one.</li>
<li> the default python package is now python26 </li>
<li>squid 3.1.1 is now in pkgsrc, with some support for IPv6 </li>
<li> php 5.3.x has been added </li>
<li> The conversion from the last teTeX distribution to texlive (currently 2009) is still in progress.</li>
<li> many, many packages have been updated to newer versions, to take advantage of fixes and improved functionality. The following
versions of packages are included in the <br />pkgsrc-2010Q1 release:
<ul>
<li> apache-2.2.15 </li>
<li> bzr-2.0.3 </li>
<li> firefox-3.6.3 </li>
<li> git-1.6.6.2 (the package is known as scmgit in pkgsrc) </li>
<li> gnome-2.28.1 </li>
<li> kde-4.3.5 </li>
<li> mercurial-1.5.1 </li>
<li> mysql-5.1.44nb2 </li>
<li> openoffice-3.1.1 and openoffice-bin-3.2.0 </li>
<li> perl-5.10.1 </li>
<li> postgresql-8.3.9nb2 and postgresql-8.4.2 </li>
<li> python-2.5.4nb5 and python-2.6.4nb4 </li>
<li> ruby-1.8.7.174nb4 </li>
<li> samba-3.3.12 </li>
<li> seamonkey-2.0.4 </li>
<li> subversion-1.6.9nb1 </li>
<li> wireshark-1.2.7 </li>
<li> zope-3.3.1 </li>
</ul>
</li>
<li> other notable changes include
<ul>
<li> we bid a fond thanks, and farewell, to some old favourites,
such as php4 and related packages, the old vmware modules
packages, sun's jdk and jre versions 1.4 and 1.5, the ISC
dhcp 3.x packages, galeon, swing, typolight-2.6 and tcl-8.3 </li>
<li> the addition of some interesting, pertinent, and shiny
packages such as tn3270 (:-) - brought over from NetBSD's
src archive), mingw, colordiff, easygit, monotone-el, swt,
fuse-bindfs, php-5.3, samba-3.3, xymon, musca, and qt4-mng </li>
<li> notable updates to packages such as bsd and gnu tar, amarok,
lame, mpg123, mysql, openldap, postgresql, sqlite, boehm-gc,
boost, doxygen, fossil, glib, libev, libffi, memcached, nspr,
nss, pango, pcre, rt3, readline, swig, xulrunner, vim, qemu,
chicken, mono, parrot, openjdk7, python, php5, squeak, clamav,
dovecot, fetchmail, getmail, gmime, linmilter, mew, sendmail,
spamassassin, squirrelmail, thunderbird, octave, pari,
calibre, dhcpcd, gupnp, nmap, rdist6, rsync, rtorrent, tnftpd,
tor, transmission, unbound, aide, netpgp, openssl, bash, osh,
tcsh, bacula, cdrtools, memtester, grub, pstree, rasqal,
openbox, firefox, ikiwiki, lighttpd, mediawiki, nginx, squid,
seamonkey, typolight, gtk2, xsnow </li>
</ul>
</li>
<li> the <b>Package of the Quarter</b> award is hereby awarded to qemu, nominated by Joerg Sonnenberger, and samba33, nominated by Matthias Scheler. </li>
</ul>
The list of platforms supported by pkgsrc is AIX, BSD/OS, Darwin (Mac OS X), DragonFly BSD, FreeBSD, HP/UX, IRIX, Interix, Linux, NetBSD, OSF1, OpenBSD, QNX and SunOS (Solaris). Haiku support is almost ready to be added to pkgsrc. We are aware that support for some platforms is at a more mature stage than others, and would like to encourage feedback from users and developers on our more esoteric platforms.
<ul>
<li> continuing engineering on the "stable" releases of pkgsrc continues
to work well, and our release engineering team has done a marvelous
job in pulling up changes to the stable release. Our thanks go to
Matthias Scheler, Lubomir Sedlacik, Tyler Retzlaff, and S.P.Zeidler
for all the hard work they do in sanity checking pullup requests, and
managing the stable releases in pkgsrc. </li>
<li> constant bulk building on a number of platforms has improved our
ability to identify potential areas of concern, and to correct them
sooner. It has also improved our ability to make binary packages
available, and we are working on ways to improve this further. For
more information, please refer to the pkgsrc-bulk mailing list,
archives available at <a rel="nofollow" href="http://mail-index.netbsd.org/pkgsrc-bulk/">http://mail-index.netbsd.org/pkgsrc-bulk/</a> </li>
<li> the number of packages has grown from 9100 to 9315; the number of supported platforms is currently 14. NetBSD, on all its supported architectures, is considered to be one pkgsrc platform. </li>
</ul>
</p>
<p>
As always, we'd like to encourage users of the packages collection to
audit for security problems at least every day using <code>pkg_admin audit</code>
- this will provide notification of any packages which are vulnerable
to exploit. <code>pkg_admin</code> is part of the pkg_install tools.
</p>
<p>
The pkgsrc-security team do a marvelous job in tracking notifications
of vulnerabilities in packages, and disseminating this information,
and our sincere thanks go to them for this essential work.
</p>
<p>
We'd also really appreciate it if people would install the
<a href="http://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/pkgtools/pkgsurvey/README.html">pkgsrc/pkgtools/pkgsurvey</a> package, and then run the pkgsurvey script
for us. This will forward us a list of the packages installed on that
machine, and the operating system and release level of the operating
system. The results will be kept confidential, but the output will
help us analyse the packages that are most used.
</p>
<p>
The source tar files for the new release can be found at:
</p>
<p align="center">
<a rel="nofollow" href="ftp://ftp.NetBSD.org/pub/pkgsrc/pkgsrc-2010Q1/pkgsrc.tar.gz">ftp://ftp.NetBSD.org/pub/pkgsrc/pkgsrc-2010Q1/pkgsrc.tar.gz</a>
<br /
<br />
or
<br />
<br />
<a rel="nofollow" href="ftp://ftp.NetBSD.org/pub/pkgsrc/pkgsrc-2010Q1/pkgsrc.tar.bz2">ftp://ftp.NetBSD.org/pub/pkgsrc/pkgsrc-2010Q1/pkgsrc.tar.bz2</a>
<p>
<p>
You can also use the <code>pkgsrc-2010Q1</code> tag to check it out yourself from <code>anoncvs.NetBSD.org</code> or any of the mirrors.
</p>
<p>
Alistair Crooks<br />
On behalf of the pkgsrc developers
</p>https://blog.netbsd.org/tnf/entry/pkgsrc_solaris_and_5000_binarypkgsrc, Solaris, and 5000 binary packagesAlistair Crooks2009-10-19T15:23:27+00:002012-04-10T21:36:09+00:00<p>
Jonathan Perkin has posted an interesting blog entry entitled
<a href="http://www.perkin.org.uk/posts/apt-get-and-5000-packages-for-solaris10x86.html">"apt-get" and 5,000 packages for Solaris10/x86</a>
about using pkgsrc and the binary package manager <a href="http://imil.net/pkgin/">pkgin</a> on Solaris 10/x86. In pkgsrc, we can get conditioned to the fact that package management, in a coherent, well-controlled way; it's nice to see this gaining further traction in Solaris circles too.
</p><p>
Jonathan Perkin has posted an interesting blog entry entitled
<a href="http://www.perkin.org.uk/posts/apt-get-and-5000-packages-for-solaris10x86.html">"apt-get" and 5,000 packages for Solaris10/x86</a>
about using pkgsrc and the binary package manager <a href="http://imil.net/pkgin/">pkgin</a> on Solaris 10/x86. In pkgsrc, we can get conditioned to the fact that package management, in a coherent, well-controlled way; it's nice to see this gaining further traction in Solaris circles too.
</p>
<p>
Oh, and on a historical note - Solaris was the first non-NetBSD platform for pkgsrc. I wonder if anyone remembers when it was committed (without resorting to cvs logs)?
</p>https://blog.netbsd.org/tnf/entry/pkgin_a_tool_to_managepkgin, a tool to manage pkgsrc binary packagesimil2009-05-27T13:38:04+00:002009-05-27T13:49:53+00:00<p>From the day I began using NetBSD I felt that there was a need for a binary
package manager. pkgsrc is great of course, I use it, love it and contribute
to, but I'm not brave enough to build my entire environment with it. Of
course the esteemed <code>pkg_add(1)</code> and <code>pkg_delete(1)</code>
can handle binary packages installation, but when it comes to upgrades,
binary packages manipulation is far from being straightforward.</p><p>
From the day I began using NetBSD I felt that there was a need for a binary
package manager. pkgsrc is great of course, I use it, love it and contribute
to, but I'm not brave enough to build my entire environment with it. Of
course the esteemed <code>pkg_add(1)</code> and <code>pkg_delete(1)</code>
can handle binary packages installation, but when it comes to upgrades,
binary packages manipulation is far from being straightforward.
</p>
<p>
For many years, Linux has had tools like <code>apt</code>, <code>yum</code>
and <code>pacman</code> that are able to handle packages installation
properly using remote repositories. With such tools, packages installation,
removal and upgrade are really simple, there is no need to <code>cvs
checkout</code>, <code>make(1)</code> or worse in order to have a usable
environment within a few minutes. I felt that this was something which might have
impacted on a persons decision to install NetBSD.
</p>
<p>
That's why I started the <a href="http://imil.net/pkgin/">pkgin</a> project
3 months ago. <a href="http://imil.net/pkgin/">Pkgin</a> (pronounced
"pay-kay-djin") is aimed at being an <code>apt / yum</code> like tool for
managing pkgsrc binary packages. It relies on <code>pkg_summary(5)</code>
for installation, removal and upgrade of packages and associated
dependencies, using a remote repository.
</p>
<p>
At the moment, <a href="http://imil.net/pkgin/">pkgin</a> is available in
<a href="http://pkgsrc-wip.sourceforge.net/">pkgsrc-wip</a>, not yet in the official <a href="http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/">pkgsrc</a> tree, as it is still in a test phase and the code obviously moves a lot. Expect the first <a href="http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/">pkgsrc</a> release to land around June.
</p>
<p>
For the brave, latest code is available via CVS, using <code>:pserver:anoncvs@cvs.gcu.info:/cvs</code> as the <i>CVSROOT</i> and <code>anoncvs</code> as the password. Please do not hesitate to give feedback, but try not to ask to change its entire behaviour as the choices have already been discussed with many NetBSD developers.
</p>https://blog.netbsd.org/tnf/entry/pkgsrc_2009q1_has_been_branchedPkgsrc-2009Q1 has been branchedAlistair Crooks2009-04-07T16:46:24+00:002009-04-07T17:21:56+00:00<p>Just a quick note to say that pkgsrc has been branched for pkgsrc-2009Q1, and has already had some pullups - thanks to the pkgsrc-releng folks.</p>
<p>We're going to sort out the release announcement now, and get that out around the same time as the binary packages get built.</p>
<p>Thanks to all involved in making this branch happen.</p>
<p>Enjoy pkgsrc-2009Q1, folks!</p>https://blog.netbsd.org/tnf/entry/pkgsrc_2009q1_freeze_starts_onPkgsrc-2009Q1 Freeze starts on March 22nd, 23:59 UTCAlistair Crooks2009-03-22T05:05:01+00:002009-03-22T05:05:01+00:00<p>In preparation for the pkgsrc-2009Q1 branch, pkgsrc will be frozen for new packages and infrastructure changes, starting on Sunday March 22nd, at 23:59 UTC.</p>
<p>Some background to our freezes: pkgsrc makes four releases a year, named after the quarter in which all the work took place, and the quarter in which the packages themselves could last have been updated. The release name is thus 2009Q1, 2009Q2, etc. So that we can stabilise packages before the branch is created, we institute a freeze on new functionality - no new packages, and the infrastructure itself does not get changed. This means that we can take a look at the results of bulk build runs, and fix up any loose ends in the packages themselves, without having to worry about the basic building blocks of pkgsrc changing from under us - we have a stable platform to build upon.</p>
<p>It always happens that third party software vendors want to release a new version of their software just after we've entered the freeze. When that happens, we ask the pkgsrc developers to make a judgement call on it - they are the ones who will be maintaining this, after all - and if they think it needs to be updated, we ask them to get approval from the pkgsrc PMC. Again, to minimise the effect on other packages, we like to limit this to leaf packages. These are packages which can be changed easily with no consequences - packages which are not pre-requisites for any other package.</p>
<p>In general, pkgsrc tries to be conservative without being out of date in the versions of the packages. Trying to stay on the bleeding edge may be great fun at times, and does ensure early access to new features, but there are consequences for others in the stability of such packages. We have some packages which are maintained like this - usually, they have a -devel suffix - but the vast majority of packages are known to be good versions. We know, because we run those versions ourselves.</p>
<p>So what does pkgsrc-2009Q1 have in store for us? New pkg_install tools, speedups for the buildlink3 infrastructure, gnome 2.26, and many more things.</p>
<p>Look for pkgsrc-2009Q1 coming to a repository near you in a couple of weeks time.</p>https://blog.netbsd.org/tnf/entry/german_perlworkshop_and_a_netbsdGerman Perlworkshop and a NetBSD related presentationrhaen2009-03-06T11:18:33+00:002009-03-06T11:37:22+00:00<p>I attended the German Perlworkshop from 25th February to 27th February in Frankfurt. The German Perlworkshop follows the tradition of the YAPC conferences. It's listed on their webpage, however the name has been changed to represent the German localisation. </p>
<p>I gave two presentations about the different usage of Perl, one was related to the NetBSD project. I maintain quite a lot of Perl packages inside the pkgsrc repository and run our Perl package update list, too. As I encounter frequent problems with our Perl modules, like missing ChangeLogs, incorrect version numbering for pkgsrc, broken dependencies, etc I decided to give a talk about these problems. It's called "Maintaining the be*st" and it deals with some of the different aspects in maintaining Perl packages for pkgsrc. I translated the slides to English, so all the English readers of this blog are able to read them. The audio recording was done during the presentation, however, it is in German.</p>
<ul>
<li><a href="http://www.netbsd.org/~rhaen/presentations/2009-DeutscherPerlWorkshop/Maintaining_the_best.pdf">Maintaining the best (German|PDF)</a></li>
<li><a href="http://www.netbsd.org/~rhaen/presentations/2009-DeutscherPerlWorkshop/Maintaining_the_best_english.pdf">Maintaining the best (English|PDF)</a></li>
<li><a href="http://www.netbsd.org/~rhaen/presentations/2009-DeutscherPerlWorkshop/Maintaining_the_best.mp3">Maintaining the best (Audio recording in German|MP3 ~21mb)</a></li>
<li><a href="http://www.netbsd.org/~rhaen/presentations/2009-DeutscherPerlWorkshop/Reverse_Testing.pdf">Reverse Testing (German|PDF)</a></li>
<li><a href="http://www.netbsd.org/~rhaen/presentations/2009-DeutscherPerlWorkshop/Reverse_Testing_english.pdf">Reverse Testing (English|PDF)</a></li>
<li><a href="http://www.netbsd.org/~rhaen/presentations/2009-DeutscherPerlWorkshop/Reverse_Testing.mp3">Reverse Testing (Audio recording in German|MP3 ~9mb)</a></li>
</ul>