GSoC 2018 Reports: Configuration files versioning in pkgsrc, Part 1

Prepared by Keivan Motavalli as part of GSoC 2018.

Packages may install code (both machine executable code and interpreted programs), documentation and manual pages, source headers, shared libraries and other resources such as graphic elements, sounds, fonts, document templates, translations and configuration files, or a combination of them.

Configuration files are usually the mean through which the behaviour of software without a user interface is specified. This covers parts of the operating systems, network daemons and programs in general that don't come with an interactive graphical or textual interface as the principal mean for setting options.

System wide configuration for operating system software tends to be kept under /etc, while configuration for software installed via pkgsrc ends up under LOCALBASE/etc (e.g., /usr/pkg/etc).

Software packaged as part of pkgsrc provides example configuration files, if any, which usually get extracted to LOCALBASE/share/examples/PKGBASE/.

After a package has been extracted pre-pending the PREFIX(/LOCALBASE?) to relative file paths as listed in the PLIST file, metadata entries (such as +BUILD_INFO, +DESC, etc) get extracted to PKG_DBDIR/PKGNAME-PKGVERSION (creating files under /usr/pkg/pkgdb/tor-0.3.2.10, as an example).

Some shell script also get extracted there, such as +INSTALL and +DEINSTALL. These incorporate further snippets that get copied out to distinct files after pkg_add executes the +INSTALL script with UNPACK as argument.

Two main frameworks exist taking care of installation and deinstallation operations: pkgtasks, still experimental, is structured as a library of POSIX-compliant shell scripts implementing functions that get included from LOCALBASE/share/pkgtasks-1 and called by the +INSTALL and +DEINSTALL scripts upon execution.

Currently pkgsrc defaults to using the pkginstall framework, which as mentioned copies out from the main file separate, monolithic scripts handling the creation and removal of directories on the system outside the PKGBASE, user accounts, shells, the setup of fonts... Among these and other duties, +FILES ADD, as called by +INSTALL, copies with correct permissions files from the PKGBASE to the system, if required by parts of the package such as init scripts and configuration files.

Files to be copied are added as comments to the script at package build time, here's an example:

# FILE: /etc/rc.d/tor cr share/examples/rc.d/tor 0755
# FILE: etc/tor/torrc c share/examples/tor/torrc.sample 0644

"c" indicates that LOCALBASE/share/examples/rc.d/tor is to be copied in place to /etc/rc.d/tor with permissions 755, "r" that it is to be handled as an rc.d script.

LOCALBASE/share/examples/tor/torrc.sample, the example file coming with default configuration options for the tor network daemon, is to be copied to LOCALBASE/etc/tor/torrc.

As of today, this only happens if the package has never been installed before and said configuration file doesn't already exist on the system, this to avoid overwriting explicit option changes made by the user (or site administrator) when upgrading or reinstalling packages.

Let's see where how it's done... actions are defined under case switches:

case $ACTION in
ADD)
        ${SED} -n "/^\# FILE: /{s/^\# FILE: //;p;}" ${SELF} | ${SORT} -u |
        while read file f_flags f_eg f_mode f_user f_group; do
	…
	case "$f_flags:$_PKG_CONFIG:$_PKG_RCD_SCRIPTS" in
	*f*:*:*|[!r]:yes:*|[!r][!r]:yes:*|[!r][!r][!r]:yes:*|*r*:yes:yes)
	if ${TEST} -f "$file"; then
		${ECHO} "${PKGNAME}: $file already exists"
	elif ${TEST} -f "$f_eg" -o -c "$f_eg"; then
		${ECHO} "${PKGNAME}: copying $f_eg to $file"
		${CP} $f_eg $file
		[...]
[...]

Programs and commands are called using variables set in the script and replaced with platform specific paths at build time, using the FILES_SUBST facility (see mk/pkginstall/bsd.pkginstall.mk) and platform tools definitions under mk/tools.

In order to also store revisions of example configuration files in a version control system, +FILES needs to be modified to always store revisions in a VCS, and to attempt merging changes non interactively when a configuration file is already installed on the system.

In order to avoid breakage, installed configuration is backed up first in the VCS, separating user-modified files from files that have been already automatically merged in the past, in order to allow the administrator to easily restore the last manually edited file in case of breakage.

Branches are deliberately not used, since not everyone may wish to get familiar with version control systems technicalities when attempting to make a broken system work again.

Here's what the modified pkginstall +FILES script does when installing spamd:

		case "$f_flags:$_PKG_CONFIG:$_PKG_RCD_SCRIPTS" in
		*f*:*:*|[!r]:yes:*|[!r][!r]:yes:*|[!r][!r][!r]:yes:*|*r*:yes:yes)
		if ${TEST} "$_PKG_RCD_SCRIPTS" = "no" -a ! -n "$NOVCS"; then

VCS functionality only applies to configuration files, not to rc.d scripts, and only if the environment variable $NOVCS is unset. Set it to any value - yes will work :) - to disable the handling of configuration file revisions.

A small note: these options could, in the future, be parsed by pkg_add from some configuration file and passed calling setenv before executing +INSTALL, without the need to pass them as arguments and thus minimizing code path changes.

$VCSDIR is used to set a working directory for VCS functionality different from the default one, VARBASE/confrepo.

VCSDIR/automergedfiles is a textual list made by the absolute paths of installed configuration files already automatically merged in the past during package upgrades.

Manually remove entries from the list when you make manual configuration changes after a package has been automatically merged!

And don't worry: automatic merging is disabled by default, set $VCSAUTOMERGE to enable it.

When a configuration file already exists on the system, if it is absent from VCSDIR/automergedfiles, it is assumed to be user edited and copied to

VCSDIR/user/path/to/installed/file is a working file REGISTERed (added and committed) to the version control system.

Check it out and restore it from there in case of breakage!

If the file is about to get automatically merged, and the operation already succeeded in the past, then you can find automatically merged revisions of installed configuration files under VCSDIR/automerged/path/to/installed/file checkout the required revision!

A new script, +VERSIONING, handles operations such as PREPARE (checks that a vcs repository is initialized), REGISTER (adds a configuration file from the working directory to the repo), COMMIT (commit multiple REGISTER actions after all configuration has been handled by the +FILES script, for VCSs that support atomic transactions), CHECKOUT (checks out the last revision of a file to the working directory) and CHECKOUT-FIRST (checks out the first revision of a file).

The version control system to be used as a backend can be set through $VCS. It default to RCS, the Revision Control System, which works only locally and doesn't support atomic transactions.

It will get setup as a tool when bootstrapping pkgsrc on platforms that don't already come with it.

Other backends such as CVS are supported and more will come; these, being used at the explicit request of the administrator, need to be already installed and placed in a directory part of $PATH.

Let's see what happens with rcs when NOVCS is unset, installing spamd (for the first time).

cd pkgsrc/mail/spamd
# bmake
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
> Fetching spamd-20060330.tar.gz
[...]
bmake install
===> Installing binary package of spamd-20060330nb2
spamd-20060330nb2: Creating group ``_spamd''
spamd-20060330nb2: Creating user ``_spamd''
useradd: Warning: home directory `/var/chroot/spamd' doesn't exist, and -m was not specified
rcs: /var/confrepo/defaults//usr/pkg/etc/RCS/spamd.conf,v: No such file or directory
/var/confrepo/defaults//usr/pkg/etc/spamd.conf,v  <--  /var/confrepo/defaults//usr/pkg/etc/spamd.conf
initial revision: 1.1
done
REGISTER /var/confrepo/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb2: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf
===========================================================================
The following files should be created for spamd-20060330nb2:

        /etc/rc.d/pfspamd (m=0755)
            [/usr/pkg/share/examples/rc.d/pfspamd]

===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $

Don't forget to add the spamd ports to /etc/services:

spamd           8025/tcp                # spamd(8)
spamd-cfg       8026/tcp                # spamd(8) configuration

===========================================================================

/usr/pkg/etc/spamd.conf didn't already exists, so in the end, as usual, the example/default configuration /usr/pkg/share/examples/spamd/spamd.conf gets copied to PKG_SYSCONFDIR/spamd.conf.

The modified +FILES script also copied the example file under the VCS working directory at /var/confrepo/default/share/examples/spamd/spamd.conf it then REGISTEREd this (initial) revision of the default configuration with RCS.

When installing an updated (ouch!) spamd package, the installed configuration at /usr/pkg/etc/spamd.conf won't get touched, but a new revision of share/examples/spamd/spamd.conf will get stored using the revision control system.

For VCSs that support them, remote repositories can also be used via $REMOTEVCS. From the +VERSIONING comment:

REMOTEVCS, if set, must contain a string that the chosen VCS understands as
an URI to a remote repository, including login credentials if not specified
through other means. This is non standard across different backends, and
additional environment variables and cryptographic material 
may need to be provided.  

So, if using CVS accessing a remote repository over ssh, one should setup keys on the systems, then set and export

VCS=cvs
CVS_RSH=/usr/bin/ssh
REMOTEVCS=user@hostname:/path/to/existing/repo

Remember to initialize (e.g., mkdir -p /path/to/repo; cd /path/to/repo; cvs init) the repository on the remote system before attempting to install new packages.

Let's try to make a configuration change to spamd.conf and reinstall it:

I will enable whitelists uncommenting

#whitelist:\
#       :white:\
#       :method=file:\
#       :file=/var/mail/whitelist.txt:

...and enable automerge:

export VCSAUTOMERGE=yes
bmake install
[...]
merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!

No conflicts get reported, diff shows no output since the installed file is already identical to the automerged one, which is installed again and contains the whitelisting options uncommented:

more /usr/pkg/etc/spamd.conf

# Whitelists are done like this, and must be added to "all" after each
# blacklist from which you want the addresses in the whitelist removed.
#
whitelist:\
        :white:\
        :method=file:\
        :file=/var/mail/whitelist.txt:

Let's simulate instead the addition of a new configuration option in a new package revision: this shouldn't generate conflicts!

bmake extract
===> Extracting for spamd-20060330nb2
vi work/spamd-20060330/etc/spamd.conf
# spamd config file, read by spamd-setup(8) for spamd(8)
#
# See spamd.conf(5)
# this is a new comment!
#

save, run bmake; bmake install:

===> Installing binary package of spamd-20060330nb2
RCS file: /var/confrepo/defaults//usr/pkg/etc/spamd.conf,v
done
/var/confrepo/defaults//usr/pkg/etc/spamd.conf,v  <--  /var/confrepo/defaults//usr/pkg/etc/spamd.conf
new revision: 1.9; previous revision: 1.8
done
REGISTER /var/confrepo/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb2: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb2: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
saving the currently installed revision to /var/confrepo/automerged//usr/pkg/etc/spamd.conf
RCS file: /var/confrepo/automerged//usr/pkg/etc/spamd.conf,v
done
/var/confrepo/automerged//usr/pkg/etc/spamd.conf,v  <--  /var/confrepo/automerged//usr/pkg/etc/spamd.conf
file is unchanged; reverting to previous revision 1.1
done
/var/confrepo/defaults//usr/pkg/etc/spamd.conf,v  -->  /var/confrepo/defaults//usr/pkg/etc/spamd.conf
revision 1.1
done
merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!
--- /usr/pkg/etc/spamd.conf     2018-07-09 22:21:47.310545283 +0200
+++ /var/confrepo/defaults//usr/pkg/etc/spamd.conf.automerge    2018-07-09 22:29:16.597901636 +0200
@@ -5,6 +5,7 @@
 # See spamd.conf(5)
 #
 # Configures whitelists and blacklists for spamd
+# this is a new comment!
 #
 # Strings follow getcap(3) convention escapes, other than you
 # can have a bare colon (:) inside a quoted string and it
revert from the last revision of /var/confrepo/automerged//usr/pkg/etc/spamd.conf if needed
===========================================================================
The following files should be created for spamd-20060330nb2:

        /etc/rc.d/pfspamd (m=0755)
            [/usr/pkg/share/examples/rc.d/pfspamd]

===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $

Don't forget to add the spamd ports to /etc/services:

spamd           8025/tcp                # spamd(8)
spamd-cfg       8026/tcp                # spamd(8) configuration

===========================================================================

more /usr/pkg/etc/spamd.conf

[...]
# See spamd.conf(5)
#
# Configures whitelists and blacklists for spamd
# this is a new comment!
#
# Strings follow getcap(3) convention escapes, other than you
[...]
# Whitelists are done like this, and must be added to "all" after each
# blacklist from which you want the addresses in the whitelist removed.
#
whitelist:\
        :white:\
        :method=file:\
        :file=/var/mail/whitelist.txt:

We're set for now. In case of conflicts merging, the user is notified, the installed configuration file is not replaced and the conflict can be manually resolved by opening the file (as an example, /var/confrepo/defaults/usr/pkg/etc/spamd.conf.automerge) in a text editor.

[0 comments]