GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 4: configuration deployment, pkgtools and future improvements

As introduced in GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 2: remote repositories (git and CVS) pkgsrc supports using some version control system repositories to search for site-specific configuration for packages being installed, and deploys it on the system.

Configuration deployment: intro to VCSCONFPULL

Support for configuration tracking is in scripts, pkginstall scripts, that get built into binary packages and are run by pkg_add upon installation. The idea behind the proposal suggested that users of the new feature should be able to store revisions of their installed configuration files, and of package-provided default, both in local or remote repositories. With this capability in place, it doesn't take much to make the scripts "pull" configuration from a VCS repository at installation time.

That's what setting VCSCONFPULL=yes in pkg_install.conf after having enabled VCSTRACK_CONF does: You are free to use official, third party prebuilt packages that have no customization in them, enable these options, and point pkgsrc to a private conf repository. If it contains custom configuration for the software you are installing, an attempt will be made to use it and install it on your system. If it fails, pkginstall will fall back to using the defaults that come inside the package. RC scripts are always deployed from the binary package, if existing and PKG_RCD_SCRIPTS=yes in pkg_install.conf or the environment.

This will be part of packages, not a separate solution like configuration management tools. It doesn't support running scripts on the target system to customize the installation, it doesn't come with its domain-specific language, it won't run as a daemon or require remote logins to work. It's quite limited in scope, but you can define a ROLE for your system in pkg_install.conf or in the environment, and pkgsrc will look for configuration you or your organization crafted for such a role (e.g., public, standalone webserver vs reverse proxy or node in a database cluster)

Configuration files should be put in branches named according to a specific convention, their paths relative to the repository is implied to be their absolute path, prepending a "/", on the target system.

Branch names define the package name they contain configuration for, the role, the version of the package the configuration is made for and a range of compatible software releases that should work with such configuration options, a best match is attemped inside of compatible ranges to the nearest software release, and roles can be undefined (both on the branch, to make it apt for any system, or on the target to ignore roles specified in branch names where they don't matter). More on that later, now some practical examples using git:

Set up a remote repository on some machine, if you are not using a public facing service or private repositories in third party services:

$ id                                                                                                          
uid=1001(vcs) gid=1001(vcs) groups=1001(vcs)
$ mkdir confpullexample.git                                                                                   
$ cd confpullexample.git/                                                                                     
$ git --bare init                                                                                             
Initialized empty Git repository in /usr/home/vcs/confpullexample.git/                                        

Check for correct settings on the target machine: (a REMOTEVCS URI is required, the VCSDIR is NOT used)

pkghost# cat /usr/pkg/etc/pkg_install.conf
VCSTRACK_CONF=yes
VCSCONFPULL=yes
REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/confpullexample.git
VCS=git

Do no reuse repositories that track existing configuration files from packages, set a new one here. It will be structured differently, with branches that can specialize between them and yet contain configuration files common to each one (from master/trunk/...). Some technicalities are needed before making practical examples. I will now cheat and cite direcly the comment in mk/pkginstall/versioning:

The remote configuration repository should contain branches named according to the following convention: category_pkgName_pkgVersion_compatRangeStart_compatRangeEnd_systemRole an optional field may exist that explicitates part of the system hostname category_pkgName_pkgVersion_compatRangeStart_compatRangeEnd_systemRole_hostname .

the branch should contain needed configuration files. Their path relative to the repository is then prepended with a "/" and files force copied to the system and chmod 0600 executed on them. Permission handling and removal upon package uninstallation are not supported.

The branch to be used, among the available ones, is chosen this way: branches named according to the convention that provide configuration for category/packageName are filtered from the VCS output; then, all branches whose ranges are compatible with the version of the package being installed are selected. The upper bound of the range is excluded as a compatible release if using sequence based identifiers. If system role is set through the ROLE environment variable, and it's different from "any", and branches exists whose role is different from "any", then their role gets compared with the one defined on the system or in pkg_add config. The last part of the branch name is optional and, if present, is compared character by character with the system hostname, finally selecting the branches that best match it. As an example, a branch named mail_postfix_3.3.0_3.0.0_3.3.20_mailrelay_ams will match with system hostname amsterdam09. A system with its ROLE unspecified or set to ANY will select branches independently of the role they are created for, scoring and using the one with the best matching optional hostname and/or nearest to the target release as explained below:

The checks now further refine the candidates: if a branch pkgVersion exactly corresponds with the version of the package being installed, that branch gets selected, otherwise the procedure uses the one which is closest to the package version being installed. Non-numerical values in package versions are accounted for when checking for an exact match, and are otherwise ignored. Only integer versions and dot-separated sequence based identifiers are understood when checking for compatible software ranges and for the closest branch, if no branch exactly matches with the package version being installed. Dates are handled provided they follow the ISO 8601 scheme: YYYY-MM-DD, YYYYMMDD

Let's suppose that an hypothetical team uses a common ssh configuration, on all systems, to disable root and passwordless logins, and enable logins from users in a specific group.

the main/master branch of the repository will then contain one custom object, etc/ssh/sshd_config that will get included when branching from there (specific removals are always possible).

$ whoami
devuser
$ file $HOME/.ssh/id_rsa
/home/devuser/.ssh/id_rsa: PEM RSA private key
$ pwd
/home/devuser/sim
$ git clone ssh://vcs@192.168.100.112/usr/home/vcs/confpullexample.git .
Cloning into '.'...
The authenticity of host '192.168.100.112 (192.168.100.112)' can't be established.
ECDSA key fingerprint is SHA256:RMkiZlYqNIKlgDQUvhFBXLUpW2qcLd1nuEi4NaROkLg.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.100.112' (ECDSA) to the list of known hosts.
warning: You appear to have cloned an empty repository.
$ mkdir -p etc/ssh
$ cat /etc/ssh/sshd_config > etc/ssh/sshd_config

customize it as needed, then

$ echo "PermitEmptyPasswords no" >> $HOME/sim/etc/ssh/sshd_config
$ echo "PermitRootLogin no" >> $HOME/sim/etc/ssh/sshd_config
$ echo "AllowGroup ops" >> $HOME/sim/etc/ssh/sshd_config

and load it in the repo:

$ git add etc/ssh/sshd_config
$ git commit -m "add common sshd_config to master"
[master (root-commit) ec91ffc] add common sshd_config to master
 1 file changed, 134 insertions(+)
 create mode 100644 etc/ssh/sshd_config
$ git push
Counting objects: 5, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (5/5), 1.89 KiB | 966.00 KiB/s, done.
Total 5 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/confpullexample.git
 * [new branch]      master -> master

With respect to this example, two different nginx configuration sets will exist, one for reverse proxies (role reverseproxy) and one for standalone webservers (role webserver). A standalone webserver will also install a default database configuration file, which will be customized for clustered postgresql instances (roles dbcluster-master and dbcluster-replicas).

Furthermore, nodes part of the same cluster will have hostnames the likes of toontowndc-node03 so that branches ending with _toontowndc-node will match and have configuration files with the addresses of nodes in the same cluster deployed.

The branch net_net-snmp_5.7.3_5.2_6_any will work on all systems you want to monitor via snmpd independently of their defined role, provided the version of net-snmp being installed is >= 5.2 and < 6

In order to keep the tutorial short, I will only show how to deploy configuration for postgresql.

I'll start with a generic configuration file, for the role webserver; the branch databases_postgresql10_10.4nb1_10.0_11_default may be created to contain default configuration from the package (the role isn't any, so it will not be accidentally selected on systems that define a role), then branch it into the specialized roles webserver, dbcluster-master, dbcluster-deplicas.

A master-node address specific to each cluster site will be included in recovery.conf by specializing the branch ...dbcluster-replicas to branches specific for earch location, e.g., dbcluster-replicas_toontowndc that will be deployed on db nodes in the "Toontown" site.

NOTE that files from the branch will get copied unconditionally on the system, replacing existing files or files coming with the binary package, even if they are not installed to location/handled by the +FILES script

This means that it's possible to write configuration files to ${pgsql_home}/data, by default /usr/pkg/pgsql/data/, where the rc.d script usually sets postgresql to look into.

$ pwd
/home/devuser
$ mkdir -p sim/usr/pkg/pgsql/data
$ cd sim
$ ls -a
.    ..   .git etc  usr
$ git checkout -b databases_postgresql10_10.4nb1_10.0_11_default
Switched to a new branch 'databases_postgresql10_10.4nb1_10.0_11_default'
$ ls
etc usr
$ cp /home/devuser/postgresql.conf.sample usr/pkg/pgsql/data/postgresql.conf                                  
$ cp /home/devuser/pg_hba.conf.sample usr/pkg/pgsql/data/pg_hba.conf                                          
$ vi usr/pkg/pgsql/data/pg_hba.conf 
$ tail -n 23 usr/pkg/pgsql/data/pg_hba.conf   

# Put your actual configuration here                                                                          
# ----------------------------------                                                                          
#                                                                                                             
# If you want to allow non-local connections, you need to add more                                            
# "host" records.  In that case you will also need to make PostgreSQL                                         
# listen on a non-local interface via the listen_addresses                                                    
# configuration parameter, or via the -i or -h command line switches.                                         
                                                                                                              
@authcomment@                                                                                                 
                                                                                                              
# TYPE  DATABASE        USER            ADDRESS                 METHOD                                        
                                                                                                              
@remove-line-for-nolocal@# "local" is for Unix domain socket connections only                                 
@remove-line-for-nolocal@local   all             all                                    ident                 
# IPv4 local connections:                                                                                     
host    all             all             127.0.0.1/32           md5                                            
# IPv6 local connections:                                                                                     
host    all             all             ::1/128                md5                                            
# Allow replication connections from localhost, by a user with the                                            
# replication privilege.                                                                                      
@remove-line-for-nolocal@local   replication     all                                    ident                 
#host    replication     all             127.0.0.1/32            md5                                           
#host    replication     all             ::1/128                 md5   

$ echo "log_destination = 'syslog'" >> usr/pkg/pgsql/data/postgresql.conf                                     
$ echo "syslog_ident = 'postgres'" >> usr/pkg/pgsql/data/postgresql.conf   
$ git add usr/pkg/pgsql/*                                                                                     
$ git commit -m "import common config for not to be deployed default role"                                    
[databases_postgresql10_10.4nb1_10.0_11_default 1ce080d] import common config for not to be deployed default e
 2 files changed, 749 insertions(+)                                                                           
 create mode 100644 usr/pkg/pgsql/data/pg_hba.conf                                                            
 create mode 100644 usr/pkg/pgsql/data/postgresql.conf                                                        
$ git push --set-upstream origin databases_postgresql10_10.4nb1_10.0_11_default                               
Counting objects: 8, done.                                                                                    
Compressing objects: 100% (5/5), done.                                                                        
Writing objects: 100% (8/8), 9.10 KiB | 4.55 MiB/s, done.                                                     
Total 8 (delta 0), reused 0 (delta 0)                                                                         
To ssh://192.168.100.112/usr/home/vcs/confpullexample.git                                                     
 * [new branch]      databases_postgresql10_10.4nb1_10.0_11_default -> databases_postgresql10_10.4nb1_10.0_11_default
Branch 'databases_postgresql10_10.4nb1_10.0_11_default' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_default'

the branch with role webserver will get deployed, and doesn't differ in configuration from the default branch:

$ git checkout -b databases_postgresql10-server_10.4nb1_10.0_11_webserver                                            
Switched to a new branch 'databases_postgresql10-server_10.4nb1_10.0_11_webserver'                                   
$ git push --set-upstream origin databases_postgresql10_10.4nb1_10.0_11_webserver
Total 0 (delta 0), reused 0 (delta 0)                                                                         
To ssh://192.168.100.112/usr/home/vcs/confpullexample.git                                                     
 * [new branch]      databases_postgresql10_10.4nb1_10.0_11_webserver -> databases_postgresql10_10.4nb1_10.0_11_webserver
Branch 'databases_postgresql10_10.4nb1_10.0_11_webserver' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_webserver'

now change it for the role dbcluster-master:

$ git checkout -b databases_postgresql10_10.4nb1_10.0_11_dbcluster-master
M       usr/pkg/pgsql/data/pg_hba.conf
M       usr/pkg/pgsql/data/postgresql.conf
Switched to a new branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-master'
$ vi usr/pkg/pgsql/data/pg_hba.conf 
$ tail -n 23 usr/pkg/pgsql/data/pg_hba.conf   

# Put your actual configuration here                                                                          
# ----------------------------------                                                                          
#                                                                                                             
# If you want to allow non-local connections, you need to add more                                            
# "host" records.  In that case you will also need to make PostgreSQL                                         
# listen on a non-local interface via the listen_addresses                                                    
# configuration parameter, or via the -i or -h command line switches.                                         
                                                                                                              
@authcomment@                                                                                                 
                                                                                                              
# TYPE  DATABASE        USER            ADDRESS                 METHOD                                        
                                                                                                              
@remove-line-for-nolocal@# "local" is for Unix domain socket connections only                                 
@remove-line-for-nolocal@local   all             all                                    ident                 
# IPv4 local connections:                                                                                     
host    all             all             127.0.0.1/32           md5                                            
# IPv6 local connections:                                                                                     
host    all             all             ::1/128                md5                                            
# Allow replication connections from localhost, by a user with the                                            
# replication privilege.                                                                                      
@remove-line-for-nolocal@local   replication     all                                    ident                 
host    replication     all             127.0.0.1/32            trust                                         
host    replication     all             ::1/128                 trust                                         

$ cat << EOF >> usr/pkg/pgsql/data/postgresql.conf 
> listen_addresses = '*'                                                                                      
> wal_level = hot_standby                                                                                     
> max_wal_senders = 10                                                                                        
> hot_standby = on                                                                                            
> archive_mode = on                                                                                           
> archive_command = 'cp %p /usr/pkg/pgsql/archive/%f'                                                         
> EOF

No automation exists to run mkdir other than creating a placeholder file. you will have to chown the archive dir! File permissions are not yet handled. This may change in the future, by reusing the +DIRS script with new input for example, if deemed necessary by the community.

$ mkdir -p usr/pkg/pgsql/archive/
$ echo "remember to chown this dir to the user pgsql runs as" > usr/pkg/pgsql/archive/placeholder
$ git add usr/pkg/pgsql/archive/placeholder 
$ git commit -m "create dbcluster-master role for postgresql10"
[databases_postgresql10_10.4nb1_10.0_11_dbcluster-master 98fe514] create dbcluster-master role for postgresql0
 3 files changed, 9 insertions(+), 2 deletions(-)
 create mode 100644 usr/pkg/pgsql/archive/placeholder
$ git push -u origin databases_postgresql10_10.4nb1_10.0_11_dbcluster-master
Counting objects: 10, done.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (10/10), 827 bytes | 827.00 KiB/s, done.
Total 10 (delta 2), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/confpullexample.git
 * [new branch]      databases_postgresql10_10.4nb1_10.0_11_dbcluster-master -> databases_postgresql10_10.4nb1_10.0_11_dbcluster-master
Branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-master' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-master'

create a generic node/replica configuration from the default config files and add the recovery.conf file...

$ git checkout databases_postgresql10_10.4nb1_10.0_11_default         
Switched to branch 'databases_postgresql10_10.4nb1_10.0_11_default'
Your branch is up to date with 'origin/databases_postgresql10_10.4nb1_10.0_11_default'.

$ mkdir -p usr/pkg/pgsql/archive/
$ echo "remember to chown this dir to the user pgsql runs as" > usr/pkg/pgsql/archive/placeholder
$ cp /home/devuser/recovery.conf.sample usr/pkg/pgsql/data/recovery.conf
$ echo "standby_mode = 'on'" >> usr/pkg/pgsql/data/recovery.conf 
$ echo "restore_command = 'cp /usr/pkg/pgsql/archive/%f \"%p%\"'" >> usr/
$ git checkout -b databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas
A       usr/pkg/pgsql/archive/placeholder
A       usr/pkg/pgsql/data/recovery.conf
Switched to a new branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas'
$ git add usr/pkg/pgsql/archive/placeholder 
$ git add usr/pkg/pgsql/data/recovery.conf 
$ git add usr/pkg/pgsql/data/postgresql.conf 
$ git add usr/pkg/pgsql/data/pg_hba.conf     
$ git commit -m "create generalized dbcluster-replica role schema"
[databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas 9260a88] create generalized dbcluster-replica role schema
 2 files changed, 161 insertions(+)
 create mode 100644 usr/pkg/pgsql/archive/placeholder
 create mode 100644 usr/pkg/pgsql/data/recovery.conf
$ git push -u origin databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas
Counting objects: 9, done.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (9/9), 2.71 KiB | 2.71 MiB/s, done.
Total 9 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/confpullexample.git
 * [new branch]      databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas -> databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas
Branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas'.

Now specialize the schema for the cluster at @ToonTown by specifying part of the hostname the machines will have in that cluster:

$ git checkout -b databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node
Switched to a new branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node'
$ echo "primary_conninfo = 'host=10.0.10.200 port=5432 user=replicauser application_name=toons'" >> usr/pkg/pgsql/data/recovery.conf
$ git add usr/pkg/pgsql/data/recovery.conf 
$ git commit -m "create specialized branch for dbcluster at ToonTown dc"
[databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node 90d9731] create specialized branch for dbcluster at ToonDown dc
 1 file changed, 1 insertion(+)
$ git push -u origin databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node
Counting objects: 7, done.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (7/7), 604 bytes | 604.00 KiB/s, done.
Total 7 (delta 2), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/confpullexample.git
 * [new branch]      databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node -> databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node
Branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node'.

Let's try to deploy it on a node!

pkghost# whoami
root
pkghost# hostname
pkghost
pkghost# cat /usr/pkg/etc/pkg_install.conf 
VCSTRACK_CONF=yes
VCSCONFPULL=yes
REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/confpullexample.git
VCS=git
pkghost# echo "ROLE=dbcluster-replicas" >> /usr/pkg/etc/pkg_install.conf 
pkghost# hostname toontowndc-node05
toontowndc-node05# #show use with pkg_add and a local package, for once
toontowndc-node05# /usr/pkg/sbin/pkg_add ./postgresql10-10.3.tgz
Trying to deploy configuration from ssh://vcs@192.168.100.112/usr/home/vcs/confpullexample.git via git
About to use remote branch databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node
Cloning into 'confpullexample'...
remote: Enumerating objects: 45, done.
remote: Counting objects: 100% (45/45), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 45 (delta 5), reused 0 (delta 0)
Receiving objects: 100% (45/45), 15.23 KiB | 12.00 KiB/s, done.
Resolving deltas: 100% (5/5), done.
/tmp/pkgsrcdeploy-17103/work/etc/ssh/sshd_config -> /etc/ssh/sshd_config
/tmp/pkgsrcdeploy-17103/work/usr/pkg/pgsql/archive/placeholder -> /usr/pkg/pgsql/archive/placeholder
/tmp/pkgsrcdeploy-17103/work/usr/pkg/pgsql/data/pg_hba.conf -> /usr/pkg/pgsql/data/pg_hba.conf
/tmp/pkgsrcdeploy-17103/work/usr/pkg/pgsql/data/postgresql.conf -> /usr/pkg/pgsql/data/postgresql.conf
/tmp/pkgsrcdeploy-17103/work/usr/pkg/pgsql/data/recovery.conf -> /usr/pkg/pgsql/data/recovery.conf

toontowndc-node05# tail /etc/ssh/sshd_config 

# Example of overriding settings on a per-user basis
#Match User anoncvs
#       X11Forwarding no
#       AllowTcpForwarding no
#       PermitTTY no
#       ForceCommand cvs server
PermitEmptyPasswords no
PermitRootLogin no
AllowGroup ops

toontowndc-node05# tail /usr/pkg/pgsql/data/recovery.conf
#---------------------------------------------------------------------------
# HOT STANDBY PARAMETERS
#---------------------------------------------------------------------------
#
# Hot Standby related parameters are listed in postgresql.conf
#
#---------------------------------------------------------------------------
standby_mode = 'on'
restore_command = 'cp /usr/pkg/pgsql/archive/%f "%p%"'
primary_conninfo = 'host=10.0.10.200 port=5432 user=replicauser application_name=toons'

toontowndc-node05# ls /tmp/pkgsrcdeploy-17103
ls: /tmp/pkgsrcdeploy-17103: No such file or directory

The same way goes if you should choose to set VCS=hg or VCS=svn in pkg_install.conf. All are supported when pulling configuration, you should adapt to managing branches the way these other VCSs expect you to work with them. It might be in scope for this tutorial, but it's getting a bit long! Also remember to set an appropriate URI in REMOTEVCS for these other Version Control Systems.

CVS is not supported: tags (branch names) cannot contain dots with CVS, and this breaks the chosen naming scheme for many software applications. It could be replaced with another special meaning character when using this vcs, but this workaround would make for possible confusion in branch naming. Furthermore, as with svn, there is no way to list remote branches before getting a local copy of the repository, with the added headcache of having to parse its log to extract branch names. I considered it not to be worth the hassle, but I'm open to adding PULL mode support for cvs if required by any user.

One word about subversion: I expect you to follow conventions and keep branches at /branches/ in your repository!

pkgconftrack store: storing changes when not upgrading packages

Even when merging changes in configuration files is not attempted because of VCSAUTOMERGE=no or unset, pkgsrc will keep track of installed configuration files by committing them to the configuration repository, as a user-modified file under user/$filePath (see mk/pkginstall/files). No VCS will commit files that haven't changed since the last revision (I hope!).

But what if you made changes to a configuration file and you want to store it in the configuration repository as a manually edited file, neither waiting for its package to be updated nor forcing a reinstallation? You could interact with your vcs of choice direcly, or just use the new script in pkgsrc at pkgtools/pkgconftrack

pkgconf# cd pkgtools/pkgconftrack/
pkgconf# make install
===> Skipping vulnerability checks.
[...]
===> Installing binary package of pkgconftrack-1.0
pkgconf# pkgconftrack
prefix: /usr/pkg, VCSDIR: /var/confrepo, VCS: svn, REMOTEVCS: no
: unknown action
Usage: pkgconftrack [-p PREFIX] [-m commit message] store packagename [... packagenames]

pkgconftrack will search for pkgsrc VCS configuration on your shell environment or in pkg_install.conf, reading it via pkg_admin config-var $VARNAME.

pkg_admin, in order to work with the correct configuration file, and look in the right package database to check if a package exists and its list of configuration files, is called relative to the prefix. By default, if unspecified, /usr/pkg is assumed and /usr/pkg/sbin/pkg_admin is executed.

You can work in packages of a different prefix by calling pkgconftrack -p /path/to/other/prefix followed by other options (you are free to chose a custom commit message with -m "my commit message") and the action to be performed.

As of now, pkgconftrack only support one action: store followed by one or more packages you wish to store configuration files from, into the configuration repository.

When storing configuration for more than one package, a unique commit is made (if using a VCS other than the old RCS, which doesn't support multi file commits and atomic transactions). This opens the way to storing config files for a service made working by a combination of software packages when these are in a known-good status, to be accessed and restored checking out one commit id.

So, let's say one last change was needed to have mail working:

pkgconf# diff /usr/pkg/etc/spamd.conf /usr/pkg/share/examples/spamd/spamd.conf 
7a8
> #
33c34
<       :method=https:\
---
>       :method=http:\
pkgconf# pkgconftrack -m "5 august 2018: mail service: blacklist only accessible via https" store spamd postfix dovecot opendkim
prefix: /usr/pkg, VCSDIR: /var/confrepo, VCS: svn, REMOTEVCS: no
Storing configuration files for opendkim
A         opendkim.conf
Package not found: dovecot in the pkgdb for /usr/pkg
Package not found: postfix in the pkgdb for /usr/pkg
Storing configuration files for spamd
Adding         usr/pkg/etc/opendkim.conf
Transmitting file data .done
Committing transaction...
Committed revision 7.
pkgconf# 

Yeah, I haven't really installed a mail server just to test pkgconftrack, so neither dovecot nor postfix are installed on the system!

future improvements

Well, it all begins with the new features being tested by end users, more bugs being found, changes made as requested. Then, once things get more stable, the versioning script could be reimplemented as part of pkgtasks, and files.subr changed there to interact with the new functions.

More VCSs could be handled, there could be more automation in switching repositories and adding remote sources, but all this is maybe best kept in a separate tool such as pkgconftrack. Speaking of which, well, it stores installed config files for one or more packages at once, but it does not restore them yet! RCS needs to be supported, being the default Version Control System, but it has no concept of an atomic transaction involving more than one file, so there would be nothing the user could reference to when asking the script to restore configuration. The script could track, by checking the log for all files, for identical commit message and restore each revision of each file having the same commit message, but what if some user mistakingly reused the same commit message as part of different transactions? Should each custom commit message be prepended with a timestamp?

All this would still lead to differences in referencing to a transaction, or the simulation thereof, and in listing available changes the user can select for restore.

pkgconftrack could also help in interactively reviewing conflicting automerge results, some scripts already do it, and in restoring the last user-installed file from the repository in case of breakage. This should really consist in a cp from vcsdir/user/path/to/file /path/to/installed/file, maybe preceded by a checkout, but there is good marging for making the tool more useful.

And yes, configuraction deployment/VCSCONFPULL does not handle permissions or the creation of empty directories, executable files: I think this would require making the way users interact with the configuration repository more complex, more akin to a configuration management and monitoring software, and widen the chances for mistakes when working across branches. Any good idea of how to implement these missing bits while still keeping things simple for users?

I'd really like to see the code tried out! it's now at Github

. [0 comments]

 

GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 3: remote repositories (SVN and Mercurial)

In GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 2: remote repositories (git and CVS) structural changes to the configuration tracking feature of pkgsrc got introduced, along with a short walkthrough in using remote Git and CVS repositories.

This third post will introduce SVN and Mercurial support, leaving configuration deployment for the final entry. Stay tuned!

SVN and remotes

Subversion, when used locally, needs to be set this way, as you can expect:

pkghost# cat /usr/pkg/etc/pkg_install.conf                                                                    
VCSTRACK_CONF=yes                                                                                            
VCS=svn                                                                                                       
VCSDIR=/var/svnconfdir                                                                                        
VCSAUTOMERGE=yes 

A local repository will be created at VCSDIR/localsvn, files will get extracted and checkedin at VCSDIR/defaults, VCSDIR/automerged, VCSDIR/user

pkghost# make   

[...]
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.                      
===> Installing for spamd-20060330nb5                                                                         
=> Generating pre-install file lists                                                                          
=> Creating installation directories                                                                          
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/man/spamd.conf.5  /ro5
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd.8  /root/8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-set8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb.8  /roo8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd.8  8
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-c
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd  /rootc
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb  /ron
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd c
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/etc/spamd.conf /root/d
=> Automatic manual page handling                                                                             
=> Generating post-install file lists                                                                         
=> Checking file-check results for spamd-20060330nb5                                                          
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb5.tgz                       
===> Building binary package for spamd-20060330nb5                                                            
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb5.tgz                                    
===> Installing binary package of spamd-20060330nb5                                                           
REGISTER /var/svnconfdir/defaults//usr/pkg/etc/spamd.conf                                                     
spamd-20060330nb5: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf                
Conf commit: pkgsrc: add spamd-20060330nb5                                                                    
===========================================================================                                   
The following files should be created for spamd-20060330nb5:                                                  
                                                                                                              
        /etc/rc.d/pfspamd (m=0755)                                                                            
            [/usr/pkg/share/examples/rc.d/pfspamd]                                                            
                                                                                                              
===========================================================================                                   
===========================================================================                                   
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $                                                    
                                                                                                              
Don't forget to add the spamd ports to /etc/services:                                                         
                                                                                                              
spamd           8025/tcp                # spamd(8)                                                            
spamd-cfg       8026/tcp                # spamd(8) configuration                                              
                                                                                                              
===========================================================================                                   

pkghost# ls -lah /var/svnconfdir/localsvn/                                                                    
total 3.2K                                                                                                    
drwxr-xr-x  6 pkgvcsconf  pkgvcsconf  512B Aug  4 07:22 .                                                     
drwx------  6 pkgvcsconf  pkgvcsconf  512B Aug  4 07:22 ..                                                    
-rw-r--r--  1 pkgvcsconf  pkgvcsconf  246B Aug  4 07:22 README.txt                                            
drwxr-xr-x  2 pkgvcsconf  pkgvcsconf  512B Aug  4 07:22 conf                                                  
drwxr-sr-x  6 pkgvcsconf  pkgvcsconf  512B Aug  4 07:22 db                                                    
-r--r--r--  1 pkgvcsconf  pkgvcsconf    2B Aug  4 07:22 format                                                
drwxr-xr-x  2 pkgvcsconf  pkgvcsconf  512B Aug  4 07:22 hooks                                                 
drwxr-xr-x  2 pkgvcsconf  pkgvcsconf  512B Aug  4 07:22 locks

pkghost# vi /usr/local/etc/spamd.conf
whitelist:\ 
        :white:\
        :method=file:\
        :file=/var/mail/whitelist.txt:                                                                        
/usr/pkg/etc/spamd.conf: 86 lines, 2767 characters                                                            
                                                  .                                                           
pkghost# make replace
[...]

and so on, you can come up with what follows. Let's test a remote repository, starting with preparations on the server:

$ hostname                                                                                                    
vers                                                                                                          
$ cd /usr/home/vcs   
$ svnadmin create svnremote

That's it, no branches or releases are used, so you don't need to create any further structure on the repository, the +VERSIONING script will take care of the remaining bits (you should migrate data from a local repository to the remote one, if you desire so, before running package installations).

Use a similarly structured REMOTEVCS URI: svn+ssh://vcs@192.168.100.112/usr/home/vcs/svnremote

(as with CVS, you can also run it as a standalone network server without ssh transport and authentication)

pkghost# cat /usr/pkg/etc/pkg_install.conf                                                                    
VCSTRACK_CONF=yes                                                                                            
VCS=svn                                                                                                       
VCSDIR=/var/svnwrkremote                                                                                      
VCSAUTOMERGE=yes                                                                                              
REMOTEVCS=svn+ssh://vcs@192.168.100.112/usr/home/vcs/svnremote   
pkghost# make                                                                                                 

[...]

pkghost# make install                                                                                         
=> Bootstrap dependency digest>=20010302: found digest-20160304                                               
===> Skipping vulnerability checks.                                                                           
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.                                                       
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.                      
===> Installing for spamd-20060330nb6                                                                         
=> Generating pre-install file lists                                                                          
=> Creating installation directories                                                                          
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/man/spamd.conf.5  /ro5
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd.8  /root/8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-set8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb.8  /roo8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd.8  8
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-c
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd  /rootc
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb  /ron
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd c
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/etc/spamd.conf /root/d
=> Automatic manual page handling                                                                             
=> Generating post-install file lists                                                                         
=> Checking file-check results for spamd-20060330nb6                                                          
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb6.tgz                       
===> Building binary package for spamd-20060330nb6                                                            
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb6.tgz                                    
===> Installing binary package of spamd-20060330nb6                                                           
svn: E155007: '/var/svnwrkremote/defaults' is not a working copy                                              
Committing transaction...                                                                                     
Committed revision 1.                                                                                         
Checked out revision 1.                                                                                       
Committing transaction...                                                                                     
Committed revision 2.                                                                                         
Checked out revision 2.                                                                                       
Committing transaction...                                                                                     
Committed revision 3.                                                                                         
Checked out revision 3.                                                                                       
A         usr                                                                                                 
A         pkg                                                                                                 
A         etc                                                                                                 
A         spamd.conf                                                                                          
REGISTER /var/svnwrkremote/defaults//usr/pkg/etc/spamd.conf                                                   
spamd-20060330nb6: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf                
Adding         usr                                                                                            
Adding         usr/pkg                                                                                        
Adding         usr/pkg/etc                                                                                    
Adding         usr/pkg/etc/spamd.conf                                                                         
Transmitting file data .done                                                                                  
Committing transaction...                                                                                     
Committed revision 4.                                                                                         
Conf commit: pkgsrc: add spamd-20060330nb6                                                                    
===========================================================================                                   
The following files should be created for spamd-20060330nb6:                                                  
                                                                                                              
        /etc/rc.d/pfspamd (m=0755)                                                                            
            [/usr/pkg/share/examples/rc.d/pfspamd]                                                            
                                                                                                              
===========================================================================                                   
===========================================================================                                   
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $                                                    
                                                                                                              
Don't forget to add the spamd ports to /etc/services:                                                         
                                                                                                              
spamd           8025/tcp                # spamd(8)                                                            
spamd-cfg       8026/tcp                # spamd(8) configuration                                              
                                                                                                              
===========================================================================                                   
pkghost# 

since I didn't show it with CVS, let's try again some simulated file edits:

pkghost# tail -n 5 /usr/pkg/etc/spamd.conf                                                                    
#                                                                                                             
#whitelist:\                                                                                                  
#       :white:\                                                                                              
#       :method=file:\                                                                                        
#       :file=/var/mail/whitelist.txt:                                                                        
pkghost# vi /usr/pkg/etc/spamd.conf; tail -n 5 /usr/pkg/etc/spamd.conf 
/usr/pkg/etc/spamd.conf: 86 lines, 2767 characters                                                            
                                                  .                                                           
#                                                                                                             
whitelist:\                                                                                                   
        :white:\                                                                                              
        :method=file:\                                                                                        
        :file=/var/mail/whitelist.txt:  
=> Bootstrap dependency digest>=20010302: found digest-20160304                                               
===> Skipping vulnerability checks.                                                                           
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.                                                       
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.                      
===> Replacing for spamd-20060330nb6                                                                          
===> Updating using binary package of spamd-20060330nb6                                                       
/usr/bin/env  /usr/pkg/sbin/pkg_add -K /var/db/pkg -U -D /root/pkgsrc/mail/spamd/work/.packages/spamd-2006033z
===========================================================================                                   
The following users are no longer being used by spamd-20060330nb6,                                            
and they can be removed if no other software is using them:                                                   
                                                                                                              
        _spamd                                                                                                
                                                                                                              
===========================================================================                                   
===========================================================================                                   
The following groups are no longer being used by spamd-20060330nb6,                                           
and they can be removed if no other software is using them:                                                   
                                                                                                              
        _spamd                                                                                                
                                                                                                              
===========================================================================                                   
===========================================================================                                   
The following files are no longer being used by spamd-20060330nb6,                                            
and they can be removed if no other packages are using them:                                                  
                                                                                                              
        /usr/pkg/etc/spamd.conf                                                                               
                                                                                                              
===========================================================================                                   
REGISTER /var/svcwrkremote/defaults//usr/pkg/etc/spamd.conf                                                   
spamd-20060330nb6: /usr/pkg/etc/spamd.conf already exists                                                     
spamd-20060330nb6: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!                             
Saving the currently user-installed revision to /var/svnwrkremote/user//usr/pkg/etc/spamd.conf                
A         usr                                                                                                 
A         pkg                                                                                                 
A         etc                                                                                                 
A         spamd.conf                                                                                          
Adding         usr                                                                                            
Adding         usr/pkg                                                                                        
Adding         usr/pkg/etc                                                                                    
Adding         usr/pkg/etc/spamd.conf                                                                         
Transmitting file data .done                                                                                  
Committing transaction...                                                                                     
Committed revision 5.                                                                                         
Conf commit: pkgsrc: backup user conf before attempting merge for spamd-20060330nb6                           
A    defaults/usr/pkg/etc/spamd.conf                                                                          
Export complete.                                                                                              
Merged with no conflicts. installing it to /usr/pkg/etc/spamd.conf!                                           
A         usr                                                                                                 
A         pkg                                                                                                 
A         etc                                                                                                 
A         spamd.conf                                                                                          
A    defaults/usr/pkg/etc/spamd.conf                                                                          
Export complete.                                                                                              
Revert from the last revision of /var/svcwrkremote/user//usr/pkg/etc/spamd.conf if needed                     
Adding         usr                                                                                            
Adding         usr/pkg                                                                                        
Adding         usr/pkg/etc                                                                                    
Adding         usr/pkg/etc/spamd.conf                                                                         
Transmitting file data .done                                                                                  
Committing transaction...                                                                                     
Committed revision 6.                                                                                         
Conf commit: pkgsrc: add spamd-20060330nb6                                                                    
===========================================================================                                   
The following files should be created for spamd-20060330nb6:                                                  
                                                                                                              
        /etc/rc.d/pfspamd (m=0755)                                                                            
            [/usr/pkg/share/examples/rc.d/pfspamd]                                                            
                                                                                                              
===========================================================================                                   
===========================================================================                                   
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $                                                    
                                                                                                              
Don't forget to add the spamd ports to /etc/services:                                                         
                                                                                                              
spamd           8025/tcp                # spamd(8)                                                            
spamd-cfg       8026/tcp                # spamd(8) configuration                                              
                                                                                                              
===========================================================================  
pkghost# cd /var/svnwrkremote/                                                                                
pkghost# ls                                                                                                   
automerged      automergedfiles defaults        user    
pkghost# cd automerged  
pkghost# svn update                                                                                           
Updating '.':                                                                                                 
At revision 6.                                                                                                
pkghost# svn log                                                                                              
------------------------------------------------------------------------                                      
r6 | vcs | 2018-08-04 13:22:29 +0000 (Sat, 04 Aug 2018) | 1 line                                              
                                                                                                              
pkgsrc: add spamd-20060330nb6                                                                                 
------------------------------------------------------------------------                                      
r1 | vcs | 2018-08-04 13:19:29 +0000 (Sat, 04 Aug 2018) | 1 line                                              
                                                                                                              
initial import                                                                                                
------------------------------------------------------------------------ 

pkghost# cd $HOME/pkgsrc/mail/spamd                                                                           
pkghost# grep REVISION Makefile                                                                               
PKGREVISION=            6                                                                                     
pkghost# sed -i "s/PKGREVISION=.*6/PKGREVISION=7/g" Makefile                                                  
pkghost# grep REVISION Makefile                                                                               
PKGREVISION=7   
pkghost# make clean; make extract; sed -i "s/http/https/g" work/spamd-20060330/etc/spamd.conf
pkghost# make update
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Deinstalling for spamd-20060330nb7
Running /usr/pkg/sbin/pkg_delete -K /var/db/pkg -r spamd-20060330nb6
===========================================================================

[...]

===> Installing for spamd-20060330nb7                                                                         
=> Generating pre-install file lists                                                                          
=> Creating installation directories                                                                          
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/man/spamd.conf.5  /ro5
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd.8  /root/8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-set8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb.8  /roo8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd.8  8
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-c
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd  /rootc
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb  /ron
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd c
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/etc/spamd.conf /root/d
=> Automatic manual page handling                                                                             
=> Generating post-install file lists                                                                         
=> Checking file-check results for spamd-20060330nb7                                                          
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb7.tgz                       
===> Building binary package for spamd-20060330nb7                                                            
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb7.tgz                                    
===> Installing binary package of spamd-20060330nb7                                                           
REGISTER /var/svcwrkremote/defaults//usr/pkg/etc/spamd.conf                                                   
spamd-20060330nb7: /usr/pkg/etc/spamd.conf already exists                                                     
spamd-20060330nb7: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!                             
A    automerged/usr/pkg/etc/spamd.conf                                                                        
Export complete.                                                                                              
Saving the currently installed revision to /var/svcwrkremote/automerged//usr/pkg/etc/spamd.conf               
Sending        usr/pkg/etc/spamd.conf                                                                         
Transmitting file data .done                                                                                  
Committing transaction...                                                                                     
Committed revision 7.                                                                                         
Conf commit: pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb7                    
A    defaults/usr/pkg/etc/spamd.conf                                                                          
Export complete.                                                                                              
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!                                            
--- /usr/pkg/etc/spamd.conf     2018-08-04 09:01:09.536149671 +0000                                           
+++ /var/svcwrkremote/defaults//usr/pkg/etc/spamd.conf.automerge        2018-08-04 09:10:52.131955252 +0000   
@@ -15,7 +15,7 @@                                                                                             
 # may be applied to each blacklist.                                                                          
 #                                                                                                            
 # As of November 2004, a place to search for black lists is                                                  
-#     http://spamlinks.net/filter-bl.htm                                                                     
+#     https://spamlinks.net/filter-bl.htm                                                                    
 #                                                                                                            
 # Some of the URLs below point to www.openbsd.org locations.  Those                                          
 # files are likely to be mirrored to other OpenBSD www mirrors located                                       
@@ -25,45 +25,45 @@                                                                                           
 all:\                                                                                                        
        :spews1:china:korea:                                                                                  
                                                                                                              
-# Mirrored from http://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2                                            
+# Mirrored from https://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2                                           
 spamhaus:\                                                                                                   
        :black:\                                                                                              
        :msg="SPAM. Your address %A is in the Spamhaus Block List\n\                                          
-       See http://www.spamhaus.org/sbl and\                                                                  
-       http://www.abuse.net/sbl.phtml?IP=%A for more details":\                                              
-       :method=http:\                                                                                        
+       See https://www.spamhaus.org/sbl and\                                                                 
+       https://www.abuse.net/sbl.phtml?IP=%A for more details":\                                             
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/SBL.cidr.gz:                                                              
                                                                                                              
-# Mirrored from http://www.spews.org/spews_list_level1.txt                                                   
+# Mirrored from https://www.spews.org/spews_list_level1.txt                                                  
 spews1:\                                                                                                     
        :black:\                                                                                              
        :msg="SPAM. Your address %A is in the spews level 1 database\n\                                       
-       See http://www.spews.org/ask.cgi?x=%A for more details":\                                             
-       :method=http:\                                                                                        
+       See https://www.spews.org/ask.cgi?x=%A for more details":\                                            
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/spews_list_level1.txt.gz:                                                 
                                                                                                              
-# Mirrored from http://www.spews.org/spews_list_level2.txt                                                   
+# Mirrored from https://www.spews.org/spews_list_level2.txt                                                  
 spews2:\                                                                                                     
        :black:\                                                                                              
        :msg="SPAM. Your address %A is in the spews level 2 database\n\                                       
-       See http://www.spews.org/ask.cgi?x=%A for more details":\                                             
-       :method=http:\                                                                                        
+       See https://www.spews.org/ask.cgi?x=%A for more details":\                                            
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/spews_list_level2.txt.gz:                                                 
                                                                                                              
-# Mirrored from http://www.okean.com/chinacidr.txt                                                           
+# Mirrored from https://www.okean.com/chinacidr.txt                                                          
 china:\                                                                                                      
        :black:\                                                                                              
        :msg="SPAM. Your address %A appears to be from China\n\                                               
-       See http://www.okean.com/asianspamblocks.html for more details":\                                     
-       :method=http:\                                                                                        
+       See https://www.okean.com/asianspamblocks.html for more details":\                                    
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/chinacidr.txt.gz:                                                         
                                                                                                              
-# Mirrored from http://www.okean.com/koreacidr.txt                                                           
+# Mirrored from https://www.okean.com/koreacidr.txt                                                          
 korea:\                                                                                                      
        :black:\                                                                                              
        :msg="SPAM. Your address %A appears to be from Korea\n\                                               
-       See http://www.okean.com/asianspamblocks.html for more details":\                                     
-       :method=http:\                                                                                        
+       See https://www.okean.com/asianspamblocks.html for more details":\                                    
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/koreacidr.txt.gz:                                                         
                                                                                                              
 #relaydb-black:\                                                                                             
Revert from the penultimate revision of /var/svcwrkremote/automerged//usr/pkg/etc/spamd.conf if needed        
Sending        usr/pkg/etc/spamd.conf                                                                         
Transmitting file data .done                                                                                  
Committing transaction...                                                                                     
Committed revision 8.                                                                                         
Sending        usr/pkg/etc/spamd.conf                                                                         
Transmitting file data .done                                                                                  
Committing transaction...                                                                                     
Committed revision 9.                                                                                         
Conf commit: pkgsrc: add spamd-20060330nb7                                                                    
===========================================================================                                   
The following files should be created for spamd-20060330nb7:                                                  
                                                                                                              
        /etc/rc.d/pfspamd (m=0755)                                                                            
            [/usr/pkg/share/examples/rc.d/pfspamd]                                                            
                                                                                                              
===========================================================================                                   
[...]
===> Cleaning for spamd-20060330nb7                                                                           

HG and remotes

Yes, mercurial. Mercurial, like git, uses a local directory (namely .hg) located under the working directory VCSDIR, it is also used to store configuration and URIs to remote repositories. to initialize and test it (as said, no repository and vcs migrations are supported by pkgsrc itself, you should take care of migrations yourself if you want to), just set pkg_install.conf to use a local mercurial repo and install a package:

pkghost# rm /usr/pkg/etc/spamd.conf                                                                           
pkghost# /usr/pkg/sbin/pkg_delete spamd
pkghost# vi /usr/pkg/etc/pkg_install.conf; cat /usr/pkg/etc/pkg_install.conf
/usr/pkg/etc/pkg_install.conf: 4 lines, 66 characters                                                         
                                                     .                                                        
VCSTRACK_CONF=yes                                                                                            
VCS=hg                                                                                                        
VCSDIR=/var/hglocaldir                                                                                        
VCSAUTOMERGE=yes                                                                                              
pkghost# make                                                                                                 
=> Bootstrap dependency digest>=20010302: found digest-20160304                                               
===> Skipping vulnerability checks.                                                                           
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.                                                       
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.                      
=> Checksum SHA1 OK for spamd-20060330.tar.gz

[...]

To save some space and time, let's just say that it works as expected:

[...]
Conf commit: pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb10                   
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!                                            
--- /usr/pkg/etc/spamd.conf     2018-08-04 09:50:00.738099463 +0000                                           
+++ /var/hglocaldir/defaults//usr/pkg/etc/spamd.conf.automerge  2018-08-04 09:52:18.467604445 +0000           
@@ -15,7 +15,7 @@                                                                                             
 # may be applied to each blacklist.                                                                          
 #                                                                                                            
 # As of November 2004, a place to search for black lists is                                                  
-#     http://spamlinks.net/filter-bl.htm                                                                     
+#     https://spamlinks.net/filter-bl.htm                                                                    
 # 
 [...]
 

One nice thing about mercurial is the simplicity enabling one to clone a local repository to a remote server. The script, when using mercurial, tries exacly that, this should succeed if the remote repository is empty.

 				if ${TEST} "$_REMOTE" != "no" -a "$_REMOTE" != "NO"; then
					execute "hg clone . \"$_REMOTE\""
					execute "hg --repository \"$_VCSDIR\" push \"$_REMOTE\""
					execute "hg --repository \"$_VCSDIR\" pull \"$_REMOTE\""
                fi

let's init an empty repository on the server and let pkgsrc clone the existing files over there!

$ cd                                                                                                          
$ hostname                                                                                                    
vers                                                                                                          
$ pwd                                                                                                         
/home/vcs                                                                                                     
$ hg init pkgconftest                                                                                         
$ ls pkgconftest/                                                                                             
$ ls pkgconftest/.hg/                                                                                         
00changelog.i   requires        store                                                                         
$ 

URIs for REMOTEVCS take the following format, should you choose to use ssh instead of hg server, http or other access methods that you'll find documented on official Mercurial resources, as with svn, git and cvs:

ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest

pkghost# vi /usr/pkg/etc/pkg_install.conf; cat /usr/pkg/etc/pkg_install.conf

/usr/pkg/etc/pkg_install.conf: 5 lines, 127 characters                                                        
                                                      .                                                       
VCSTRACK_CONF=yes                                                                                            
VCS=hg                                                                                                        
VCSDIR=/var/hglocaldir                                                                                        
VCSAUTOMERGE=yes                                                                                              
REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest                                                  

pkghost# make replace  

[...]

===> Updating using binary package of spamd-20060330nb10                                                      
/usr/bin/env  /usr/pkg/sbin/pkg_add -K /var/db/pkg -U -D /root/pkgsrc/mail/spamd/work/.packages/spamd-2006033z
===========================================================================                                   
The following users are no longer being used by spamd-20060330nb10,                                           
and they can be removed if no other software is using them:                                                   
                                                                                                              
        _spamd                                                                                                
                                                                                                              
===========================================================================                                   
===========================================================================                                   
The following groups are no longer being used by spamd-20060330nb10,                                          
and they can be removed if no other software is using them:                                                   
                                                                                                              
        _spamd                                                                                                
                                                                                                              
===========================================================================                                   
===========================================================================                                   
The following files are no longer being used by spamd-20060330nb10,                                           
and they can be removed if no other packages are using them:                                                  
                                                                                                              
        /usr/pkg/etc/spamd.conf                                                                               
                                                                                                              
===========================================================================                                   
searching for changes                                                                                         
remote: adding changesets                                                                                     
remote: adding manifests                                                                                      
remote: adding file changes                                                                                   
remote: added 5 changesets with 6 changes to 3 files                                                          
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest                                                 
searching for changes                                                                                         
no changes found                                                                                              
pulling from ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest                                               
searching for changes                                                                                         
no changes found                                                                                              
defaults/usr/pkg/etc/spamd.conf already tracked!                                                              
REGISTER /var/hglocaldir/defaults//usr/pkg/etc/spamd.conf                                                     
spamd-20060330nb10: /usr/pkg/etc/spamd.conf already exists                                                    
spamd-20060330nb10: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!                            
Saving the currently installed revision to /var/hglocaldir/automerged//usr/pkg/etc/spamd.conf                 
automerged/usr/pkg/etc/spamd.conf already tracked!                                                            
Failed to commit conf: backup preexisting conf before attempting merge for spamd-20060330nb10                 
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest                                                 
searching for changes                                                                                         
no changes found                                                                                              
hg: failed to push changes to the remote repository ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest        
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!                                            
Revert from the penultimate revision of /var/hglocaldir/automerged//usr/pkg/etc/spamd.conf if needed          
Failed to commit conf: add spamd-20060330nb10                                                                 
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest                                                 
searching for changes                                                                                         
no changes found                                                                                              
hg: failed to push changes to the remote repository ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest        
===========================================================================                                   
The following files should be created for spamd-20060330nb10:                                                 
                                                                                                              
        /etc/rc.d/pfspamd (m=0755)                                                                            
            [/usr/pkg/share/examples/rc.d/pfspamd]                                                            
[...]

Yes, hg exits in error when there are no changes to be pushed. Everything is working fine.

Let's simulate some more package-provided changes in defaults

pkghost# make clean; sed -i "s/PKGREVISION=10/PKGREVISION=11/g" Makefile; make extract
pkghost# sed -i "s/http/https/g" work/spamd-20060330/etc/spamd.conf 
pkghost# vi work/spamd-20060330/etc/spamd.conf; head work/spamd-20060330/etc/spamd.conf
work/spamd-20060330/etc/spamd.conf: 87 lines, 2812 characters                                                 
                                                             .                                                
# $OpenBSD: spamd.conf,v 1.17 2006/02/01 20:22:43 dhartmei Exp $                                              
#                                                                                                             
# spamd config file, read by spamd-setup(8) for spamd(8)                                                      
#                                                                                                             
# See spamd.conf(5)                                                                                           
#THIS IS A NEW COMMENT!                                                                                       
#                                                                                                             
# Configures whitelists and blacklists for spamd                                                              
#                                                                                                             
# Strings follow getcap(3) convention escapes, other than you                                                 
pkghost# make update                                                                                          
===> Skipping vulnerability checks.                                                                           
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.                                                       
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.                      
===> Deinstalling for spamd-20060330nb11                                                                      
Running /usr/pkg/sbin/pkg_delete -K /var/db/pkg -r spamd-20060330nb10                                         
===========================================================================                                   
The following users are no longer being used by spamd-20060330nb10,                                           
and they can be removed if no other software is using them:  

[...]

===> Installing for spamd-20060330nb11                                                                        
[...]
=> Automatic manual page handling                                                                             
=> Generating post-install file lists                                                                         
=> Checking file-check results for spamd-20060330nb11                                                         
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb11.tgz                      
===> Building binary package for spamd-20060330nb11                                                           
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb11.tgz                                   
===> Installing binary package of spamd-20060330nb11                                                          
abort: repository usr/home/vcs/pkgconftest already exists!                                                    
abort: could not create remote repo!                                                                          
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest                                                 
searching for changes                                                                                         
no changes found                                                                                              
pulling from ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest                                               
searching for changes                                                                                         
no changes found                                                                                              
defaults/usr/pkg/etc/spamd.conf already tracked!                                                              
REGISTER /var/hglocaldir/defaults//usr/pkg/etc/spamd.conf                                                     
spamd-20060330nb11: /usr/pkg/etc/spamd.conf already exists                                                    
spamd-20060330nb11: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!                            
Saving the currently installed revision to /var/hglocaldir/automerged//usr/pkg/etc/spamd.conf                 
automerged/usr/pkg/etc/spamd.conf already tracked!                                                            
Conf commit: pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb11                   
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest                                                 
searching for changes                                                                                         
remote: adding changesets                                                                                     
remote: adding manifests                                                                                      
remote: adding file changes                                                                                   
remote: added 1 changesets with 1 changes to 1 files                                                          
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!                                            
--- /usr/pkg/etc/spamd.conf     2018-08-04 10:43:37.554182167 +0000                                           
+++ /var/hglocaldir/defaults//usr/pkg/etc/spamd.conf.automerge  2018-08-04 10:50:23.582806647 +0000           
@@ -3,6 +3,7 @@                                                                                               
 # spamd config file, read by spamd-setup(8) for spamd(8)                                                     
 #                                                                                                            
 # See spamd.conf(5)                                                                                          
+#THIS IS A NEW COMMENT!                                                                                      
 #                                                                                                            
 # Configures whitelists and blacklists for spamd                                                             
 #                                                                                                            
Revert from the penultimate revision of /var/hglocaldir/automerged//usr/pkg/etc/spamd.conf if needed          
Conf commit: pkgsrc: add spamd-20060330nb11                                                                   
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest                                                 
searching for changes                                                                                         
remote: adding changesets                                                                                     
remote: adding manifests                                                                                      
remote: adding file changes                                                                                   
remote: added 1 changesets with 2 changes to 2 files                                                          
===========================================================================                                   
The following files should be created for spamd-20060330nb11:                                                 
[...]

Also see GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 4: configuration deployment, pkgtools and possible improvements

[0 comments]

 

GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 2: remote repositories (git and CVS)

Prepared by Keivan Motavalli as part of GSoC 2018.

In Configuration files versioning in pkgsrc, Part 1 basic pkgsrc support for tracking package configuration changes was introduced, as were the features that attempt to automatically merge installed configuration files with new defaults, all shown using RCS as the backing Version Control System and passing options by setting environment variables.

Some of them have changed (conf tracking now needs to be explicitly enabled), support is now also in pkg_install.conf and aside from RCS, pkgsrc now supports CVS, Git, mercurial and SVN, both locally and connecting to remote resources.

Furthermore, pkgsrc is now able to deploy configuration from packages being installed from a remote, site-specific vcs repository.

User modified files are always tracked even if automerge functionality is not enabled, and a new tool, pkgconftrack(1), exists to manually store user changes made outside of package upgrade time.

Version Control software is executed as the same user running pkg_add or make install, unless the user is "root". In this case, a separate, unprivileged user, pkgvcsconf, gets created with its own home directory and a working login shell (but no password). The home directory is not strictly necessary, it exists to facilitate migrations betweens repositories and vcs changes; it also serves to store keys used to access remote repositories.

Files, objects and other data in the working directory are only accessible by UID 0 and pkgvcsconf, but be aware, if you want to login to remote repositories via ssh, keys will now need to be placed at pkgvcsconf own $HOME/.ssh/ directory, and ssh-agent, if you need to use it, also should have its socket, as defined by SSH_AUTH_SOCK, accessible by pkgvcsconf:pkgvcsconf.

New pkgsrc options

NOVCS, which was used to disable configuration tracking via Version Control Systems, doesn't exist anymore as an option or environment variable. In its place, VCSTRACK_CONF now needs to be set truthfully in order to enable configuration files version tracking. This can be done temporarily via environment variables before using pkgsrc or installing binary packages, such as by exporting

export VCSTRACK_CONF=yes

on the tty, or permanently in pkg_install.conf located under the PREFIX in use.

After bootstrapping a pkgsrc branch that supports the new features, check new options by calling, if you are using /usr/pkg as the PREFIX:

man -M /usr/pkg/man/ pkg_install.conf

Remember to edit /usr/pkg/etc/pkg_install.conf in order to set VCSTRACK_CONF=yes. Also try out VCSAUTOMERGE=yes to attempt merging installed configuration files with new defaults, as described in the first blog post. Installed configuration files are always backed up before an attempt is made to merge changes, and if conflicts are reported no changes get automatically installed in place of the existing configuration files, so that the user can manually review them.

Both VCSTRACK_CONF and VCSAUTOMERGE variables, if set on the environment, take precedence over values defined in pkg_install.conf in order to allow for an easy override at runtime. All other vcs-related options defined in pkg_install.conf replace those defined as environment variables.

These are:

VCS, sets the backend used to track configuration files. If unset, rcs gets called. pkgsrc now also supports Git, SVN, HG and CVS. More on that will get introduced later in the post.

VCSDIR, the path used as a working directory by pkginstall scripts run at package installation by pkg_add and looked up by pkgconftrack(1). Files are copied there before being tracked in a VCS or merged, this directory also stores a local configuration repository or objects and metadata associated with remote repositories, according to the solution in use. By default, /var/confrepo is assumed as the default working directory and initialized.

REMOTEVCS, an URI describing how to access a remote repository, according to the format required by the chosen VCS. Remote repositories are disabled if this variable is unset, or if it is set to "no".

VCSCONFPULL, set it to "yes" to have pkgscr attempt to deploy configuration files from a remote repository upon package installation, instead of using package provided defaults. It now works with git, svn and mercurial.

ROLE, the role defined for the system, used when deploying configuration from a remote, site repository for packages being installed.

More details on system roles and configuration deployment are down in this post.

rcs vs network-capable systems

RCS, as stated, is the default backend because of its simplicity and widespread presence. When packages are being compiled from source, system rcs is used if present, otherwise it gets installed automatically by pkgsrc as a TOOL when the first package gets built, being an mk/pkginstall/bsd.pkginstall.mk tool dependency.

Please note that binaries defined as pkgsrc TOOLs will get their path, as on the build system, hardcoded in binary packages install scripts. This shouldn't be a problem for NetBSD binary package users since RCS is part of the base system installation and paths will match. Users who choose binary packages on other platforms and got pkgtools/pkg_install (e.g., pkg_add) by running pkgsrc bootstrap/bootstrap script should also be ok, since packages get built as part of the bootstrap procedure, making bsd.pkginstall.mk tool dependency on RCS tick. Also note that even when using other systems, automerge functionality depends on merge, part of Revision Control System.

Other software, such as git, svn, cvs and hg is not installed or called in pkginstall scripts as TOOL dependencies: they get searched in the PATH defined on the environment, and must be manually installed by the user via pkgsrc itself or other means before setting VCS= to the desired system.

Each will get briefly introduced, with an example setting up and using remote repositories.

pkgsrc will try to import existing and new files into the chosen VCS when it gets switched, and a previously undefined REMOTEVCS URI may be added to an existing repository, already existing files may get imported and pushed to the repository. It all depends on the workflow of the chosen system, but users SHOULD NOT expect pkgsrc to handle changes and the addition of remote repositories for them. At best there will be inconsistencies and the loss of revisions tracked in the old solution: the handling of repository switches is out of scope for this project.

Users who decide to switch to a different Version Control System should manually convert their local repository and start clean. Users who want to switch from local to using a remote repository should also, when setting it up, take care in populating it with local data and then start over, preferably by removing the path specified by VCSDIR or by setting it to a new path. pkgsrc will try its best to ensure new package installations don't fail because of changes made without taking the aforementioned steps, but no consistency with information stored in the old system should then be expected.

Some words on REMOTEVCS

REMOTEVCS, if defined and different from "no", points to the URI of a remote repository in a way understandable by the chosen version control system. It may include the access method, the user name, the remote FQDN or IP address, and the path to the repository. if a password gets set via userName:password@hostname, care should be taken to only set REMOTEVCS as an environment variable, leaving out login information from pkg_install.conf.

The preferred way to access remote repositories is to use ssh with asymmetrical keys. These should be placed under $HOME/.ssh/ for the user pkgsrc runs under, or under pkgvcsconf home directory if you are installing packages as root. If keys are password protected or present in non-standard search paths, ssh-agent should be started and the appropriate key unlocked via ssh-add before package installations or upgrades are run.

If running ssh-agent, SSH_AUTH_SOCK and SSH_AGENT_PID variables must be appropriately exported on the environment the package installations will take place in; CVS_RSH should be set to the path of the ssh executable and exported on the environment when using CVS to access a remote repository over ssh.

Git and remotes

git, pretty straightforwardly, will store objects, metadata and configuration at VCSDIR/.git, while VCSDIR is defined as the working-tree files get checked out to.

The configuration under VCSDIR/.git includes the definition of the remote repository, which the +VERSIONING script will set once in the PREPARE phase; the script can also switch the remote to a new one, when using git. Data consistency should be ensured by actions taken beforehand by the user.

Using git instead of rcs is simply done by setting VCS=git in pkg_install.conf

.

pkgsrc will inizialize the repository.

Please do this before installing packages that come with configuration files, then remove the old VCSDIR (e.g., by running rm -fr /var/confrepo) and let pkgsrc take care of the rest, or set VCSDIR to a new path.

If you wish to migrate an existing repository to git, seek documentation for scripts such as rcs-fast-export and git fast-import, but initialize it under a new working directory. Running git on top of an older, different repository without migrating data first will allow you to store new files, but automerge may break in strange ways when it tries to extract the first revision of a configuration file from the repository, if it is only tracked in the previous version control system.

Now, some steps needed to setup a remote git repository are also common to cvs, svn and mercurial: I will run a server at 192.168.100.112 with ssh access enabled for the user vcs. An ssh key has been generated under /root/.ssh and added to vcs@192.168.100.112 authorized_keys list for the pkgsrc host to automatically login as vcs on the remote instance.

In order to create a remote repository, login as vcs on the server and run

$ pwd                                                                                                         
/usr/home/vcs                                                                                                 
$ mkdir gitconftrack.git                                                                                      
$ cd gitconftrack.git/                                                                                        
$ git --bare init                                                                                             
Initialized empty Git repository in /usr/home/vcs/gitconftrack.git/                                           
$ cat /usr/home/vcs/.ssh/authorized_keys                                                                      
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABA[...]
$ /etc/rc.d/sshd status                                                                                       
sshd is running as pid 1343.                                                                                  

now, on the client, set the URI as ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git (full path here! hg and svn use paths relative to the user's home directory

pkghost# echo "VCSTRACK_CONF=yes" >> /usr/pkg/etc/pkg_install.conf
pkghost# echo "VCS=git" >> /usr/pkg/etc/pkg_install.conf
pkghost# echo "VCSDIR=/var/gitandremote" >> /usr/pkg/etc/pkg_install.conf
pkghost# echo "VCSAUTOMERGE=yes" >> /usr/pkg/etc/pkg_install.conf
pkghost# echo "REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git
pkghost# cat /usr/pkg/etc/pkg_install.conf 
VCSTRACK_CONF=yes
VCS=git
VCSDIR=/var/gitandremote
VCSAUTOMERGE=yes
REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git
pkghost# 

and try to install spamd as with the examples in the first post (it comes with a configuration file and is written in shell scripts that don't need to be compiled or many dependencies installed when testing things out)

pkghost# cd pkgsrc/mail/spamd
pkghost# make
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
=> Fetching spamd-20060330.tar.gz

[...]

pkghost# make install
pkghost# make install
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Installing binary package of spamd-20060330nb2
fatal: Not a git repository: '/var/gitandremote/.git'
Initialized empty Git repository in /var/gitandremote/.git/
fatal: Couldn't find remote ref master
fatal: The remote end hung up unexpectedly
REGISTER /var/gitandremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb2: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf
[master (root-commit) f84f8ee] pkgsrc: add spamd-20060330nb2
 1 file changed, 86 insertions(+)
 create mode 100644 defaults/usr/pkg/etc/spamd.conf
Conf commit: pkgsrc: add spamd-20060330nb2
Counting objects: 7, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (7/7), 1.34 KiB | 688.00 KiB/s, done.
Total 7 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git
 * [new branch]      master -> master
===========================================================================
The following files should be created for spamd-20060330nb2:

        /etc/rc.d/pfspamd (m=0755)
            [/usr/pkg/share/examples/rc.d/pfspamd]

===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $

Don't forget to add the spamd ports to /etc/services:

spamd           8025/tcp                # spamd(8)
spamd-cfg       8026/tcp                # spamd(8) configuration

===========================================================================
pkghost# 

Let's see what's at the VCSDIR now...

pkghost# cd /var/gitandremote/                                                                                
pkghost# ls                                                                                                   
.git       automerged defaults   user
pkghost# find defaults/ -type f -print                                                                        
defaults/usr/pkg/etc/spamd.conf
pkghost# git log                                                                                              
commit f84f8ee278ec53f55d652e5c6bf52554216cfd49 (HEAD -> master, origin/master)                               
Author: kmotavalli                                                                       
Date:   Fri Aug 3 15:00:26 2018 +0000                                                                         
                                                                                                              
    pkgsrc: add spamd-20060330nb2                                                                             
pkghost# git remote -v                                                                                        
origin  ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git (fetch)                                       
origin  ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git (push)                                        

on the server:

$ hostname                                                                                                    
vers                                                                                                          
$ pwd                                                                                                         
/usr/home/vcs/gitconftrack.git                                                                                
$ git log                                                                                                     
commit f84f8ee278ec53f55d652e5c6bf52554216cfd49 (HEAD -> master)                                              
Author: kmotavalli                                                                       
Date:   Fri Aug 3 15:00:26 2018 +0000                                                                         
                                                                                                              
    pkgsrc: add spamd-20060330nb2                                                                             
$

All fine, let's simulate automerge:

pkghost# vi /usr/pkg/etc/spamd.conf 

[...]

# Whitelists are done like this, and must be added to "all" after each                                        
# blacklist from which you want the addresses in the whitelist removed.                                       
#                                                                                                             
#whitelist:\
#       :white:\                                                                                              
#       :method=file:\                                                                                        
#       :file=/var/mail/whitelist.txt:

whitelist:\ 
        :white:\                                                                                              
        :method=file:\                                                                                        
        :file=/var/mail/whitelist.txt:
:x
/usr/pkg/etc/spamd.conf: 86 lines, 2767 characters
pkghost# cd $HOME/pkgsrc/mail/spamd; make replace
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Replacing for spamd-20060330nb2
===> Updating using binary package of spamd-20060330nb2
/usr/bin/env  /usr/pkg/sbin/pkg_add -K /var/db/pkg -U -D /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb2.tgz
===========================================================================
The following users are no longer being used by spamd-20060330nb2,
and they can be removed if no other software is using them:

        _spamd

===========================================================================
===========================================================================
The following groups are no longer being used by spamd-20060330nb2,
and they can be removed if no other software is using them:

        _spamd

===========================================================================
===========================================================================
The following files are no longer being used by spamd-20060330nb2,
and they can be removed if no other packages are using them:

        /usr/pkg/etc/spamd.conf

===========================================================================
Everything up-to-date
From ssh://192.168.100.112/usr/home/vcs/gitconftrack
 * branch            master     -> FETCH_HEAD
Already up to date.
REGISTER /var/gitandremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb2: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb2: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
Saving the currently user-installed revision to /var/gitandremote/user//usr/pkg/etc/spamd.conf
[master eacdef9] pkgsrc: backup user conf before attempting merge for spamd-20060330nb2
 1 file changed, 86 insertions(+)
 create mode 100644 user/usr/pkg/etc/spamd.conf
Conf commit: pkgsrc: backup user conf before attempting merge for spamd-20060330nb2
Counting objects: 7, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (7/7), 1.44 KiB | 1.44 MiB/s, done.
Total 7 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git
   f84f8ee..eacdef9  master -> master
Merged with no conflicts. installing it to /usr/pkg/etc/spamd.conf!
Revert from the last revision of /var/gitandremote/user//usr/pkg/etc/spamd.conf if needed
[master 4f1d514] pkgsrc: add spamd-20060330nb2
 1 file changed, 86 insertions(+)
 create mode 100644 automerged/usr/pkg/etc/spamd.conf
Conf commit: pkgsrc: add spamd-20060330nb2
Counting objects: 2, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 280 bytes | 280.00 KiB/s, done.
Total 2 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git
   eacdef9..4f1d514  master -> master
===========================================================================
The following files should be created for spamd-20060330nb2:

        /etc/rc.d/pfspamd (m=0755)
            [/usr/pkg/share/examples/rc.d/pfspamd]

===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $

Don't forget to add the spamd ports to /etc/services:

spamd           8025/tcp                # spamd(8)
spamd-cfg       8026/tcp                # spamd(8) configuration

===========================================================================

A new file backing up the installed configuration, as edited by the user, has been created in the working directory at /var/gitandremote/user/usr/pkg/etc/spamd.conf and pushed to the remote repository.

You can check it out and rever from it in case something breaks:

pkghost# cd /var/gitandremote/user/                                                                           
pkghost# ls                                                                                                   
usr                                                                                                           
pkghost# git checkout usr/pkg/etc/spamd.conf
pkghost# tail /var/gitandremote/user/usr/pkg/etc/spamd.conf 
#       :method=exec:\                                                                                        
#       :file=/usr/local/bin/relaydb -4lw:                                                                    
                                                                                                              
# Whitelists are done like this, and must be added to "all" after each                                        
# blacklist from which you want the addresses in the whitelist removed.                                       
#                                                                                                             
whitelist:\                                                                                                   
        :white:\                                                                                              
        :method=file:\                                                                                        
        :file=/var/mail/whitelist.txt:                                                                        
pkghost# 

running git log on the server, we can see two new commits: one pushed when backing up the installed configuration file, one after the automerge succeded and the package got installed.

$ hostname                                                                                                    
vers                                                                                                          
$ pwd                                                                                                         
/usr/home/vcs/gitconftrack.git                                                                                
$ git log                                                                                                     
commit 4f1d514159860906154eeaaa24a10ec572a4d062 (HEAD -> master)                                              
Author: kmotavalli                                                                       
Date:   Fri Aug 3 15:12:45 2018 +0000                                                                         
                                                                                                              
    pkgsrc: add spamd-20060330nb2                                                                             
                                                                                                              
commit eacdef9f52b49e3ce6f544008dc7c75dc89cbd67                                                               
Author: kmotavalli                                                                       
Date:   Fri Aug 3 15:12:43 2018 +0000                                                                         
                                                                                                              
    pkgsrc: backup user conf before attempting merge for spamd-20060330nb2                                    
                                                                                                              
commit f84f8ee278ec53f55d652e5c6bf52554216cfd49                                                               
Author: kmotavalli                                                                       
Date:   Fri Aug 3 15:00:26 2018 +0000                                                                         
                                                                                                              
    pkgsrc: add spamd-20060330nb2                                                                             
$ 

Furthermore, I'll simulate the addition of a new comment to the package provided configuration file, then the change of a variable, which should generate an automerge conflict that needs manual user review. (Note: you don't need to change configuration files in pkgsrc package working directory! This is only done to simulate a change coming with a package upgrade!

pkghost# cd $HOME/pkgsrc/mail/spamd; make clean                                                               
===> Cleaning for spamd-20060330nb2
pkghost# make fetch                                                                                           
pkghost# sed -i 's/PKGREVISION=.*2/PKGREVISION=3/g' Makefile
pkghost# make extract                                                                                         
=> Bootstrap dependency digest>=20010302: found digest-20160304                                               
===> Skipping vulnerability checks.                                                                           
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.                                                       
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.                      
=> Checksum SHA1 OK for spamd-20060330.tar.gz                                                                 
=> Checksum RMD160 OK for spamd-20060330.tar.gz                                                               
=> Checksum SHA512 OK for spamd-20060330.tar.gz                                                               
===> Installing dependencies for spamd-20060330nb3                                                            
=> Build dependency cwrappers>=20150314: found cwrappers-20180325                                             
===> Overriding tools for spamd-20060330nb3                                                                   
===> Extracting for spamd-20060330nb3                                                                         
pkghost# head work/spamd-20060330/etc/spamd.conf 
# spamd config file, read by spamd-setup(8) for spamd(8)
#
# See spamd.conf(5)
#
# Configures whitelists and blacklists for spamd
#
# Strings follow getcap(3) convention escapes, other than you
# can have a bare colon (:) inside a quoted string and it
# will deal with it. See spamd-setup(8) for more details.

pkghost# sed -i 's/method=http/method=https/g' work/spamd-20060330/etc/spamd.conf

                                                             .                                                
pkghost# make update                                                                                          
===> Skipping vulnerability checks.                                                                           
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.                                                       
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.                      
===> Deinstalling for spamd-20060330nb4                                                                       
Running /usr/pkg/sbin/pkg_delete -K /var/db/pkg -r spamd-20060330nb3                                          
===========================================================================                                   
The following users are no longer being used by spamd-20060330nb3,                                            
and they can be removed if no other software is using them:                                                   
                                                                                                              
        _spamd                                                                                                
                                                                                                              
===========================================================================                                   
===========================================================================                                   
The following groups are no longer being used by spamd-20060330nb3,                                           
and they can be removed if no other software is using them:                                                   
                                                                                                              
        _spamd                                                                                                
                                                                                                              
===========================================================================                                   
===========================================================================                                   
The following files are no longer being used by spamd-20060330nb3,                                            
and they can be removed if no other packages are using them:                                                  
                                                                                                              
        /usr/pkg/etc/spamd.conf                                                                               
                                                                                                              
===========================================================================                                   
=> Bootstrap dependency digest>=20010302: found digest-20160304                                               
===> Skipping vulnerability checks.                                                                           
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.                                                       
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.                      
===> Patching for spamd-20060330nb4                                                                           
=> Applying pkgsrc patches for spamd-20060330nb4                                                              
=> Fixing configuration paths.                                                                                
===> Creating toolchain wrappers for spamd-20060330nb4                                                        
===> Building for spamd-20060330nb4

[...]

=> Creating /root/pkgsrc/mail/spamd/work/.rc.d/pfspamd                                                        
===> Installing for spamd-20060330nb4                                                                         
=> Generating pre-install file lists                                                                          
=> Creating installation directories                                                                          
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/man/spamd.conf.5  /ro5
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd.8  /root/8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-set8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb.8  /roo8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd.8  8
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-c
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd  /rootc
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb  /ron
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd c
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/etc/spamd.conf /root/d
=> Automatic manual page handling                                                                             
=> Generating post-install file lists                                                                         
=> Checking file-check results for spamd-20060330nb4                                                          
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb4.tgz                       
===> Building binary package for spamd-20060330nb4                                                            
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb4.tgz                                    
===> Installing binary package of spamd-20060330nb4                                                           
Everything up-to-date                                                                                         
From ssh://192.168.100.112/usr/home/vcs/gitconftrack                                                          
 * branch            master     -> FETCH_HEAD                                                                 
Already up to date.                                                                                           
REGISTER /var/gitretest/defaults//usr/pkg/etc/spamd.conf                                                      
spamd-20060330nb4: /usr/pkg/etc/spamd.conf already exists                                                     
spamd-20060330nb4: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!                             
Saving the currently installed revision to /var/gitretest/automerged//usr/pkg/etc/spamd.conf                  
[master 8970767] pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb4                
 1 file changed, 17 insertions(+), 17 deletions(-)                                                            
Conf commit: pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb4                    
Counting objects: 7, done.                                                                                    
Compressing objects: 100% (3/3), done.                                                                        
Writing objects: 100% (7/7), 599 bytes | 599.00 KiB/s, done.                                                  
Total 7 (delta 1), reused 0 (delta 0)                                                                         
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git                                                        
   8dd7368..8970767  master -> master                                                                         
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!                                            
--- /usr/pkg/etc/spamd.conf     2018-08-04 06:13:39.017744154 +0000                                           
+++ /var/gitretest/defaults//usr/pkg/etc/spamd.conf.automerge   2018-08-04 06:15:09.799758773 +0000           
@@ -15,7 +15,7 @@                                                                                             
 # may be applied to each blacklist.                                                                          
 #                                                                                                            
 # As of November 2004, a place to search for black lists is                                                  
-#     http://spamlinks.net/filter-bl.htm                                                                     
+#     https://spamlinks.net/filter-bl.htm                                                                    
 #                                                                                                            
 # Some of the URLs below point to www.openbsd.org locations.  Those                                          
 # files are likely to be mirrored to other OpenBSD www mirrors located                                       
@@ -25,45 +25,45 @@                                                                                           
 all:\                                                                                                        
        :spews1:china:korea:                                                                                  
                                                                                                              
-# Mirrored from http://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2                                            
+# Mirrored from https://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2                                           
 spamhaus:\                                                                                                   
        :black:\                                                                                              
        :msg="SPAM. Your address %A is in the Spamhaus Block List\n\                                          
-       See http://www.spamhaus.org/sbl and\                                                                  
-       http://www.abuse.net/sbl.phtml?IP=%A for more details":\                                              
-       :method=http:\                                                                                        
+       See https://www.spamhaus.org/sbl and\                                                                 
+       https://www.abuse.net/sbl.phtml?IP=%A for more details":\                                             
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/SBL.cidr.gz:                                                              
                                                                                                              
-# Mirrored from http://www.spews.org/spews_list_level1.txt                                                   
+# Mirrored from https://www.spews.org/spews_list_level1.txt                                                  
 spews1:\                                                                                                     
        :black:\                                                                                              
        :msg="SPAM. Your address %A is in the spews level 1 database\n\                                       
-       See http://www.spews.org/ask.cgi?x=%A for more details":\                                             
-       :method=http:\                                                                                        
+       See https://www.spews.org/ask.cgi?x=%A for more details":\                                            
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/spews_list_level1.txt.gz:                                                 
                                                                                                              
-# Mirrored from http://www.spews.org/spews_list_level2.txt                                                   
+# Mirrored from https://www.spews.org/spews_list_level2.txt                                                  
 spews2:\                                                                                                     
        :black:\                                                                                              
        :msg="SPAM. Your address %A is in the spews level 2 database\n\                                       
-       See http://www.spews.org/ask.cgi?x=%A for more details":\                                             
-       :method=http:\                                                                                        
+       See https://www.spews.org/ask.cgi?x=%A for more details":\                                            
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/spews_list_level2.txt.gz:                                                 
                                                                                                              
-# Mirrored from http://www.okean.com/chinacidr.txt                                                           
+# Mirrored from https://www.okean.com/chinacidr.txt                                                          
 china:\                                                                                                      
        :black:\                                                                                              
        :msg="SPAM. Your address %A appears to be from China\n\                                               
-       See http://www.okean.com/asianspamblocks.html for more details":\                                     
-       :method=http:\                                                                                        
+       See https://www.okean.com/asianspamblocks.html for more details":\                                    
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/chinacidr.txt.gz:                                                         
                                                                                                              
-# Mirrored from http://www.okean.com/koreacidr.txt                                                           
+# Mirrored from https://www.okean.com/koreacidr.txt                                                          
 korea:\                                                                                                      
        :black:\                                                                                              
        :msg="SPAM. Your address %A appears to be from Korea\n\                                               
-       See http://www.okean.com/asianspamblocks.html for more details":\                                     
-       :method=http:\                                                                                        
+       See https://www.okean.com/asianspamblocks.html for more details":\                                    
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/koreacidr.txt.gz:                                                         
                                                                                                              
 #relaydb-black:\                                                                                             
Revert from the penultimate revision of /var/gitretest/automerged//usr/pkg/etc/spamd.conf if needed           
[master 1eef6d8] pkgsrc: add spamd-20060330nb4                                                                
 1 file changed, 17 insertions(+), 17 deletions(-)                                                            
Conf commit: pkgsrc: add spamd-20060330nb4                                                                    
Counting objects: 7, done.                                                                                    
Compressing objects: 100% (3/3), done.                                                                        
Writing objects: 100% (7/7), 563 bytes | 563.00 KiB/s, done.                                                  
Total 7 (delta 1), reused 0 (delta 0)                                                                         
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git                                                        
   8970767..1eef6d8  master -> master                                                                         
===========================================================================                                   
The following files should be created for spamd-20060330nb4:                                                  
                                                                                                              
        /etc/rc.d/pfspamd (m=0755)                                                                            
            [/usr/pkg/share/examples/rc.d/pfspamd]                                                            
                                                                                                              
===========================================================================                                   
===========================================================================                                   
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $                                                    
                                                                                                              
Don't forget to add the spamd ports to /etc/services:                                                         
                                                                                                              
spamd           8025/tcp                # spamd(8)                                                            
spamd-cfg       8026/tcp                # spamd(8) configuration                                              
                                                                                                              
===========================================================================                                   
===> Cleaning for spamd-20060330nb4                                                                           
[...]

pkghost# cat /var/gitandremote/automergedfiles 
/usr/pkg/etc/spamd.conf
pkghost# 

Should conflicts arise, these will be highlited as per rcs 3-way merge syntax in the file /var/gitandremote/defaults/usr/pkg/etc/spamd.conf.automerge and no action taken.

In that case, manually review the conflict opening spamd.conf.automerge in your favorite text editor, then manually copy it in place of the existing installed configuration file for spamd at /usr/pkg/etc/spamd.conf

CVS and remotes

CVS is used by having a CVSROOT directory at VCSDIR/CVSROOT, with VCSDIR as the working directory. The same precautions treated illustrating Git usage also apply when chosing to use CVS (and svn, mercurial). To use CVS, simply set VCS=cvs in PREFIX/etc/pkg_install.conf and change the VCSDIR to a new path.

pkgsrc will try to import existing files in CVS if the VCSDIR is not empty, but past files revision will not get migrated from whatever version control system was in use, nor will files present in the old repository but not checked out in the working directory. This will likely make automerge malfunction, so it's up to the user to migrate preeexisting repositories to CVS or to chose a new, empty path as the VCS working directory, if automerge has never been used up to this change (VCSAUTOMERGE not set to yes or VCSDIR/automergedfiles not existing/empty).

See CVS documentation for a description of how to form remote URIs (that will get passed to cvs with the -d flag)

In this example, I will use :ext:vcs@192.168.100.112:/usr/home/vcs/cvsconftrack as the REMOTEVCS uri

pkghost# cat /usr/pkg/etc/pkg_install.conf 
VCSTRACK_CONF=yes
VCS=cvs
VCSDIR=/var/cvsandremote
VCSAUTOMERGE=yes
REMOTEVCS=:ext:vcs@192.168.100.112:/usr/home/vcs/cvsconftrack
pkghost# echo "export CVS_RSH=ssh" >> $HOME/.profile
pkghost# export CVS_RSH=ssh                                                                                    
pkghost# 

and initialize a remote cvs repository on the server at 192.168.100.112:

$ hostname                                                                                                    
vers                                                                                                          
$ pwd                                                                                                         
/usr/home/vcs                                                                                                 
$ cvs -d/usr/home/vcs/cvsconftrack init                                                                       

manually migrate existing data from other systems to the new repository if needed, then try installing some package:

pkghost# pwd
/root/pkgsrc/mail/spamd
pkghost# make install
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Installing binary package of spamd-20060330nb5
cvs rlog: cannot find module `defaults' - ignored
cvs checkout: Updating automerged
cvs checkout: Updating defaults
cvs checkout: Updating user
cvs update: Updating automerged
cvs update: Updating defaults
cvs update: Updating user
? usr/pkg
Directory /usr/home/vcs/cvsconftrack/defaults/usr added to the repository
? pkg/etc
Directory /usr/home/vcs/cvsconftrack/defaults/usr/pkg added to the repository
? etc/spamd.conf
Directory /usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc added to the repository
cvs add: scheduling file `spamd.conf' for addition
cvs add: use 'cvs commit' to add this file permanently
REGISTER /var/cvsandremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb5: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf
RCS file: /usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/spamd.conf,v
done
Checking in defaults/usr/pkg/etc/spamd.conf;
/usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/spamd.conf,v  <--  spamd.conf
initial revision: 1.1
done
Conf commit: pkgsrc: add spamd-20060330nb5
===========================================================================
The following files should be created for spamd-20060330nb5:

        /etc/rc.d/pfspamd (m=0755)
            [/usr/pkg/share/examples/rc.d/pfspamd]

===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $

Don't forget to add the spamd ports to /etc/services:

spamd           8025/tcp                # spamd(8)
spamd-cfg       8026/tcp                # spamd(8) configuration

===========================================================================
pkghost# 

On a successive run, the remote repository will get used without files being imported and checked out:

pkghost# cd /root/pkgsrc/pkgtools/pkgin/
pkghost# make install
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Installing for pkgin-0.9.4nb8
=> Generating pre-install file lists
=> Creating installation directories
[...]
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for pkgin-0.9.4nb8
=> Creating binary package /root/pkgsrc/pkgtools/pkgin/work/.packages/pkgin-0.9.4nb8.tgz
===> Building binary package for pkgin-0.9.4nb8
=> Creating binary package /root/pkgsrc/packages/All/pkgin-0.9.4nb8.tgz
===> Installing binary package of pkgin-0.9.4nb8
cvs rlog: Logging defaults
cvs rlog: Logging defaults/usr
cvs rlog: Logging defaults/usr/pkg
cvs rlog: Logging defaults/usr/pkg/etc
cvs update: Updating automerged
cvs update: Updating defaults
cvs update: Updating defaults/usr
cvs update: Updating defaults/usr/pkg
cvs update: Updating defaults/usr/pkg/etc
cvs update: Updating user
cvs [add aborted]: there is a version in usr already
cvs [add aborted]: there is a version in pkg already
cvs [add aborted]: there is a version in etc already
? pkgin/repositories.conf
Directory /usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/pkgin added to the repository
cvs add: scheduling file `repositories.conf' for addition
cvs add: use 'cvs commit' to add this file permanently
REGISTER /var/cvsandremote/defaults//usr/pkg/etc/pkgin/repositories.conf
pkgin-0.9.4nb8: copying /usr/pkg/share/examples/pkgin/repositories.conf.example to /usr/pkg/etc/pkgin/repositories.conf
RCS file: /usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/pkgin/repositories.conf,v
done
Checking in defaults/usr/pkg/etc/pkgin/repositories.conf;
/usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/pkgin/repositories.conf,v  <--  repositories.conf
initial revision: 1.1
done
Conf commit: pkgsrc: add pkgin-0.9.4nb8
===========================================================================
$NetBSD: MESSAGE,v 1.3 2010/06/10 08:05:00 is Exp $

First steps before using pkgin.

. Modify /usr/pkg/etc/pkgin/repositories.conf to suit your platform
. Initialize the database :

        # pkgin update

===========================================================================
pkghost# 

Yes, I know, it's a bit too chatty. Once the script gets used (and tested) some more cvs should be invoked in a more quiet way.

See the next post for a walkthrough in using SVN and Mercurial!

[0 comments]

 

GSoC 2018 Reports: Configuration files versioning in pkgsrc, Part 1

Starting with this post I will describe how, as part of the Google Summer of Code 2018, support for configuration files versioning is shaping up in pkgsrc.

[Read More] [0 comments]

 

Report from pkgsrcCon 2018

On July 7th and 8th there was pkgsrcCon 2018 in Berlin, Germany. It was my first pkgsrcCon and it was really really nice... So, let's share a report about it, what we have done, the talk presented and everything else!

[Read More] [1 comment]

 

GSoC 2017 Reports: Add SUBPACKAGES support to pkgsrc, part 1

In this blog post series I will discuss about SUBPACKAGES work done during Google Summer of Code 2017.

In this first part I'll briefly introduce what are SUBPACKAGES, why and when can be useful and finally we'll give a quick look to a trivial pkgsrc package that uses them. At the end we'll also dive a bit on parts of the pkgsrc infrastructure that needed to be adjusted for implementing that.

[Read More] [0 comments]

 

GSoC 2016 Reports: Split debug symbols for pkgsrc builds, part 2

Google Summer of Code 2016 is now over. We have polished the code and documentation and submitted the final term evaluation on 23rd of August 2016. The mentors evaluated us in the following week.

If you are impatient (and this time impatience is definitely a virtue!) please take a look to all The NetBSD Foundation GSoC 2016 projects' code submissions!

[Read More] [0 comments]

 

GSoC 2016 Reports: Split debug symbols for pkgsrc builds, part 1

For the 10th time The NetBSD Foundation was selected for the GSoC 2016!

Now that we're near the first mid-term evaluation and have written the code during these weeks it's also the right time to start writing some reports regarding our projects in this series of blog posts.

[Read More] [0 comments]

 

pkgsrc 50th release interviews - Joerg Sonnenberger

The pkgsrc team has prepared the 50th release of their package management system, with the 2016Q1 version. It's infrequent event, as the 100th release will be held after 50 quarters.

The NetBSD team has prepared series of interviews with the authors. We started with Joerg Sonnenberger, a developer well known for his LLVM on NetBSD contribution.

[Read More] [0 comments]

 

pkgsrc-2016Q1 - the Fiftieth pkgsrc Release

A perspective, looking back 19 years over pkgsrc, to celebrate the 50th quarterly pkgsrc release.[Read More] [0 comments]

 

pkgsrc-wip migrating to NetBSD.org, git

If everything goes as planned, the pkgsrc-wip CVS repository will be converted to git and hosted on NetBSD.org by end of September.

In July we cleaned up the repository so it can be converted easily; since then we've been working on the infrastructure and details of the conversion. The main tasks are now finished. We have set up a server for it which hosts a preliminary git conversion (on wip.pkgsrc.org) of the CVS repository, created a mailing list for the commit messages, pkgsrc-wip-changes, and prepared a list of authors for the conversion.

We've also provided a conversion of pkgsrc-wip based on data from July so that it can be tested on (nearly) live data. If you are interested in beta-testing the setup, send a suggestion for a username and an SSH public key to me. Details on how to test are on the NetBSD wiki but will probably change some more over time.

We still need help for the conversion: if you are or were a wip contributor, please let me know by September 15 what name and email to use for the conversion from CVS to git. This conversion will not be done again, so after that date, the commit data will be final. [0 comments]

 

pkgsrc for Illumos

The Illumos project is "a community maintained derivative of the OpenSolaris ON source, including open source replacements for closed bits, and additional changes" (from http://www.illumos.org/). A couple of month ago, the Illumos community launched "The Illumos pkgsrc project", thus communicating for the first time on pkgsrc being officially supported by a SunOS derivative.

[Read More] [2 comments]