![[NetBSD Logo]](/tnf/resource/NetBSD-headerlogo.png)
The process of upstreaming support to LLVM sanitizers has been finalized
I've finished the process of upstreaming patches to LLVM sanitizers (almost 2000LOC of local code) and submitted to upstream new improvements for the NetBSD support. Today out of the box (in unpatched version) we have support for a variety of compiler-rt LLVM features: ASan (finds unauthorized memory access), UBSan (finds unspecified code semantics), TSan (finds threading bugs), MSan (finds uninitialized memory use), SafeStack (double stack hardening), Profile (code coverage), XRay (dynamic code tracing); while other ones such as Scudo (hardened allocator) or DFSan (generic data flow sanitizer) are not far away from completeness.
The NetBSD support is no longer visibly lacking behind Linux in sanitizers, although there are still failing tests on NetBSD that are not observed on Linux. On the other hand there are features working on NetBSD that are not functional on Linux, like sanitizing programs during early initialization process of OS (this is caused by /proc dependency on Linux that is mounted by startup programs, while NetBSD relies on sysctl(3) interfaces that is always available).[Read More] [2 comments]
NetBSD entering 2019 with more complete LLVM support
Prepared by Michał Górny (mgorny AT gentoo.org).
I'm recently helping the NetBSD developers to improve the support for this operating system in various LLVM components. As you can read in my previous report, I've been focusing on fixing build and test failures for the purpose of improving the buildbot coverage.
Previously, I've resolved test failures in LLVM, Clang, LLD, libunwind, openmp and partially libc++. During the remainder of the month, I've been working on the remaining libc++ test failures, improving the NetBSD clang driver and helping Kamil Rytarowski with compiler-rt.
[Read More] [0 comments]
The NetBSD support update before the LLVM-8.0 branching point
Prepared by Michał Górny (mgorny AT gentoo.org).
I'm recently helping the NetBSD developers to improve the support for this operating system in various LLVM components. My first task in this endeavor was to fix build and test issues in as many LLVM projects as timely possible, and get them all covered by the NetBSD LLVM buildbot.
Including more projects in the continuous integration builds is important as it provides the means to timely catch regressions and new issues in NetBSD support. It is not only beneficial because it lets us find offending commits easily but also because it makes other LLVM developers aware of NetBSD porting issues, and increases the chances that the patch authors will fix their mistakes themselves.
[Read More] [0 comments]
Reducing the delta with upstream version of sanitizers
I've been actively working on reducing the delta with the local copy of sanitizers with upstream LLVM sources. Their diff has been reduced to less than 2000 Lines Of Code. I've pushed to review almost all of the local code and I'm working on addressing comments from upstream developers. [Read More] [0 comments]
NetBSD LLVM Sanitizers in The Bay Area
I have presented the state of NetBSD sanitizers during two conferences in the San Francisco Bay Area: Google Summer of Code Mentor Summit (Mountain View) and MeetBSDCa (Santa Clara, Intel Campus SC12). I've also made progress in upstreaming of our local patches to LLVM sanitizers and introducing generic NetBSD enhancements there.[Read More] [1 comment]
EuroBSDCon 2018 and NetBSD sanitizers
I presented the state of NetBSD sanitizers during EuroBSDCon 2018 held in Bucharest, Romania.
I gave two talks, one covered userland sanitizers and the other one kernel sanitizers. Unfortunately video recordings from the conference are not available, but I've uploaded my slides online:
- LLVM Sanitizers in the NetBSD userland
- Taking NetBSD kernel bug roast to the next level: Kernel Sanitizers
EuroBSDCon 2018 travel report and obligatory pics
This was my first big BSD conference. We also planned - planned might be a big word - thought about doing a devsummit on Friday. Since the people who were in charge of that had a change of plans, I was sure it'd go horribly wrong.
The day before the devsummit and still in the wrong country, I mentioned the hours and venue on the wiki, and booked a reservation for a restaurant.
It turns out that everything was totally fine, and since the devsummit was at the conference venue (that was having tutorials that day), they even had signs pointing at the room we were given. Thanks EuroBSDCon conference organizers!
At the devsummit, we spent some time hacking. A few people came with "travel laptops" without access to anything, like Riastradh, so I gave him access to my own laptop. This didn't hold very long and I kinda forgot about it, but for a few moments he had access to a NetBSD source tree and an 8 thread, 16GB RAM machine with which to build things.
We had a short introduction and I suggested we take some pictures, so here's the ones we got. A few people were concerned about privacy, so they're not pictured. We had small team to hold the camera :-)
At the actual conference days, I stayed at the speaker hotel with the other speakers.
I've attempted to make conversation with some visibly FreeBSD/OpenBSD people, but
didn't have plans to talk about anything, so there was a lot of just following
people silently.
Perhaps for the next conference I'll prepare a list of questions to random BSD people
and then very obviously grab a piece of paper and ask, "what was...", read a bit from
it, and say, "your latest kernel panic?", I'm sure it'll be a great conversation
starter.
At the conference itself, was pretty cool to have folks like Kirk McKusick give first person accounts of some past events (Kirk gave a talk about governance at FreeBSD), or the second keynote by Ron Broersma.
My own talk was hastily prepared, it was difficult to bring the topic together into a coherent talk. Nevertheless, I managed to talk about stuff for a while 40 minutes, though usually I skip over so many details that I have trouble putting together a sufficiently long talk.
I mentioned some of my coolest bugs to solve (I should probably make a separate article about some!). A few people asked for the slides after the talk, so I guess it wasn't totally incoherent.
It was really fun to meet some of my favourite NetBSD people. I got to show off my now fairly well working laptop (it took a lot of work by all of us!).
After the conference I came back with a conference cold, and it took a few days to recover from it. Hopefully I didn't infect too many people on the way back.
[0 comments]
Using acme.sh for Let's Encrypt certificates on pkgsrc.org servers
Peter Wemm's writeup about using acme.sh for FreeBSD.org served as inspiration, but I chose to do a few things different:
- using DNS alias mode with sub-domains dedicated to ACME verification
- delegating the sub-domains to the servers where the certificate will be needed
- using bind on the servers where the certificate will be needed (where it was running as resolver already anyway)
- using dns_nsupdate (i.e. dynamic DNS) to add the challenge to the ACME subzone.
options {
....
allow-update { localhost; };
....
};
zone "acme-www.pkgsrc.org" {
type master;
file "acme/acme-www.pkgsrc.org";
update-policy {
grant acme-ddns name _acme-challenge.acme-www.pkgsrc.org. TXT;
};
};
And last but not least, deployment of certificates via make, i.e. completely independent of acme.sh.
Due to all of the above, acme.sh does not need to tentacle about in the filesystem and can run as a plain user in a chroot. It's not a tiny chroot, though (20M), since acme.sh needs a bunch of common shell tools:
- awk basename cat chmod cp curl cut date egrep/grep head mkdir mktemp mv nsupdate od openssl printf readlink rm sed sh sleep stat tail touch tr uname, and their shared libs, /libexec/ld.elf_so and /usr/libexec/ld.elf_so;
- under the chroot /etc a resolv.conf, the CA cert for Let's Encrypt (mozilla-rootcert-60.pem) and to make openssl complain less an empty openssl.cnf
- and in the chroot /dev: null, random and urandom.
I call both the acme.sh --cron job and the certificate deployment make from daily.local, which adds the output to the daily mail and makes it easy to keep an eye on things. [0 comments]
Finishing leftover tasks from Google Summer of Code
Over the past month, I was coordinating and coding the remaining post-GSoC tasks. This mostly covers work around honggfuzz and sanitizers.[Read More] [0 comments]
NetBSD 7.2 released
The NetBSD 7.2 release is available now.
[Read More] [0 comments]
GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 4: configuration deployment, pkgtools and future improvements
As introduced in GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 2: remote repositories (git and CVS) pkgsrc supports using some version control system repositories to search for site-specific configuration for packages being installed, and deploys it on the system.
Configuration deployment: intro to VCSCONFPULL
Support for configuration tracking is in scripts, pkginstall scripts, that get built into binary packages and are run by pkg_add upon installation. The idea behind the proposal suggested that users of the new feature should be able to store revisions of their installed configuration files, and of package-provided default, both in local or remote repositories. With this capability in place, it doesn't take much to make the scripts "pull" configuration from a VCS repository at installation time.
That's what setting VCSCONFPULL=yes in pkg_install.conf after having enabled VCSTRACK_CONF does:
You are free to use official, third party prebuilt packages that have no customization in them, enable these options, and point pkgsrc to a private conf repository. If it contains custom configuration for the software you are installing, an attempt will be made to use it and install it on your system. If it fails, pkginstall will fall back to using the defaults that come inside the package. RC scripts are always deployed from the binary package, if existing and PKG_RCD_SCRIPTS=yes in pkg_install.conf or the environment.
This will be part of packages, not a separate solution like configuration management tools. It doesn't support running scripts on the target system to customize the installation, it doesn't come with its domain-specific language, it won't run as a daemon or require remote logins to work. It's quite limited in scope, but you can define a ROLE for your system in pkg_install.conf or in the environment, and pkgsrc will look for configuration you or your organization crafted for such a role (e.g., public, standalone webserver vs reverse proxy or node in a database cluster)
Configuration files should be put in branches named according to a specific convention, their paths relative to the repository is implied to be their absolute path, prepending a "/", on the target system.
Branch names define the package name they contain configuration for, the role, the version of the package the configuration is made for and a range of compatible software releases that should work with such configuration options, a best match is attemped inside of compatible ranges to the nearest software release, and roles can be undefined (both on the branch, to make it apt for any system, or on the target to ignore roles specified in branch names where they don't matter). More on that later, now some practical examples using git:
Set up a remote repository on some machine, if you are not using a public facing service or private repositories in third party services:
$ id uid=1001(vcs) gid=1001(vcs) groups=1001(vcs) $ mkdir confpullexample.git $ cd confpullexample.git/ $ git --bare init Initialized empty Git repository in /usr/home/vcs/confpullexample.git/
Check for correct settings on the target machine: (a REMOTEVCS URI is required, the VCSDIR is NOT used)
pkghost# cat /usr/pkg/etc/pkg_install.conf VCSTRACK_CONF=yes VCSCONFPULL=yes REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/confpullexample.git VCS=git
Do no reuse repositories that track existing configuration files from packages, set a new one here. It will be structured differently, with branches that can specialize between them and yet contain configuration files common to each one (from master/trunk/...). Some technicalities are needed before making practical examples. I will now cheat and cite direcly the comment in mk/pkginstall/versioning:
The remote configuration repository should contain branches named
according to the following convention:
category_pkgName_pkgVersion_compatRangeStart_compatRangeEnd_systemRole
an optional field may exist that explicitates part of the system hostname
category_pkgName_pkgVersion_compatRangeStart_compatRangeEnd_systemRole_hostname
.
the branch should contain needed configuration files. Their path relative to the repository is then prepended with a "/" and files force copied to the system and chmod 0600 executed on them. Permission handling and removal upon package uninstallation are not supported.
The branch to be used, among the available ones, is chosen this way:
branches named according to the convention that provide configuration
for category/packageName are filtered from the VCS output;
then, all branches whose ranges are compatible with the version of the
package being installed are selected. The upper bound of the range is
excluded as a compatible release if using sequence based identifiers.
If system role is set through the ROLE environment variable,
and it's different from "any",
and branches exists whose role is different from "any", then their
role gets compared with the one defined on the system or in pkg_add config.
The last part of the branch name is optional and, if present, is compared
character by character with the system hostname,
finally selecting the branches that best match it.
As an example, a branch named mail_postfix_3.3.0_3.0.0_3.3.20_mailrelay_ams
will match with system hostname amsterdam09.
A system with its ROLE unspecified or set to ANY will select branches
independently of the role they are created for, scoring and using the one
with the best matching optional hostname and/or nearest to the target release
as explained below:
Let's suppose that an hypothetical team uses a common ssh configuration, on all systems, to disable root and passwordless logins, and enable logins from users in a specific group.
the main/master branch of the repository will then contain one custom object,
etc/ssh/sshd_config
that will get included when branching from there (specific removals are always possible).
$ whoami devuser $ file $HOME/.ssh/id_rsa /home/devuser/.ssh/id_rsa: PEM RSA private key $ pwd /home/devuser/sim $ git clone ssh://vcs@192.168.100.112/usr/home/vcs/confpullexample.git . Cloning into '.'... The authenticity of host '192.168.100.112 (192.168.100.112)' can't be established. ECDSA key fingerprint is SHA256:RMkiZlYqNIKlgDQUvhFBXLUpW2qcLd1nuEi4NaROkLg. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.100.112' (ECDSA) to the list of known hosts. warning: You appear to have cloned an empty repository. $ mkdir -p etc/ssh $ cat /etc/ssh/sshd_config > etc/ssh/sshd_config
customize it as needed, then
$ echo "PermitEmptyPasswords no" >> $HOME/sim/etc/ssh/sshd_config $ echo "PermitRootLogin no" >> $HOME/sim/etc/ssh/sshd_config $ echo "AllowGroup ops" >> $HOME/sim/etc/ssh/sshd_config
and load it in the repo:
$ git add etc/ssh/sshd_config $ git commit -m "add common sshd_config to master" [master (root-commit) ec91ffc] add common sshd_config to master 1 file changed, 134 insertions(+) create mode 100644 etc/ssh/sshd_config $ git push Counting objects: 5, done. Compressing objects: 100% (2/2), done. Writing objects: 100% (5/5), 1.89 KiB | 966.00 KiB/s, done. Total 5 (delta 0), reused 0 (delta 0) To ssh://192.168.100.112/usr/home/vcs/confpullexample.git * [new branch] master -> master
With respect to this example, two different nginx configuration sets will exist, one for reverse proxies (role reverseproxy) and one for standalone webservers (role webserver). A standalone webserver will also install a default database configuration file, which will be customized for clustered postgresql instances (roles dbcluster-master and dbcluster-replicas).
Furthermore, nodes part of the same cluster will have hostnames the likes of toontowndc-node03 so that branches ending with _toontowndc-node will match and have configuration files with the addresses of nodes in the same cluster deployed.
The branch net_net-snmp_5.7.3_5.2_6_any will work on all systems you want to monitor via snmpd independently of their defined role, provided the version of net-snmp being installed is >= 5.2 and < 6
In order to keep the tutorial short, I will only show how to deploy configuration for postgresql.
I'll start with a generic configuration file, for the role webserver;
the branch databases_postgresql10_10.4nb1_10.0_11_default may be created to contain default configuration from the package (the role isn't any, so it will not be accidentally selected on systems that define a role), then branch it into the specialized roles webserver, dbcluster-master, dbcluster-deplicas.
A master-node address specific to each cluster site will be included in recovery.conf by specializing the branch ...dbcluster-replicas to branches specific for earch location, e.g., dbcluster-replicas_toontowndc that will be deployed on db nodes in the "Toontown" site.
NOTE that files from the branch will get copied unconditionally on the system, replacing existing files or files coming with the binary package, even if they are not installed to location/handled by the +FILES script
This means that it's possible to write configuration files to ${pgsql_home}/data, by default /usr/pkg/pgsql/data/, where the rc.d script usually sets postgresql to look into.
$ pwd
/home/devuser
$ mkdir -p sim/usr/pkg/pgsql/data
$ cd sim
$ ls -a
. .. .git etc usr
$ git checkout -b databases_postgresql10_10.4nb1_10.0_11_default
Switched to a new branch 'databases_postgresql10_10.4nb1_10.0_11_default'
$ ls
etc usr
$ cp /home/devuser/postgresql.conf.sample usr/pkg/pgsql/data/postgresql.conf
$ cp /home/devuser/pg_hba.conf.sample usr/pkg/pgsql/data/pg_hba.conf
$ vi usr/pkg/pgsql/data/pg_hba.conf
$ tail -n 23 usr/pkg/pgsql/data/pg_hba.conf
# Put your actual configuration here
# ----------------------------------
#
# If you want to allow non-local connections, you need to add more
# "host" records. In that case you will also need to make PostgreSQL
# listen on a non-local interface via the listen_addresses
# configuration parameter, or via the -i or -h command line switches.
@authcomment@
# TYPE DATABASE USER ADDRESS METHOD
@remove-line-for-nolocal@# "local" is for Unix domain socket connections only
@remove-line-for-nolocal@local all all ident
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
@remove-line-for-nolocal@local replication all ident
#host replication all 127.0.0.1/32 md5
#host replication all ::1/128 md5
$ echo "log_destination = 'syslog'" >> usr/pkg/pgsql/data/postgresql.conf
$ echo "syslog_ident = 'postgres'" >> usr/pkg/pgsql/data/postgresql.conf
$ git add usr/pkg/pgsql/*
$ git commit -m "import common config for not to be deployed default role"
[databases_postgresql10_10.4nb1_10.0_11_default 1ce080d] import common config for not to be deployed default e
2 files changed, 749 insertions(+)
create mode 100644 usr/pkg/pgsql/data/pg_hba.conf
create mode 100644 usr/pkg/pgsql/data/postgresql.conf
$ git push --set-upstream origin databases_postgresql10_10.4nb1_10.0_11_default
Counting objects: 8, done.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (8/8), 9.10 KiB | 4.55 MiB/s, done.
Total 8 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/confpullexample.git
* [new branch] databases_postgresql10_10.4nb1_10.0_11_default -> databases_postgresql10_10.4nb1_10.0_11_default
Branch 'databases_postgresql10_10.4nb1_10.0_11_default' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_default'
the branch with role webserver will get deployed, and doesn't differ in configuration from the default branch:
$ git checkout -b databases_postgresql10-server_10.4nb1_10.0_11_webserver Switched to a new branch 'databases_postgresql10-server_10.4nb1_10.0_11_webserver' $ git push --set-upstream origin databases_postgresql10_10.4nb1_10.0_11_webserver Total 0 (delta 0), reused 0 (delta 0) To ssh://192.168.100.112/usr/home/vcs/confpullexample.git * [new branch] databases_postgresql10_10.4nb1_10.0_11_webserver -> databases_postgresql10_10.4nb1_10.0_11_webserver Branch 'databases_postgresql10_10.4nb1_10.0_11_webserver' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_webserver'
now change it for the role dbcluster-master:
$ git checkout -b databases_postgresql10_10.4nb1_10.0_11_dbcluster-master
M usr/pkg/pgsql/data/pg_hba.conf
M usr/pkg/pgsql/data/postgresql.conf
Switched to a new branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-master'
$ vi usr/pkg/pgsql/data/pg_hba.conf
$ tail -n 23 usr/pkg/pgsql/data/pg_hba.conf
# Put your actual configuration here
# ----------------------------------
#
# If you want to allow non-local connections, you need to add more
# "host" records. In that case you will also need to make PostgreSQL
# listen on a non-local interface via the listen_addresses
# configuration parameter, or via the -i or -h command line switches.
@authcomment@
# TYPE DATABASE USER ADDRESS METHOD
@remove-line-for-nolocal@# "local" is for Unix domain socket connections only
@remove-line-for-nolocal@local all all ident
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
@remove-line-for-nolocal@local replication all ident
host replication all 127.0.0.1/32 trust
host replication all ::1/128 trust
$ cat << EOF >> usr/pkg/pgsql/data/postgresql.conf
> listen_addresses = '*'
> wal_level = hot_standby
> max_wal_senders = 10
> hot_standby = on
> archive_mode = on
> archive_command = 'cp %p /usr/pkg/pgsql/archive/%f'
> EOF
No automation exists to run mkdir other than creating a placeholder file. you will have to chown the archive dir! File permissions are not yet handled. This may change in the future, by reusing the +DIRS script with new input for example, if deemed necessary by the community.
$ mkdir -p usr/pkg/pgsql/archive/ $ echo "remember to chown this dir to the user pgsql runs as" > usr/pkg/pgsql/archive/placeholder $ git add usr/pkg/pgsql/archive/placeholder $ git commit -m "create dbcluster-master role for postgresql10" [databases_postgresql10_10.4nb1_10.0_11_dbcluster-master 98fe514] create dbcluster-master role for postgresql0 3 files changed, 9 insertions(+), 2 deletions(-) create mode 100644 usr/pkg/pgsql/archive/placeholder $ git push -u origin databases_postgresql10_10.4nb1_10.0_11_dbcluster-master Counting objects: 10, done. Compressing objects: 100% (7/7), done. Writing objects: 100% (10/10), 827 bytes | 827.00 KiB/s, done. Total 10 (delta 2), reused 0 (delta 0) To ssh://192.168.100.112/usr/home/vcs/confpullexample.git * [new branch] databases_postgresql10_10.4nb1_10.0_11_dbcluster-master -> databases_postgresql10_10.4nb1_10.0_11_dbcluster-master Branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-master' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-master'
create a generic node/replica configuration from the default config files and add the recovery.conf file...
$ git checkout databases_postgresql10_10.4nb1_10.0_11_default Switched to branch 'databases_postgresql10_10.4nb1_10.0_11_default' Your branch is up to date with 'origin/databases_postgresql10_10.4nb1_10.0_11_default'. $ mkdir -p usr/pkg/pgsql/archive/ $ echo "remember to chown this dir to the user pgsql runs as" > usr/pkg/pgsql/archive/placeholder $ cp /home/devuser/recovery.conf.sample usr/pkg/pgsql/data/recovery.conf $ echo "standby_mode = 'on'" >> usr/pkg/pgsql/data/recovery.conf $ echo "restore_command = 'cp /usr/pkg/pgsql/archive/%f \"%p%\"'" >> usr/ $ git checkout -b databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas A usr/pkg/pgsql/archive/placeholder A usr/pkg/pgsql/data/recovery.conf Switched to a new branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas' $ git add usr/pkg/pgsql/archive/placeholder $ git add usr/pkg/pgsql/data/recovery.conf $ git add usr/pkg/pgsql/data/postgresql.conf $ git add usr/pkg/pgsql/data/pg_hba.conf $ git commit -m "create generalized dbcluster-replica role schema" [databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas 9260a88] create generalized dbcluster-replica role schema 2 files changed, 161 insertions(+) create mode 100644 usr/pkg/pgsql/archive/placeholder create mode 100644 usr/pkg/pgsql/data/recovery.conf $ git push -u origin databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas Counting objects: 9, done. Compressing objects: 100% (6/6), done. Writing objects: 100% (9/9), 2.71 KiB | 2.71 MiB/s, done. Total 9 (delta 0), reused 0 (delta 0) To ssh://192.168.100.112/usr/home/vcs/confpullexample.git * [new branch] databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas -> databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas Branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas'.
Now specialize the schema for the cluster at @ToonTown by specifying part of the hostname the machines will have in that cluster:
$ git checkout -b databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node Switched to a new branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node' $ echo "primary_conninfo = 'host=10.0.10.200 port=5432 user=replicauser application_name=toons'" >> usr/pkg/pgsql/data/recovery.conf $ git add usr/pkg/pgsql/data/recovery.conf $ git commit -m "create specialized branch for dbcluster at ToonTown dc" [databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node 90d9731] create specialized branch for dbcluster at ToonDown dc 1 file changed, 1 insertion(+) $ git push -u origin databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node Counting objects: 7, done. Compressing objects: 100% (5/5), done. Writing objects: 100% (7/7), 604 bytes | 604.00 KiB/s, done. Total 7 (delta 2), reused 0 (delta 0) To ssh://192.168.100.112/usr/home/vcs/confpullexample.git * [new branch] databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node -> databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node Branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node' set up to track remote branch 'databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node'.
Let's try to deploy it on a node!
pkghost# whoami root pkghost# hostname pkghost pkghost# cat /usr/pkg/etc/pkg_install.conf VCSTRACK_CONF=yes VCSCONFPULL=yes REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/confpullexample.git VCS=git pkghost# echo "ROLE=dbcluster-replicas" >> /usr/pkg/etc/pkg_install.conf pkghost# hostname toontowndc-node05 toontowndc-node05# #show use with pkg_add and a local package, for once toontowndc-node05# /usr/pkg/sbin/pkg_add ./postgresql10-10.3.tgz Trying to deploy configuration from ssh://vcs@192.168.100.112/usr/home/vcs/confpullexample.git via git About to use remote branch databases_postgresql10_10.4nb1_10.0_11_dbcluster-replicas_toontowndc-node Cloning into 'confpullexample'... remote: Enumerating objects: 45, done. remote: Counting objects: 100% (45/45), done. remote: Compressing objects: 100% (28/28), done. remote: Total 45 (delta 5), reused 0 (delta 0) Receiving objects: 100% (45/45), 15.23 KiB | 12.00 KiB/s, done. Resolving deltas: 100% (5/5), done. /tmp/pkgsrcdeploy-17103/work/etc/ssh/sshd_config -> /etc/ssh/sshd_config /tmp/pkgsrcdeploy-17103/work/usr/pkg/pgsql/archive/placeholder -> /usr/pkg/pgsql/archive/placeholder /tmp/pkgsrcdeploy-17103/work/usr/pkg/pgsql/data/pg_hba.conf -> /usr/pkg/pgsql/data/pg_hba.conf /tmp/pkgsrcdeploy-17103/work/usr/pkg/pgsql/data/postgresql.conf -> /usr/pkg/pgsql/data/postgresql.conf /tmp/pkgsrcdeploy-17103/work/usr/pkg/pgsql/data/recovery.conf -> /usr/pkg/pgsql/data/recovery.conf toontowndc-node05# tail /etc/ssh/sshd_config # Example of overriding settings on a per-user basis #Match User anoncvs # X11Forwarding no # AllowTcpForwarding no # PermitTTY no # ForceCommand cvs server PermitEmptyPasswords no PermitRootLogin no AllowGroup ops toontowndc-node05# tail /usr/pkg/pgsql/data/recovery.conf #--------------------------------------------------------------------------- # HOT STANDBY PARAMETERS #--------------------------------------------------------------------------- # # Hot Standby related parameters are listed in postgresql.conf # #--------------------------------------------------------------------------- standby_mode = 'on' restore_command = 'cp /usr/pkg/pgsql/archive/%f "%p%"' primary_conninfo = 'host=10.0.10.200 port=5432 user=replicauser application_name=toons' toontowndc-node05# ls /tmp/pkgsrcdeploy-17103 ls: /tmp/pkgsrcdeploy-17103: No such file or directory
The same way goes if you should choose to set VCS=hg or VCS=svn in pkg_install.conf. All are supported when pulling configuration, you should adapt to managing branches the way these other VCSs expect you to work with them. It might be in scope for this tutorial, but it's getting a bit long!
Also remember to set an appropriate URI in REMOTEVCS for these other Version Control Systems.
CVS is not supported: tags (branch names) cannot contain dots with CVS, and this breaks the chosen naming scheme for many software applications. It could be replaced with another special meaning character when using this vcs, but this workaround would make for possible confusion in branch naming. Furthermore, as with svn, there is no way to list remote branches before getting a local copy of the repository, with the added headcache of having to parse its log to extract branch names. I considered it not to be worth the hassle, but I'm open to adding PULL mode support for cvs if required by any user.
One word about subversion: I expect you to follow conventions and keep branches at /branches/ in your repository!
pkgconftrack store: storing changes when not upgrading packages
Even when merging changes in configuration files is not attempted because of VCSAUTOMERGE=no or unset, pkgsrc will keep track of installed configuration files by committing them to the configuration repository, as a user-modified file under user/$filePath (see mk/pkginstall/files).
No VCS will commit files that haven't changed since the last revision (I hope!).
But what if you made changes to a configuration file and you want to store it in the configuration repository as a manually edited file, neither waiting for its package to be updated nor forcing a reinstallation? You could interact with your vcs of choice direcly, or just use the new script in pkgsrc at pkgtools/pkgconftrack
pkgconf# cd pkgtools/pkgconftrack/ pkgconf# make install ===> Skipping vulnerability checks. [...] ===> Installing binary package of pkgconftrack-1.0 pkgconf# pkgconftrack prefix: /usr/pkg, VCSDIR: /var/confrepo, VCS: svn, REMOTEVCS: no : unknown action Usage: pkgconftrack [-p PREFIX] [-m commit message] store packagename [... packagenames]
pkgconftrack will search for pkgsrc VCS configuration on your shell environment or in pkg_install.conf, reading it via pkg_admin config-var $VARNAME.
pkg_admin, in order to work with the correct configuration file, and look in the right package database to check if a package exists and its list of configuration files, is called relative to the prefix.
By default, if unspecified, /usr/pkg is assumed and /usr/pkg/sbin/pkg_admin is executed.
You can work in packages of a different prefix by calling pkgconftrack -p /path/to/other/prefix followed by other options (you are free to chose a custom commit message with -m "my commit message") and the action to be performed.
As of now, pkgconftrack only support one action: store followed by one or more packages you wish to store configuration files from, into the configuration repository.
When storing configuration for more than one package, a unique commit is made (if using a VCS other than the old RCS, which doesn't support multi file commits and atomic transactions). This opens the way to storing config files for a service made working by a combination of software packages when these are in a known-good status, to be accessed and restored checking out one commit id.
So, let's say one last change was needed to have mail working:
pkgconf# diff /usr/pkg/etc/spamd.conf /usr/pkg/share/examples/spamd/spamd.conf 7a8 > # 33c34 < :method=https:\ --- > :method=http:\ pkgconf# pkgconftrack -m "5 august 2018: mail service: blacklist only accessible via https" store spamd postfix dovecot opendkim prefix: /usr/pkg, VCSDIR: /var/confrepo, VCS: svn, REMOTEVCS: no Storing configuration files for opendkim A opendkim.conf Package not found: dovecot in the pkgdb for /usr/pkg Package not found: postfix in the pkgdb for /usr/pkg Storing configuration files for spamd Adding usr/pkg/etc/opendkim.conf Transmitting file data .done Committing transaction... Committed revision 7. pkgconf#
Yeah, I haven't really installed a mail server just to test pkgconftrack, so neither dovecot nor postfix are installed on the system!
future improvements
Well, it all begins with the new features being tested by end users, more bugs being found, changes made as requested. Then, once things get more stable, the versioning script could be reimplemented as part of pkgtasks, and files.subr changed there to interact with the new functions.
More VCSs could be handled, there could be more automation in switching repositories and adding remote sources, but all this is maybe best kept in a separate tool such as pkgconftrack. Speaking of which, well, it stores installed config files for one or more packages at once, but it does not restore them yet! RCS needs to be supported, being the default Version Control System, but it has no concept of an atomic transaction involving more than one file, so there would be nothing the user could reference to when asking the script to restore configuration. The script could track, by checking the log for all files, for identical commit message and restore each revision of each file having the same commit message, but what if some user mistakingly reused the same commit message as part of different transactions? Should each custom commit message be prepended with a timestamp?
All this would still lead to differences in referencing to a transaction, or the simulation thereof, and in listing available changes the user can select for restore.
pkgconftrack could also help in interactively reviewing conflicting automerge results, some scripts already do it, and in restoring the last user-installed file from the repository in case of breakage. This should really consist in a cp from vcsdir/user/path/to/file /path/to/installed/file, maybe preceded by a checkout, but there is good marging for making the tool more useful.
And yes, configuraction deployment/VCSCONFPULL does not handle permissions or the creation of empty directories, executable files: I think this would require making the way users interact with the configuration repository more complex, more akin to a configuration management and monitoring software, and widen the chances for mistakes when working across branches. Any good idea of how to implement these missing bits while still keeping things simple for users?
I'd really like to see the code tried out! it's now at Github
. [0 comments]
GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 3: remote repositories (SVN and Mercurial)
In GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 2: remote repositories (git and CVS) structural changes to the configuration tracking feature of pkgsrc got introduced, along with a short walkthrough in using remote Git and CVS repositories.
This third post will introduce SVN and Mercurial support, leaving configuration deployment for the final entry. Stay tuned!
SVN and remotes
Subversion, when used locally, needs to be set this way, as you can expect:
pkghost# cat /usr/pkg/etc/pkg_install.conf VCSTRACK_CONF=yes VCS=svn VCSDIR=/var/svnconfdir VCSAUTOMERGE=yes
A local repository will be created at VCSDIR/localsvn, files will get extracted and checkedin at VCSDIR/defaults, VCSDIR/automerged, VCSDIR/user
pkghost# make
[...]
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Installing for spamd-20060330nb5
=> Generating pre-install file lists
=> Creating installation directories
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/man/spamd.conf.5 /ro5
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd.8 /root/8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-set8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb.8 /roo8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd.8 8
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-c
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd /rootc
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb /ron
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd c
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/etc/spamd.conf /root/d
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for spamd-20060330nb5
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb5.tgz
===> Building binary package for spamd-20060330nb5
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb5.tgz
===> Installing binary package of spamd-20060330nb5
REGISTER /var/svnconfdir/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb5: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf
Conf commit: pkgsrc: add spamd-20060330nb5
===========================================================================
The following files should be created for spamd-20060330nb5:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $
Don't forget to add the spamd ports to /etc/services:
spamd 8025/tcp # spamd(8)
spamd-cfg 8026/tcp # spamd(8) configuration
===========================================================================
pkghost# ls -lah /var/svnconfdir/localsvn/
total 3.2K
drwxr-xr-x 6 pkgvcsconf pkgvcsconf 512B Aug 4 07:22 .
drwx------ 6 pkgvcsconf pkgvcsconf 512B Aug 4 07:22 ..
-rw-r--r-- 1 pkgvcsconf pkgvcsconf 246B Aug 4 07:22 README.txt
drwxr-xr-x 2 pkgvcsconf pkgvcsconf 512B Aug 4 07:22 conf
drwxr-sr-x 6 pkgvcsconf pkgvcsconf 512B Aug 4 07:22 db
-r--r--r-- 1 pkgvcsconf pkgvcsconf 2B Aug 4 07:22 format
drwxr-xr-x 2 pkgvcsconf pkgvcsconf 512B Aug 4 07:22 hooks
drwxr-xr-x 2 pkgvcsconf pkgvcsconf 512B Aug 4 07:22 locks
pkghost# vi /usr/local/etc/spamd.conf
whitelist:\
:white:\
:method=file:\
:file=/var/mail/whitelist.txt:
/usr/pkg/etc/spamd.conf: 86 lines, 2767 characters
.
pkghost# make replace
[...]
and so on, you can come up with what follows. Let's test a remote repository, starting with preparations on the server:
$ hostname vers $ cd /usr/home/vcs $ svnadmin create svnremote
That's it, no branches or releases are used, so you don't need to create any further structure on the repository, the +VERSIONING script will take care of the remaining bits (you should migrate data from a local repository to the remote one, if you desire so, before running package installations).
Use a similarly structured REMOTEVCS URI: svn+ssh://vcs@192.168.100.112/usr/home/vcs/svnremote
(as with CVS, you can also run it as a standalone network server without ssh transport and authentication)
pkghost# cat /usr/pkg/etc/pkg_install.conf
VCSTRACK_CONF=yes
VCS=svn
VCSDIR=/var/svnwrkremote
VCSAUTOMERGE=yes
REMOTEVCS=svn+ssh://vcs@192.168.100.112/usr/home/vcs/svnremote
pkghost# make
[...]
pkghost# make install
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Installing for spamd-20060330nb6
=> Generating pre-install file lists
=> Creating installation directories
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/man/spamd.conf.5 /ro5
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd.8 /root/8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-set8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb.8 /roo8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd.8 8
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-c
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd /rootc
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb /ron
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd c
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/etc/spamd.conf /root/d
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for spamd-20060330nb6
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb6.tgz
===> Building binary package for spamd-20060330nb6
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb6.tgz
===> Installing binary package of spamd-20060330nb6
svn: E155007: '/var/svnwrkremote/defaults' is not a working copy
Committing transaction...
Committed revision 1.
Checked out revision 1.
Committing transaction...
Committed revision 2.
Checked out revision 2.
Committing transaction...
Committed revision 3.
Checked out revision 3.
A usr
A pkg
A etc
A spamd.conf
REGISTER /var/svnwrkremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb6: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf
Adding usr
Adding usr/pkg
Adding usr/pkg/etc
Adding usr/pkg/etc/spamd.conf
Transmitting file data .done
Committing transaction...
Committed revision 4.
Conf commit: pkgsrc: add spamd-20060330nb6
===========================================================================
The following files should be created for spamd-20060330nb6:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $
Don't forget to add the spamd ports to /etc/services:
spamd 8025/tcp # spamd(8)
spamd-cfg 8026/tcp # spamd(8) configuration
===========================================================================
pkghost#
since I didn't show it with CVS, let's try again some simulated file edits:
pkghost# tail -n 5 /usr/pkg/etc/spamd.conf
#
#whitelist:\
# :white:\
# :method=file:\
# :file=/var/mail/whitelist.txt:
pkghost# vi /usr/pkg/etc/spamd.conf; tail -n 5 /usr/pkg/etc/spamd.conf
/usr/pkg/etc/spamd.conf: 86 lines, 2767 characters
.
#
whitelist:\
:white:\
:method=file:\
:file=/var/mail/whitelist.txt:
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Replacing for spamd-20060330nb6
===> Updating using binary package of spamd-20060330nb6
/usr/bin/env /usr/pkg/sbin/pkg_add -K /var/db/pkg -U -D /root/pkgsrc/mail/spamd/work/.packages/spamd-2006033z
===========================================================================
The following users are no longer being used by spamd-20060330nb6,
and they can be removed if no other software is using them:
_spamd
===========================================================================
===========================================================================
The following groups are no longer being used by spamd-20060330nb6,
and they can be removed if no other software is using them:
_spamd
===========================================================================
===========================================================================
The following files are no longer being used by spamd-20060330nb6,
and they can be removed if no other packages are using them:
/usr/pkg/etc/spamd.conf
===========================================================================
REGISTER /var/svcwrkremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb6: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb6: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
Saving the currently user-installed revision to /var/svnwrkremote/user//usr/pkg/etc/spamd.conf
A usr
A pkg
A etc
A spamd.conf
Adding usr
Adding usr/pkg
Adding usr/pkg/etc
Adding usr/pkg/etc/spamd.conf
Transmitting file data .done
Committing transaction...
Committed revision 5.
Conf commit: pkgsrc: backup user conf before attempting merge for spamd-20060330nb6
A defaults/usr/pkg/etc/spamd.conf
Export complete.
Merged with no conflicts. installing it to /usr/pkg/etc/spamd.conf!
A usr
A pkg
A etc
A spamd.conf
A defaults/usr/pkg/etc/spamd.conf
Export complete.
Revert from the last revision of /var/svcwrkremote/user//usr/pkg/etc/spamd.conf if needed
Adding usr
Adding usr/pkg
Adding usr/pkg/etc
Adding usr/pkg/etc/spamd.conf
Transmitting file data .done
Committing transaction...
Committed revision 6.
Conf commit: pkgsrc: add spamd-20060330nb6
===========================================================================
The following files should be created for spamd-20060330nb6:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $
Don't forget to add the spamd ports to /etc/services:
spamd 8025/tcp # spamd(8)
spamd-cfg 8026/tcp # spamd(8) configuration
===========================================================================
pkghost# cd /var/svnwrkremote/
pkghost# ls
automerged automergedfiles defaults user
pkghost# cd automerged
pkghost# svn update
Updating '.':
At revision 6.
pkghost# svn log
------------------------------------------------------------------------
r6 | vcs | 2018-08-04 13:22:29 +0000 (Sat, 04 Aug 2018) | 1 line
pkgsrc: add spamd-20060330nb6
------------------------------------------------------------------------
r1 | vcs | 2018-08-04 13:19:29 +0000 (Sat, 04 Aug 2018) | 1 line
initial import
------------------------------------------------------------------------
pkghost# cd $HOME/pkgsrc/mail/spamd
pkghost# grep REVISION Makefile
PKGREVISION= 6
pkghost# sed -i "s/PKGREVISION=.*6/PKGREVISION=7/g" Makefile
pkghost# grep REVISION Makefile
PKGREVISION=7
pkghost# make clean; make extract; sed -i "s/http/https/g" work/spamd-20060330/etc/spamd.conf
pkghost# make update
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Deinstalling for spamd-20060330nb7
Running /usr/pkg/sbin/pkg_delete -K /var/db/pkg -r spamd-20060330nb6
===========================================================================
[...]
===> Installing for spamd-20060330nb7
=> Generating pre-install file lists
=> Creating installation directories
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/man/spamd.conf.5 /ro5
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd.8 /root/8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-set8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb.8 /roo8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd.8 8
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-c
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd /rootc
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb /ron
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd c
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/etc/spamd.conf /root/d
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for spamd-20060330nb7
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb7.tgz
===> Building binary package for spamd-20060330nb7
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb7.tgz
===> Installing binary package of spamd-20060330nb7
REGISTER /var/svcwrkremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb7: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb7: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
A automerged/usr/pkg/etc/spamd.conf
Export complete.
Saving the currently installed revision to /var/svcwrkremote/automerged//usr/pkg/etc/spamd.conf
Sending usr/pkg/etc/spamd.conf
Transmitting file data .done
Committing transaction...
Committed revision 7.
Conf commit: pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb7
A defaults/usr/pkg/etc/spamd.conf
Export complete.
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!
--- /usr/pkg/etc/spamd.conf 2018-08-04 09:01:09.536149671 +0000
+++ /var/svcwrkremote/defaults//usr/pkg/etc/spamd.conf.automerge 2018-08-04 09:10:52.131955252 +0000
@@ -15,7 +15,7 @@
# may be applied to each blacklist.
#
# As of November 2004, a place to search for black lists is
-# http://spamlinks.net/filter-bl.htm
+# https://spamlinks.net/filter-bl.htm
#
# Some of the URLs below point to www.openbsd.org locations. Those
# files are likely to be mirrored to other OpenBSD www mirrors located
@@ -25,45 +25,45 @@
all:\
:spews1:china:korea:
-# Mirrored from http://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2
+# Mirrored from https://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2
spamhaus:\
:black:\
:msg="SPAM. Your address %A is in the Spamhaus Block List\n\
- See http://www.spamhaus.org/sbl and\
- http://www.abuse.net/sbl.phtml?IP=%A for more details":\
- :method=http:\
+ See https://www.spamhaus.org/sbl and\
+ https://www.abuse.net/sbl.phtml?IP=%A for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/SBL.cidr.gz:
-# Mirrored from http://www.spews.org/spews_list_level1.txt
+# Mirrored from https://www.spews.org/spews_list_level1.txt
spews1:\
:black:\
:msg="SPAM. Your address %A is in the spews level 1 database\n\
- See http://www.spews.org/ask.cgi?x=%A for more details":\
- :method=http:\
+ See https://www.spews.org/ask.cgi?x=%A for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/spews_list_level1.txt.gz:
-# Mirrored from http://www.spews.org/spews_list_level2.txt
+# Mirrored from https://www.spews.org/spews_list_level2.txt
spews2:\
:black:\
:msg="SPAM. Your address %A is in the spews level 2 database\n\
- See http://www.spews.org/ask.cgi?x=%A for more details":\
- :method=http:\
+ See https://www.spews.org/ask.cgi?x=%A for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/spews_list_level2.txt.gz:
-# Mirrored from http://www.okean.com/chinacidr.txt
+# Mirrored from https://www.okean.com/chinacidr.txt
china:\
:black:\
:msg="SPAM. Your address %A appears to be from China\n\
- See http://www.okean.com/asianspamblocks.html for more details":\
- :method=http:\
+ See https://www.okean.com/asianspamblocks.html for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/chinacidr.txt.gz:
-# Mirrored from http://www.okean.com/koreacidr.txt
+# Mirrored from https://www.okean.com/koreacidr.txt
korea:\
:black:\
:msg="SPAM. Your address %A appears to be from Korea\n\
- See http://www.okean.com/asianspamblocks.html for more details":\
- :method=http:\
+ See https://www.okean.com/asianspamblocks.html for more details":\
+ :method=https:\
:file=www.openbsd.org/spamd/koreacidr.txt.gz:
#relaydb-black:\
Revert from the penultimate revision of /var/svcwrkremote/automerged//usr/pkg/etc/spamd.conf if needed
Sending usr/pkg/etc/spamd.conf
Transmitting file data .done
Committing transaction...
Committed revision 8.
Sending usr/pkg/etc/spamd.conf
Transmitting file data .done
Committing transaction...
Committed revision 9.
Conf commit: pkgsrc: add spamd-20060330nb7
===========================================================================
The following files should be created for spamd-20060330nb7:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
===========================================================================
[...]
===> Cleaning for spamd-20060330nb7
HG and remotes
Yes, mercurial.
Mercurial, like git, uses a local directory (namely .hg) located under the working directory VCSDIR, it is also used to store configuration and URIs to remote repositories.
to initialize and test it (as said, no repository and vcs migrations are supported by pkgsrc itself, you should take care of migrations yourself if you want to), just set pkg_install.conf to use a local mercurial repo and install a package:
pkghost# rm /usr/pkg/etc/spamd.conf
pkghost# /usr/pkg/sbin/pkg_delete spamd
pkghost# vi /usr/pkg/etc/pkg_install.conf; cat /usr/pkg/etc/pkg_install.conf
/usr/pkg/etc/pkg_install.conf: 4 lines, 66 characters
.
VCSTRACK_CONF=yes
VCS=hg
VCSDIR=/var/hglocaldir
VCSAUTOMERGE=yes
pkghost# make
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
=> Checksum SHA1 OK for spamd-20060330.tar.gz
[...]
To save some space and time, let's just say that it works as expected:
[...] Conf commit: pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb10 Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf! --- /usr/pkg/etc/spamd.conf 2018-08-04 09:50:00.738099463 +0000 +++ /var/hglocaldir/defaults//usr/pkg/etc/spamd.conf.automerge 2018-08-04 09:52:18.467604445 +0000 @@ -15,7 +15,7 @@ # may be applied to each blacklist. # # As of November 2004, a place to search for black lists is -# http://spamlinks.net/filter-bl.htm +# https://spamlinks.net/filter-bl.htm # [...]
One nice thing about mercurial is the simplicity enabling one to clone a local repository to a remote server. The script, when using mercurial, tries exacly that, this should succeed if the remote repository is empty.
if ${TEST} "$_REMOTE" != "no" -a "$_REMOTE" != "NO"; then
execute "hg clone . \"$_REMOTE\""
execute "hg --repository \"$_VCSDIR\" push \"$_REMOTE\""
execute "hg --repository \"$_VCSDIR\" pull \"$_REMOTE\""
fi
let's init an empty repository on the server and let pkgsrc clone the existing files over there!
$ cd $ hostname vers $ pwd /home/vcs $ hg init pkgconftest $ ls pkgconftest/ $ ls pkgconftest/.hg/ 00changelog.i requires store $
URIs for REMOTEVCS take the following format, should you choose to use ssh instead of hg server, http or other access methods that you'll find documented on official Mercurial resources, as with svn, git and cvs:
ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
pkghost# vi /usr/pkg/etc/pkg_install.conf; cat /usr/pkg/etc/pkg_install.conf
/usr/pkg/etc/pkg_install.conf: 5 lines, 127 characters
.
VCSTRACK_CONF=yes
VCS=hg
VCSDIR=/var/hglocaldir
VCSAUTOMERGE=yes
REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
pkghost# make replace
[...]
===> Updating using binary package of spamd-20060330nb10
/usr/bin/env /usr/pkg/sbin/pkg_add -K /var/db/pkg -U -D /root/pkgsrc/mail/spamd/work/.packages/spamd-2006033z
===========================================================================
The following users are no longer being used by spamd-20060330nb10,
and they can be removed if no other software is using them:
_spamd
===========================================================================
===========================================================================
The following groups are no longer being used by spamd-20060330nb10,
and they can be removed if no other software is using them:
_spamd
===========================================================================
===========================================================================
The following files are no longer being used by spamd-20060330nb10,
and they can be removed if no other packages are using them:
/usr/pkg/etc/spamd.conf
===========================================================================
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 5 changesets with 6 changes to 3 files
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
no changes found
pulling from ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
no changes found
defaults/usr/pkg/etc/spamd.conf already tracked!
REGISTER /var/hglocaldir/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb10: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb10: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
Saving the currently installed revision to /var/hglocaldir/automerged//usr/pkg/etc/spamd.conf
automerged/usr/pkg/etc/spamd.conf already tracked!
Failed to commit conf: backup preexisting conf before attempting merge for spamd-20060330nb10
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
no changes found
hg: failed to push changes to the remote repository ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!
Revert from the penultimate revision of /var/hglocaldir/automerged//usr/pkg/etc/spamd.conf if needed
Failed to commit conf: add spamd-20060330nb10
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
no changes found
hg: failed to push changes to the remote repository ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
===========================================================================
The following files should be created for spamd-20060330nb10:
/etc/rc.d/pfspamd (m=0755)
[/usr/pkg/share/examples/rc.d/pfspamd]
[...]
Yes, hg exits in error when there are no changes to be pushed. Everything is working fine.
Let's simulate some more package-provided changes in defaults
pkghost# make clean; sed -i "s/PKGREVISION=10/PKGREVISION=11/g" Makefile; make extract
pkghost# sed -i "s/http/https/g" work/spamd-20060330/etc/spamd.conf
pkghost# vi work/spamd-20060330/etc/spamd.conf; head work/spamd-20060330/etc/spamd.conf
work/spamd-20060330/etc/spamd.conf: 87 lines, 2812 characters
.
# $OpenBSD: spamd.conf,v 1.17 2006/02/01 20:22:43 dhartmei Exp $
#
# spamd config file, read by spamd-setup(8) for spamd(8)
#
# See spamd.conf(5)
#THIS IS A NEW COMMENT!
#
# Configures whitelists and blacklists for spamd
#
# Strings follow getcap(3) convention escapes, other than you
pkghost# make update
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Deinstalling for spamd-20060330nb11
Running /usr/pkg/sbin/pkg_delete -K /var/db/pkg -r spamd-20060330nb10
===========================================================================
The following users are no longer being used by spamd-20060330nb10,
and they can be removed if no other software is using them:
[...]
===> Installing for spamd-20060330nb11
[...]
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for spamd-20060330nb11
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb11.tgz
===> Building binary package for spamd-20060330nb11
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb11.tgz
===> Installing binary package of spamd-20060330nb11
abort: repository usr/home/vcs/pkgconftest already exists!
abort: could not create remote repo!
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
no changes found
pulling from ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
no changes found
defaults/usr/pkg/etc/spamd.conf already tracked!
REGISTER /var/hglocaldir/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb11: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb11: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
Saving the currently installed revision to /var/hglocaldir/automerged//usr/pkg/etc/spamd.conf
automerged/usr/pkg/etc/spamd.conf already tracked!
Conf commit: pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb11
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!
--- /usr/pkg/etc/spamd.conf 2018-08-04 10:43:37.554182167 +0000
+++ /var/hglocaldir/defaults//usr/pkg/etc/spamd.conf.automerge 2018-08-04 10:50:23.582806647 +0000
@@ -3,6 +3,7 @@
# spamd config file, read by spamd-setup(8) for spamd(8)
#
# See spamd.conf(5)
+#THIS IS A NEW COMMENT!
#
# Configures whitelists and blacklists for spamd
#
Revert from the penultimate revision of /var/hglocaldir/automerged//usr/pkg/etc/spamd.conf if needed
Conf commit: pkgsrc: add spamd-20060330nb11
pushing to ssh://vcs@192.168.100.112/usr/home/vcs/pkgconftest
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 2 changes to 2 files
===========================================================================
The following files should be created for spamd-20060330nb11:
[...]
[0 comments]