Puppetserver migration

As mentioned in Infrastructure SIG Meeting notes 2023-01-12 I’m going to start the migration of puppetmaster.theforeman.org. The plan is described at:

I’ll post here with any notes of the process and when I’m finished.

There should be no disruption for CI, Discourse, Redmine or any other services during the migration.

During the migration I ran into the problem that after the DB dump the puppetserver was not registered as a Smart Proxy. That meant Puppetserver couldn’t retrieve node definitions. I hacked around this by changing /etc/puppetlabs/puppet/node.rb on puppet01 to:

echo "{}"

Then ran puppet agent -t, which registered and placed the correct node.rbagain. Then I could run puppet onforeman01`.

After that, I proceeded with switching all hosts but forgot to copy the REX SSH private key from the old Foreman server, leading to access denied errors… After copying that I found out that my script failed:

  • puppet was not (always) in $PATH, which I worked around by using absolute paths.
  • mirror01.koji.aws.theforeman.org didn’t have REX set up; ignored at first since it’s not production yet.
  • I had a mistake in the puppet config server puppet.theforeman.org command: notice I missed set

I reran a REX task to change the Puppet config and bootstrap.

After applying that I found I forgot to unassign Puppet classes in the ENC, leading to a duplicate class declaration. On master02.jenkins I also noticed Fact 'jenkins_plugins' can exceed value length limit: 4096 · Issue #1048 · voxpupuli/puppet-jenkins · GitHub.

I also forgot to change the puppet server config in Jenkins, so the ENC had the wrong master. So I reran the setting of the server (old puppetserver was still off, so no harm done).

Then I remembered it would be smart to put all nodes in noop mode too, so changes could be spotted.

Also noticed the REX SSH key created a new entry, so we should clean up the old entry. Probably also a good chance to rotate the SSH key:

On Redmine01 I noticed that I didn’t transfer the cached data, which would have changed the Redmine DB password. To avoid a service interruption I copied the cached data (though I wonder if the password is even used, or socket auth is used).

After that I noticed ntp was on EL7 where previously chrony ran:

Additionally for Jenkins nodes the Koji certificate was passed via ENC, so if Puppet gets out of noop mode the Koji client certificate would be removed. This means that for now the Jenkins nodes are still in noop. Otherwise everything has been migrated.

A last note is that due to the Puppet 6 (Ruby 2.5) → 7 (Ruby 2.7) upgrade we now get warnings:

/opt/puppetlabs/puppet/cache/lib/puppet/type/gnupg_key.rb:91: warning: URI.escape is obsolete
/opt/puppetlabs/puppet/cache/lib/puppet/type/gnupg_key.rb:102: warning: URI.escape is obsolete
/opt/puppetlabs/puppet/cache/lib/puppet/type/gnupg_key.rb:91: warning: URI.escape is obsolete
/opt/puppetlabs/puppet/cache/lib/puppet/type/gnupg_key.rb:102: warning: URI.escape is obsolete
/opt/puppetlabs/puppet/cache/lib/puppet/type/gnupg_key.rb:91: warning: URI.escape is obsolete
/opt/puppetlabs/puppet/cache/lib/puppet/type/gnupg_key.rb:102: warning: URI.escape is obsolete

I already started on this in Modernize the module by ekohl · Pull Request #38 · dgolja/golja-gnupg · GitHub but didn’t complete it.

Now I’ll migrate the secrets. After that I’ll shut down the old server while keeping it around for a while.

Also noticed I forgot to copy the SSH key (which is only used for libvirt), but I rotated it while I was at it.