Puppetserver migration

ekohl · January 13, 2023, 11:04am

As mentioned in Infrastructure SIG Meeting notes 2023-01-12 I’m going to start the migration of puppetmaster.theforeman.org. The plan is described at:

github.com/theforeman/foreman-infra

Comment by ekohl to Rebuild Foreman infrastructure

theforeman:master ← ekohl:new-foreman

What works: * Foreman is installed and operational on an IPv6-only server * Pu…ppetserver runs on a dual stack server and reports back * There's a (documented) path to bootstrap the whole infrastructure from scratch I also copied the SSH keys from the existing Puppetserver because it allowed me to verify deployment. `/usr/share/foreman/config/initializers/encryption_key.rb` was also needed. What remains to be done: * <del>Today SSH runs on a non-standard port to avoid a lot of noise. Do we want to do the same again?</del> * <del>Current server also houses the shared secret storage. Move that to puppet01 now?</del> * <del>Service name `puppet.theforeman.org` or `puppetserver.theforeman.org`?</del> Keep `puppet.` * Run the actual migration Actual migration plan: * [x] Merge https://github.com/theforeman/jenkins-jobs/pull/283 * [x] Merge this PR * [x] Stop services on puppetmaster.theforeman.org: `systemctl stop httpd foreman.service foreman.socket dynflow\* foreman-proxy puppetserver` * [x] Disable puppet on puppetmaster.theforeman.org: `puppet agent --disable 'Migrating to new infrastructure'` * [x] Copy Foreman DB * On old server: `sudo -u postgres pg_dump -Fc foreman > foreman.pg_dump` * On new Foreman server: ``` sudo /opt/puppetlabs/bin/puppet agent --disable 'Migrating to new infrastructure' sudo systemctl stop foreman.service foreman.socket dynflow\* sudo -u postgres dropdb foreman sudo -u postgres createdb --owner foreman --encoding UTF-8 --locale en_US.UTF-8 foreman sudo -u postgres pg_restore -d foreman foreman.pg_dump sudo foreman-rake db:migrate sudo foreman-rake db:seed sudo /opt/puppetlabs/bin/puppet config set --section agent environment production sudo /opt/puppetlabs/bin/puppet agent --enable sudo /opt/puppetlabs/bin/puppet agent -t ``` * [x] Change the environment on the new Puppetserver: ``` sudo /opt/puppetlabs/bin/puppet config set --section agent environment production sudo /opt/puppetlabs/bin/puppet agent -t ``` * [x] Verify the host reports correctly. * [ ] Migrate secrets * Copy `/srv/secretsgit` to new Puppetserver * Merge https://github.com/theforeman/theforeman-rel-eng/pull/171 * [x] Set up autosign records for all hosts * [x] Migrate machines by running a REX job (from the new Foreman): ```bash if [[ -f /etc/redhat-release ]] ; then yum -y remove puppet6-release yum -y install https://yum.puppet.com/puppet7-release-el-7.noarch.rpm yum -y update puppet-agent elif [[ -f /etc/debian_release ]] ; then apt -y remove puppet6-release wget https://apt.puppet.com/puppet7-release-bullseye.deb apt -y install ./puppet7-release-bullseye.deb apt update apt -y upgrade fi rm -rf /etc/puppetlabs/puppet/ssl puppet config server puppet.theforeman.org puppet ssl bootstrap puppet agent -t ``` * [x] Verify all nodes have reported in * [x] Remove autosign records * [x] Close https://github.com/theforeman/foreman-infra/issues/1685 & https://github.com/theforeman/foreman-infra/issues/1686 once it's considered stable. Now the old server can be considered old and should only remain as a fallback. It shouldn't be used by anything anymore. It's best to shut it down, but keep the old server for a while (2 - 4 weeks). Also keep the 1.24 database dump since we may want to use it for automated upgrade testing. Foreman is running on 3.1. https://github.com/theforeman/foreman-infra/issues/1676 is a separate issue, but can be done after this is done.

I’ll post here with any notes of the process and when I’m finished.

There should be no disruption for CI, Discourse, Redmine or any other services during the migration.

ekohl · January 13, 2023, 2:52pm

During the migration I ran into the problem that after the DB dump the puppetserver was not registered as a Smart Proxy. That meant Puppetserver couldn’t retrieve node definitions. I hacked around this by changing /etc/puppetlabs/puppet/node.rb on puppet01 to:

#!/bin/sh
echo "{}"

Then ran puppet agent -t, which registered and placed the correct node.rbagain. Then I could run puppet onforeman01`.

After that, I proceeded with switching all hosts but forgot to copy the REX SSH private key from the old Foreman server, leading to access denied errors… After copying that I found out that my script failed:

puppet was not (always) in $PATH, which I worked around by using absolute paths.
mirror01.koji.aws.theforeman.org didn’t have REX set up; ignored at first since it’s not production yet.
I had a mistake in the puppet config server puppet.theforeman.org command: notice I missed set

I reran a REX task to change the Puppet config and bootstrap.

After applying that I found I forgot to unassign Puppet classes in the ENC, leading to a duplicate class declaration. On master02.jenkins I also noticed Fact 'jenkins_plugins' can exceed value length limit: 4096 · Issue #1048 · voxpupuli/puppet-jenkins · GitHub.

I also forgot to change the puppet server config in Jenkins, so the ENC had the wrong master. So I reran the setting of the server (old puppetserver was still off, so no harm done).

Then I remembered it would be smart to put all nodes in noop mode too, so changes could be spotted.

Also noticed the REX SSH key created a new entry, so we should clean up the old entry. Probably also a good chance to rotate the SSH key:

github.com/theforeman/foreman-infra

Rotate REX SSH key on puppet01 and client old puppetmaster entry

opened 02:45PM - 13 Jan 23 UTC

closed 03:47PM - 13 Apr 23 UTC

ekohl

Right now I've copied the key from the old `puppetmaster.theforeman.org` machine… to `puppet01.conova.theforeman.org`. However, it's good to rotate. This can be done by moving the key to a different file location (keeping a backup for now). This should then propagate to Foreman, which Puppet will then apply automatically. After this is rotated everywhere we should purge the old ssh public key for `puppetmaster.theforeman.org`.

On Redmine01 I noticed that I didn’t transfer the cached data, which would have changed the Redmine DB password. To avoid a service interruption I copied the cached data (though I wonder if the password is even used, or socket auth is used).

After that I noticed ntp was on EL7 where previously chrony ran:

Additionally for Jenkins nodes the Koji certificate was passed via ENC, so if Puppet gets out of noop mode the Koji client certificate would be removed. This means that for now the Jenkins nodes are still in noop. Otherwise everything has been migrated.

A last note is that due to the Puppet 6 (Ruby 2.5) → 7 (Ruby 2.7) upgrade we now get warnings:

/opt/puppetlabs/puppet/cache/lib/puppet/type/gnupg_key.rb:91: warning: URI.escape is obsolete
/opt/puppetlabs/puppet/cache/lib/puppet/type/gnupg_key.rb:102: warning: URI.escape is obsolete
/opt/puppetlabs/puppet/cache/lib/puppet/type/gnupg_key.rb:91: warning: URI.escape is obsolete
/opt/puppetlabs/puppet/cache/lib/puppet/type/gnupg_key.rb:102: warning: URI.escape is obsolete
/opt/puppetlabs/puppet/cache/lib/puppet/type/gnupg_key.rb:91: warning: URI.escape is obsolete
/opt/puppetlabs/puppet/cache/lib/puppet/type/gnupg_key.rb:102: warning: URI.escape is obsolete

I already started on this in Modernize the module by ekohl · Pull Request #38 · dgolja/golja-gnupg · GitHub but didn’t complete it.

Now I’ll migrate the secrets. After that I’ll shut down the old server while keeping it around for a while.

ekohl · January 13, 2023, 3:45pm

Also noticed I forgot to copy the SSH key (which is only used for libvirt), but I rotated it while I was at it.