Katello upgrade task failed during ssl certificate update

jowig · November 20, 2024, 3:15pm

Problem:
When updating the ssl certificates on the foreman server (also smart proxy) with the foreman-installer command the error “failed upgrade task: katello:correct_repositories, see logs for more information”. The ssl certificate is an officially signed certificate an we provided a valid ca chain during installation. Finally the foreman web interface is working with the new certificate.

Expected outcome:
forman-installer will finish without errors.

Foreman and Proxy versions:
foreman-3.12.0-1.el9.noarch
katello-4.14.0-1.el9.noarch

Foreman and Proxy plugin versions:
foreman-proxy-3.12.0-1.el9.noarch

Distribution and version:
Rocky Linux 9 x86_64

Other relevant data:
foreman-installer command

foreman-installer --scenario katello \
--certs-server-cert "/etc/ssl/intra.company.local.crt" \
--certs-server-key "/etc/ssl/intra.company.local.key" \
--certs-server-ca-cert "/etc/ssl/globalsign_r6_alphassl_ca_chain.crt" \
--certs-update-server \
--certs-update-server-ca

Installer output

2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Using /usr/share/foreman-installer/parser_cache/katello.yaml cache with parsed modules"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/boot/01-kafo-hook-extensions.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/boot/02-message-helpers.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/boot/03-foreman-maintain-extensions.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/boot/04-services.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/boot/05-environment.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/boot/06-postgresql-upgrade-extensions.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/boot/09-version_locking.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/boot/10-reset_data.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/boot/11-detailed_exitcodes.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/boot/13-tuning.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/boot/20-certs_update.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre_validations/01-reset_data.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre_validations/12-check_certs_tar.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre_validations/30-el8_upgrade_postgresql.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre_validations/34-pulpcore_directory_layout.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre_commit/05-puppet_certs_exist.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre_commit/09-version_locking.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre_commit/13-tuning.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre_commit/20-certs_update.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre_commit/33-pulpcore_assets_permissions.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre/10-reset_data.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre/20-certs_update.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre/20-check-hammer-credentials.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre/25-remove_apache_from_foreman_group.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre/30-el8_upgrade_postgresql.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre/31-puppet_agent_oauth.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre/31-puppet_puppet_server_invalid_java.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre/31-puppet_server_migrate_ca.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre/32-install_selinux_packages.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre/33-pulpcore_assets_permissions.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre/34-pulpcore_directory_layout.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/post/30-upgrade.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/post/34-pulpcore_directory_layout.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/post/99-post_install_message.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/post/99-version_locking.rb"]
2024-11-20 15:48:41 [DEBUG ] [root] <Array> ["Loading hook /usr/share/foreman-installer/hooks/pre_exit/20-certs_regenerate.rb"]

...

2024-11-20 15:51:22 [NOTICE] [configure] System configuration has finished.
2024-11-20 15:51:22 [INFO  ] [post] Executing hooks in group post
Executing: foreman-rake upgrade:run
2024-11-20 15:51:22 [DEBUG ] [root] Executing: foreman-rake upgrade:run
=============================================
2024-11-20 15:51:53 [DEBUG ] [root] =============================================
Upgrade Step 1/2: katello:correct_repositories. This may take a long while.
2024-11-20 15:51:53 [DEBUG ] [root] Upgrade Step 1/2: katello:correct_repositories. This may take a long while.
Processing Repository 1/614: Rocky Linux 9 BaseOS (9)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 1/614: Rocky Linux 9 BaseOS (9)
Processing Repository 2/614: Rocky Linux 9 AppStream (10)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 2/614: Rocky Linux 9 AppStream (10)
Processing Repository 3/614: Rocky Linux 9 Extras (11)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 3/614: Rocky Linux 9 Extras (11)
Processing Repository 4/614: Rocky Linux 9 Foreman Client (12)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 4/614: Rocky Linux 9 Foreman Client (12)
Processing Repository 5/614: EPEL 9 (13)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 5/614: EPEL 9 (13)
Processing Repository 6/614: Rocky Linux 8 Base OS (14)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 6/614: Rocky Linux 8 Base OS (14)
Processing Repository 7/614: Rocky Linux 8 AppStream (15)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 7/614: Rocky Linux 8 AppStream (15)
Processing Repository 8/614: Rocky Linux 8 Extras (16)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 8/614: Rocky Linux 8 Extras (16)
Processing Repository 9/614: Rocky Linux 8 Foreman Client (17)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 9/614: Rocky Linux 8 Foreman Client (17)
Processing Repository 10/614: EPEL 8 (18)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 10/614: EPEL 8 (18)

...

2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 146/614: Remi 9 php 7.4 x86_64 (254)
Processing Repository 147/614: Remi 9 x86_64 (255)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 147/614: Remi 9 x86_64 (255)
Processing Repository 148/614: Percona MySQL Xtrabackup Tools EL9 x86_64 (256)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 148/614: Percona MySQL Xtrabackup Tools EL9 x86_64 (256)
Processing Repository 149/614: EPEL 9 (257)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 149/614: EPEL 9 (257)
Processing Repository 150/614: Rocky Linux 9 BaseOS (258)
2024-11-20 15:51:53 [DEBUG ] [root] Processing Repository 150/614: Rocky Linux 9 BaseOS (258)
Failed upgrade task: katello:correct_repositories, see logs for more information.
2024-11-20 15:51:53 [DEBUG ] [root] Failed upgrade task: katello:correct_repositories, see logs for more information.
=============================================
2024-11-20 15:51:53 [DEBUG ] [root] =============================================
Upgrade Step 2/2: katello:clean_backend_objects. This may take a long while.
2024-11-20 15:51:53 [DEBUG ] [root] Upgrade Step 2/2: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
2024-11-20 15:51:53 [DEBUG ] [root] 0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []
2024-11-20 15:51:53 [DEBUG ] [root] Candlepin orphaned consumers: []
2024-11-20 15:51:53 [DEBUG ] [post] Hook /usr/share/foreman-installer/hooks/post/30-upgrade.rb returned nil
2024-11-20 15:51:53 [DEBUG ] [post] Hook /usr/share/foreman-installer/hooks/post/34-pulpcore_directory_layout.rb returned nil
  Success!
  * Foreman is running at https://foreman.intra.company.local
  * To install an additional Foreman proxy on separate machine continue by running:

      foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar.gz"
  * Foreman Proxy is running at https://foreman.intra.company.local:9090

The full log is at /var/log/foreman-installer/katello.log
2024-11-20 15:51:53 [DEBUG ] [post] Hook /usr/share/foreman-installer/hooks/post/99-post_install_message.rb returned nil
2024-11-20 15:51:53 [DEBUG ] [post] Hook /usr/share/foreman-installer/hooks/post/99-version_locking.rb returned nil
2024-11-20 15:51:53 [INFO  ] [post] All hooks in group post finished
2024-11-20 15:51:53 [DEBUG ] [pre_exit] Hook /usr/share/foreman-installer/hooks/pre_exit/20-certs_regenerate.rb returned nil
2024-11-20 15:51:53 [DEBUG ] [root] Exit with status code: 2 (signal was 2)
2024-11-20 15:51:53 [DEBUG ] [root] Cleaning /tmp/kafo_installation20241120-2296962-hdfgs1
2024-11-20 15:51:53 [DEBUG ] [root] Cleaning /tmp/kafo_installation20241120-2296962-oy3ivz
2024-11-20 15:51:53 [DEBUG ] [root] Cleaning /tmp/default_values.yaml
2024-11-20 15:51:53 [DEBUG ] [root] Installer finished in 185.969898247 seconds

Additionally there are error messages when visiting the smart proxy page.

Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-7885 [ProxyAPI::ProxyException]: Unable to fetch logs ([Errno::ECONNRESET]: Connection reset by peer) for proxy https://foreman.intra.company.local:9090/logs)

Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-7885 [ProxyAPI::ProxyException]: Unable to fetch logs ([OpenSSL::SSL::SSLError]: SSL_read: tlsv1 alert unknown ca) for proxy https://foreman.intra.company.local:9090/logs)

How can we fix this?

Thanks Jorg

iballou · November 25, 2024, 8:04pm

Hi @jowig ,

Are there any relevant logs in /var/log/foreman-installer, or maybe /var/log/foreman that you could show us?

It looks like correct_repositories worked for a number of repos but then failed on one of them. I’m curious if there was a better log output for the repository that failed.

Another option would be to run the task directly to see if any more information is shown:

foreman-rake katello:correct_repositories

If you’re really stuck, there’s a decent chance that step could be hacked out without causing issue. However, I am curious why it’s failing, since it could be related to the cert change, or potentially a broken repository.

jowig · November 26, 2024, 2:04pm

Thanks for your response.

it all started when i just wanted to renew the ssl certificate used by forman, using the following comand

jowig:

foreman-installer --scenario katello \
--certs-server-cert "/etc/ssl/intra.company.local.crt" \
--certs-server-key "/etc/ssl/intra.company.local.key" \
--certs-server-ca-cert "/etc/ssl/globalsign_r6_alphassl_ca_chain.crt" \
--certs-update-server \
--certs-update-server-ca

After running the command the error occured:

Failed upgrade task: katello:correct_repositories
Errors on smart proxy page (See Screenshot)
Clients cannot pull RPM Packages ( - Curl error (60): SSL peer certificate or SSH remote key was not OK for https://foreman.intra…/repomd.xml [SSL certificate problem: self-signed certificate in certificate chain])

We want to use the same certificate on the foreman server, katallo and smart proxy. Puppet uses its own certificate. The certificate is an officially signed certificate (key, cert and chain containing intermediate and root certificate).

What would the the command to configure the server using this certificates. We don’t want to update every katello client to use the repos after a certifiicate update…

Thanks
Jorg