Problem with Foreman after certificates update

Support
,
1

Problem:
I updated the certificates on my foreman machine because the old one had expired. I had some problems with foreman-installer, but after many attempts it finally worked. The working command looked as follows (Instead of domain, the full domain name was entered, and there was IPv4 addres in the last line):

foreman-installer --scenario katello
–certs-server-cert “/etc/pki/sig-foreman.domain/sig-foreman.pem”
–certs-server-key “/etc/pki/sig-foreman.domain/private.key”
–certs-server-ca-cert “/etc/pki/sig-foreman.domain/ca_bundle.pem”
–foreman-proxy-ssl-ca “/etc/pki/sig-foreman.domain/ca_bundle.pem”
–foreman-proxy-ssl-cert “/etc/pki/sig-foreman.domain/sig-foreman.pem”
–foreman-proxy-ssl-key “/etc/pki/sig-foreman.domain/private.key”
–foreman-plugin-puppetdb-ssl-ca-file “/etc/pki/sig-foreman.domain/ca_bundle.pem”
–foreman-plugin-puppetdb-ssl-certificate “/etc/pki/sig-foreman.domain/sig-foreman.pem”
–foreman-plugin-puppetdb-ssl-private-key “/etc/pki/sig-foreman.domain/private.key”
–foreman-proxy-foreman-ssl-ca “/etc/pki/sig-foreman.domain/ca_bundle.pem”
–foreman-proxy-foreman-ssl-cert “/etc/pki/sig-foreman.domain/sig-foreman.pem”
–foreman-proxy-foreman-ssl-key “/etc/pki/sig-foreman.domain/private.key”
–foreman-proxy-puppet-ssl-ca “/etc/pki/sig-foreman.domain/ca_bundle.pem”
–foreman-proxy-puppet-ssl-cert “/etc/pki/sig-foreman.domain/sig-foreman.pem”
–foreman-proxy-puppet-ssl-key “/etc/pki/sig-foreman.domain/private.key”
–foreman-client-ssl-ca “/etc/pki/sig-foreman.domain/ca_bundle.pem”
–foreman-client-ssl-cert “/etc/pki/sig-foreman.domain/sig-foreman.pem”
–foreman-client-ssl-key “/etc/pki/sig-foreman.domain/private.key”
–foreman-server-ssl-ca “/etc/pki/sig-foreman.domain/ca_bundle.pem”
–foreman-server-ssl-cert “/etc/pki/sig-foreman.domain/sig-foreman.pem”
–foreman-server-ssl-key “/etc/pki/sig-foreman.domain/private.key”
–foreman-plugin-puppetdb-ssl-ca-file “/etc/pki/sig-foreman.domain/ca_bundle.pem”
–foreman-plugin-puppetdb-ssl-certificate “/etc/pki/sig-foreman.domain/sig-foreman.pem”
–foreman-plugin-puppetdb-ssl-private-key “/etc/pki/sig-foreman.domain/private.key”
–puppet-server-foreman-ssl-ca “/etc/pki/sig-foreman.domain/ca_bundle.pem”
–puppet-server-foreman-ssl-cert “/etc/pki/sig-foreman.domain/sig-foreman.pem”
–puppet-server-foreman-ssl-key “/etc/pki/sig-foreman.domain/private.key”
–certs-update-server --certs-update-server-ca
–certs-update-all
–foreman-proxy-trusted-hosts sig-foreman.domain
–foreman-proxy-trusted-hosts domain
–foreman-proxy-trusted-hosts *.domain
–foreman-proxy-trusted-hosts .domain
–foreman-proxy-trusted-hosts (foreman machine IP) --foreman-proxy-trusted-hosts 127.0.0.1

After that on one client I’ve updated package:

yum remove katello-ca-consumer-sig-foreman.domain-1.0-12.noarch
curl --insecure --output katello-ca-consumer-latest.noarch.rpm sig-foreman.domain/pub/katello-ca-consumer-latest.noarch.rpm
yum install katello-ca-consumer-latest.noarch.rpm

However when I tried to perform: subscription-manager refresh
I recived: Unable to verify server’s identity: tlsv1 alert unknown ca

Expected outcome:
Ability to upgrade machines
Foreman and Proxy versions:
foreman-release-3.6.1-1.el8.noarch
foreman-proxy-3.6.1-1.el8.noarch

Distribution and version:
Rocky Linux 8.7

2

Check this: subscription-manager commands fail with the error message: "Unable to verify server's identity: tlsv1 alert unknown ca" - Red Hat Customer Portal

3

Katello 4.8 hasn’t been officially released, yet. You should really mention, which versions you have installed exactly.

Why did you not follow the docs to update the certificate? Installing Foreman 3.6 Server with Katello 4.8 Plugin on RHEL/CentOS

With your command you have replaced all internal certificates which foreman uses for internal communication. Most of those are really only used internally and based on self-signed certs. You have changed those and I think that means that you have to change that for the communication for everything: communication between smart proxy and main server, between puppet agents and puppet server, etc. I think that’s a pretty good way to break the whole thing beyond repair.

If you just would have followed the docs it would replace the certificate for the web interface of the foreman server, i.e. the public interface, while your clients would still access the API using the self signed certs which are valid much longer.

In addition: I guess that your cert in sig-foreman.pem doesn’t have the CA flag set, i.e. it’s not allowed to issue certs but that is what foreman does and that’s why those internal certs are self signed with CA flag.

IMHO you should revert back if you can and then make the certificate update following the docs. You can simply use katello-certs-check and run the command it prints out and that updates the necessary certs which you usually access with your web browser or where the clients download the packages. That worked fine for me in the past…

4

Thank you for an answer. As you suggested I reverted my foreman vm from snapshot and update certificates as it’s described in docs.
It worked fine, but now I have a problem with my task.
They’re all stuck in pending state. I checked both production.log and proxy.log. I noticed errors in them as below:
proxy.log

2023-04-11T11:28:37 011812b6 [I] Finished POST /dynflow/tasks/6d84375f-e4d3-408e-b819-c9acfd2c08a5/cancel with 500 (1.79 ms)
2023-04-11T11:28:37 011812b6 [I] Started POST /dynflow/tasks/488bb45a-67af-4569-a4a8-a9f30ac27980/cancel
2023-04-11T11:28:37 011812b6 [W] Error processing request '011812b6-21ca-48cf-808a-938d3475136b: <KeyError>: searching: execution_plan by: {:uuid=>"488bb45a-
67af-4569-a4a8-a9f30ac27980"}

production.log

2023-04-11T11:15:05 [E|dyn|] unexpected error when invalidating execution plan 9e04ab1c-4497-42f3-a14e-460703649d51, skipping
2023-04-11T11:15:05 [W|for|011812b6] Could not load execution plan a94d425c-d88a-4773-bc54-3ec062e809cc for task 30aaae04-0635-42b6-87cf-4ee5ea030db7
2023-04-11T11:15:05 [I|for|011812b6] Backtrace for 'Could not load execution plan a94d425c-d88a-4773-bc54-3ec062e809cc for task 30aaae04-0635-42b6-87cf-4ee5ea030db7' error (Dynflow::Errors::PersistenceError): caused by Sequel::PoolTimeout: timeout: 5.0, elapsed: 5.04030008899997