Expired Certs and Kat3.17 Foreman 2.2.0

Problem:
My certificates went out during - I generated new pairs and i try to change them in Foreman/Katello scenario and trying to swap them for working:

katello-certs-check -c /root/foreman_cert_20/foreman.cer -k /root/foreman_cert_20/serverkey.key -b /root/foreman_cert_20/foreman.pem

Checking server certificate encoding:
[OK]

Checking expiration of certificate:
[OK]

Checking expiration of CA bundle:
[OK]

Checking if server certificate has CA:TRUE flag
[OK]

Checking for private key passphrase:
[OK]

Checking to see if the private key matches the certificate:
[OK]

Checking CA bundle against the certificate file:
[OK]

Checking CA bundle size:
[OK]

Checking Subject Alt Name on certificate
[OK]

Checking Key Usage extension on certificate for Key Encipherment
[OK]

Validation succeeded

To install the Katello server with the custom certificates, run:

foreman-installer --scenario katello \
                  --certs-server-cert "/root/foreman_cert_20/foreman.cer" \
                  --certs-server-key "/root/foreman_cert_20/serverkey.key" \
                  --certs-server-ca-cert "/root/foreman_cert_20/foreman.pem"

To update the certificates on a currently running Katello installation, run:

foreman-installer --scenario katello \
                  --certs-server-cert "/root/foreman_cert_20/foreman.cer" \
                  --certs-server-key "/root/foreman_cert_20/serverkey.key" \
                  --certs-server-ca-cert "/root/foreman_cert_20/foreman.pem" \
                  --certs-update-server --certs-update-server-ca

To use them inside a NEW $FOREMAN_PROXY, rerun this command with -t foreman-proxy

foreman-installer --scenario katello \
                  --certs-server-cert "/root/foreman_cert_20/foreman.cer" \
                  --certs-server-key "/root/foreman_cert_20/serverkey.key" \
                  --certs-server-ca-cert "/root/foreman_cert_20/foreman.pem" \
                  --certs-update-server --certs-update-server-ca

Expected outcome:
Installing Done [100%] […]
Executing: foreman-rake upgrade:run

Upgrade Step 1/3: katello:correct_repositories. This may take a long while.

Processing Repository 31/90: x86_64 (45)
Processing Repository 32/90: Ambari 2.7.1.0 (48)
Processing Repository 33/90: vflov 0.6.5 (52)
Failed upgrade task: katello:correct_repositories, see logs for more information.

Upgrade Step 2/3: katello:correct_puppet_environments. This may take a long while.

Upgrade Step 3/3: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers:
0 orphaned consumer id(s) found in pulp.
Pulp orphaned consumers:
foreman-rake upgrade:run finished successfully!
Success!

After this - Certs are changed - but foreman clients did not see any content.

[root@vmsvlforeman ~]# grep -i error /var/log/foreman-installer/katello.log

[ERROR 2020-12-11T12:09:57 main] foreman-maintain packages is-locked --assumeyes failed! Check the output for error!
[DEBUG 2020-12-11T12:10:04 main] Facter: Error reading file: No such file or directory
[DEBUG 2020-12-11T12:10:04 main] -D DEFAULT_ERRORLOG=“logs/error_log”
[DEBUG 2020-12-11T12:10:04 main] -D DEFAULT_ERRORLOG=“logs/error_log”
[DEBUG 2020-12-11T12:10:04 main] Error Detecting Method: None
[DEBUG 2020-12-11T12:10:04 main] Error Correcting Capabilities:
[DEBUG 2020-12-11T12:10:04 main] Enabled Error Correcting Capabilities:
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Status: OK
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Error Correction Type: Unknown
[DEBUG 2020-12-11T12:10:04 main] Descriptor 1: POST error
[DEBUG 2020-12-11T12:10:04 main] Descriptor 2: Single-bit ECC memory error
[DEBUG 2020-12-11T12:10:04 main] Descriptor 3: Multi-bit ECC memory error

[ERROR 2020-12-11T12:13:17 main] Errors encountered during run:
[ERROR 2020-12-11T12:13:17 main] foreman-maintain packages is-locked --assumeyes failed! Check the output for error!

Foreman and Proxy versions:
Foreman version: 2.2.0

Foreman and Proxy plugin versions:
Foreman version: 2.2.0
Plugins:

  • foreman-tasks 3.0.1
  • foreman_ansible 6.0.0
  • foreman_docker 5.0.0
  • foreman_openscap 4.0.4
  • foreman_remote_execution 4.1.0
  • foreman_virt_who_configure 0.5.3
  • katello 3.17.0

Distribution and version:
RHEL 7.9

Other relevant data:

Looking at your output, it appears almost like you had some memory issues during the run? e.g. [DEBUG 2020-12-11T12:10:04 main] Descriptor 3: Multi-bit ECC memory error

Not Physical server - virtual in vmware farm. I may migrate our foreman to other physical host, a try there. but been there - done that. :frowning: