Problem:
Unable to upgrade to Katello 4.13
Expected outcome:
Successful upgrade of server and proxies
Foreman and Proxy versions:
Foreman 3.10
Katello 4.12
Foreman and Proxy plugin versions:
foreman-tasks 9.1.1
foreman_discovery 24.0.1
foreman_remote_execution 13.0.0
katello 4.12.1
Distribution and version:
Red Hat Enterprise Linux release 8.10 (Ootpa)
Other relevant data:
foreman-installer --scenario katello
–certs-server-cert “/etc/pki/tls/certs/katello.example.com.crt”
–certs-server-key “/etc/pki/tls/private/katello.example.com.key”
–certs-server-ca-cert “/etc/pki/tls/certs/minimal-ca-bundle.pem”
–certs-update-server --certs-update-server-ca
2024-10-02 15:01:07 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-10-02 15:01:10 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-10-02 15:01:10 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
Marking certificate /root/ssl-build/katello.example.com/katello.example.com-apache for update
Marking certificate /root/ssl-build/katello.example.com/katello.example.com-foreman-proxy for update
Marking certificate /root/ssl-build/katello-server-ca for update
2024-10-02 15:01:18 [NOTICE] [configure] Starting system configuration.
2024-10-02 15:01:31 [NOTICE] [configure] 250 configuration steps out of 1483 steps complete.
2024-10-02 15:01:34 [NOTICE] [configure] 500 configuration steps out of 1485 steps complete.
2024-10-02 15:01:37 [NOTICE] [configure] 750 configuration steps out of 1487 steps complete.
2024-10-02 15:01:38 [NOTICE] [configure] 1000 configuration steps out of 1493 steps complete.
2024-10-02 15:02:02 [NOTICE] [configure] 1250 configuration steps out of 1493 steps complete.
2024-10-02 15:02:10 [ERROR ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-katello.example.com]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: excessive message size in get request to: https://katello.example.com/api/v2/hosts?search=name%3D"katello.example.com"
2024-10-02 15:02:10 [ERROR ] [configure] Wrapped exception:
2024-10-02 15:02:10 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: excessive message size
2024-10-02 15:02:12 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-katello.example.com]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: excessive message size in get request to: https://katello.example.com/api/v2/hosts?search=name%3D"katello.example.com"
2024-10-02 15:02:12 [ERROR ] [configure] Wrapped exception:
2024-10-02 15:02:12 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: excessive message size
2024-10-02 15:02:12 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[katello.example.com]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: excessive message size in get request to: https://katello.example.com/api/v2/smart_proxies?search=name%3D"katello.example.com"
2024-10-02 15:02:12 [ERROR ] [configure] Wrapped exception:
2024-10-02 15:02:12 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: excessive message size
2024-10-02 15:02:12 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[katello.example.com]: Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: excessive message size in get request to: https://katello.example.com/api/v2/smart_proxies?search=name%3D"katello.example.com"
2024-10-02 15:02:12 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[katello.example.com]: Exception SSL_connect returned=1 errno=0 state=error: excessive message size in get request to: https://katello.example.com/api/v2/smart_proxies?search=name%3D"katello.example.com"
2024-10-02 15:02:12 [ERROR ] [configure] Wrapped exception:
2024-10-02 15:02:12 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: excessive message size
2024-10-02 15:02:14 [NOTICE] [configure] System configuration has finished.
Typically, when this has happened when there are too many certs in the CA bundle. I had only 16 certs which I then limited to only Entrust certs as we use Entrust as our CA. So, that reduced it down to 6. I have also run “katello-certs-check -t foreman”, and there are no failures; “Validation succeeded”. So, something else is causing the excessive message size error.
The ca file should contain the chain for the server certificate including the root. That’s should require 16 certs and not even 6. Why do you need 6? Are you sure?
I tried with just the root and intermediate, but still was having the issue. I eventually had success. What I noticed was that something was causing issues with the certs on the web server during the install. I’m sure that during the install, since we start with everything off, the web server is being configured and brought up.
What I found was that when I got this error, if I checked the web interface, it said the site wasn’t secure, and I got this error: “Error code: SSL_ERROR_RX_MALFORMED_HANDSHAKE”. So, I tried running “foreman-maintain service restart” and running the installer again. This was successful. So, it appears that their is something not working right with restarting httpd the first go around.
Also, I frequently find that nearly 150 CA are added to the ca bundle whenever I run the installer. I usually have to clear all the added ones from the file, and rerun. For some reason, it doesn’t happen on the second run.
Did you check with katello-certs-check that it’s all O.K.?
The file to --certs-server-ca-cert should be the chain, starting with the issuing ca of the server cert, continuing the intermediates ending with the root ca cert.
If some other certs/ca pop up sometimes, I would say it’s because at some point in the past you have configured different files using some of the other cert options. As foreman-installer remembers all previous settings in its answer files, omitting an option does not undo a configuration done before…
You could check the answers file /etc/foreman-installer/scenarios.d/ for some of those changes or with foreman-installer --full-help. Usually, the various certificates, keys and chains are all stored in /root/ssl-build. If you look there, maybe you’ll find the file containing those 150 CAs. If not, check the foreman-installer log. It should somewhere there explain why it did what it did.
Yes, I ran that prior to the install, and everything was “OK” and verification succeeded.