The idea was to add more SAN names to the self-signed certificate.
The update (using foreman-installer --scenario katello --certs-update-server --certs-update-server-ca) went wrong, and Foreman now doesn’t work anymore as a result. The webserver doesn’t respond at all.
It seems that one of the main errors is:
A smart proxy seems to have been refreshed without pulpcore being running. Please refresh the smart proxy after ensuring that pulpcore services are running.
or in more detail:
# foreman-installer --scenario katello --certs-server-cert "/root/foreman_cert/foreman_ssl_cert.pem" --certs-server-key "/root/foreman_cert/foreman_ssl_key.pem" --certs-server-ca-cert "/root/foreman_cert/foreman_ssl_cert.pem" --certs-update-server --certs-update-server-ca
2023-02-02 15:56:06 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-02-02 15:56:08 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-02-02 15:56:08 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
Marking certificate /root/ssl-build/foreman.company.org/foreman.company.org-apache for update
Marking certificate /root/ssl-build/foreman.company.org/foreman.company.org-foreman-proxy for update
Marking certificate /root/ssl-build/katello-server-ca for update
2023-02-02 15:56:13 [NOTICE] [configure] Starting system configuration.
2023-02-02 15:56:22 [NOTICE] [configure] 250 configuration steps out of 1383 steps complete.
2023-02-02 15:56:25 [NOTICE] [configure] 500 configuration steps out of 1385 steps complete.
2023-02-02 15:56:28 [NOTICE] [configure] 750 configuration steps out of 1390 steps complete.
2023-02-02 15:56:28 [NOTICE] [configure] 1000 configuration steps out of 1394 steps complete.
2023-02-02 15:57:01 [ERROR ] [configure] Systemd start for foreman failed!
2023-02-02 15:57:01 [ERROR ] [configure] journalctl log for foreman:
2023-02-02 15:57:01 [ERROR ] [configure] -- Logs begin at Thu 2023-02-02 15:39:32 CET, end at Thu 2023-02-02 15:57:01 CET. --
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:56:53 foreman.company.org systemd[1]: Starting Foreman...
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:56:56 foreman.company.org foreman[17803]: => Booting Puma
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:56:56 foreman.company.org foreman[17803]: => Rails 6.1.7 application starting in production
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:56:56 foreman.company.org foreman[17803]: => Run `bin/rails server --help` for more startup options
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:57:00 foreman.company.org foreman[17803]: Exiting
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:57:00 foreman.company.org foreman[17803]: /usr/share/gems/gems/katello-4.7.1/app/services/katello/repository_type_manager.rb:29:in `fix_pulp3_capabilities': A smart proxy seems to have been refreshed without pulpcore being running. Please refresh the smart proxy after ensuring that pulpcore services are running. (Katello::Errors::PulpcoreMissingCapabilities)
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:57:00 foreman.company.org foreman[17803]: from /usr/share/gems/gems/katello-4.7.1/app/services/katello/repository_type_manager.rb:35:in `enabled_repository_types'
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:57:00 foreman.company.org foreman[17803]: from /usr/share/gems/gems/katello-4.7.1/app/services/katello/repository_type_manager.rb:79:in `generic_content_types'
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:57:00 foreman.company.org foreman[17803]: from /usr/share/gems/gems/katello-4.7.1/app/controllers/katello/api/v2/generic_content_units_controller.rb:5:in `block in <class:GenericContentUnitsController>'
In the /var/log/foreman-proxy/proxy.log, I can see lots of errors:
2023-02-03T11:05:58 [I] WEBrick::HTTPServer#start: pid=1093 port=9090
2023-02-03T11:05:58 [I] Smart proxy has launched on 1 socket(s), waiting for requests
2023-02-03T11:06:19 9dcee2a2 [I] Started GET /v2/features
2023-02-03T11:06:19 b60eef43 [I] Started GET /v2/features
2023-02-03T11:06:19 b60eef43 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 b60eef43 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 9dcee2a2 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 b60eef43 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 9dcee2a2 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 9dcee2a2 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 b60eef43 [I] Finished GET /v2/features with 200 (19.42 ms)
2023-02-03T11:06:19 b60eef43 [I] Finished GET /v2/features with 200 (19.42 ms)
2023-02-03T11:06:19 b60eef43 [I] Finished GET /v2/features with 200 (19.42 ms)
2023-02-03T11:06:19 9dcee2a2 [I] Finished GET /v2/features with 200 (19.7 ms)
2023-02-03T11:06:19 9dcee2a2 [I] Finished GET /v2/features with 200 (19.7 ms)
2023-02-03T11:06:19 9dcee2a2 [I] Finished GET /v2/features with 200 (19.7 ms)
2023-02-03T11:06:19 0aa0bf82 [I] Started GET /v2/features
2023-02-03T11:06:19 0aa0bf82 [I] Started GET /v2/features
2023-02-03T11:06:19 0aa0bf82 [I] Started GET /v2/features
2023-02-03T11:06:19 0aa0bf82 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 0aa0bf82 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 0aa0bf82 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 0aa0bf82 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
The log fills very quickly (10-20GB per day) and the disk volume is filled up 100%.
Having a software which can fill the log space so quickly is a risk on its own.
Is there a foreman-native way to truncate the log, or should we configure logrotate to protect the system?
2023-02-17 15:05:16 [NOTICE] [configure] 1250 configuration steps out of 1426 steps complete.
2023-02-17 15:05:17 [ERROR ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-vmforeman-dev.dev-farm.something.net]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://vmforeman-dev.dev-farm.something.net/api/v2/hosts?search=name%3D"vmforeman-dev.dev-farm.something.net"
2023-02-17 15:05:17 [ERROR ] [configure] Wrapped exception:
2023-02-17 15:05:17 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2023-02-17 15:05:24 [NOTICE] [configure] System configuration has finished.
There were errors detected during install.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.