Foreman broken after updating to new self-signed certificates

ToniF · February 2, 2023, 3:27pm

Yesterday, I have installed a new Foreman server, and I am relatively new to Foreman.

I tried to replace Foreman’s existing self-signed certificate using these instructions:
https://docs.theforeman.org/3.4/Installing_Server/index-katello.html#Configuring_Server_with_a_Custom_SSL_Certificate_foreman

The idea was to add more SAN names to the self-signed certificate.

The update (using foreman-installer --scenario katello --certs-update-server --certs-update-server-ca) went wrong, and Foreman now doesn’t work anymore as a result. The webserver doesn’t respond at all.

It seems that one of the main errors is:

A smart proxy seems to have been refreshed without pulpcore being running. Please refresh the smart proxy after ensuring that pulpcore services are running.

or in more detail:

# foreman-installer --scenario katello --certs-server-cert "/root/foreman_cert/foreman_ssl_cert.pem" --certs-server-key "/root/foreman_cert/foreman_ssl_key.pem" --certs-server-ca-cert "/root/foreman_cert/foreman_ssl_cert.pem" --certs-update-server --certs-update-server-ca
2023-02-02 15:56:06 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-02-02 15:56:08 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-02-02 15:56:08 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
Marking certificate /root/ssl-build/foreman.company.org/foreman.company.org-apache for update
Marking certificate /root/ssl-build/foreman.company.org/foreman.company.org-foreman-proxy for update
Marking certificate /root/ssl-build/katello-server-ca for update
2023-02-02 15:56:13 [NOTICE] [configure] Starting system configuration.
2023-02-02 15:56:22 [NOTICE] [configure] 250 configuration steps out of 1383 steps complete.
2023-02-02 15:56:25 [NOTICE] [configure] 500 configuration steps out of 1385 steps complete.
2023-02-02 15:56:28 [NOTICE] [configure] 750 configuration steps out of 1390 steps complete.
2023-02-02 15:56:28 [NOTICE] [configure] 1000 configuration steps out of 1394 steps complete.
2023-02-02 15:57:01 [ERROR ] [configure] Systemd start for foreman failed!
2023-02-02 15:57:01 [ERROR ] [configure] journalctl log for foreman:
2023-02-02 15:57:01 [ERROR ] [configure] -- Logs begin at Thu 2023-02-02 15:39:32 CET, end at Thu 2023-02-02 15:57:01 CET. --
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:56:53 foreman.company.org systemd[1]: Starting Foreman...
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:56:56 foreman.company.org foreman[17803]: => Booting Puma
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:56:56 foreman.company.org foreman[17803]: => Rails 6.1.7 application starting in production
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:56:56 foreman.company.org foreman[17803]: => Run `bin/rails server --help` for more startup options
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:57:00 foreman.company.org foreman[17803]: Exiting
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:57:00 foreman.company.org foreman[17803]: /usr/share/gems/gems/katello-4.7.1/app/services/katello/repository_type_manager.rb:29:in `fix_pulp3_capabilities': A smart proxy seems to have been refreshed without pulpcore being running. Please refresh the smart proxy after ensuring that pulpcore services are running. (Katello::Errors::PulpcoreMissingCapabilities)
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:57:00 foreman.company.org foreman[17803]:         from /usr/share/gems/gems/katello-4.7.1/app/services/katello/repository_type_manager.rb:35:in `enabled_repository_types'
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:57:00 foreman.company.org foreman[17803]:         from /usr/share/gems/gems/katello-4.7.1/app/services/katello/repository_type_manager.rb:79:in `generic_content_types'
2023-02-02 15:57:01 [ERROR ] [configure] Feb 02 15:57:00 foreman.company.org foreman[17803]:         from /usr/share/gems/gems/katello-4.7.1/app/controllers/katello/api/v2/generic_content_units_controller.rb:5:in `block in <class:GenericContentUnitsController>'

Foreman and Proxy versions:
3.5.1

Foreman and Proxy plugin versions:
Katello 4.7.1

Distribution and version:
CentOS 8 Stream

How to I get Foreman back running?

ToniF · February 3, 2023, 11:13am

In the /var/log/foreman-proxy/proxy.log, I can see lots of errors:

2023-02-03T11:05:58  [I] WEBrick::HTTPServer#start: pid=1093 port=9090
2023-02-03T11:05:58  [I] Smart proxy has launched on 1 socket(s), waiting for requests
2023-02-03T11:06:19 9dcee2a2 [I] Started GET /v2/features 
2023-02-03T11:06:19 b60eef43 [I] Started GET /v2/features 
2023-02-03T11:06:19 b60eef43 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 b60eef43 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 9dcee2a2 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 b60eef43 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 9dcee2a2 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 9dcee2a2 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 b60eef43 [I] Finished GET /v2/features with 200 (19.42 ms)
2023-02-03T11:06:19 b60eef43 [I] Finished GET /v2/features with 200 (19.42 ms)
2023-02-03T11:06:19 b60eef43 [I] Finished GET /v2/features with 200 (19.42 ms)
2023-02-03T11:06:19 9dcee2a2 [I] Finished GET /v2/features with 200 (19.7 ms)
2023-02-03T11:06:19 9dcee2a2 [I] Finished GET /v2/features with 200 (19.7 ms)
2023-02-03T11:06:19 9dcee2a2 [I] Finished GET /v2/features with 200 (19.7 ms)
2023-02-03T11:06:19 0aa0bf82 [I] Started GET /v2/features 
2023-02-03T11:06:19 0aa0bf82 [I] Started GET /v2/features 
2023-02-03T11:06:19 0aa0bf82 [I] Started GET /v2/features 
2023-02-03T11:06:19 0aa0bf82 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 0aa0bf82 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 0aa0bf82 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)
2023-02-03T11:06:19 0aa0bf82 [E] Could not fetch capabilities: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate)

The log fills very quickly (10-20GB per day) and the disk volume is filled up 100%.
Having a software which can fill the log space so quickly is a risk on its own.
Is there a foreman-native way to truncate the log, or should we configure logrotate to protect the system?

ToniF · February 8, 2023, 2:19pm

I wasn’t able to find a solution.
As this host was relatively new, I chose to re-install it from scratch.

ogghi · February 22, 2023, 2:44pm

OK, same issue here, can’t figure it out. Restored from a clean snapshot to start fresh with:

foreman-installer
–foreman-ssl=“true”
–enable-foreman-plugin-discovery
–enable-foreman-plugin-ansible
–enable-foreman-cli-ansible
–enable-foreman-proxy
–foreman-server-ssl-port=“443”
–foreman-server-ssl-protocol=“TLSv1.3”
–foreman-server-ssl-ca="/etc/puppetlabs/puppet/ssl/ca/ca_crt.pem"
–foreman-server-ssl-key="/etc/puppetlabs/puppet/ssl/private_keys/vmforeman-dev.dev-farm.something.net.pem"
–foreman-server-ssl-cert="/etc/puppetlabs/puppet/ssl/certs/vmforeman-dev.dev-farm.something.net.pem"
–foreman-server-ssl-chain="/etc/puppetlabs/puppet/ssl/certs/vmforeman-dev.dev-farm.something.net.pem"
–foreman-proxy-tftp=“true”
–foreman-proxy-tftp-servername=“10.1.0.1”
–foreman-proxy-dhcp=“true”
–foreman-proxy-dhcp-interface=“ens160”
–foreman-proxy-dhcp-gateway=“10.1.0.1”
–foreman-proxy-dhcp-range=“10.1.0.160 10.1.0.250”
–foreman-proxy-dhcp-nameservers=“10.1.0.1”
–foreman-proxy-dhcp-key-name=“dev-farm-omapi”
–foreman-proxy-dhcp-key-secret=“rLVpx8abtJ6GKGZxZ/J3zQ==”
–foreman-proxy-dns=“true”
–foreman-proxy-dns-interface=“ens160”
–foreman-proxy-dns-zone=“dev-farm.something.net”
–foreman-proxy-dns-reverse=“0.1.10.in-addr-arpa”
–foreman-proxy-dns-forwarders=“172.16.0.10”
–foreman-proxy-foreman-base-url=“https://vmforeman-dev.dev-farm.something.net”
–foreman-proxy-plugin-discovery-install-images=“true”
–enable-foreman-proxy-plugin-discovery
–foreman-proxy-dhcp-server=“10.1.0.1”
–foreman-initial-admin-password=“l8Zc6n3z1Gb0Lo527x4F”
–foreman-proxy-dhcp-managed=“true”
–foreman-proxy-trusted-hosts=“vmforeman-dev.dev-farm.something.net”
–enable-foreman-plugin-puppet
–enable-puppet
–enable-foreman-cli-puppet
–foreman-proxy-puppetca=“true”
–puppet-server-ca=“true”
–puppet-server-foreman-url=“https://vmforeman-dev.dev-farm.something.net”
–foreman-proxy-oauth-consumer-key=“zwbb2DB2DFjPR3RQLN5fmsuhRprb77AX”
–foreman-proxy-oauth-consumer-secret=“VzwDRDuqhRsJrc98JAs5uS9ntEmhGdFL”
–foreman-oauth-active=“true”
–foreman-proxy-ssl=“true”
–foreman-proxy-ssl-ca="/etc/puppetlabs/puppet/ssl/ca/ca_crt.pem"
–foreman-proxy-ssl-cert="/etc/puppetlabs/puppet/ssl/certs/vmforeman-dev.dev-farm.something.net.pem"
–foreman-proxy-ssl-key="/etc/puppetlabs/puppet/ssl/private_keys/vmforeman-dev.dev-farm.something.net.pem"

Giving me this…

2023-02-17 15:05:16 [NOTICE] [configure] 1250 configuration steps out of 1426 steps complete.
2023-02-17 15:05:17 [ERROR ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-vmforeman-dev.dev-farm.something.net]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://vmforeman-dev.dev-farm.something.net/api/v2/hosts?search=name%3D"vmforeman-dev.dev-farm.something.net"
2023-02-17 15:05:17 [ERROR ] [configure] Wrapped exception:
2023-02-17 15:05:17 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2023-02-17 15:05:24 [NOTICE] [configure] System configuration has finished.
There were errors detected during install.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.