Hopefully I have all the detail required @gvde (sorry for any prior confusion I was adding to the matter).
I’ve reverted our VM snapshot (with the fstab detailed above, LVM still enabled) and attempted a fresh install of 2.5/4.1
foreman-proxy systemd status:
# systemctl status foreman-proxy -l
● foreman-proxy.service - Foreman Proxy
Loaded: loaded (/usr/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/foreman-proxy.service.d
└─90-limits.conf
Active: active (running) since Mon 2021-08-02 10:56:07 UTC; 6min ago
Main PID: 58916 (ruby)
CGroup: /system.slice/foreman-proxy.service
└─58916 ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize
Aug 02 10:56:05 xv-sat001.uk2.scp.bskyb.com smart-proxy[58916]: Your Gemfile lists the gem rsec (< 1) more than once.
Aug 02 10:56:05 xv-sat001.uk2.scp.bskyb.com smart-proxy[58916]: You should probably keep only one of them.
Aug 02 10:56:05 xv-sat001.uk2.scp.bskyb.com smart-proxy[58916]: Remove any duplicate entries and specify the gem only once.
Aug 02 10:56:05 xv-sat001.uk2.scp.bskyb.com smart-proxy[58916]: While it's not a problem now, it could cause errors if you change the version of one of them later.
Aug 02 10:56:07 xv-sat001.uk2.scp.bskyb.com systemd[1]: Started Foreman Proxy.
Aug 02 10:56:15 xv-sat001.uk2.scp.bskyb.com smart-proxy[58916]: /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/sqlite.rb:114: warning: rb_check_safe_obj will be removed in Ruby 3.0
Aug 02 10:58:15 xv-sat001.uk2.scp.bskyb.com smart-proxy[58916]: 10.64.184.13 - - [02/Aug/2021:10:56:15 UTC] "GET /v2/features HTTP/1.1" 200 0
Aug 02 10:58:15 xv-sat001.uk2.scp.bskyb.com smart-proxy[58916]: - -> /v2/features
Aug 02 10:59:15 xv-sat001.uk2.scp.bskyb.com smart-proxy[58916]: 10.64.184.13 - - [02/Aug/2021:10:57:15 UTC] "GET /v2/features HTTP/1.1" 200 0
Aug 02 10:59:15 xv-sat001.uk2.scp.bskyb.com smart-proxy[58916]: - -> /v2/features
netstat -lntp | fgrep :9090:
# netstat -lntp | fgrep :9090
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 58916/ruby
tcp6 0 0 :::9090 :::* LISTEN 58916/ruby
curl from foreman for :9090/features:
# curl https://xv-sat001.uk2.scp.bskyb.com:9090/features
["ansible","dhcp","dynflow","logs","pulpcore","registration","ssh","tftp"]
curl remote for :9090/features:
$ curl https://xv-sat001.uk2.scp.bskyb.com:9090/features
["ansible","dhcp","dynflow","logs","pulpcore","registration","ssh","tftp"]
curl from foreman :9090/v2/features with key and cert (takes a while to complete, time added to illustrate):
# time curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem 'https://xv-sat001.uk2.scp.bskyb.com:9090/v2/features'
{"dynflow":{"http_enabled":false,"https_enabled":true,"settings":{},"state":"running","capabilities":[]},"pulpcore":{"http_enabled":false,"https_enabled":true,"settings":{"pulp_url":"https://xv-sat001.uk2.scp.bskyb.com","mirror":false,"content_app_url":"https://xv-sat001.uk2.scp.bskyb.com/pulp/content","username":null,"password":null},"state":"running","capabilities":[]},"ssh":{"http_enabled":false,"https_enabled":true,"settings":{},"state":"running","capabilities":[]},"ansible":{"http_enabled":false,"https_enabled":true,"settings":{},"state":"running","capabilities":[]},"facts":{"http_enabled":false,"https_enabled":false,"settings":{},"state":"disabled","capabilities":[]},"dns":{"http_enabled":false,"https_enabled":false,"settings":{"use_provider":null},"state":"disabled","capabilities":[]},"templates":{"http_enabled":false,"https_enabled":false,"settings":{"template_url":null},"state":"disabled","capabilities":[]},"tftp":{"http_enabled":false,"https_enabled":true,"settings":{"tftp_servername":null},"state":"running","capabilities":[]},"dhcp":{"http_enabled":false,"https_enabled":true,"settings":{"use_provider":"dhcp_isc"},"state":"running","capabilities":["dhcp_filename_hostname","dhcp_filename_ipv4"]},"puppetca":{"http_enabled":false,"https_enabled":false,"settings":{"use_provider":null},"state":"disabled","capabilities":[]},"puppet":{"http_enabled":false,"https_enabled":false,"settings":{"use_provider":null},"state":"disabled","capabilities":[]},"bmc":{"http_enabled":false,"https_enabled":false,"settings":{},"state":"disabled","capabilities":["redfish","shell","ssh"]},"realm":{"http_enabled":false,"https_enabled":false,"settings":{"use_provider":null},"state":"disabled","capabilities":[]},"logs":{"http_enabled":false,"https_enabled":true,"settings":{},"state":"running","capabilities":[]},"httpboot":{"http_enabled":false,"https_enabled":false,"settings":{"http_port":null,"https_port":null},"state":"disabled","capabilities":[]},"registration":{"http_enabled":false,"https_enabled":true,"settings":{},"state":"running","capabilities":[]}}
real 2m0.461s
user 0m0.198s
sys 0m0.099s
katello.log ERROR grep:
# grep -B 5 ERROR /var/log/foreman-installer/katello.log
2021-08-02 10:29:08 [INFO ] [root] ... finished loading default values from puppet modules.
2021-08-02 10:29:08 [INFO ] [pre_values] Executing hooks in group pre_values
2021-08-02 10:29:08 [INFO ] [pre_values] All hooks in group pre_values finished
2021-08-02 10:29:08 [DEBUG ] [root] Running installer with args [["--scenario", "katello", "--foreman-initial-admin-password", "SECRET-HERE", "--puppet-runmode", "none", "--puppet-server", "false", "--foreman-proxy-puppet", "false", "--foreman-proxy-puppetca", "false", "--foreman-initial-organization", "Sky", "--foreman-initial-location", "UK2", "--certs-server-cert", "/etc/pki/tls/certs/xv-sat001.uk2.scp.bskyb.com.cer", "--certs-server-key", "/etc/pki/tls/private/xv-sat001.uk2.scp.bskyb.com.key", "--certs-server-ca-cert", "/etc/pki/ca-trust/source/anchors/NEW-BSKYB-CA-CHAIN.cer", "--foreman-proxy-dns", "false", "--foreman-proxy-tftp", "true", "--foreman-proxy-tftp-managed", "true", "--foreman-proxy-dhcp", "true", "--foreman-proxy-dhcp-managed", "true", "--foreman-proxy-dhcp-provider", "isc", "--foreman-proxy-dhcp-config", "/etc/dhcp/dhcpd.conf", "--foreman-proxy-dhcp-leases", "/var/lib/dhcpd/dhcpd.leases", "--foreman-proxy-dhcp-gateway", "10.64.187.254", "--foreman-proxy-dhcp-nameservers", "10.64.133.93,10.64.132.32", "--enable-foreman-plugin-ansible", "--enable-foreman-proxy-plugin-ansible", "--enable-foreman-plugin-remote-execution", "--enable-foreman-proxy-plugin-remote-execution-ssh"]]
2021-08-02 10:29:08 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2021-08-02 10:29:08 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
--
2021-08-02 10:38:56 [INFO ] [configure] +pinsetter.org.candlepin.pinsetter.tasks.ExpiredPoolsJob.schedule=0 0 0 * * ?
2021-08-02 10:38:56 [INFO ] [configure] +
2021-08-02 10:38:56 [INFO ] [configure] +pinsetter.org.candlepin.pinsetter.tasks.CertificateRevocationListTask.schedule=0 0 0 1 1 ?
2021-08-02 10:38:56 [INFO ] [configure] +
2021-08-02 10:38:56 [INFO ] [configure] +# Required for https://hibernate.atlassian.net/browse/HHH-12927
2021-08-02 10:38:56 [INFO ] [configure] +log4j.logger.org.hibernate.internal.SessionImpl=ERROR
--
2021-08-02 10:38:58 [INFO ] [configure] # Log configuration
2021-08-02 10:38:58 [INFO ] [configure] # Uncomment and modify if you want to change the location of the log file or use STDOUT or SYSLOG values
2021-08-02 10:38:58 [INFO ] [configure] -#:log_file: /var/log/foreman-proxy/proxy.log
2021-08-02 10:38:58 [INFO ] [configure] +:log_file: /var/log/foreman-proxy/proxy.log
2021-08-02 10:38:58 [INFO ] [configure] # Uncomment and modify if you want to change the log level
2021-08-02 10:38:58 [INFO ] [configure] # WARN, DEBUG, ERROR, FATAL, INFO, UNKNOWN
--
2021-08-02 10:56:07 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[xv-sat001.uk2.scp.bskyb.com]: Starting to evaluate the resource (2431 of 2455)
2021-08-02 10:56:07 [DEBUG ] [configure] Foreman_smartproxy[xv-sat001.uk2.scp.bskyb.com](provider=rest_v3): Making get request to https://xv-sat001.uk2.scp.bskyb.com/api/v2/smart_proxies?search=name%3D%22xv-sat001.uk2.scp.bskyb.com%22
2021-08-02 10:56:15 [DEBUG ] [configure] Foreman_smartproxy[xv-sat001.uk2.scp.bskyb.com](provider=rest_v3): Received response 200 from request to https://xv-sat001.uk2.scp.bskyb.com/api/v2/smart_proxies?search=name%3D%22xv-sat001.uk2.scp.bskyb.com%22
2021-08-02 10:56:15 [DEBUG ] [configure] Foreman_smartproxy[xv-sat001.uk2.scp.bskyb.com](provider=rest_v3): Making post request to https://xv-sat001.uk2.scp.bskyb.com/api/v2/smart_proxies
2021-08-02 10:58:15 [DEBUG ] [configure] Foreman_smartproxy[xv-sat001.uk2.scp.bskyb.com](provider=rest_v3): Received response 422 from request to https://xv-sat001.uk2.scp.bskyb.com/api/v2/smart_proxies
2021-08-02 10:58:15 [ERROR ] [configure] Error making POST request to Foreman at https://xv-sat001.uk2.scp.bskyb.com/api/v2/smart_proxies: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::Exceptions::ReadTimeout]: Timed out reading data from server) for proxy https://xv-sat001.uk2.scp.bskyb.com:9090/v2/features Please check the proxy is configured and running on the host.
2021-08-02 10:58:15 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[xv-sat001.uk2.scp.bskyb.com]/ensure: change from 'absent' to 'present' failed: Error making POST request to Foreman at https://xv-sat001.uk2.scp.bskyb.com/api/v2/smart_proxies: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::Exceptions::ReadTimeout]: Timed out reading data from server) for proxy https://xv-sat001.uk2.scp.bskyb.com:9090/v2/features Please check the proxy is configured and running on the host
We’re defining a custom-hiera.yml too that I can share if helpful
