Foreman-Proxy - certificate verify failed (self signed certificate in certificate chain)

Problem:
When running salt on a host from foreman, the request is denied on the proxy side.

Expected outcome:
The request is successful.

Foreman and Proxy versions:
Latest.

Foreman and Proxy plugin versions:
Latest.

Distribution and version:
Alma8

Other relevant data:
Im not really sure what Im doing wrong here as Ive set up quite a few proxies throughout the years.

I am running the following on the main Foreman server:

puppetserver ca generate --certname proxy.ssnc-corp.cloud 

I then take those certs and place them on the proxy server under

/etc/puppetlabs/puppet/ssl/....

Foreman connects to the proxy and reads all its features just fine. However, when I initiate a salt run against a host, I get:

2024-08-01T13:57:50 61b939ce [E] <OpenSSL::SSL::SSLError> SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)

I have also tried deleting all certs, and requesting them from the proxy itself via

puppet agent --test --waitforcert=60 --debug

Then signing it on the puppetserver side (which is really just the foreman server).

Any ideas what is going on? Ive been at this for a good day.

Full output:

2024-08-01T14:27:29 0fb4ab44 [E] <OpenSSL::SSL::SSLError> SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
        /usr/share/ruby/net/protocol.rb:44:in `connect_nonblock'
        /usr/share/ruby/net/protocol.rb:44:in `ssl_socket_connect'
        /usr/share/ruby/net/http.rb:1009:in `connect'
        /usr/share/ruby/net/http.rb:943:in `do_start'
        /usr/share/ruby/net/http.rb:932:in `start'
        /usr/share/ruby/net/http.rb:1483:in `request'
        /usr/share/foreman-proxy/lib/proxy/request.rb:48:in `send_request'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.2/lib/smart_proxy_dynflow/callback.rb:15:in `callback'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.2/lib/smart_proxy_dynflow/callback.rb:9:in `send_to_foreman_tasks'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.2/lib/smart_proxy_dynflow/callback.rb:31:in `run'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/progress.rb:15:in `run'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.2/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:17:in `block in run'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.2/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:51:in `restore_current_request_id'
        /usr/share/gems/gems/smart_proxy_dynflow-0.9.2/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:17:in `run'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:33:in `run'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/world.rb:31:in `execute'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:588:in `catch'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:588:in `block in execute_run'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `block in with_error_handling'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `catch'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `with_error_handling'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:583:in `execute_run'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:304:in `execute'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/director.rb:70:in `execute'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/parallel/worker.rb:16:in `block in on_message'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors.rb:18:in `run_user_code'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/parallel/worker.rb:15:in `on_message'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/actor.rb:122:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
        /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/actor.rb:56:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `block in synchronize'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
        /usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
        /usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'
        /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-08-01T14:27:29 0fb4ab44 [D]          Step bde42f54-fdb7-478b-8cd6-fba321ba28ac: 5   running >>     error in phase      Run Proxy::Dynflow::Callback::Action

Since the certs will almost always be self signed, due to requesting the certs from the puppetserver, which is also the foreman server, I edited this line to ignore checking:

vim +82 /usr/share/foreman-proxy/lib/proxy/request.rb

Before:

 75         if ca_file && !ca_file.to_s.empty?
 76           http.ca_file     = ca_file
 77           http.verify_mode = OpenSSL::SSL::VERIFY_PEER
 78         end
 75         if ca_file && !ca_file.to_s.empty?
 76           http.ca_file     = ca_file
 77           http.verify_mode = OpenSSL::SSL::VERIFY_NONE
 78         end

Well, “Latest” doesn’t tell which version it really is nor does it say if it’s a katello server or a server without katello.

Generally, it’s best to follow the docs and use foreman-installer etc. to configure the system. Putting, replacing or editing files usually won’t really work or potentially break the next time you run foreman-installer.

So all considering, I just guess and say to look here

Ya, sorry that was lazy… Was going on like hour 50 of trying to fix this issue.
Non Katello.

Foreman:

Collecting plugin information
Foreman plugin: foreman-tasks, 9.1.1, Ivan NeÄŤas, The goal of this plugin is to unify the way of showing task statuses across the Foreman instance.
It defines Task model for keeping the information about the tasks and Lock for assigning the tasks
to resources. The locking allows dealing with preventing multiple colliding tasks to be run on the
same resource. It also optionally provides Dynflow infrastructure for using it for managing the tasks.
Foreman plugin: foreman_default_hostgroup, 7.0.0, Greg Sutcliffe, Adds the option to specify a default hostgroup for new hosts created from facts/reports
Foreman plugin: foreman_kubevirt, 0.1.9, Moti Asayag, Provision and manage Kubevirt Virtual Machines from Foreman.
Foreman plugin: foreman_puppet, 6.2.0, Ondřej Ezr and Shira Maximov, Allow assigning Puppet environments and classes to the Foreman Hosts.
Foreman plugin: foreman_remote_execution, 13.0.0, Foreman Remote Execution team, A plugin bringing remote execution to the Foreman, completing the config management functionality with remote management functionality.
Foreman plugin: foreman_salt, 16.0.2, Stephen Benjamin, Foreman Plug-in for Salt
Foreman plugin: foreman_statistics, 2.1.0, Ondrej Ezr, Statistics and Trends for Foreman gives users overview of their infrastructure.
Foreman plugin: foreman_templates, 9.4.0, Greg Sutcliffe, Engine to synchronise provisioning templates from GitHub
Foreman plugin: foreman_vault, 2.0.0, dmTECH GmbH,
foreman-release-3.10.0-1.el9.noarch
foreman-installer-3.10.0-1.el9.noarch
foreman-selinux-3.10.0-1.el9.noarch
foreman-3.10.0-1.el9.noarch
foreman-postgresql-3.10.0-1.el9.noarch
foreman-service-3.10.0-1.el9.noarch
foreman-dynflow-sidekiq-3.10.0-1.el9.noarch
foreman-redis-3.10.0-1.el9.noarch
foreman-proxy-3.10.0-1.el9.noarch
rubygem-hammer_cli_foreman-3.10.0-1.el9.noarch
foreman-cli-3.10.0-1.el9.noarch
foreman-libvirt-3.10.0-1.el9.noarch
foreman-ovirt-3.10.0-1.el9.noarch
foreman-vmware-3.10.0-1.el9.noarch
rubygem-hammer_cli_foreman_kubevirt-0.1.5-1.el9.noarch
rubygem-hammer_cli_foreman_puppet-0.0.7-1.el9.noarch
rubygem-hammer_cli_foreman_tasks-0.0.20-1.el9.noarch
rubygem-hammer_cli_foreman_remote_execution-0.3.0-1.el9.noarch
rubygem-hammer_cli_foreman_ssh-0.0.3-1.el9.noarch
rubygem-foreman_default_hostgroup-7.0.0-1.fm3_10.el9.noarch
rubygem-foreman_kubevirt-0.1.9-5.fm3_10.el9.noarch
rubygem-foreman_puppet-6.2.0-1.fm3_10.el9.noarch
rubygem-foreman-tasks-9.1.1-1.fm3_11.el9.noarch
rubygem-foreman_remote_execution-13.0.0-1.fm3_11.el9.noarch
rubygem-foreman_salt-16.0.2-1.fm3_11.el9.noarch
rubygem-foreman_statistics-2.1.0-2.fm3_10.el9.noarch
rubygem-foreman_templates-9.4.0-2.fm3_10.el9.noarch
rubygem-foreman_vault-2.0.0-1.fm3_11.el9.noarch

Proxy:

# rpm -qa | grep foreman-proxy
foreman-proxy-3.11.0-1.el8.noarch
# gem list | grep smart_proxy
smart_proxy_dynflow (0.9.2)
smart_proxy_salt (5.1.0)

Im fairly certain the certs are correct. Else I wouldnt be able to curl, or connect to the proxy, from the foreman server I assume? Something seems off in the certificate verification in the dynflow callback.

Hmm it seems that foremans client ca cert for the connection to the proxy or the proxies client ca cert to talk to foreman have a problem.

How do you handle the certs an are they trusted on both sides by foreman?

Hello,

same here…

2024-08-08 06:17:31 [NOTICE] [configure] Starting system configuration.
2024-08-08 06:17:37 [NOTICE] [configure] 250 configuration steps out of 1114 steps complete.
2024-08-08 06:17:37 [NOTICE] [configure] 500 configuration steps out of 1116 steps complete.
2024-08-08 06:17:38 [NOTICE] [configure] 750 configuration steps out of 1121 steps complete.
2024-08-08 06:17:47 [NOTICE] [configure] 1000 configuration steps out of 1122 steps complete.
2024-08-08 06:17:48 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-server.net]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://server.net/api/v2/hosts?search=name%3D%22server.net%22
2024-08-08 06:17:48 [ERROR ] [configure] Wrapped exception:
2024-08-08 06:17:48 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-08 06:17:48 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[server.net]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://server.net/api/v2/smart_proxies?search=name%3D%22server.net%22
2024-08-08 06:17:48 [ERROR ] [configure] Wrapped exception:
2024-08-08 06:17:48 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-08 06:17:51 [NOTICE] [configure] System configuration has finished.

CentOS Stream 9 Server
Foreman 3.11.1

ansible-collection-theforeman-foreman-4.0.0-2.el9.noarch
candlepin-4.4.12-1.el9.noarch
candlepin-selinux-4.4.12-1.el9.noarch
dynflow-utils-1.6.3-1.el9.x86_64
foreman-3.11.1-1.el9.noarch
foreman-cli-3.11.1-1.el9.noarch
foreman-dynflow-sidekiq-3.11.1-1.el9.noarch
foreman-installer-3.11.1-1.el9.noarch
foreman-installer-katello-3.11.1-1.el9.noarch
foreman-postgresql-3.11.1-1.el9.noarch
foreman-proxy-3.11.1-1.el9.noarch
foreman-redis-3.11.1-1.el9.noarch
foreman-selinux-3.11.1-1.el9.noarch
foreman-service-3.11.1-1.el9.noarch
katello-4.13.1-1.el9.noarch
katello-certs-tools-2.10.0-1.el9.noarch
katello-client-bootstrap-1.7.9-2.el9.noarch
katello-common-4.13.1-1.el9.noarch
katello-selinux-5.0.2-1.el9.noarch
pulpcore-selinux-2.0.1-1.el9.x86_64
python3.11-pulp-ansible-0.21.6-1.el9.noarch
python3.11-pulp-cli-0.25.6-1.el9.noarch
python3.11-pulp-container-2.20.1-1.el9.noarch
python3.11-pulp-deb-3.2.0-1.el9.noarch
python3.11-pulp-glue-0.25.6-1.el9.noarch
python3.11-pulp-python-3.11.1-1.el9.noarch
python3.11-pulp-rpm-3.26.1-1.el9.noarch
python3.11-pulpcore-3.49.10-1.el9.noarch
rubygem-dynflow-1.8.4-1.el9.noarch
rubygem-foreman-tasks-9.1.1-1.fm3_11.el9.noarch
rubygem-foreman_ansible-14.0.0-2.fm3_11.el9.noarch
rubygem-foreman_bootdisk-21.2.3-2.fm3_11.el9.noarch
rubygem-foreman_discovery-24.0.1-2.fm3_11.el9.noarch
rubygem-foreman_maintain-1.6.9-1.el9.noarch
rubygem-foreman_remote_execution-13.1.0-1.fm3_11.el9.noarch
rubygem-hammer_cli-3.11.0-1.el9.noarch
rubygem-hammer_cli_foreman-3.11.0-1.el9.noarch
rubygem-hammer_cli_foreman_ansible-0.7.0-1.fm3_11.el9.noarch
rubygem-hammer_cli_foreman_remote_execution-0.3.0-1.el9.noarch
rubygem-hammer_cli_foreman_tasks-0.0.21-1.fm3_11.el9.noarch
rubygem-hammer_cli_katello-1.13.0-0.2.pre.master.el9.noarch
rubygem-katello-4.13.1-1.el9.noarch
rubygem-pulp_ansible_client-0.21.3-1.el9.noarch
rubygem-pulp_certguard_client-3.49.6-1.el9.noarch
rubygem-pulp_container_client-2.20.0-1.el9.noarch
rubygem-pulp_deb_client-3.2.0-1.el9.noarch
rubygem-pulp_file_client-3.49.6-1.el9.noarch
rubygem-pulp_ostree_client-2.3.0-1.el9.noarch
rubygem-pulp_python_client-3.11.1-1.el9.noarch
rubygem-pulp_rpm_client-3.26.1-1.el9.noarch
rubygem-pulpcore_client-3.49.6-1.el9.noarch
rubygem-smart_proxy_dynflow-0.9.2-1.fm3_11.el9.noarch
rubygem-smart_proxy_pulp-3.3.0-1.el9.noarch

Hopefully this will be fixed soon!

Best regards,
Thomas

Not really:

So the existing docs generally apply.

Beyond that, you didn’t even post what you did exactly before that nor the exact foreman-installer command you are running to update the cert. Nor if you are running this on the main server or a proxy like in the original post.

OK, sorry, what I did is the following:

Installed a main Foreman/Katello Server from scratch.

  • CentOS Stream 9 server
  • Foreman/Katello 3.11.1
dnf install foreman-installer-katello
foreman-installer --scenario katello \
    --foreman-initial-organization "My Organization" \
    --foreman-initial-admin-username admin \
    --foreman-initial-admin-password xxxxxxx
    --tuning development

Created a custom certificate as the following:

Afterwards
openssl verify -CAfile CertificateChain.pem mainserver.net-cer.pem

foreman-installer --scenario katello \
  --certs-server-cert "/root/foreman_cert/mainserver.net-cer.pem" \
  --certs-server-key "/root/foreman_cert/mainserver.net.cert_key.pem" \
  --certs-server-ca-cert "/root/foreman_cert/CertificateChain.pem" \
  --certs-update-server --certs-update-server-ca
  
foreman-installer \
  --foreman-server-ssl-cert /root/foreman_cert/mainserver.net-cer.pem \
  --foreman-server-ssl-key /root/foreman_cert/mainserver.net.cert_key.pem \
  --foreman-server-ssl-chain /root/foreman_cert/CertificateChain.pem

Opened the new Foreman webinterface with https://mainserver.net and did all the configuratio steps

  • user
  • roles
  • subnet
  • organization
  • http proxies
  • domains
  • etc…

Then installed a smart-proxy server from scratch

  • CentOS stream 9 server
  • yum install foreman-proxy-content

On the main server:

foreman-proxy-certs-generate \
--foreman-proxy-fqdn "smartproxyserver.net" \
--certs-tar "/root/smartproxyserver.net-certs.tar" \
--server-cert /root/foreman_cert/mainserver.net-cer.pem \
--server-cert-req /root/foreman_cert/mainserver.net-req.pem \
--server-key /root/foreman_cert/mainserver.net.cert_key.pem \
--server-ca-cert /root/foreman_cert/CertificateChain.pem

Copied the tar file to the new smart-proxy server

foreman-installer \
  --scenario foreman-proxy-content \
  --certs-tar-file "/root/smartproxyserver.net-certs.tar" \
  --foreman-proxy-register-in-foreman "true" \
  --foreman-proxy-foreman-base-url "https://mainserver.net" \
  --foreman-proxy-trusted-hosts "mainserver.net" \
  --foreman-proxy-trusted-hosts "smartproxyserver.net" \
  --foreman-proxy-oauth-consumer-key "xxx" \
  --foreman-proxy-oauth-consumer-secret "xxx"

Result:

2024-08-08 06:17:31 [NOTICE] [configure] Starting system configuration.
2024-08-08 06:17:37 [NOTICE] [configure] 250 configuration steps out of 1114 steps complete.
2024-08-08 06:17:37 [NOTICE] [configure] 500 configuration steps out of 1116 steps complete.
2024-08-08 06:17:38 [NOTICE] [configure] 750 configuration steps out of 1121 steps complete.
2024-08-08 06:17:47 [NOTICE] [configure] 1000 configuration steps out of 1122 steps complete.
2024-08-08 06:17:48 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-mainserver.net]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://mainserver.net/api/v2/hosts?search=name%3D%22mainserver.net%22
2024-08-08 06:17:48 [ERROR ] [configure] Wrapped exception:
2024-08-08 06:17:48 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-08 06:17:48 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[mainserver.net]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://mainserver.net/api/v2/smart_proxies?search=name%3D%22mainserver.net%22
2024-08-08 06:17:48 [ERROR ] [configure] Wrapped exception:
2024-08-08 06:17:48 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-08 06:17:51 [NOTICE] [configure] System configuration has finished.

Info: Server names are replaced by fantasy names

I did exactly the same with Foreman 3.9 without any problem.

Error 1: Puppet Foreman_host resource 'smartproxyserver.net' failed. Logs:
  /Stage[main]/Foreman_proxy::Register/Foreman_host[smartproxyserver.net]
    Adding autorequire relationship with Anchor[foreman::providers::oauth]
    Starting to evaluate the resource (1108 of 1123)
    Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://mainserver.net/api/v2/hosts?search=name%3D%22smartproxyserver.net%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Evaluated in 0.01 seconds
  Foreman_host[smartproxyserver.net](provider=rest_v3)
    Making get request to https://smartproxyserver.net/api/v2/hosts?search=name%3D%22smartproxyserver.net%22
Error 2: Puppet Foreman_smartproxy resource 'smartproxyserver.net' failed. Logs:
  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[smartproxyserver.net]
    Adding autorequire relationship with Anchor[foreman::providers::oauth]
    Starting to evaluate the resource (1110 of 1123)
    Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://smartproxyserver.net/api/v2/smart_proxies?search=name%3D%22smartproxyserver.net%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Evaluated in 0.01 seconds
  Foreman_smartproxy[smartproxyserver.net](provider=rest_v3)
    Making get request to https://smartproxyserver.net/api/v2/smart_proxies?search=name%3D%22smartproxyserver.net%22

You should use the official foreman documentation for your version and not the redhat satellite documentation for an older version.

See Installing Foreman Server with Katello 4.13 plugin on Enterprise Linux

You did not run katello-certs-check to check if the files are O.K.

This is nowhere mentioned in the docs. You are using non-default paths. That’s a bad idea. Why do you use different paths? Who knows what happens when you point foreman directly to the files in /root. foreman-installer usually copies the right files into the rights places if you let it.

This again isn’t in the docs: Installing a Smart Proxy Server 3.11 on Enterprise Linux.

You are configuring your smart proxy to use the certificate of the main server. That doesn’t make any sense.

Your smart proxy has the name smartproxyserver but is using the certificate mainserver. Obviously, that won’t work.

I would suggest you start over and follow the docs.

As I told you, the names are changed. We don’t want to show our origingal names here. The certificate is a SAN cert which include all our foreman/foreman-proxy names as alternate names.

And yes the Katello cert check was successful.

I’m running into the same issue while trying to install a Smart Proxy w/ 3.11. We’ve been over the docs many times, trying to follow them exactly. But still this:

2024-08-16 16:14:15 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-08-16 16:14:18 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-08-16 16:14:18 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-08-16 16:15:22 [NOTICE] [configure] Starting system configuration.
2024-08-16 16:16:01 [NOTICE] [configure] 250 configuration steps out of 1120 steps complete.
2024-08-16 16:16:34 [NOTICE] [configure] 500 configuration steps out of 1122 steps complete.
2024-08-16 16:16:43 [NOTICE] [configure] 750 configuration steps out of 1147 steps complete.
2024-08-16 16:18:09 [NOTICE] [configure] 1000 configuration steps out of 1148 steps complete.
2024-08-16 16:18:31 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-gpnixfor02.ipa.medforest.org]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/hosts?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] Wrapped exception:
2024-08-16 16:18:31 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-16 16:18:31 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] Wrapped exception:
2024-08-16 16:18:31 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-16 16:18:31 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] Wrapped exception:
2024-08-16 16:18:31 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-16 16:18:34 [NOTICE] [configure] System configuration has finished.

Can anyone shed any light on what the issue might be?

Hi Luke,

Did you get an answer? I think the problem is with 3.11 and not with the certificate or something like that.

So please, Foreman developers, please help us!

Best regards,
Thomas

Hi Thomas,

I did not get an answer, no. This weekend I tried for a few hours to figure it out. Still no luck. My team has all tried as well, and we’ve looked at our ca cert bundle and custom certs over and over again. We can’t find any issues with them. And we’re doing exactly the same thing we did as the last time we did a fresh install (earlier 3.x release).

I’m starting to suspect the same thing-- an issue with 3.11. I’m gong to try to set up a proxy w/ 3.10 today and see if I have the same problem.

I agree, if one of the developers could weigh in, that would be extremely helpful. I know their time is valuable, so I’d appreciate it.

1 Like

I am having the same problem, and it looks to be a 3.11 problem.

I have just installed a foreman content proxy on EL9 following the docs Installing a Smart Proxy Server 3.11 on Enterprise Linux with custom certificates. It works fine. No errors registering.

Running on the main server:

# foreman-proxy-certs-generate --foreman-proxy-fqdn foreman9-content.example.com \
         --certs-tar /root/foreman9-content.example.com-certs.tar \
         --server-cert /root/certs/foreman9-content/2024/foreman9-content_cert.cer \
         --server-key /root/certs/foreman9-content/2024/foreman9-content-2024.key \
         --server-ca-cert /root/certs/foreman9-content/2024/foreman9-content_interm.cer

Certificate and key in pem format. ca-cert is the chain starting with the issuer of the server cert ending with the root ca.

Then running the exact command printed on the new content proxy.

This is a new content proxy. There was no previous foreman installation nor any foreman-installer run before.

There’s a known problem in foreman 3.11, see Foreman 3.11/Katello4.13 - Smartproxy install/update issues with custom CA if it is not your case