Added proxy has broken certificates when Katello has custom cert

Support
1

[ Smart Proxy installation with custom cert still broken :slight_smile: ]

Hi all,

I’m having an issue with Katello 3.3 … [ We use a custom certificate and when we try to add a new proxy using the online instructions the proxy install fails with a certificate error. ]

What I was expecting to happen is … [ a clean proxy install ]

Here’s the debugging I’ve done so far … **[ Basically as mentioned above I’m following the official docs found here: theforeman dot org site section 1.2

Using this command:
foreman-installer --scenario foreman-proxy-content --foreman-proxy-content-parent-fqdn "katello.rcc.yyy.edu" --foreman-proxy-register-in-foreman "true" --foreman-proxy-foreman-base-url "webs://katello.rcc.yyy.edu" --foreman-proxy-trusted-hosts "katello.rcc.yyy.edu" --foreman-proxy-trusted-hosts "puppet4-proxy-bob.rcc.yyy.edu" --foreman-proxy-oauth-consumer-key "WnQqgNhkNs9x8zRUsagocAkmdTRtAD8Q" --foreman-proxy-oauth-consumer-secret "LqiNe5RyYhGEbh8AV6kqxXeiNCsyz7um" --foreman-proxy-content-pulp-oauth-secret "5rdFmrpSZASCbNsxHXJXacjyn9NCcAKi" --foreman-proxy-content-certs-tar "/root/puppet4-proxy-bob.rcc.yyy.edu-certs.tar"

I get this error:

/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:23:in `create'
/usr/share/ruby/vendor_ruby/puppet/property/ensure.rb:16:in `block in defaultvalues'
/usr/share/ruby/vendor_ruby/puppet/property.rb:197:in `call_valuemethod'
/usr/share/ruby/vendor_ruby/puppet/property.rb:498:in `set'
/usr/share/ruby/vendor_ruby/puppet/property.rb:581:in `sync'
/usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in `sync'
/usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
/usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
/usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
/usr/share/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
/usr/share/ruby/benchmark.rb:296:in `realtime'
/usr/share/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
/usr/share/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/share/gems/gems/kafo-1.0.5/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
/usr/share/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/share/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/share/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/share/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/share/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/share/ruby/benchmark.rb:296:in `realtime'
/usr/share/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/share/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/share/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/share/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/share/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[puppet4-proxy-bob.rcc.yyy.edu]/ensure: change from absent to present failed: Proxy puppet4-proxy-bob.rcc.yyy.edu cannot be registered: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verif...) for proxy webs://puppet4-proxy-bob.rcc.yyy.edu:9090/features Please check the proxy is configured and running on the host.
 Could not start Service[httpd]: Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait start httpd' returned 1: Redirecting to /bin/systemctl start httpd.service
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
/usr/share/ruby/vendor_ruby/puppet/util/execution.rb:219:in `execute'
/usr/share/ruby/vendor_ruby/puppet/provider.rb:115:in `execute'
/usr/share/ruby/vendor_ruby/puppet/provider/service/service.rb:25:in `texecute'
/usr/share/ruby/vendor_ruby/puppet/provider/service/service.rb:39:in `ucommand'
/usr/share/ruby/vendor_ruby/puppet/provider/service/base.rb:65:in `start'
/usr/share/ruby/vendor_ruby/puppet/type/service.rb:83:in `block (3 levels) in <module:Puppet>'
/usr/share/ruby/vendor_ruby/puppet/property.rb:197:in `call_valuemethod'
/usr/share/ruby/vendor_ruby/puppet/property.rb:498:in `set'
/usr/share/ruby/vendor_ruby/puppet/property.rb:581:in `sync'
/usr/share/ruby/vendor_ruby/puppet/type/service.rb:94:in `sync'
/usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in `sync'
/usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
/usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
/usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
/usr/share/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
/usr/share/ruby/benchmark.rb:296:in `realtime'
/usr/share/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
/usr/share/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/share/gems/gems/kafo-1.0.5/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
/usr/share/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/share/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/share/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/share/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/share/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/share/ruby/benchmark.rb:296:in `realtime'
/usr/share/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/share/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/share/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/share/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/share/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
 /Stage[main]/Apache::Service/Service[httpd]/ensure: change from stopped to running failed: Could not start Service[httpd]: Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait start httpd' returned 1: Redirecting to /bin/systemctl start httpd.service
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.```
 ]**

We've been having this issue since Katello 3.1
The fix for it seems to be that we first had to run:
foreman-proxy-certs-generate --foreman-proxy-fqdn "puppet4-proxy-bob.rcc.yyy.edu"\
                --certs-update-all
                --server-ca-cert "/xxx/keys/rcc.yyy.edu_wildcard-2016-2018/rcc.yyy.edu_apache_chain.cer"\
                --server-cert "/xxx/keys/rcc.yyy.edu_wildcard-2016-2018/rcc.yyy.edu_apache.cer"\
                --server-cert-req "/xxx/keys/rcc.yyy.edu_wildcard-2016-2018/rcc.yyy.edu.csr"\
                --server-key "/xxx/keys/rcc.yyy.edu_wildcard-2016-2018/rcc.yyy.edu.key"\
                --certs-tar "/root/puppet4-proxy-bob.rcc.yyy.edu-certs.tar"

Thanks!
**[ Edson Manners ]**
2

Do you still get the issue with Katello 3.5, or just 3.3?

3

Katello 3.3. We’re in the process of upgrading which is why we’re running into this. But I can test on 3.5 and let you know.

4

Have you tried using katello-certs-check?

5

Yip. Works fine. See below.

Validating the certificate subject= /CN=katello3.aaa.bbb
Check private key matches the certificate: [OK]
Check ca bundle verifies the cert file: [OK]

Validation succeeded.

To install the Katello main server with the custom certificates, run:
...```

 Like I said everything works fine except for when we follow the instructions to add a new proxy. And  the way to make it work is to run the following which is not included in instructions. So I guess I should say SOLVED but i wanted this out there so that others in the community and/or the devs can fix it in upcoming releases if not already done.

```foreman-proxy-certs-generate --foreman-proxy-fqdn "puppet4-proxy.aaa.bbb"\
                --certs-update-all
                --server-ca-cert "/xxx/keys/aaa.bbb_wildcard-2016-2018/aaa.bbb_apache_chain.cer"\
                --server-cert "/xxx/keys/aaa.bbb_wildcard-2016-2018/aaa.bbb_apache.cer"\
                --server-cert-req "/xxx/keys/aaa.bbb_wildcard-2016-2018/aaa.bbb.csr"\
                --server-key "/xxx/keys/aaa.bbb_wildcard-2016-2018/aaa.bbb.key"\
                --certs-tar "/root/puppet4-proxy.aaa.bbb-certs.tar"```

But I'll be testing in 3.5 as soon as I can get a new 3.5 Katello built.
6

Sounds first of all like Apache is unhappy. Can you post what errors httpd is reporting trying to start?