Foreman-Proxy is not importing hosts from Foreman-Master

ahoiroman · February 21, 2020, 11:45am

Problem:

I created a foreman main host (foreman.example.com) and a foreman-proxy (foreman-proxy.example.com).

The foreman main host is running on an isolated network where only port 443 and 8443 are open to foreman-proxy.example.com.

Now I created a host (new-host.example.com) that is using the Puppet-CA of foreman-proxy.example.com. If I then run puppet agent -t I am getting this result:

[root@new-host ~]# puppet agent --test
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Server Error: Failed to find new-host.example.com via exec: Execution of '/etc/puppetlabs/puppet/node.rb new-host.example.com' returned 1:
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node new-host.example.com: Failed to find new-host.example.com.de via exec: Execution of '/etc/puppetlabs/puppet/node.rb new-host.example.com' returned 1:
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

On the foreman-proxy I get this in my log:

Failed to open TCP connection to foreman-proxy.example.com:443 (Connection refused - connect(2) for "foreman-proxy.example.com" port 443)
["org/jruby/ext/socket/RubyTCPSocket.java:144:in `initialize'", "org/jruby/RubyIO.java:1156:in `open'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:941:in `block in connect'", "org/jruby/ext/timeout/Timeout.java:99:in `timeout'", "org/jruby/ext/timeout/Timeout.java:75:in `timeout'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:939:in `connect'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in `do_start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:913:in `start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1465:in `request'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:69:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:54:in `block in processors'", "org/jruby/RubyArray.java:1800:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:316:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:199:in `do_save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'", "org/jruby/RubyArray.java:1800:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:87:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:70:in `block in with_request_profiling'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:66:in `with_request_profiling'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:93:in `respond_to_errors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `process'", "uri:classloader:/puppetserver-lib/puppet/server/master.rb:64:in `block in handleRequest'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'", "uri:classloader:/puppetserver-lib/puppet/server/master.rb:63:in `handleRequest'"]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:75:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:54:in `block in processors'
org/jruby/RubyArray.java:1800:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:316:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:199:in `do_save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'
org/jruby/RubyArray.java:1800:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:87:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:70:in `block in with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:66:in `with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:93:in `respond_to_errors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `process'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:64:in `block in handleRequest'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:63:in `handleRequest'

I am wondering why this call is being made:

foreman-proxy.example.com:443 - on the foreman-proxy there’s no service running on 443. Shouldn’t it be `foreman.example.com:443?

On the main foreman host the server new-host.example.com is existing.

This is how I installed the foreman-proxy:

foreman-installer \
  --no-enable-foreman \
  --no-enable-foreman-cli \
  --enable-foreman-proxy \
  --enable-foreman-proxy-plugin-remote-execution-ssh \
  --enable-foreman-proxy-plugin-discovery \
  --foreman-proxy-plugin-discovery-install-images=true \
  --foreman-proxy-templates=true \
  --foreman-proxy-template-url=http://foreman-proxy.example.com:8000 \
  --foreman-proxy-templates-listen-on=both \
  --foreman-proxy-puppetca=true \
  --foreman-proxy-tftp=true \
  --foreman-proxy-http=true \
  --foreman-proxy-foreman-ssl-ca=/etc/foreman-proxy/ca.pem \
  --foreman-proxy-foreman-ssl-cert=/etc/foreman-proxy/cert.pem \
  --foreman-proxy-foreman-ssl-key=/etc/foreman-proxy/key.pem \
  --foreman-proxy-foreman-base-url=https://foreman.example.com \
  --foreman-proxy-trusted-hosts=foreman.example.com \
  --foreman-proxy-oauth-consumer-key=... \
  --foreman-proxy-oauth-consumer-secret=...

Foreman and Proxy versions:

Foreman main host

Discovery

Version

1.0.5

Dynflow

false

Templates

Version

1.24.2

areyus · February 21, 2020, 12:05pm

Hi,

you are right, that call should go to foreman.example.com
I did not see any obvious errors/missing params in you installer call, but there might be one I missed.
Please check /etc/puppetlabs/puppet/foreman.yaml for the url parameter, that should point to https://foreman.example.com. From that file the Puppet ENC script fetches it’s config.

ekohl · February 21, 2020, 12:07pm

Looks like you forgot --puppet-server-foreman-url https://foreman.example.com.

See Foreman :: Manual as well.

ahoiroman · February 21, 2020, 1:24pm

Thank you for your hints.

Actually it’s not working yet, I assume my certs are faulty.

This is what I did on foreman-proxy

[root@foreman-proxy ~]# foreman-installer \
>   --no-enable-foreman \
>   --no-enable-foreman-cli \
>   --enable-foreman-proxy \
>   --enable-foreman-proxy-plugin-remote-execution-ssh \
>   --enable-foreman-proxy-plugin-discovery \
>   --foreman-proxy-puppet=true \
>   --puppet-server-foreman-url=https://foreman.example.com \
>   --foreman-proxy-plugin-discovery-install-images=true \
>   --foreman-proxy-templates=true \
>   --foreman-proxy-template-url=http://foreman-proxy.example.com:8000 \
>   --foreman-proxy-templates-listen-on=both \
>   --foreman-proxy-puppetca=true \
>   --foreman-proxy-tftp=true \
>   --foreman-proxy-http=true \
>   --foreman-proxy-foreman-ssl-ca=/etc/foreman-proxy/ca.pem \
>   --foreman-proxy-foreman-ssl-cert=/etc/foreman-proxy/cert.pem \
>   --foreman-proxy-foreman-ssl-key=/etc/foreman-proxy/key.pem \
>   --foreman-proxy-foreman-base-url=https://foreman.example.com \
>   --foreman-proxy-trusted-hosts=foreman.example.com \
>   --foreman-proxy-oauth-consumer-key=... \
>   --foreman-proxy-oauth-consumer-secret=...

This is the output:

 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman-proxy.example.com]: Failed to call refresh: Proxy foreman-proxy.example.com cannot be refreshed: unknown error (response 500)
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman-proxy.example.com]: Proxy foreman-proxy.example.com cannot be refreshed: unknown error (response 500)
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:53:in `refresh_features!'

This is the installation log:

[DEBUG 2020-02-21T14:08:35 main]  /File[/etc/puppetlabs/puppet/ssl]: Adding autorequire relationship with File[/etc/puppetlabs/puppet]
[DEBUG 2020-02-21T14:08:35 main]  /File[/etc/puppetlabs/puppet/ssl/public_keys]: Adding autorequire relationship with File[/etc/puppetlabs/puppet/ssl]
[DEBUG 2020-02-21T14:08:35 main]  /File[/etc/puppetlabs/puppet/ssl/certificate_requests]: Adding autorequire relationship with File[/etc/puppetlabs/puppet/ssl]
[DEBUG 2020-02-21T14:08:35 main]  /File[/etc/puppetlabs/puppet/ssl/private_keys]: Adding autorequire relationship with File[/etc/puppetlabs/puppet/ssl]
[DEBUG 2020-02-21T14:08:35 main]  /File[/etc/puppetlabs/puppet/ssl/private]: Adding autorequire relationship with File[/etc/puppetlabs/puppet/ssl]
[DEBUG 2020-02-21T14:08:35 main]  /File[/etc/puppetlabs/puppet/ssl/certs/foreman-proxy.example.com.pem]: Adding autorequire relationship with File[/etc/puppetlabs/puppet/ssl/certs]
[DEBUG 2020-02-21T14:08:35 main]  /File[/etc/puppetlabs/puppet/ssl/private_keys/foreman-proxy.example.com.pem]: Adding autorequire relationship with File[/etc/puppetlabs/puppet/ssl/private_keys]
[DEBUG 2020-02-21T14:08:35 main]  /File[/etc/puppetlabs/puppet/ssl/public_keys/foreman-proxy.example.com.pem]: Adding autorequire relationship with File[/etc/puppetlabs/puppet/ssl/public_keys]
[DEBUG 2020-02-21T14:08:35 main]  /File[/etc/puppetlabs/puppet/ssl/certs/ca.pem]: Adding autorequire relationship with File[/etc/puppetlabs/puppet/ssl/certs]
[DEBUG 2020-02-21T14:08:35 main]  /File[/etc/puppetlabs/puppet/ssl/crl.pem]: Adding autorequire relationship with File[/etc/puppetlabs/puppet/ssl]
[DEBUG 2020-02-21T14:08:35 main]  /File[/opt/puppetlabs/puppet/cache/facts.d]: Adding autorequire relationship with File[/opt/puppetlabs/puppet/cache]
[DEBUG 2020-02-21T14:08:35 main]  /File[/opt/puppetlabs/puppet/cache/locales]: Adding autorequire relationship with File[/opt/puppetlabs/puppet/cache]
[DEBUG 2020-02-21T14:08:35 main]  Finishing transaction 33235140
[DEBUG 2020-02-21T14:08:35 main]  Received report to process from foreman-proxy.example.com
[ INFO 2020-02-21T14:08:35 main] Puppet has finished, bye!
[ INFO 2020-02-21T14:08:35 main] Executing hooks in group post
[DEBUG 2020-02-21T14:08:35 main] Hook /usr/share/foreman-installer/hooks/post/10-post_install_message.rb returned nil
[ INFO 2020-02-21T14:08:35 main] All hooks in group post finished
[DEBUG 2020-02-21T14:08:35 main] Exit with status code: 6 (signal was 6)
[ERROR 2020-02-21T14:08:35 main] Errors encountered during run:
[ERROR 2020-02-21T14:08:35 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman-proxy.example.com]: Failed to call refresh: Proxy foreman-proxy.example.com cannot be refreshed: unknown error (response 500)
[ERROR 2020-02-21T14:08:35 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman-proxy.example.com]: Proxy foreman-proxy.example.com cannot be refreshed: unknown error (response 500)
[ERROR 2020-02-21T14:08:35 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:53:in `refresh_features!'
[ERROR 2020-02-21T14:08:35 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/type/foreman_smartproxy.rb:73:in `refresh'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:149:in `process_callback'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:34:in `block in process_events'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:121:in `block in queued_events'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `each'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `queued_events'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:33:in `process_events'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:288:in `eval_resource'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:521:in `block in thinmark'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:520:in `thinmark'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-02-21T14:08:35 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'

After this, the smart proxy shows an error on foreman’s web interface with the following messages:


Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-7885 [ProxyAPI::ProxyException]: Unable to fetch logs ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)) for proxy https://foreman-proxy.example.com:8443/logs)

and

 Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA certificates ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)) for proxy https://foreman-proxy.example.com:8443/puppet/ca)

The foreman-proxy shows some errors in proxy.log:

2020-02-21T14:21:04  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:21:04  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:21:04  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:21:04  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:21:04  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:21:04  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:21:04  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:21:04  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:21:04  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:21:04  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:21:04  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:21:04  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'

ekohl · February 21, 2020, 1:39pm

The status code 500 comes from Foreman. I’d look in /var/log/foreman/production.log on foreman.example.com.

ahoiroman · February 21, 2020, 1:49pm

Thanks again for your response.

This is the result:

2020-02-21T14:42:45 [I|app|17774185]   Rendered api/v2/smart_proxies/index.json.rabl within api/v2/layouts/index_layout (26.4ms)
2020-02-21T14:42:45 [I|app|17774185] Completed 200 OK in 122ms (Views: 26.8ms | ActiveRecord: 13.1ms)
2020-02-21T14:42:45 [I|app|a6407526] Started PUT "/api/v2/smart_proxies/10/refresh" for 10.11.50.50 at 2020-02-21 14:42:45 +0100
2020-02-21T14:42:45 [I|app|a6407526] Processing by Api::V2::SmartProxiesController#refresh as JSON
2020-02-21T14:42:45 [I|app|a6407526]   Parameters: {"apiv"=>"v2", "id"=>"10", "smart_proxy"=>{}}
2020-02-21T14:42:45 [I|app|a6407526] Authorized user foreman_api_admin(API Admin)
2020-02-21T14:42:45 [W|app|a6407526] Action failed
2020-02-21T14:42:45 [I|app|a6407526]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2020-02-21T14:42:45 [I|app|a6407526]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (1.1ms)
2020-02-21T14:42:45 [I|app|a6407526] Completed 500 Internal Server Error in 59ms (Views: 3.8ms | ActiveRecord: 8.1ms)

if you re-run the command, the error does not appear - but still all the cert-warnings… On re-running the installer, the smart-proxy host also shows this:

[ERROR 2020-02-21T14:46:44 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman-proxy.example.com]/ensure: change from 'absent' to 'present' failed: Proxy foreman-proxy.example.com cannot be registered: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)) for proxy https://foreman-proxy.example.com:8443/v2/features Please check the proxy is configured and running on the host.

The proxy-log on foreman-proxy.example.com shows

2020-02-21T14:45:27  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:45:50  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:46:16  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-21T14:46:43  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'

ekohl · February 22, 2020, 11:57am

It sounds like your CA certificates don’t match. Could it be that both Foreman and Foreman Proxy are a PuppetCA and you’re treating it as a single CA? In general I’d recommend a single Puppet CA in your infrastructure (unless you know what you’re doing) and only let that generate certificates for all machines.

ahoiroman · February 24, 2020, 6:54am

Alright, so I disabled puppetca on the foreman-proxy like that:

foreman-installer \
  --no-enable-foreman \
  --no-enable-foreman-cli \
  --enable-puppet \
  --puppet-server-ca=false \
  --puppet-server-foreman-url=https://foreman.example.com \
  --enable-foreman-proxy \
  --foreman-proxy-puppetca=false \
  --foreman-proxy-tftp=false \
  --foreman-proxy-foreman-base-url=https://foreman.example.com \
  --foreman-proxy-trusted-hosts=foreman.example.com \
  --foreman-proxy-oauth-consumer-key=... \
  --foreman-proxy-oauth-consumer-secret=... \
  --no-enable-foreman-proxy-plugin-remote-execution-ssh \
  --no-enable-foreman-proxy-plugin-discovery

I am getting this error:

[ERROR 2020-02-24T07:42:09 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman-proxy.example.com]/ensure: change from 'absent' to 'present' failed: Proxy foreman-proxy.example.com cannot be registered: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)) for proxy https://foreman-proxy.example.com:8443/v2/features Please check the proxy is configured and running on the host.
[DEBUG 2020-02-24T07:42:09 main] Cleaning /tmp/kafo_installation20200224-19830-1kpma3y
[DEBUG 2020-02-24T07:42:09 main] Cleaning /tmp/kafo_installation20200224-19830-5htw19
[DEBUG 2020-02-24T07:42:09 main] Cleaning /tmp/default_values.yaml
[ INFO 2020-02-24T07:42:09 main] Installer finished in 12.937376452 seconds

So the discovery of features on the smart proxy itself is failing: https://foreman-proxy.example.com:8443/v2/features - I checked the proxy.log:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2 (0x2)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Puppet CA: foreman-proxy.example.com
        Validity
            Not Before: Feb 20 10:47:14 2020 GMT
            Not After : Feb 19 10:47:14 2025 GMT
        Subject: CN=foreman-proxy.example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:db:1b:95:9a:98:37:f9:0a:02:66:99:94:8a:fd:
                    [...]
                    4a:32:25:3f:e7:65:64:f4:34:08:40:37:53:96:e8:
                    00:16:91
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            Netscape Comment:
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier:
                keyid:3A:91:01:35:B6:F4:29:99:03:23:90:86:E1:7F:30:EA:34:D0:A8:8F

            X509v3 Subject Key Identifier:
                6A:E4:B1:70:C9:33:B8:36:F5:7B:77:95:51:92:FA:8F:81:46:F6:EE
            1.3.6.1.4.1.34380.1.3.39:
                ..true
            X509v3 Subject Alternative Name:
                DNS:puppet, DNS:foreman-proxy.example.com
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
    Signature Algorithm: sha256WithRSAEncryption
         5b:7d:bd:0b:c0:94:41:cb:d4:bd:c3:9f:0b:48:a7:a6:15:c1:
		 [...]
         8c:74:62:46:cb:9c:e5:64

2020-02-24T07:40:59  [I] WEBrick::HTTPServer#start: pid=19750 port=8443
2020-02-24T07:40:59  [I] Smart proxy has launched on 2 socket(s), waiting for requests
2020-02-24T07:41:00  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
2020-02-24T07:41:00  [I] Finished puppet class cache initialization
2020-02-24T07:42:08  [E] OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'

So now I added the cert that has been issued by the Puppet CA on the Foreman Main-Server:

foreman-installer \
  --no-enable-foreman \
  --no-enable-foreman-cli \
  --enable-puppet \
  --puppet-server-ca=false \
  --puppet-server-foreman-url=https://foreman.example.com \
  --enable-foreman-proxy \
  --foreman-proxy-puppetca=false \
  --foreman-proxy-tftp=false \
  --foreman-proxy-foreman-base-url=https://foreman.example.com \
  --foreman-proxy-trusted-hosts=foreman.example.com \
  --foreman-proxy-oauth-consumer-key=... \
  --foreman-proxy-oauth-consumer-secret=... \
  --no-enable-foreman-proxy-plugin-remote-execution-ssh \
  --foreman-proxy-foreman-ssl-ca=/etc/foreman-proxy/ca.pem \
  --foreman-proxy-foreman-ssl-cert=/etc/foreman-proxy/cert.pem \
  --foreman-proxy-foreman-ssl-key=/etc/foreman-proxy/key.pem \
  --no-enable-foreman-proxy-plugin-discovery

Again:

[ERROR 2020-02-24T07:50:44 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman-proxy.example.com]/ensure: change from 'absent' to 'present' failed: Proxy foreman-proxy.example.com cannot be registered: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)) for proxy https://foreman-proxy.example.com:8443/v2/features Please check the proxy is configured and running on the host.
[DEBUG 2020-02-24T07:50:44 main] Cleaning /tmp/kafo_installation20200224-20500-1w0p13z
[DEBUG 2020-02-24T07:50:44 main] Cleaning /tmp/kafo_installation20200224-20500-1u10k4e

The certificate has been generated by the Puppet CA on foreman.example.com using puppetserver ca generate --certname foreman-proxy.example.com

ahoiroman · February 24, 2020, 7:04am

This is how I created the cert on the master:

[root@foreman ~]# puppetserver ca generate --certname foreman-proxy.example.com
Successfully saved private key for foreman-proxy.example.com to /etc/puppetlabs/puppet/ssl/private_keys/foreman-proxy.example.com.pem
Successfully saved public key for foreman-proxy.example.com to /etc/puppetlabs/puppet/ssl/public_keys/foreman-proxy.example.com.pem
Successfully submitted certificate request for foreman-proxy.example.com
Error:
    Signed certificate foreman-proxy.example.com could not be found on the CA
Successfully signed certificate request for foreman-proxy.example.com
Successfully saved certificate for foreman-proxy.example.com to /etc/puppetlabs/puppet/ssl/certs/foreman-proxy.example.com.pem

After this I copied these files to /etc/foreman-proxy/ca.pem/cert.pem/key.pem

ahoiroman · February 24, 2020, 7:23am

A strange thing: If I restore a snapshot and run

foreman-installer \
  --no-enable-foreman \
  --no-enable-foreman-cli \
  --enable-foreman-proxy \
  --enable-foreman-proxy-plugin-remote-execution-ssh \
  --enable-foreman-proxy-plugin-discovery \
  --foreman-proxy-puppet=true \
  --foreman-proxy-plugin-discovery-install-images=true \
  --foreman-proxy-templates=true \
  --foreman-proxy-template-url=http://foreman-proxy.example.com:8000 \
  --foreman-proxy-templates-listen-on=both \
  --foreman-proxy-puppetca=true \
  --foreman-proxy-tftp=true \
  --foreman-proxy-http=true \
  --foreman-proxy-foreman-ssl-ca=/etc/foreman-proxy/ca.pem \
  --foreman-proxy-foreman-ssl-cert=/etc/foreman-proxy/cert.pem \
  --foreman-proxy-foreman-ssl-key=/etc/foreman-proxy/key.pem \
  --foreman-proxy-foreman-base-url=https://foreman.example.com \
  --foreman-proxy-trusted-hosts=foreman.example.com \
  --foreman-proxy-oauth-consumer-key=...\
  --foreman-proxy-oauth-consumer-secret=...

The smart proxy installation is working fine. As soon as I change --foreman-proxy-puppetca=true \ to false, it crashes. Even new installation attempts using the old working setup fails after running puppetca=false

ahoiroman · February 24, 2020, 7:54am

Solved this issue by adding:

>   --foreman-proxy-ssl-ca=/etc/foreman-proxy/ca.pem\
>   --foreman-proxy-ssl-cert=/etc/foreman-proxy/cert.pem \
>   --foreman-proxy-ssl-key=/etc/foreman-proxy/key.pem