Help Troubleshooting SSL issue

Support
,
1

Hi there

I need to debug this issue, how can I debug cert issues on foreman? Im
trying to send puppet report to foreman.

The error on puppetserver.log

certificate verify failed ["org/jruby/ext/openssl/SSLSocket.java:217:in `connect'"

Packaged installed:

tfm-rubygem-foreman_setup-5.0.0-1.fm1_13.el7.noarch
foreman-release-1.15.6-1.el7.noarch
foreman-proxy-1.15.6-1.el7.noarch
foreman-postgresql-1.15.6-1.el7.noarch
foreman-cli-1.15.6-1.el7.noarch
foreman-selinux-1.15.6-1.el7.noarch
foreman-release-scl-3-1.el7.noarch
foreman-1.15.6-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-0.10.2-1.el7.noarch
foreman-installer-1.15.6-2.el7.noarch

Foreman configuration:

cat foreman.yaml

:url: "https://puppetserver.domain.lan"
:ssl_ca: "/etc/puppetlabs/puppet/ssl/certs/ca.pem"
:ssl_cert: "/etc/puppetlabs/puppet/ssl/certs/puppetserver.domain.lan.pem"
:ssl_key: "/etc/puppetlabs/puppet/ssl/private_keys/puppetserver.domain.lan.pem"
:user: ""
:password: ""
:puppetdir: "/opt/puppetlabs/puppet/cache"
:puppetuser: "puppet"
:facts: true
:timeout: 10
:threads: null

How I installed Foreman.

foreman-installer --foreman-db-host=x.x.x.x  --foreman-db-type=postgresql 
--foreman-db-database=foreman --foreman-db-adapter=postgresql 
--foreman-db-username=foreman --foreman-db-password=dbpassword
--foreman-admin-email=user@domain.com --foreman-admin-password=secretpassoword
--foreman-admin-username=foremanadmin --no-enable-puppet --foreman-db-manage=false

If you need more details just let me know

Thanks
Regards

2

You specified no-enable-puppet, and normally Foreman would use the Puppet certs for its web interface. Two questions arise from this:

  • is the Puppet server on the same host as Foreman?
  • what certs is Foreman using (check the Apache config)?
3

Hi,

Yes, I installed that way because the puppet server including postgresql are installed on the same server.
You mentioned apache config and after checking the certs listed were the same as in foreman.yaml I restarted the service just in case and now the error is gone, this is good!!

Now the issue is that I do not see the report being sent to foreman but only to puppetdb and I do not see the host
listed on Foreman dashboard

store_report’ command for testinstance_i-0e7c5e1b886615560.domain.lan submitted to PuppetDB

This is how I configured puppet.conf ( server side )

storeconfigs = true
storeconfigs_backend = puppetdb
reports = store,puppetdb,foreman

Client side

[agent]
environment = production

[main]
server = puppet.domain.com
#environment = test
#logdest = /var/log/puppetlabs/puppet/puppet-client.log
#log_level = info
runinterval = 30
report = true

How can I troubleshoot this?

Thanks for your time and support.
Regards

4

This is what I see on the production.log, is the smart proxy the culprit? how can I fix this?

2018-01-07 14:46:56 e192ab9c [app] [I] Started POST "/api/config_reports" for 10.87.23.91 at 2018-01-07 14:46:56 +0000
2018-01-07 14:46:56 e192ab9c [app] [I] Processing by Api::V2::ConfigReportsController#create as JSON
2018-01-07 14:46:56 e192ab9c [app] [I]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2018-01-07 14:46:56 e192ab9c [app] [W] No smart proxy server found on ["puppetserver.domain.lan", "puppet", "puppetserver.domain.lan", "puppet.domain.com", "puppetserver-f799dfc2a5c4c927.elb.us-east-1.amazonaws.com"] and is not in trusted_puppetmaster_hosts
2018-01-07 14:46:56 e192ab9c [app] [I]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (0.5ms)
2018-01-07 14:46:56 e192ab9c [app] [I] Filter chain halted as #<Proc:0x00000006d245a0@/usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb:14> rendered or redirected
2018-01-07 14:46:56 e192ab9c [app] [I] Completed 403 Forbidden in 9ms (Views: 1.3ms | ActiveRecord: 0.5ms)

Thanks
Regards

5

Hi,
I forgot to start the foreman-proxy service, this is the configuration and seems to ve using the same certificates so this is good.

:settings_directory: /etc/foreman-proxy/settings.d

:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/puppetserver.domain.lan.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/puppetserver.domain.lan.pem


:trusted_hosts:
  - puppetserver.domain.lan

:foreman_url: https://puppetserver.domain.lan
:daemon: true
:bind_host: '*'
:https_port: 8443
:log_file: /var/log/foreman-proxy/proxy.log
:log_level: INFO
:log_buffer: 2000
:log_buffer_errors: 1000

Now I see this on the log.

I, [2018-01-07T15:06:39.836762 ]  INFO -- : Started puppet class cache initialization
I, [2018-01-07T15:06:39.837494 ]  INFO -- : Successfully initialized 'puppet_proxy_puppet_api'
I, [2018-01-07T15:06:39.837815 ]  INFO -- : Successfully initialized 'puppet'
I, [2018-01-07T15:06:39.837921 ]  INFO -- : Successfully initialized 'logs'
I, [2018-01-07T15:06:39.860489 ]  INFO -- : WEBrick 1.3.1
I, [2018-01-07T15:06:39.860596 ]  INFO -- : ruby 2.0.0 (2015-12-16) [x86_64-linux]
W, [2018-01-07T15:06:39.861756 ]  WARN -- : TCPServer Error: Address already in use - bind(2)
I, [2018-01-07T15:06:39.862730 ]  INFO -- : 

I, [2018-01-07T15:06:39.865016 ]  INFO -- : WEBrick::HTTPServer#start: pid=23530 port=8443
E, [2018-01-07T15:06:39.999486 ] ERROR -- : Error while retrieving puppet classes for 'production' environment
W, [2018-01-07T15:06:40.000206 ]  WARN -- : Failed to initialize puppet class cache, deferring initialization. Is puppetserver running?

And still seeing:

[W] No smart proxy server found on ["puppetserver.domain.lan", "puppet", "puppetserver.domain.lan", "puppet.domain.com", "puppetserver-f799dfc2a5c4c927.elb.us-east-1.amazonaws.com"] and is not in trusted_puppetmaster_hosts

Thanks
Regards

6

Sounds like you haven’t added the proxy to the Foreman UI - do you have a proxy under Smart Proxies, with a URL that matches the certificate, and has the Puppet feature enabled?

7

Hi
While trying to create the proxy I see this on the logs.

2018-01-07 16:32:46 9cc17610 [app] [I] Failed to save: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::RequestTimeout]: Request Timeout) for proxy https://puppetserver.domain.lan/features, Please check the proxy is configured and running on the host.
2018-01-07 16:32:46 9cc17610 [app] [I]   Rendered taxonomies/_loc_org_tabs.html.erb (0.0ms)
2018-01-07 16:32:46 9cc17610 [app] [I]   Rendered smart_proxies/_form.html.erb (2.8ms)
2018-01-07 16:32:46 9cc17610 [app] [I]   Rendered smart_proxies/new.html.erb (3.1ms)
2018-01-07 16:32:46 9cc17610 [app] [I] Completed 200 OK in 10017ms (Views: 3.8ms | ActiveRecord: 1.7ms)

The service is running

foreman+ 30073 1 0 15:33 ? 00:00:00 ruby /usr/share/foreman-proxy/bin/smart-proxy
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN

proxy.log

I, [2018-01-07T15:33:14.188370 ]  INFO -- : WEBrick::HTTPServer#start: pid=30073 port=8443
E, [2018-01-07T15:33:14.305108 ] ERROR -- : Error while retrieving puppet classes for 'production' environment
W, [2018-01-07T15:33:14.305904 ]  WARN -- : Failed to initialize puppet class cache, deferring initialization. Is puppetserver running?

What I do not see is a configuration file under /etc/http/conf.d/ for the proxy itself, there should be something ? is that created automatically when creating the proxy from the foreman UI?

Any idea what’s could be wrong?

Thanks for your time and support, really appreciated.

Regards

8

Timeout suggests that either the hostname is wrong, the DNS can’t resolve it, or there’s a firewall in the way.

Whilst this is only a warning, it’s probably worth checking. The proxy doesn’t run in Apache - you’ll find its config in /etc/foreman-proxy - can you post the contents of settings.d/puppet.yml?

9 
---
# Puppet management
:enabled: https
# valid providers:
#   puppet_proxy_puppetrun   (for puppetrun/kick, deprecated in Puppet 3)
#   puppet_proxy_mcollective (uses mco puppet)
#   puppet_proxy_ssh         (run puppet over ssh)
#   puppet_proxy_salt        (uses salt puppet.run)
#   puppet_proxy_customrun   (calls a custom command with args)
#:use_provider: puppet_proxy_puppetrun

:puppet_version: 5.3.3
10

2018-01-07 16:32:46 9cc17610 [app] [I] Failed to save: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::RequestTimeout]: Request Timeout) for proxy https://puppetserver.domain.lan/features, Please check the proxy is configured and running on the host.

:8443 is missing in the URL.

1 Like
11

You are right! that did the trick

smartproxy https://puppetserver.domain.lan:8443 Logs, Puppet, Puppet CA, and TFTP Edit

2018-01-08 12:29:49 07b5a75b [app] [I] Started GET "/notification_recipients" for 10.x.x.x at 2018-01-08 12:29:49 +0000
2018-01-08 12:29:49 07b5a75b [app] [I] Processing by NotificationRecipientsController#index as JSON
2018-01-08 12:29:49 07b5a75b [app] [I] Current user: foremanadmin (administrator)
2018-01-08 12:29:49 07b5a75b [app] [I] Completed 200 OK in 4ms (Views: 0.1ms | ActiveRecord: 0.5ms)
2018-01-08 12:30:07  [app] [I] Current user: foreman_admin (administrator)
2018-01-08 12:30:07  [app] [I] Current user: foreman_admin (administrator)
2018-01-08 12:30:25 1f0bddd2 [app] [I] Started POST "/api/config_reports" for 10.x.x.x at 2018-01-08 12:30:25 +0000
2018-01-08 12:30:25 1f0bddd2 [app] [I] Processing by Api::V2::ConfigReportsController#create as JSON
2018-01-08 12:30:25 1f0bddd2 [app] [I]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2018-01-08 12:30:25 1f0bddd2 [app] [I] Current user: foreman_api_admin (administrator)
2018-01-08 12:30:25 1f0bddd2 [app] [I] Imported report for testinstance_i-0e7c5e1b886615560.domain.lan in 0.08 seconds, status refreshed in 0.01 seconds
2018-01-08 12:30:26 1f0bddd2 [app] [I]   Rendered api/v2/config_reports/create.json.rabl (24.5ms)
2018-01-08 12:30:26 1f0bddd2 [app] [I] Completed 201 Created in 138ms (Views: 21.4ms | ActiveRecord: 43.2ms)
2018-01-08 12:30:45 998d815e [app] [I] Started GET "/notification_recipients" for 10.x.x.x  at 2018-01-08 12:30:45 +0000
2018-01-08 12:30:45 998d815e [app] [I] Processing by NotificationRecipientsController#index as JSON
2018-01-08 12:30:45 998d815e [app] [I] Current user: foremanadmin (administrator)
2018-01-08 12:30:45 998d815e [app] [I] Completed 200 OK in 6ms (Views: 0.2ms | ActiveRecord: 0.7ms)

And now I can see the host in the dashboard!!
A big thanx to all of you for your time and support

Regards

1 Like