Puppet reports from standalone puppet servers

Hello,

I’d like to add multiple puppetservers to my infrastructure and have them report back to foreman/katello.

The workflow is:

  1. deploy a vm via foreman with no puppetmaster selected.
  2. puppet apply on the new vm from the main puppet environment repo, this includes configs for the foreman_proxy and foreman-puppet modules which installs a full puppetserver and new CA and joins the puppet agent to itself. It also includes certmonger to retrieve SSL certificates from an IPA realm. The new host will use these certs for foreman-proxy connection to/from foreman.
  3. After puppet apply is complete I can add the host to foreman as a smart proxy and configure host groups to use the new puppet server, etc.

Where this all breaks:

  1. The foreman modules install the report function on the new puppetserver, but when the puppetserver tries to send reports it gets an error: “SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca”.

This is a bit confusing as I’m using custom certificates on the foreman side signed by the aforementioned IPA server, and on the new puppetserver I’ve defined the new IPA certs in the hiera data for /etc/puppetlabs/puppet/foreman.yaml:

:ssl_ca: "/etc/ipa/ca.crt"
:ssl_cert: "/etc/pki/tls/certs/foreman-puppet-test.internal.domain.crt"
:ssl_key: "/etc/pki/tls/private/foreman-puppet-test.internal.domain.key"
:user: ""
:password: ""
:puppetdir: "/opt/puppetlabs/server/data/puppetserver"
:puppetuser: "puppet"
:facts: true
:timeout: 60
:report_timeout: 60
:threads: null

My guess is somehow the foreman server is using the katello signed certs but I’m not sure which part of the apache config dictates this.
/etc/httpd/conf.d/05-foreman-ssl.conf:

  ServerName foreman001.internal.domain
...
  ## Server aliases
  ServerAlias foreman

  ## SSL directives
  SSLEngine on
  SSLCertificateFile      "/etc/pki/tls/certs/foreman001.internal.domain.crt"
  SSLCertificateKeyFile   "/etc/pki/tls/private/foreman001.internal.domain.key"
  SSLCertificateChainFile "/etc/ipa/ca.crt"
  SSLVerifyClient         optional
#  SSLCACertificateFile    "/etc/ipa/ca.crt"
  SSLCACertificateFile    "/etc/pki/katello/certs/katello-default-ca.crt"
  SSLVerifyDepth          3
  SSLOptions +StdEnvVars +ExportCertData

My guess is the ‘SSLCACertificateFile “/etc/pki/katello/certs/katello-default-ca.crt”’ has something to do with it, but if I change to the commented out IPA CA I’m not sure what the knockon effects will be or if this is actually the problem. The SSLVerifyDepth 3 also leaves me in doubt, perhaps changing this to 2 would only verify the IPA ca and the host cert instead of including the katello CA?

I think the katello CA needs to be here as new content hosts use the same foreman url to register.

To be clear, by using IPA certs I’m trying to avoid creating katello certs and deploying them throughout the infra but maybe this isn’t possible? It seems like the puppet reports shouldn’t have anything to do with the katello side.

  1. node.rb on the new puppetserver /var/log/puppetserver/puppetserver.log:
ERROR [qtp257792657-68] [puppetserver] Puppet Server Error: Failed when searching for node foreman-puppet-test.internal.domain: Failed to find foreman-puppet-test.internal.domain via exec: Execution of '/etc/puppetlabs/puppet/node.rb foreman-puppet-test.internal.domain' returned 1: 

This might be the same root cause as the errors in problem 1?

Versions:
Centos 7.4
Katello 3.4
Foreman 1.15.6

Thanks,
Steve

Ok, looking a bit closer at the foreman host /etc/puppetlabs/puppet/foreman.yaml file I see the certs are not only signed by katello, but specific in their use with puppet:
/etc/puppetlabs/puppet/foreman.yaml

:ssl_ca: "/etc/pki/katello/puppet/puppet_client_ca.crt"
:ssl_cert: "/etc/pki/katello/puppet/puppet_client.crt"
:ssl_key: "/etc/pki/katello/puppet/puppet_client.key"

So I ran

foreman-proxy-certs-generate --foreman-proxy-fqdn "foreman-puppet-test.internal.domain" --certs-tar "/root/ssl-build/foreman001.internal.domain/foreman-puppet-test.internal.domain.tar"

Extracted the rpms from the tarball on the new puppetserver, extracted the puppet_client certs and referenced them in /etc/puppetlabs/puppet/foreman.yaml. I don’t get tls connection errors which is great, but:

  1. I get permission errors when running node.rb.
/etc/puppetlabs/puppet/node.rb test011.internal.domain
During the fact upload the server responded with: 403 Forbidden. Error is ignored and the execution continues.
{
  "error": {"message":"Access denied","details":"Missing one of the required permissions: upload_facts"}
}
Error retrieving node test011.internal.domain: Net::HTTPForbidden
Check Foreman's /var/log/foreman/production.log for more information.

/var/log/foreman/production.log:

2018-02-15 10:41:12 edc77375 [app] [I] Started GET "/node/test011.internal.domain?format=yml" for 10.2.202.150 at 2018-02-15 10:41:12 -0800
2018-02-15 10:41:12 edc77375 [app] [I] Processing by HostsController#externalNodes as YML
2018-02-15 10:41:12 edc77375 [app] [I]   Parameters: {"name"=>"test011.internal.domain"}
2018-02-15 10:41:12 edc77375 [app] [W] No SSL cert with CN supplied - request from 10.2.202.150, 
2018-02-15 10:41:12 edc77375 [app] [I] Redirected to https://foreman001.internal.domain/users/login
2018-02-15 10:41:12 edc77375 [app] [I] Filter chain halted as #<Proc:0x0000000adef508@/usr/share/foreman/app/controllers/concerns/foreman/co
ntroller/smart_proxy_auth.rb:14> rendered or redirected
2018-02-15 10:41:12 edc77375 [app] [I] Completed 403 Forbidden in 24ms (ActiveRecord: 0.0ms)
  1. Whenever 1 is resolved I still want to replace these certs with my IPA CA managed certs so I need to know where certs are configured on the foreman side for reports.

False alarm on the perms. I had changed the 05-foreman-ssl.conf which mangled that.

But I’m still stuck not knowing how I can replace the reports certs with my IPA signed certs.

Can anyone from @katello help out here? SSL isn’t my strongest area :slight_smile: