So a curious problem was discovered today and I"m not entirely sure if it's
a puppet or a foreman issue, so I figured I'd break out the layout:
The Design
We have a foreman/puppet all-in-one server. It has been setup in the
domain samplecompany.com. We are leveraging Foreman as an ENC, so if you
follow the class roles/profiles structure, our profiles are defined in
puppet, but our roles are handled by Foreman groups. So we join computer
to a group in foreman and the appropriate profiles are applied. Simple
enough.
The Culprit
enter the wildcard. a wildcard entry was added to the domain as a CNAME.
So *.samplecompany.com resolved to someserver.samplecompany.com. In this
case, said server had puppet and ll the fixings in place.
Triggering the Problem
enter the new computer. Someone from somewhere deploys a new computer from
an image, which then fires up on DHCP and the puppet agent kicks in. Lets
call it newserver. Below is what I see happening:
- Newserver comes online and requests a certificate with the name
newserver.samplecompany.com (cool)
- foreman issues said certificate
- puppet run kicks off
- foreman/puppet applies the configuration for
someserver.samplecompany.com (!)
- foreman reports for the run show up in someserver.samplecompany.com
(!)
So basically, despite the correct certificate being issued, the server runs
the wrong file set of profiles and the results get reported to the
wrong server. The actual server (newserver) basically never appears in
as a host, and never gets audited properly until a DNS A record is added.
So the big question: what? why is a system successfully getting a
certificate with the correct name but then apparently doing some kind of
ip/name validation? Isn't the entire point of that certificate for
identity? Also, is that validation happening from puppet or foreman (as it
determines the profiles as the ENC)? Finally: is this maybe a cname
thing? would flipping it to a records fix this (doubt it … but hey. lets
ask).
Anyone run into this?
>
> So a curious problem was discovered today and I"m not entirely sure if
> it's a puppet or a foreman issue, so I figured I'd break out the layout:
>
> The Design
> We have a foreman/puppet all-in-one server. It has been setup in the
> domain samplecompany.com. We are leveraging Foreman as an ENC, so if you
> follow the class roles/profiles structure, our profiles are defined in
> puppet, but our roles are handled by Foreman groups. So we join computer
> to a group in foreman and the appropriate profiles are applied. Simple
> enough.
>
> The Culprit
> enter the wildcard. a wildcard entry was added to the domain as a CNAME.
> So *.samplecompany.com resolved to someserver.samplecompany.com. In
> this case, said server had puppet and ll the fixings in place.
>
> Triggering the Problem
> enter the new computer. Someone from somewhere deploys a new computer
> from an image, which then fires up on DHCP and the puppet agent kicks in.
> Lets call it newserver. Below is what I see happening:
>
>
> 1. Newserver comes online and requests a certificate with the name
> newserver.samplecompany.com (cool)
> 2. foreman issues said certificate
> 3. puppet run kicks off
> 4. foreman/puppet applies the configuration for
> someserver.samplecompany.com (!)
> 5. foreman reports for the run show up in someserver.samplecompany.com
> (!)
>
> So basically, despite the correct certificate being issued, the server
> runs the wrong file set of profiles and the results get reported to the
> wrong server. The actual server (newserver) basically never appears in
> as a host, and never gets audited properly until a DNS A record is added.
>
> So the big question: what? why is a system successfully getting a
> certificate with the correct name but then apparently doing some kind of
> ip/name validation? Isn't the entire point of that certificate for
> identity? Also, is that validation happening from puppet or foreman (as it
> determines the profiles as the ENC)? Finally: is this maybe a cname
> thing? would flipping it to a records fix this (doubt it … but hey. lets
> ask).
>
> Anyone run into this?
>
···
On Thursday, September 14, 2017 at 9:20:48 PM UTC-7, Justin DynamicD wrote:
Puppet decides what config to give to the host by running the following
script (puppet 4 path given as an example) on the puppetserver.
/etc/puppetlabs/puppet/node.rb newserver.samplecompany.com
You can run this yourself in a shell on someserver.samplecompany.com and
also tail /var/log/foreman/production.log as newserver checks in or you run
the script manually, to see what's going on.
The same script is responsible for delivering facts to foreman too.
The certname is created by puppet on the client (defaults to the host's
FQDN) before it ever talks to puppetserver or foreman.
The reason adding an A record fixes the problem is because a wildcard CNAME
only applies to names that aren't otherwise defined in the zonefile. see http://www.ietf.org/rfc/rfc1912.txt
section 2.7
So at a guess I'd say the likely scenario is
- if you ask newserver.samplecompany.com what it's FQDN is it'll say
newserver.samplecompany.com, hence the certname being correct (probably
due to /etc/hosts)
- if you ask someserver.samplecompany.com who
newserver.samplecompany.com is, It'll ask DNS and get the answer you
have told DNS to give which is someserver.samplecompany.com
- Something in node.rb is doing a DNS query for newserver.samplecompany.com
and using the answer to get the config from foreman and deliver the
facts. See no. 2.
The solution would be to either explicitly add discrete records for any
hosts you want to work with puppet/foreman or remove the wildcard CNAME
(and then add records for everything).
Unless there is a really compelling reason I'd suggest the latter.
That's not quite right, I'm afraid. We treat the incoming argument to
node.rb as canonical, and don't do any further lookups, see [1]. Once
we have the certname, we use it directly in the the call to Foreman,
see [2]
The test I would make is this - if you call
"/etc/puppetlabs/puppet/node.rb newserver.samplecompany.com" yourself,
what answer do you get from Foreman? I would strongly hope it's not
the result for "someserver.samplecompany.com".
Assuming so, then the focus needs to be on how Puppet decides what
argument to use with node.rb …
[1] https://github.com/theforeman/puppet-foreman/blob/master/files/exte
rnal_node_v2.rb#L361
[2] https://github.com/theforeman/puppet-foreman/blob/master/files/exte
rnal_node_v2.rb#L197
···
On Tue, 2017-09-26 at 07:10 -0700, Matt Cahill wrote:
> Something in node.rb is doing a DNS query
> for newserver.samplecompany.com and using the answer to get the
> config from foreman and deliver the facts. See no. 2.
Ah, thanks for the clarification Greg, sorry about that.
