Foreman/Puppet deleting client certs after build, keeps certs in memory until restart

John_Hughes · February 26, 2015, 4:36pm

Hi all,

I am running into something a bit strange after an upgrade to 1.7.2(may or
may not be related):

Create new host
1. host gets built
2. registers with Foreman/Puppet
3. puppet certs get put into /var/lib/puppet/ssl on foreman server.
Then a few seconds later
1. Client SSL certs for the new host get removed by a puppet node
  clean <hostname> on the Foreman/Puppet server(seen in the logs).
  1. SSL certificates removed.
2. However, the build finishes
3. host works fine
4. shows up in foreman
5. puppet on new host works fine.
Until Foreman/Puppet(Apache/Passenger) is restarted
1. Then the new host's puppet gets rejected with a "certificate
  revoked" error.
2. Can never check in with foreman again.

Anyone seen this before? This system was working just fine for a while. I
have other pre-existing hosts working fine on it(ssl certs exist/persist).
This is really driving me crazy.

Many thanks for your time,
JTH

John_Hughes · February 26, 2015, 4:59pm

A little more info, from the smart proxy log:

D, [2015-02-26T15:53:07.926652 #4589] DEBUG – : Added
foobar.do.clearleap.com (10.201.10.105 / 52:54:00:42:68:30) to
10.201.10.0/255.255.255.0
D, [2015-02-26T15:53:07.930857 #4589] DEBUG – : omshell: executed - set
name = "foobar.do.clearleap.com"
D, [2015-02-26T15:53:07.930926 #4589] DEBUG – : true
D, [2015-02-26T15:53:07.931002 #4589] DEBUG – : omshell: executed - set
ip-address = 10.201.10.105
D, [2015-02-26T15:53:07.931040 #4589] DEBUG – : true
D, [2015-02-26T15:53:07.931099 #4589] DEBUG – : omshell: executed - set
hardware-address = 52:54:00:42:68:30
D, [2015-02-26T15:53:07.931141 #4589] DEBUG – : true
D, [2015-02-26T15:53:07.931187 #4589] DEBUG – : omshell: executed - set
hardware-type = 1
D, [2015-02-26T15:53:07.931226 #4589] DEBUG – : true
D, [2015-02-26T15:53:07.931430 #4589] DEBUG – : omshell: executed - set
statements = "filename = "pxelinux.0"; option host-name =
"foobar.do.clearleap.com";"
D, [2015-02-26T15:53:07.931481 #4589] DEBUG – : true
D, [2015-02-26T15:53:07.931568 #4589] DEBUG – : omshell: executed - create
D, [2015-02-26T15:53:07.931576 #4589] DEBUG – : true
I, [2015-02-26T15:53:07.941136 #4589] INFO – : Added DHCP reservation for
foobar.do.clearleap.com (10.201.10.105 / 52:54:00:42:68:30)
10.201.10.11 - - [26/Feb/2015 15:53:07] "POST /dhcp/10.201.10.0 HTTP/1.1"
200 - 0.0251
D, [2015-02-26T15:53:08.714378 #4589] DEBUG – : running /usr/bin/nsupdate
-k /etc/bind/rndc.key
D, [2015-02-26T15:53:08.720555 #4589] DEBUG – : nsupdate: executed -
server 10.201.10.5
D, [2015-02-26T15:53:08.726385 #4589] DEBUG – : nsupdate: executed -
update add foobar.do.clearleap.com. 86400 A 10.201.10.105
10.201.10.11 - - [26/Feb/2015 15:53:08] "POST /dns/ HTTP/1.1" 200 - 0.0241
D, [2015-02-26T15:53:09.382059 #4589] DEBUG – : running /usr/bin/nsupdate
-k /etc/bind/rndc.key
D, [2015-02-26T15:53:09.388510 #4589] DEBUG – : nsupdate: executed -
server 10.201.10.5
D, [2015-02-26T15:53:09.391170 #4589] DEBUG – : nsupdate: executed -
update add 105.10.201.10.in-addr.arpa. 86400 IN PTR foobar.do.clearleap.com
10.201.10.11 - - [26/Feb/2015 15:53:09] "POST /dns/ HTTP/1.1" 200 - 0.0247
D, [2015-02-26T15:53:22.480896 #4589] DEBUG – : Found puppetca at
/usr/bin/puppet
D, [2015-02-26T15:53:22.481033 #4589] DEBUG – : Found sudo at /usr/bin/sudo
D, [2015-02-26T15:53:22.481086 #4589] DEBUG – : Executing /usr/bin/sudo -S
/usr/bin/puppet cert --ssldir /var/lib/puppet/ssl --clean
foobar.do.clearleap.com
I, [2015-02-26T15:53:23.638243 #4589] INFO – : Attempt to remove
nonexistant client certificate for foobar.do.clearleap.com
E, [2015-02-26T15:53:23.639023 #4589] ERROR – : Attempt to remove
nonexistant client certificate for foobar.do.clearleap.com
10.201.10.11 - - [26/Feb/2015 15:53:23] "DELETE
/puppet/ca/foobar.do.clearleap.com HTTP/1.1" 404 76 1.1602
I, [2015-02-26T15:53:23.719049 #4589] INFO – : Added
foobar.do.clearleap.com to autosign
10.201.10.11 - - [26/Feb/2015 15:53:23] "POST
/puppet/ca/autosign/foobar.do.clearleap.com HTTP/1.1" 200 - 0.0009
D, [2015-02-26T15:53:51.263564 #4589] DEBUG – : Found puppetca at
/usr/bin/puppet
D, [2015-02-26T15:53:51.263671 #4589] DEBUG – : Found sudo at /usr/bin/sudo
D, [2015-02-26T15:53:51.263728 #4589] DEBUG – : Executing /usr/bin/sudo -S
/usr/bin/puppet cert --ssldir /var/lib/puppet/ssl --clean
foobar.do.clearleap.com
I, [2015-02-26T15:53:52.374690 #4589] INFO – : cleaned puppet certificate
for foobar.do.clearleap.com
10.201.10.11 - - [26/Feb/2015 15:53:52] "DELETE
/puppet/ca/foobar.do.clearleap.com HTTP/1.1" 200 - 1.1123
I, [2015-02-26T15:53:52.412909 #4589] INFO – : Added
foobar.do.clearleap.com to autosign
10.201.10.11 - - [26/Feb/2015 15:53:52] "POST
/puppet/ca/autosign/foobar.do.clearleap.com HTTP/1.1" 200 - 0.0010
I, [2015-02-26T15:53:52.714729 #4589] INFO – : Removed
foobar.do.clearleap.com from autosign
10.201.10.11 - - [26/Feb/2015 15:53:52] "DELETE
/puppet/ca/autosign/foobar.do.clearleap.com HTTP/1.1" 200 - 0.0011

···

On Thursday, February 26, 2015 at 11:53:58 AM UTC-5, John Hughes wrote: > > Hi all, > > I am running into something a bit strange after an upgrade to 1.7.2(may > or may not be related): > > > 1. Create new host > 1. host gets built > 2. registers with Foreman/Puppet > 3. puppet certs get put into /var/lib/puppet/ssl on foreman server. > 2. Then a few seconds later > 1. Client SSL certs for the new host get removed by a puppet node > clean on the Foreman/Puppet server(seen in the logs). > 1. SSL certificates removed. > 3. However, the build finishes > 1. host works fine > 2. shows up in foreman > 3. puppet on new host works fine. > 4. Until Foreman/Puppet(Apache/Passenger) is restarted > 1. Then the new host's puppet gets rejected with a "certificate > revoked" error. > 2. Can never check in with foreman again. > > Anyone seen this before? This system was working just fine for a while. I > have other pre-existing hosts working fine on it(ssl certs exist/persist). > This is really driving me crazy. > > Many thanks for your time, > JTH >

John_Hughes · February 27, 2015, 1:25pm

After digging through the log(/var/log/foreman/production.log) I found this
error:

Operation FAILED: undefined method `path' for nil:NilClass
Completed 500 Internal Server Error in 1779.9ms

ArgumentError (There was no default layout for UnattendedController in
#<ActionView::PathSet:0x000000092e2328
@paths=[/usr/share/foreman/app/views,
/usr/share/foreman/vendor/ruby/1.9.1/gems/foreman_setup-2.1.0/app/views,
/usr/share/foreman/vendor/ruby/1
.9.1/gems/foreman_bootdisk-4.0.2/app/views,
/usr/share/foreman/vendor/ruby/1.9.1/gems/puppetdb_foreman-0.1.2/app/views,
/usr/share/foreman/vendor/ruby/1.9.1/gems/apipie-rails-0.2.6/app/views]>):

app/controllers/application_controller.rb:307:in `generic_exception'*
lib/middleware/catch_json_parse_errors.rb:9:in `call'*

Which led me to this
post: https://groups.google.com/forum/#!msg/foreman-users/t9jLeW2RmO8/Vh0x87Exw3cJ

And though we are not using user_data templates, I tried the modification
of the foreman_url as per this reply:

*Ah, are you using foreman_url with no options in your template? That's *
*a known bug with the user_data templates, try using *
*foreman_url('built') instead. *

And that solved the issue. The original install of this forman was 1.6.X,
then to 1.7.1, and recently to 1.7.2 on Debian Wheezy. It would seem our
modified templates were missing some stuff that has evolved. I am answering
my own post with the solution as a matter of reference for anyone else,
since no one had seemed to have run into this the way we did. I hope it can
be helpful.

Also, thank you Greg Sutcliffe for providing an answer almost a year ago
that solved my issue.

John Hughes

···

On Thursday, February 26, 2015 at 11:53:58 AM UTC-5, John Hughes wrote: > > Hi all, > > I am running into something a bit strange after an upgrade to 1.7.2(may > or may not be related): > > > 1. Create new host > 1. host gets built > 2. registers with Foreman/Puppet > 3. puppet certs get put into /var/lib/puppet/ssl on foreman server. > 2. Then a few seconds later > 1. Client SSL certs for the new host get removed by a puppet node > clean on the Foreman/Puppet server(seen in the logs). > 1. SSL certificates removed. > 3. However, the build finishes > 1. host works fine > 2. shows up in foreman > 3. puppet on new host works fine. > 4. Until Foreman/Puppet(Apache/Passenger) is restarted > 1. Then the new host's puppet gets rejected with a "certificate > revoked" error. > 2. Can never check in with foreman again. > > Anyone seen this before? This system was working just fine for a while. I > have other pre-existing hosts working fine on it(ssl certs exist/persist). > This is really driving me crazy. > > Many thanks for your time, > JTH >