Puppet certificate when provisioning or deleting a host

I've installed Foreman 1.4 on 64-bit RHEL 6.5 using foreman-installer. That
installed puppet 2.7.23 from EPEL.

When trying to create my first foreman-managed host, it was unable to fetch
the kickstart. The logs show foreman failing to remove the puppet
certificate. If I try to delete the host, I get the same error. I've found
references to this being a problem with combinations of earlier versions of
foreman and puppet, but those discussions suggest it should be fixed now.
Does anyone have suggestions on what could be wrong or how I should
troubleshoot this?

The full log entries are

Started GET
"/unattended/provision?token=6890eecb-5363-4b76-ba13-6f45b66a1df8" for
192.168.32.100 at 2014-02-14 11:14:10 -0600
Processing by UnattendedController#provision as /
Parameters: {"token"=>"6890eecb-5363-4b76-ba13-6f45b66a1df8"}
Found test1.nuc
Remove puppet certificate for test1.nuc
Failed to remove test1.nuc's puppet certificate: 406 Not Acceptable
Rendered text template (0.0ms)
Filter chain halted as :handle_ca rendered or redirected
Completed 500 Internal Server Error in 114ms (Views: 2.1ms | ActiveRecord:
1.5ms)

Started DELETE "/hosts/test1.nuc" for 128.206.21.109 at 2014-02-14 11:34:53
-0600
Processing by HostsController#destroy as HTML
Parameters:
{"authenticity_token"=>"3cOE9D7fSuhl8uW6DQ6BLHZe74ne7rw2C3YvTbwfFEA=",
"id"=>"test1.nuc"}
Delete the DNS PTR record for 192.168.32.100/test1.nuc
Delete the DNS A record for test1.nuc/192.168.32.100
Delete DHCP reservation for test1.nuc-00:1d:09:2e:8a:e1/192.168.32.100
Delete the TFTP configuration for test1.nuc
Remove puppet certificate for test1.nuc
Failed to remove test1.nuc's puppet certificate: 406 Not Acceptable
Rolling back due to a problem: [Delete PuppetCA certificates for
test1.nuc 50 failed [#<Host::Managed id: 2, name: "test1.nuc",
ip: "192.168.32.100", last_compile: nil, last_freshcheck: nil, last_report:
nil, updated_at: "2014-02-14 17:13:28", source_file_id: nil, created_at:
"2014-02-14 17:12:57", mac: "00:1d:09:2e:8a:e1", root_pass: nil, serial:
nil, puppet_status: 0, domain_id: 2, architecture_id: 1,
operatingsystem_id: 2, environment_id: 1, subnet_id: 1, ptable_id: 6,
medium_id: 3, build: true, comment: "testing provisioning", disk: "",
installed_at: nil, model_id: nil, hostgroup_id: nil, owner_id: 1,
owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true,
use_image: nil, image_file: nil, uuid: nil, compute_resource_id: nil,
puppet_proxy_id: 1, certname: nil, image_id: nil, organization_id: nil,
location_id: nil, type: "Host::Managed", compute_profile_id: nil>,
:delCertificate]]
Add the TFTP configuration for test1.nuc
Create DHCP reservation for test1.nuc-00:1d:09:2e:8a:e1/192.168.32.100
Add DNS A record for test1.nuc/192.168.32.100
Add DNS PTR record for 192.168.32.100/test1.nuc
Failed to save: Failed to remove test1.nuc's puppet certificate: 406 Not
Acceptable
Redirected to https://datascience-3.rnet.missouri.edu/hosts
Completed 302 Found in 1486ms (ActiveRecord: 13.8ms)

This is a bad response from the smart proxy, which is running puppet
commands to remove the certificate. Its logs are at
/var/log/foreman-proxy/proxy.log.

It's very likely an issue with sudoers rules if you've upgraded or
downgraded versions of Puppet. For Puppet 2,
/etc/sudoers.d/foreman-proxy should reference /usr/sbin/puppetca and for
Puppet 3 it should reference /usr/bin/puppet cert.

You'll find the full files further down in this section:
http://theforeman.org/manuals/1.4/index.html#4.3.2SmartProxySettings

··· On 14/02/14 17:57, Brian Pitts wrote: > I've installed Foreman 1.4 on 64-bit RHEL 6.5 using foreman-installer. > That installed puppet 2.7.23 from EPEL. > > When trying to create my first foreman-managed host, it was unable to > fetch the kickstart. The logs show foreman failing to remove the puppet > certificate. If I try to delete the host, I get the same error. I've > found references to this being a problem with combinations of earlier > versions of foreman and puppet, but those discussions suggest it should > be fixed now. Does anyone have suggestions on what could be wrong or how > I should troubleshoot this? > > The full log entries are > > Started GET > "/unattended/provision?token=6890eecb-5363-4b76-ba13-6f45b66a1df8" for > 192.168.32.100 at 2014-02-14 11:14:10 -0600 > Processing by UnattendedController#provision as */* > Parameters: {"token"=>"6890eecb-5363-4b76-ba13-6f45b66a1df8"} > Found test1.nuc > Remove puppet certificate for test1.nuc > Failed to remove test1.nuc's puppet certificate: 406 Not Acceptable


Dominic Cleal
Red Hat Engineering

That was it. The proxy logs showed

D, [2014-02-14T11:34:54.539426 #24146] DEBUG – : Found puppetca at
/usr/sbin/puppetca
D, [2014-02-14T11:34:54.539556 #24146] DEBUG – : Found sudo at
/usr/bin/sudo
D, [2014-02-14T11:34:54.539609 #24146] DEBUG – : Executing
/usr/bin/sudo -S /usr/sbin/puppetca --ssldir /var/lib/puppet/ssl --clean
test1.nuc
W, [2014-02-14T11:34:54.558675 #24146] WARN – : Failed to run
puppetca: [sudo] password for foreman-proxy:
E, [2014-02-14T11:34:54.559148 #24146] ERROR – : Failed to remove
certificate(s) for test1.nuc: Execution of puppetca failed, check log files

and when I checked the sudoers config it was referencing the wrong commands.

foreman-proxy ALL = NOPASSWD : /usr/bin/puppet cert *, /usr/bin/puppet
kick *
Defaults:foreman-proxy !requiretty

I changed the first line to

foreman-proxy ALL = NOPASSWD : /usr/sbin/puppeca *, /usr/bin/puppet kick *

and that resolved the problem. Thanks for your help!

··· On 02/14/2014 12:08 PM, Dominic Cleal wrote: > On 14/02/14 17:57, Brian Pitts wrote: >> I've installed Foreman 1.4 on 64-bit RHEL 6.5 using foreman-installer. >> That installed puppet 2.7.23 from EPEL. >> >> When trying to create my first foreman-managed host, it was unable to >> fetch the kickstart. The logs show foreman failing to remove the puppet >> certificate. If I try to delete the host, I get the same error. I've >> found references to this being a problem with combinations of earlier >> versions of foreman and puppet, but those discussions suggest it should >> be fixed now. Does anyone have suggestions on what could be wrong or how >> I should troubleshoot this? >> >> The full log entries are >> >> Started GET >> "/unattended/provision?token=6890eecb-5363-4b76-ba13-6f45b66a1df8" for >> 192.168.32.100 at 2014-02-14 11:14:10 -0600 >> Processing by UnattendedController#provision as */* >> Parameters: {"token"=>"6890eecb-5363-4b76-ba13-6f45b66a1df8"} >> Found test1.nuc >> Remove puppet certificate for test1.nuc >> Failed to remove test1.nuc's puppet certificate: 406 Not Acceptable > > This is a bad response from the smart proxy, which is running puppet > commands to remove the certificate. Its logs are at > /var/log/foreman-proxy/proxy.log. > > It's very likely an issue with sudoers rules if you've upgraded or > downgraded versions of Puppet. For Puppet 2, > /etc/sudoers.d/foreman-proxy should reference /usr/sbin/puppetca and for > Puppet 3 it should reference /usr/bin/puppet cert. > > You'll find the full files further down in this section: > http://theforeman.org/manuals/1.4/index.html#4.3.2SmartProxySettings


All the best,
Brian Pitts