Upgraded Foreman 1.8 to 1.9.3, smart-proxy certificates UI retrieval error ERF12-5356

Support
1

I updated my foreman install from 1.8 to 1.9.3 about 20 days ago and was
able to discover new hosts/deploy/sign discovered host certificates through
the web UI.

Today I tried to add a new discovered host, and am getting an error when
clicking the web UI Infrastructure -> Smart Proxies -> 'Certificates'
button: "ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA
certificates ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy
https://foreman.domain:8443/puppet/ca"

I've tried restarting foreman-proxy service and the host with the same
results.

tail proxy.log

I, [2016-02-03T13:45:38.493622 #9939] INFO – : 'puppet' settings were
initialized with default values: :puppet_provider: puppetrun, :puppetdir:
/etc/puppet, :salt_puppetrun_cmd: puppet.run, :use_cache: true
I, [2016-02-03T13:45:38.496173 #9939] INFO – : 'bmc' module is disabled.
I, [2016-02-03T13:45:38.496522 #9939] INFO – : 'realm' module is disabled.
D, [2016-02-03T13:45:59.839756 #9959] DEBUG – : verifying remote client
142.104.194.18 against trusted_hosts ["foreman.domain"]
D, [2016-02-03T13:45:59.842077 #9959] DEBUG – : Found puppetca at
/usr/bin/puppet
D, [2016-02-03T13:45:59.842192 #9959] DEBUG – : Found sudo at /usr/bin/sudo
D, [2016-02-03T13:45:59.842239 #9959] DEBUG – : Executing /usr/bin/sudo -S
/usr/bin/puppet cert --ssldir /var/lib/puppet/ssl --list --all
W, [2016-02-03T13:46:00.519101 #9959] WARN – : Failed to run puppetca:
E, [2016-02-03T13:46:00.519551 #9959] ERROR – : Failed to list
certificates: Execution of puppetca failed, check log files
142.104.194.18 - - [03/Feb/2016 13:46:00] "GET /puppet/ca HTTP/1.1" 406 74
0.6808

su - foreman-proxy

Last login: Wed Feb 3 13:52:09 PST 2016 on pts/0
-bash-4.2$ /usr/bin/sudo -S /usr/bin/puppet cert --ssldir
/var/lib/puppet/ssl --list --all
/usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in encode': "\xC5" on US-ASCII (Encoding::InvalidByteSequenceError) from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:ininitialize'
from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in new' from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:inparse'
from /usr/share/ruby/vendor_ruby/puppet/module.rb:62:in has_metadata?' from /usr/share/ruby/vendor_ruby/puppet/module.rb:49:ininitialize'
from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:355:in new' from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:355:inblock
(2 levels) in <class:Environment>'
from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:353:in collect&#39; from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:353:inblock
in <class:Environment>'
from /usr/share/ruby/vendor_ruby/puppet/util/cacher.rb:55:in cached_value&#39; from /usr/share/ruby/vendor_ruby/puppet/util/cacher.rb:29:inblock in
cached_attr'
from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:296:in
each_plugin_directory&#39; from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:137:inrun'
from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:92:in execute&#39; from /usr/bin/puppet:8:in<main>'

If I run the above as root, or my own user with sudo, I get a return of all
the current certificates listed with fingerprints.

cat /etc/sudoers.d/foreman-proxy

foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet cert *
foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet kick *
Defaults:foreman-proxy !requiretty

/var/log/secure:
Feb 3 13:52:50 foreman su: pam_unix(su-l:session): session opened for user
foreman-proxy by sdainard(uid=0)
Feb 3 13:52:57 foreman sudo: foreman-proxy : TTY=pts/0 ;
PWD=/usr/share/foreman-proxy ; USER=root ; COMMAND=/usr/bin/puppet cert
–ssldir /var/lib/puppet/ssl --list --all
Feb 3 13:55:21 foreman su: pam_unix(su-l:session): session closed for user
foreman-proxy

su - foreman-proxy

Last login: Wed Feb 3 13:57:29 PST 2016 on pts/0
-bash-4.2$ sudo -l
Matching Defaults entries for foreman-proxy on this host:
requiretty, !visiblepw, always_set_home, env_reset, env_keep="COLORS
DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS",
env_keep+="MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE",
env_keep+="LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES",
env_keep+="LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE",
env_keep+="LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY",
secure_path=/sbin:/bin:/usr/sbin:/usr/bin, !requiretty

User foreman-proxy may run the following commands on this host:
(root) NOPASSWD: /usr/bin/puppet cert *
(root) NOPASSWD: /usr/bin/puppet kick *

ps aux | grep foreman-proxy

foreman+ 2959 0.0 0.4 369800 51540 ? Sl 14:25 0:00 ruby
/usr/share/foreman-proxy/bin/smart-proxy

I can't think of anything that has changed on the Foreman host since the
upgrade to 1.9.3, and I remember rebooting after the upgrade and testing if
everything was working.

Any help is appreciated.

2

Seeing as user foreman-proxy and root have different results running:
/usr/bin/sudo -S /usr/bin/puppet cert --ssldir /var/lib/puppet/ssl --list
–all

I thought it might be a good test to see if foreman-proxy running as root
could retrieve certificates, but that returned the same error.

··· On Wednesday, 3 February 2016 14:42:20 UTC-8, steved0ca wrote: > > I updated my foreman install from 1.8 to 1.9.3 about 20 days ago and was > able to discover new hosts/deploy/sign discovered host certificates through > the web UI. > > Today I tried to add a new discovered host, and am getting an error when > clicking the web UI Infrastructure -> Smart Proxies -> 'Certificates' > button: "ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA > certificates ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy > https://foreman.domain:8443/puppet/ca" > > I've tried restarting foreman-proxy service and the host with the same > results. > > # tail proxy.log > I, [2016-02-03T13:45:38.493622 #9939] INFO -- : 'puppet' settings were > initialized with default values: :puppet_provider: puppetrun, :puppetdir: > /etc/puppet, :salt_puppetrun_cmd: puppet.run, :use_cache: true > I, [2016-02-03T13:45:38.496173 #9939] INFO -- : 'bmc' module is disabled. > I, [2016-02-03T13:45:38.496522 #9939] INFO -- : 'realm' module is > disabled. > D, [2016-02-03T13:45:59.839756 #9959] DEBUG -- : verifying remote client > 142.104.194.18 against trusted_hosts ["foreman.domain"] > D, [2016-02-03T13:45:59.842077 #9959] DEBUG -- : Found puppetca at > /usr/bin/puppet > D, [2016-02-03T13:45:59.842192 #9959] DEBUG -- : Found sudo at > /usr/bin/sudo > D, [2016-02-03T13:45:59.842239 #9959] DEBUG -- : Executing /usr/bin/sudo > -S /usr/bin/puppet cert --ssldir /var/lib/puppet/ssl --list --all > W, [2016-02-03T13:46:00.519101 #9959] WARN -- : Failed to run puppetca: > E, [2016-02-03T13:46:00.519551 #9959] ERROR -- : Failed to list > certificates: Execution of puppetca failed, check log files > 142.104.194.18 - - [03/Feb/2016 13:46:00] "GET /puppet/ca HTTP/1.1" 406 74 > 0.6808 > > > > # su - foreman-proxy > Last login: Wed Feb 3 13:52:09 PST 2016 on pts/0 > -bash-4.2$ /usr/bin/sudo -S /usr/bin/puppet cert --ssldir > /var/lib/puppet/ssl --list --all > /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in `encode': "\xC5" > on US-ASCII (Encoding::InvalidByteSequenceError) > from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in `initialize' > from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in `new' > from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in `parse' > from /usr/share/ruby/vendor_ruby/puppet/module.rb:62:in `has_metadata?' > from /usr/share/ruby/vendor_ruby/puppet/module.rb:49:in `initialize' > from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:355:in `new' > from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:355:in `block > (2 levels) in ' > from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:353:in > `collect' > from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:353:in `block > in ' > from /usr/share/ruby/vendor_ruby/puppet/util/cacher.rb:55:in `cached_value' > from /usr/share/ruby/vendor_ruby/puppet/util/cacher.rb:29:in `block in > cached_attr' > from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:296:in > `each_plugin_directory' > from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run' > from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:92:in > `execute' > from /usr/bin/puppet:8:in `' > > If I run the above as root, or my own user with sudo, I get a return of > all the current certificates listed with fingerprints. > > # cat /etc/sudoers.d/foreman-proxy > foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet cert * > foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet kick * > Defaults:foreman-proxy !requiretty > > /var/log/secure: > Feb 3 13:52:50 foreman su: pam_unix(su-l:session): session opened for > user foreman-proxy by sdainard(uid=0) > Feb 3 13:52:57 foreman sudo: foreman-proxy : TTY=pts/0 ; > PWD=/usr/share/foreman-proxy ; USER=root ; COMMAND=/usr/bin/puppet cert > --ssldir /var/lib/puppet/ssl --list --all > Feb 3 13:55:21 foreman su: pam_unix(su-l:session): session closed for > user foreman-proxy > > # su - foreman-proxy > Last login: Wed Feb 3 13:57:29 PST 2016 on pts/0 > -bash-4.2$ sudo -l > Matching Defaults entries for foreman-proxy on this host: > requiretty, !visiblepw, always_set_home, env_reset, env_keep="COLORS > DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS", > env_keep+="MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE", > env_keep+="LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES", > env_keep+="LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE", > env_keep+="LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY", > secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin, !requiretty > > User foreman-proxy may run the following commands on this host: > (root) NOPASSWD: /usr/bin/puppet cert * > (root) NOPASSWD: /usr/bin/puppet kick * > > # ps aux | grep foreman-proxy > foreman+ 2959 0.0 0.4 369800 51540 ? Sl 14:25 0:00 ruby > /usr/share/foreman-proxy/bin/smart-proxy > > > I can't think of anything that has changed on the Foreman host since the > upgrade to 1.9.3, and I remember rebooting after the upgrade and testing if > everything was working. > > Any help is appreciated. >
3

I've reverted to the 1.8.2 install, and noticed a couple things.

If I update to 1.8.4 using yum, I run into the same certificate issues.

But I can successfully update from 1.8.2 to 1.8.4 by running:

yum update --exclude=ruby*

After this I don't run into the certificates issue.

So it appears that one of the following packages is the offender:

··· ============================================================================================================================================ Package Arch Version Repository Size ============================================================================================================================================ Updating: ruby193-rubygem-rabl noarch 0.11.6-1.el7 foreman 154 k rubygem-bundler noarch 1.7.8-3.el7 base 147 k rubygem-rack-protection noarch 1.5.3-3.el7 epel 15 k rubygem-rkerberos x86_64 0.1.3-5.el7 epel 28 k rubygem-thor noarch 0.19.1-1.el7 base 52 k

Feel like I’ve been talking to myself, but hopefully this is useful for
someone else searching.

On Wednesday, 3 February 2016 14:42:20 UTC-8, steved0ca wrote:

I updated my foreman install from 1.8 to 1.9.3 about 20 days ago and was
able to discover new hosts/deploy/sign discovered host certificates through
the web UI.

Today I tried to add a new discovered host, and am getting an error when
clicking the web UI Infrastructure → Smart Proxies → ‘Certificates’
button: “ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA
certificates ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy
https://foreman.domain:8443/puppet/ca”

I’ve tried restarting foreman-proxy service and the host with the same
results.

tail proxy.log

I, [2016-02-03T13:45:38.493622 #9939] INFO – : ‘puppet’ settings were
initialized with default values: :puppet_provider: puppetrun, :puppetdir:
/etc/puppet, :salt_puppetrun_cmd: puppet.run, :use_cache: true
I, [2016-02-03T13:45:38.496173 #9939] INFO – : ‘bmc’ module is disabled.
I, [2016-02-03T13:45:38.496522 #9939] INFO – : ‘realm’ module is
disabled.
D, [2016-02-03T13:45:59.839756 #9959] DEBUG – : verifying remote client
142.104.194.18 against trusted_hosts [“foreman.domain”]
D, [2016-02-03T13:45:59.842077 #9959] DEBUG – : Found puppetca at
/usr/bin/puppet
D, [2016-02-03T13:45:59.842192 #9959] DEBUG – : Found sudo at
/usr/bin/sudo
D, [2016-02-03T13:45:59.842239 #9959] DEBUG – : Executing /usr/bin/sudo
-S /usr/bin/puppet cert --ssldir /var/lib/puppet/ssl --list --all
W, [2016-02-03T13:46:00.519101 #9959] WARN – : Failed to run puppetca:
E, [2016-02-03T13:46:00.519551 #9959] ERROR – : Failed to list
certificates: Execution of puppetca failed, check log files
142.104.194.18 - - [03/Feb/2016 13:46:00] “GET /puppet/ca HTTP/1.1” 406 74
0.6808

su - foreman-proxy

Last login: Wed Feb 3 13:52:09 PST 2016 on pts/0
-bash-4.2$ /usr/bin/sudo -S /usr/bin/puppet cert --ssldir
/var/lib/puppet/ssl --list --all
/usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in encode': "\xC5" on US-ASCII (Encoding::InvalidByteSequenceError) from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in initialize’
from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in new' from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in parse’
from /usr/share/ruby/vendor_ruby/puppet/module.rb:62:in has_metadata?' from /usr/share/ruby/vendor_ruby/puppet/module.rb:49:in initialize’
from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:355:in new' from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:355:in block
(2 levels) in class:Environment’
from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:353:in
collect' from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:353:in block
in class:Environment’
from /usr/share/ruby/vendor_ruby/puppet/util/cacher.rb:55:in cached_value' from /usr/share/ruby/vendor_ruby/puppet/util/cacher.rb:29:in block in
cached_attr’
from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:296:in
each_plugin_directory' from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:137:in run’
from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:92:in
execute' from /usr/bin/puppet:8:in ’

If I run the above as root, or my own user with sudo, I get a return of
all the current certificates listed with fingerprints.

cat /etc/sudoers.d/foreman-proxy

foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet cert *
foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet kick *
Defaults:foreman-proxy !requiretty

/var/log/secure:
Feb 3 13:52:50 foreman su: pam_unix(su-l:session): session opened for
user foreman-proxy by sdainard(uid=0)
Feb 3 13:52:57 foreman sudo: foreman-proxy : TTY=pts/0 ;
PWD=/usr/share/foreman-proxy ; USER=root ; COMMAND=/usr/bin/puppet cert
–ssldir /var/lib/puppet/ssl --list --all
Feb 3 13:55:21 foreman su: pam_unix(su-l:session): session closed for
user foreman-proxy

su - foreman-proxy

Last login: Wed Feb 3 13:57:29 PST 2016 on pts/0
-bash-4.2$ sudo -l
Matching Defaults entries for foreman-proxy on this host:
requiretty, !visiblepw, always_set_home, env_reset, env_keep=“COLORS
DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS”,
env_keep+=“MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE”,
env_keep+=“LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES”,
env_keep+=“LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE”,
env_keep+=“LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY”,
secure_path=/sbin:/bin:/usr/sbin:/usr/bin, !requiretty

User foreman-proxy may run the following commands on this host:
(root) NOPASSWD: /usr/bin/puppet cert *
(root) NOPASSWD: /usr/bin/puppet kick *

ps aux | grep foreman-proxy

foreman+ 2959 0.0 0.4 369800 51540 ? Sl 14:25 0:00 ruby
/usr/share/foreman-proxy/bin/smart-proxy

I can’t think of anything that has changed on the Foreman host since the
upgrade to 1.9.3, and I remember rebooting after the upgrade and testing if
everything was working.

Any help is appreciated.

4

Without any system updates this same issue occurred.

-bash-4.2$ /usr/bin/sudo /usr/bin/puppet cert --ssldir /var/lib/puppet/ssl
–list --all
/usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in encode&#39;: &quot;\xC5&quot; on US-ASCII (Encoding::InvalidByteSequenceError) from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:ininitialize'
from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in new&#39; from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:inparse'
from /usr/share/ruby/vendor_ruby/puppet/module.rb:62:in has_metadata?&#39; from /usr/share/ruby/vendor_ruby/puppet/module.rb:49:ininitialize'
from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:355:in new&#39; from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:355:inblock
(2 levels) in <class:Environment>'
from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:353:in collect&#39; from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:353:inblock
in <class:Environment>'
from /usr/share/ruby/vendor_ruby/puppet/util/cacher.rb:55:in cached_value&#39; from /usr/share/ruby/vendor_ruby/puppet/util/cacher.rb:29:inblock in
cached_attr'
from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:296:in
each_plugin_directory&#39; from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:137:inrun'
from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:92:in execute&#39; from /usr/bin/puppet:8:in<main>'

Ensuring /etc/locale.conf was configured was the fix.

cat /etc/locale.conf

LANG=en_CA.utf8

··· On Wednesday, 17 February 2016 14:43:19 UTC-8, steved0ca wrote: > > I've reverted to the 1.8.2 install, and noticed a couple things. > > If I update to 1.8.4 using yum, I run into the same certificate issues. > > But I can successfully update from 1.8.2 to 1.8.4 by running: > > yum update --exclude=ruby* > > After this I don't run into the certificates issue. > > So it appears that one of the following packages is the offender: > > ============================================================================================================================================ > Package Arch > Version Repository Size > > ============================================================================================================================================ > Updating: > ruby193-rubygem-rabl noarch > 0.11.6-1.el7 foreman 154 k > rubygem-bundler noarch > 1.7.8-3.el7 base 147 k > rubygem-rack-protection noarch > 1.5.3-3.el7 epel 15 k > rubygem-rkerberos x86_64 > 0.1.3-5.el7 epel 28 k > rubygem-thor noarch > 0.19.1-1.el7 base 52 k > > > Feel like I've been talking to myself, but hopefully this is useful for > someone else searching. > > On Wednesday, 3 February 2016 14:42:20 UTC-8, steved0ca wrote: >> >> I updated my foreman install from 1.8 to 1.9.3 about 20 days ago and was >> able to discover new hosts/deploy/sign discovered host certificates through >> the web UI. >> >> Today I tried to add a new discovered host, and am getting an error when >> clicking the web UI Infrastructure -> Smart Proxies -> 'Certificates' >> button: "ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA >> certificates ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy >> https://foreman.domain:8443/puppet/ca" >> >> I've tried restarting foreman-proxy service and the host with the same >> results. >> >> # tail proxy.log >> I, [2016-02-03T13:45:38.493622 #9939] INFO -- : 'puppet' settings were >> initialized with default values: :puppet_provider: puppetrun, :puppetdir: >> /etc/puppet, :salt_puppetrun_cmd: puppet.run, :use_cache: true >> I, [2016-02-03T13:45:38.496173 #9939] INFO -- : 'bmc' module is disabled. >> I, [2016-02-03T13:45:38.496522 #9939] INFO -- : 'realm' module is >> disabled. >> D, [2016-02-03T13:45:59.839756 #9959] DEBUG -- : verifying remote client >> 142.104.194.18 against trusted_hosts ["foreman.domain"] >> D, [2016-02-03T13:45:59.842077 #9959] DEBUG -- : Found puppetca at >> /usr/bin/puppet >> D, [2016-02-03T13:45:59.842192 #9959] DEBUG -- : Found sudo at >> /usr/bin/sudo >> D, [2016-02-03T13:45:59.842239 #9959] DEBUG -- : Executing /usr/bin/sudo >> -S /usr/bin/puppet cert --ssldir /var/lib/puppet/ssl --list --all >> W, [2016-02-03T13:46:00.519101 #9959] WARN -- : Failed to run puppetca: >> E, [2016-02-03T13:46:00.519551 #9959] ERROR -- : Failed to list >> certificates: Execution of puppetca failed, check log files >> 142.104.194.18 - - [03/Feb/2016 13:46:00] "GET /puppet/ca HTTP/1.1" 406 >> 74 0.6808 >> >> >> >> # su - foreman-proxy >> Last login: Wed Feb 3 13:52:09 PST 2016 on pts/0 >> -bash-4.2$ /usr/bin/sudo -S /usr/bin/puppet cert --ssldir >> /var/lib/puppet/ssl --list --all >> /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in `encode': >> "\xC5" on US-ASCII (Encoding::InvalidByteSequenceError) >> from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in >> `initialize' >> from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in `new' >> from /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in `parse' >> from /usr/share/ruby/vendor_ruby/puppet/module.rb:62:in `has_metadata?' >> from /usr/share/ruby/vendor_ruby/puppet/module.rb:49:in `initialize' >> from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:355:in `new' >> from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:355:in `block >> (2 levels) in ' >> from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:353:in >> `collect' >> from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:353:in `block >> in ' >> from /usr/share/ruby/vendor_ruby/puppet/util/cacher.rb:55:in >> `cached_value' >> from /usr/share/ruby/vendor_ruby/puppet/util/cacher.rb:29:in `block in >> cached_attr' >> from /usr/share/ruby/vendor_ruby/puppet/node/environment.rb:296:in >> `each_plugin_directory' >> from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run' >> from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:92:in >> `execute' >> from /usr/bin/puppet:8:in `' >> >> If I run the above as root, or my own user with sudo, I get a return of >> all the current certificates listed with fingerprints. >> >> # cat /etc/sudoers.d/foreman-proxy >> foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet cert * >> foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet kick * >> Defaults:foreman-proxy !requiretty >> >> /var/log/secure: >> Feb 3 13:52:50 foreman su: pam_unix(su-l:session): session opened for >> user foreman-proxy by sdainard(uid=0) >> Feb 3 13:52:57 foreman sudo: foreman-proxy : TTY=pts/0 ; >> PWD=/usr/share/foreman-proxy ; USER=root ; COMMAND=/usr/bin/puppet cert >> --ssldir /var/lib/puppet/ssl --list --all >> Feb 3 13:55:21 foreman su: pam_unix(su-l:session): session closed for >> user foreman-proxy >> >> # su - foreman-proxy >> Last login: Wed Feb 3 13:57:29 PST 2016 on pts/0 >> -bash-4.2$ sudo -l >> Matching Defaults entries for foreman-proxy on this host: >> requiretty, !visiblepw, always_set_home, env_reset, env_keep="COLORS >> DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS", >> env_keep+="MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE", >> env_keep+="LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES", >> env_keep+="LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE", >> env_keep+="LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY", >> secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin, !requiretty >> >> User foreman-proxy may run the following commands on this host: >> (root) NOPASSWD: /usr/bin/puppet cert * >> (root) NOPASSWD: /usr/bin/puppet kick * >> >> # ps aux | grep foreman-proxy >> foreman+ 2959 0.0 0.4 369800 51540 ? Sl 14:25 0:00 ruby >> /usr/share/foreman-proxy/bin/smart-proxy >> >> >> I can't think of anything that has changed on the Foreman host since the >> upgrade to 1.9.3, and I remember rebooting after the upgrade and testing if >> everything was working. >> >> Any help is appreciated. >> >