Puppetca proxy issues starting after upgrade 2.0 -> 2.1

Problem:
After upgrading from 2.0 to 2.1 one of my proxies fails to start on boot. But if I systemctl restart foreman-proxy then it starts fine. My other proxy runs fine.

Log from failed start on boot:

2020-07-11T23:38:28  [E] Disabling all modules in the group ['puppetca_http_api', 'puppetca_hostname_whitelisting', 'puppetca'] due to a failure in one of them: File at '/etc/puppetlabs/puppet/ssl/certs/ca.pem' defined in 'puppet_ssl_ca' 
parameter doesn't exist or is unreadable
2020-07-11T23:38:28  [E] Disabling all modules in the group ['puppet_proxy_puppet_api', 'puppet'] due to a failure in o
ne of them: File at '/etc/puppetlabs/puppet/ssl/certs/ca.pem' defined in 'puppet_ssl_ca' parameter doesn't exist or is 
unreadable
2020-07-11T23:38:28  [I] Successfully initialized 'foreman_proxy'
2020-07-11T23:38:28  [I] Successfully initialized 'logs'
2020-07-11T23:38:28  [E] Unable to load private SSL key. Are the values correct in settings.yml and do permissions allo
w reading?
2020-07-11T23:38:28  [W] Error details for Unable to load private SSL key. Are the values correct in settings.yml and d
o permissions allow reading?: <Errno::ENOENT>: No such file or directory @ rb_sysopen - /etc/puppetlabs/puppet/ssl/private_keys/puppet.XXX.pem
/usr/share/foreman-proxy/lib/launcher.rb:111:in `read'
/usr/share/foreman-proxy/lib/launcher.rb:111:in `load_ssl_private_key'
/usr/share/foreman-proxy/lib/launcher.rb:101:in `https_app'
/usr/share/foreman-proxy/lib/launcher.rb:174:in `launch'
/usr/share/foreman-proxy/bin/smart-proxy:6:in `<main>'
2020-07-11T23:38:28  [E] Error during startup, terminating
2020-07-11T23:38:28  [W] Error details for Error during startup, terminating: <Errno::ENOENT>: No such file or director
y @ rb_sysopen - /etc/puppetlabs/puppet/ssl/private_keys/puppet.XXX.pem
/usr/share/foreman-proxy/lib/launcher.rb:111:in `read'
/usr/share/foreman-proxy/lib/launcher.rb:111:in `load_ssl_private_key'
/usr/share/foreman-proxy/lib/launcher.rb:101:in `https_app'
/usr/share/foreman-proxy/lib/launcher.rb:174:in `launch'
/usr/share/foreman-proxy/bin/smart-proxy:6:in `<main>'

The permissions for for the .pem are correct:

# ll /etc/puppetlabs/puppet/ssl/private_keys
total 12
drwxr-x--- 2 puppet puppet 6144 Oct 30  2019 ./
drwxrwx--x 8 puppet puppet 6144 Jul 11 23:39 ../
-rw-r----- 1 puppet puppet 3243 Oct 30  2019 puppet.XXX.pem
# groups foreman-proxy 
foreman-proxy : foreman-proxy puppet

Expected outcome:
The proxy starts on boot.

Foreman and Proxy versions:
Foreman: 2.1.0
Proxy: 2.1.0

Foreman and Proxy plugin versions:
foreman_bootdisk: 17.0.2
foreman_setup: 6.0.0

Distribution and version:
Ubuntu 18.04

Other relevant data:
As above.

That’s odd. Some things to check:

  • Is the complete path to /etc/puppetlabs/puppet/ssl/private_keys readable (so also all parent directories)?
  • Is /etc/puppetlabs/puppet/ssl/certs/ca.pem also readable?
  • Is there a casing issue by any chance? uppercase vs lowercase?
  • AFAIK Ubuntu has no SELinux but perhaps AppArmor is getting in the way?

OK, found it. Was my mistake sorry. I am running this in AWS and /etc/puppetlabs is on EFS and mounted through fstab. That mount isn’t up when the foreman-proxy service first tries to start.

So I need to add RequiresMountsFor=/etc/puppetlabs into a systemd drop-in file.

Thanks for your help.

1 Like