I have a couple questions on how Foreman manages puppet CA stuff.
1). Autosign: In previous versions of Foreman, I could set Autosign to a domain name wildcard and that’s all that got put into the autosign.conf file. Now, for every new instance, a FQDN entry is put into the Autosign.conf file. The problem is, since we’re not using Foreman for provisioning or building those entries never go away.
I still want to use Foreman to be able to delete certs from the CA, though. Is there a way to get that configured? ATM, I’ve had to "chattr +i the Autosign.conf file, which is not ideal
2). Actual certificate management: Why does Foreman still use ssh to the puppet CA and “puppet cert list --all” “puppet cert clean $fqdn”? The CA has an API that can be used that seems like it would be better: Puppet HTTP API: Certificate Status
The other issue with this, is we have 15k+ instances. “puppet cert list --all” takes a long time and a lot of CPU. We have a 4c 12G ram instance just for CA functionality and it’s not uncommon for cert deletion to completely kill the instance, particularly when we’re deleting hundreds a day (lots of flex up/flex downs).
The other issue with this, and it might just be a bug, but if we refresh our puppet environments in Foreman via the web UI, and a few cert deletions come in during that process, the environment refresh errors out with an SSL error. Hammer works fine in the same situation. And the environment import works fine if not cert deletions are triggered.
Hopefully someone can help provide some insight into this because it’s killing me
For the first part, you might want to look at the manual for PuppetCA. The puppetca_token_whitelisting provider is probably closer to what you want.
For the second part the HTTP API in Puppet 5 is incomplete. In Puppet 6 it provides all the fields. You can set the version to 6 and trick it into using the API. Then you can manage certs, but don’t get fields like expiration. The UI might look odd here and there, but all actions work.
While I can not help with your original question, running a newer proxy version with an older Foreman version is generally a bad idea and will most likely breake. Foreman and foreman-proxy should always be the same version. If anything an older version of the proxy might work, but this should also only happen during upgrades.
I’ve been using Foreman since 0.5 AFAIK it has always worked with specific FQDNs. You could have manually put in unmanaged entries to autosign.conf and that should still work, but that should still work.
The reason I suggested the puppetca_token_whitelisting provider is that it doesn’t write to autosign.conf. However, Puppetserver will still read from it. You don’t have to set up the Puppetserver to actually read the tokens.
AFAIK the APIs didn’t drop anything so it should be compatible.
This is always a bit of an interesting area. While in Katello the common workflow is to first upgrade the Foreman/Katello server, I know that the previous maintainer of the Foreman Proxy advocated for first upgrading the proxies. That’s because it’s easier to keep its REST API’s compatible than Foreman’s code to deal with multiple versions.
Right now it should generally work with a pretty wide range since the API’s are quite stable now, except for some modules (HTTPBoot has seen the most changes).
However, I would recommend to upgrade the server to a supported version. With time it becomes more difficult and it’s a good practice. Also since there are known security issues building up.
Thanks for all that, I do appreciate it. I was looking at the API’s after being intrigued by your comment about the API in v5 being incomplete and found out where they’re not complete.
It seems that yes, you can “revoke” and “delete” a cert via the API, but the API only removes the file from the ca/signed directory, and not any of the other files. So…if you try to recreate a cert for a FQDN, it failed because according to the CA, a cert already exists, lol! So using proxy version 1.22 wouldn’t matter anyway.
However, after writing a script to call the API and delete the remaining files (if they exist) and then modifying the proxy code to call my script[1], I think I have a solution that works. Certs are getting deleted and there are no huge cpu spikes while it’s happening.
[1] It’s not ideal. It’d be nice if there was a parameter in a settings file that could be used to specify the command to run.
