RFC: Redesign Certificate Handling within Foreman Deployments

Development RFCs
41

In general looks good, only question I have is regarding the puppet usecases and how do they fit in here?

42

I didn’t forget your question :slight_smile: But I did need to do some further digging and thinking around designs to come back and propose some technical follow up.

Certificate Management

First, here are two options around how I have been thinking about how to approach certificates input to the installer and propagated to the puppet modules from a strategy perspective. My overarching design principle is:

Separate creation of certificates from the management of the certificates

Option 1: Copy and manage

Steps

  • User provides certificates as input
  • Module copies those certificates to the “correct” location and manages them (e.g. user, group, perms)

Notes

  • If a user removes the certificates installation will fail next run as it won’t find the original certificate
  • If users removes the certificates, services will continue to function through restarts (since copies will remain in tact)
  • Any certificate changes require installer re-run

Option 2: Manage the files directly

Steps

  • User provides certificates as input
  • Modules manages the input certificate paths directly (e.g. user, group, perms)

Notes

  • Could gate behind a manage_certificates flag for advanced users
  • One downside is user has to make they put certificates in an accessible location
  • One downside is the location of certificates become unpredictable on a given installation
  • One downside is potentially competing services (e.g. need multiple groups, different permissions)
  • One upside/downside is user has more direct control, when certificates change they need only restart services
    • However, for services that use certificate stores those stores have to be re-generated anyway
  • If a user removes the certificates, services will fail on any restarts, installation will fail next run

Certificate store Management

For some services, there are advanced certificate storage mechanisms in use such as keystores and NSS databases. The design here is around moving the creation and management of those stores into the modules for the services themselves and focus on the input to the module being the needed set of certificates. This keeps the user from having to create and manage these more complex entities and focus on just certificates. Further, given these stores need properties set on them like user and group for example it is easier for the module to manage this and ensure everything is correctly set up for the service to operate correctly.

I’ve opened a proof of concept PR to show this in action: POC: Manage truststore completely within module by ehelms · Pull Request #195 · theforeman/puppet-candlepin · GitHub

How does this apply to Puppet certificates (and Katello)?

Today Foreman defaults to using certificates generated by Puppet and using the paths on disk that Puppet creates. Management of the certificate access is handled by ensuring the foreman user is part of the puppet group. In Option 1 above, we would copy these Puppet certificates to a standard location (e.g. /etc/foreman) and set user/group/perms on them accordingly. In Option 2, we would continue the model we have today but based on user input rather than the current defaults.

That gets me to some of my thinking around how do we split apart the Foreman default to Puppet certificate paths and level set Puppet certificates with user defined certificates or some other provider such as Katello’s default certificate setup or something new in the future.

I am not sure yet on that point. For Puppet, the creation of the certificates and the paths are handled within the theforeman-puppet module and thus could be used as inputs to Foreman’s parameters for SSL. This same problem applies to Katello where we hard-code the certificate paths in the answers file.

How do we orchestrate that and make it easier for the user inside the installer?

I have opened a draft POC PR with this question to help try to gather some design discussions closer to the code as I think that can be easier to think and talk about.

43

Long time, no update to this RFC. Given it’s been 3 years, and I have been thinking on this topic again. We previously had extensive discussion and ideas on how to tackle this problem. Some of the ideas we implemented and simplified, others felt hard to go from where we are today to that future.

This update to the RFC is aimed at defining a set of high level goals to target regarding how we generate, manage and think about the certificates used when operating Foreman, Katello and related services. I believe it is valuable to align Foreman and Katello certificate expectations, remove technical debt and open up the ability for users to provide Katello based installations with full certificate infrastructure if they desire. In order to do this I believe we need to tease apart layer by layer in order to make sure we do not de-stabilize a user’s install by forcing re-deployment of certificates.

That being said, I believe as long as we protect existing installations, and the CA certificate and key we can consider more radical approaches if they get us to a better future faster tha what I propose below.

#1 - Remove katello-certs-tool

The katello-certs-tool project is old, and full of both outdated methods (e.g. using nsCertType) and large chunk of unused functionality. The code is hard to read and understand.

Proposal: Eliminate the use of katello-certs-tool under the hood of puppet-certs via a refactor and instead rely on native methods using openssl directly via abstractions from puppet-openssl. This is presented here.

#2 - Separate generation from deployment

Today Katello uses puppet-certs to generate certificates in /root/ssl-build and handles the deployment and management of those certificates to locations for use by services.

Proposal: Add certificate deployment to the appropriate service modules and remove the deployment handling code from puppet-certs.

#3 - Define required certificates and locations

Currently Foreman defaults to using puppet certificates or allows a user to present them as inputs. Katello relies in puppet-certs to put certificates in various locations that are inconsistent across services.

Proposal: This comes in two parts. The first is to clearly define the set of certificates required and naming conventions that clearly identify them (e.g. apache.crt). The second part is to define a pattern and location for each set of certificates for a given service (e.g. /etc/foreman/pki, /etc/httpd/pki).

#4 - Remove generation from core installer execution

For Katello the generation of a default set of certificates happens within the puppet modules and thus core installer execution. This tightly couples the generation of the default set of certificates with installation making it hard to cleanly provide your own certificates.

Proposal: Remove generation from within the puppet modules and instead make it a step that happens within installer hooks prior to the core installation run. Users can either provide their own certificates or the installer will generate a default set of certificates.

1 Like
44

I’m a big fan of the automated style that Let’s Encrypt has made popular. With their ACME protocol it’s now easier than ever to request certificates.

Various tools have also integrated this, making it even easier. For example, Apache has mod_md which requests certificates from a CA and ensures renewal happens.

It’s also possible to run your own private CA this way. There’s smallstep but also FreeIPA can serve as an ACME compatible CA server (using dogtag. I haven’t used either in that role, but it feels to me like the ideal solution.

Now for Apache this will work very well, but I can see challenges for others. The Smart Proxy runs on a non-standard port and the http-01 challenge is always on port 80 while the TLS-ALPN challenge is always on port 443. There are ways around this, but they won’t be pretty.

Another issue I can see is the certificates we use to identify with other services (client certificates).

Having said that: smallstep also implements other mechanisms, such as JWT.

Candlepin will also have its own CA as well as Puppet (though the latter can use an intermediate CA issues by your CA).

So while I like the removal of katello-certs-tools, I’m not sure puppet-openssl is the best alternative.

2 Likes
45

We can save the replacement discussion for the final step and instead tackle the other required changes ahead of time. To take advantage of any of this more modern methods I believe we have to firstly split generation from deployment. If we can standardize on certificates as inputs to the installation model we get a lot more flexibility on where those certificates come from. We have to keep in mind that along the way we must keep the current self-signed certificates in place to not force users into updating everything.

That is, we can ignore #1 for now, and focus on these:

#2 - Separate generation from deployment
#3 - Define required certificates and locations
#4 - Remove generation from core installer execution

1 Like
46

Last Thursday and a bit of Friday I spent time investigating systemd credentials. Today I wrote up my lessons learned in Understanding systemd credentials - Partial Solutions and GitHub - ekohl/foreman-credentials has a bit more code (the service files themselves are untested now).

I’ve opened Support reading SSL credentials from default locations by ekohl · Pull Request #898 · theforeman/smart-proxy · GitHub with an example of how we can implement CREDENTIALS_DIRECTORY support.

The reason I bring this up specifically is that in order to separate the deployment, this can greatly help. Today we have this monstrosity to make sure foreman-proxy can read Puppet’s SSL files:

And for Foreman there’s a similar hack:

What I implied in my blog is that we can implement is so a foreman-credentials service reads Puppet’s files (as root) and passes them to the application. We can even implement it to read subscription-manager’s certificates. Then the workflow becomes:

  • Register the machine (Puppet or subscription-manager)
  • Run the installer

The installer will configure foreman-credentials. It may even have a certain default (like Katello uses subman where Foreman uses Puppet). I could also see some common way to read files from certbot’s directory structure.

One complication I can see is that a self-registered server is unsupported, so for Katello specifically we may need something special.

Having said all of that, I do wonder about cronjobs or CLI commands. They won’t have credentials loaded. We could switch to systemd timers. I also don’t know about Apache and CREDENTIAL_DIRECTORY.

In short: let’s look a bit further beyond the horizon to see where we’re going.

47

I don’t follow where subscription-manager plays a role in installation through this method.

I read through all of the material and there is an aspect that is either subtle or still present but glossed over. In the case of the smart-proxy example, are the files still being read from disk? Does this solve file level permissions user/group/mode or do those have to remain the same as they are today for the application to access the credentials?

I ask as that would imply that we still need to solve the issue of deployment where the puppet modules themselves should ensure the certificates are in the correct locations with the correct user/group/mode for the application to consume.

48

In a Puppet world (where Foreman started) it sort of “just works” when using the installer. Foreman and Foreman Proxy are configured to use it, if the machine has Puppet certificates. My thoughts were that subscription-manager does essentially the same: register gives you a set of (client) certificates. What if we could reuse those so the whole installer sort of “just works”. No need to provide the foreman-proxy-content installer scenario with a tarball that includes certificates; reuse the subman certs instead.

It’s in there, but perhaps hidden.

If you use LoadCredential=cred:/path/to/file.pem it’s read on application start up and kept in memory (in a non-swappable location). No reload, even if you reread it multiple times. I found Reload SSL certificates with systemd - iBug to reload services, but I’m not sure I like that very much.

If it’s LoadCredential=cred:/path/to/sock then it calls the service every time. That’s why I implemented the Python application. That can then do the right thing.

Of course, the application itself also needs to support reloading. Foreman Proxy doesn’t support it for its server certificates and you must restart. Client ones are reread every time so no reload is needed, but if you remove the files it also breaks. Foreman only has client certs (except websockify, but let’s ignore that for now) and relies on Apache for server certs. Apache can reload, but doesn’t appear to support systemd credentials.

It is my understanding (and I’ll need to verify this) that systemd reads credentials as root and presents it readable to the application.

If you use a dynamic service, the user that runs the service needs to be able to read the files. So it at least partially solves the permission problem.

I would like to move to a model where the actual certificate rotation is outside of our Puppet modules. Some system is responsible for that. Puppet can ensure the certificate system is initialized, but that’s it.

49

This is a great topic and something our users struggles with a lot. Some time ago, certificates was top theme in Support category.

I like the proposal to start with #2, #3 and #4, these are really the initial steps that need to done. However I think it’s good to look further as it may inform the definition for #3 and part of #4. So I’d say let’s continue the discussion on how the ideal solution should look like before we proceed. Eric, you made a good point about the necessity of keeping the current self-signed certificates in place. Whatever ideal solution we’ll find needs to play nicely with today’s setups too.

2 Likes