RFC: Redesign Certificate Handling within Foreman Deployments

I can see that we’d want to integrate parts of this tooling in Foreman or Smart-Proxy, so I believe we should stick with Ruby.

Another thought I have is, that currently Candlepin issues certificates to Katello clients. Is that something we want to take a look at in this RFC as well?
I could see a workflow where Foreman deploys a new server which should act as a smart-proxy, the smart-proxy registers in Foreman, an administrator approves the new proxy, the proxy is issued a signed certificate and can start operating. Is this a goal we want to reach?

My take is that we will define and control all this logic. I’m happy to have some sort of plugin architecture if folks out there want to add their plugin/service support without contributing it to the tool directly. I want as much as possible everything to live in the tool to avoid sprawl.

If I dig into this statement more, do you forsee the network option as being something built in to Foreman or the Smart Proxy?

There is a lot to this concept that I we can explore separately, but I am not wanting to try to redefine this workflow at the moment. I’d like to stick with Candlepin needs a CA that should either be the same root CA or a subordinate CA and handles management of those certificates.

I think this captures the base level networked workflow that @ekohl has mentioned. We would need support still for being able to get signed certificates for services running on the smart proxy that need them. For example, if I deploy a Pulp there I need certificates for Apache running in front of Pulp and a certificate for that Pulp to sync from the master.

The way we did this at a previous employer was using Puppet CA.

Every provisioned host was provisioned with Puppet. Foreman used the Puppet CA infrastructure. We used the Foreman Puppet Modules to deploy a Smart Proxy. Our Puppetmaster had the oauth credentials to register a new smart proxy so deploying a new Smart Proxy was as simple as going to the Host form, select the right hostgroup and press provision.

Note that this is all about 7 years ago so I might miss some details but I know many users use our Puppet modules directly without the installer so I’m guessing that we have users that follow the same approach today. IMHO this should continue being a first class citizen.

1 Like

I believe this could open up some great possibilities. We need to be aware though, that we’re reinventing PuppetCA. :wink:

The tool (or smart-proxy) could request those, Foreman could be the CA that signs these certificates.

We just tested this workflow today, so 7 years later this still works well.

One workflow I think should be supported is the ability to provide third party certificates directly to the right location (/etc/pki/<service> for me because I grew up on RHL and not on debian :wink: ), with maybe a tool to just ensure that provided certificates are correct. We use FreeIPA+certmonger to deploy all our certificates (and automatically renew them), except for some foreman/katello components. Today it’s a pain point for our katello/smart proxies instances management.

I like the idea and proposal. Whatever we do, let’s make sure that:

  • It’s explained why the change, I think the OP briefly touched this. I do understand the motivation in a sense that Foreman would like to go its own path being the tool that enables other technologies. Current state where puppet certs are reused, others are deployed next to it is confusing. But that’s my take.
  • It works fine on non-redhat systems (e.g. Debian), there might be different paths. I think the tool should not try to unify across platforms and build on what’s expected on those. I like having copies in /var and /etc that feels sane.
  • Whatever language is picked, make sure it has minimum dependencies. I’d vote for Bash or Golang but I am not going to be the one writing and maintaining this. Note that with shift to containers, distributions tend to strip down minimal installations, e.g. in RHEL8 there is no Python by default (there is a “hidden” one that might get deleted down the road).
  • Let’s encrypt is considered as the first citizen and fully integrated into this from the day one because that’s essentially the first things people in non-enterprise will be doing.

Whenever I think about this, I always stumble upon this question: Why does this need to be a separate tool? If this were part of Foreman core and Foreman could be the manager of the Foreman CA, we’d just need to figure out a way to initialise the CA and bootstrap the main Foreman app. All other services could then request a certificate either via built in code (smart-proxy) or via a small cli client (e.g. for candlepin, we could use hammer cli for this). A user could then approve signing requests, revoke certificates, … via Foreman’s UI, hammer-cli etc.
A nice workflow could be: A user installs a smart-proxy on some host, it signs in to Foreman, an Admin gets a UI notification, approves the request, the proxy is automatically added as a proper smart-proxy in Foreman. I believe this would be very, very useful to improve the UX of Foreman.
And in the future we could use this CA to identify Foreman managed hosts, e.g. when they check in for something like foreman_dlm.

1 Like

If we go this way, I hope users won’t start (ab)using Foreman as another ACME provider or their first CA. I don’t think we’d have capacity for supporting non-Foreman related use cases. Bugs are usually reported against the top layer, Foreman in this case. However, this would be super cool feature.

I voted for foreman_maintain as there are already a lot of tools to run foreman. Additionally, I think that functionality of foreman_mainain can maybe re-used sooner or later. If its not available for debian right now, its the best reason to get it in to debian asap - as it makes totally sense that foreman-maintain on debian / ubuntu exists, too.

I get your point, but e.g. in our deployment we haven’t had any success in using that as it does too many assumptions. I think it’s a good tool for the average “I-buy-software-with-support-and-have-no-clue-what-I’m-doing-and-call-support-for-everything” user and it’s good that we have it. But I don’t see any real user value in reusing the tool.

I am curious to see the result of the questions, but apparently I need to vote before I can see the results. :worried:

Well, have users done this with the subscription manager certificates?

I believe we need to solve these two use cases

  • Identify a host (which can be a smart-proxy) towards a Foreman service
  • Have certificates for foreman internal services (candlepin, …)

How would we do this with a standalone tool?

I presume you mean if there is no networking involved (i.e. no ability to reach out to some service and request a signed certificate). This would be done the way Katello does this today. All certificates are generated on the server and must be copied over to their respective locations for use. Which really only happens when spinning up an external smart proxy. I often think its fair to ask how much this happens within an organization? How much are we trying to optimize for a rarer operation?

This conversation does interweave two certificate concepts I believe so I do want to separate them:

  • infra certificates
  • client host certificates

For infra as of today:

  • Foreman base installs use Puppet CA
  • Katello base installs create and use a self-signed certificate

For host certificates connecting back to Foreman:

  • Foreman base installs tend to use Puppet CA
  • Katello uses Candlepin generated certificates from the self-signed CA

This raises two questions I am less familiar with:

  1. What does a Salt or Chef or Ansible only installation prefer for client certificates? (relates to the optional Puppet RFC)
  2. I assume if you had a base Foreman installation, with hosts able to connect back without a config management provider in a more inventory focused deployment. Those hosts would need a client certificate generated from somewhere that Foreman is able to authenticate them with?

Sorry for asking again, but I believe the external smart proxy should generate a key and a CSR. The CSR is not a secret, so a user does not need to make sure it stays private. The CA can then issue a certificate (also public data) that the user transfers to the external proxy.
If we implemented the CA as part of Foreman, the user would install the external proxy, copy the CSR into the browser, Foreman would issue a certificate that the user can download and transfer it to the smart-proxy. We could even embed some data into the CSR so that Foreman automatically creates a SmartProxy object or something. Sounds like a secure and user-friendly use case. Is this feasible?

I can’t answer this, but if the goal is to provide an agnostic tool, I guess we have to solve this ourselves. Ansible uses SSH for transport and not HTTP, so I believe it does not have this challenge. SSH can use certificates, though.

Yep. I believe that’s what the Katello use-case currently does with the candlepin issued certificates you mentioned. An activation key basically approves a host to get a certificate. Is this secure enough?

We still need something that handles formatting for other services like Candlepin where keystores are required but I think what you noted as the base workflow and support is feasible. In the end, I forsee us spending more time discussing and hammering out the design than implementaion.

Indeed. I’m thinking more along the lines of how hosts tell Foreman they should be added to the inventory. Today that’s through a variety of tools right?

  • puppet agent
  • subscription-manager
  • Ansible callback

Do we aim to simply support those with “common” APIs, one for each, or do we also support a generic method for a host to “register” itself for lack of a better term?

The subscription-manager register process is what handles this. Activation keys provide a way to bring a host into the inventory without needing to know any credentials but they are not secure. If you know the name of the key you can add it the inventory.

I’ve played with this idea in a small plugin called foreman_register (naming is hard, right? :wink: ). The workflow idea is that a user is presented with a curl http://foreman.example.com/bla/bla/bla | bash type command that is supposed to be run in order to register or enroll a host with Foreman. Part of the URL is a JWT that acts as a secret and basically makes the process secure. The JWT includes the hostname so Foreman can present a host-specific enrollment script. The script is a Foreman template.
This is might be a pattern that is worth investigating so the CA could issue a certificate. If we send some fact data along the way, maybe with the new ufacter tool, we could also create a host even if it’s unknown to Foreman.

I believe this is mostly based on fact processing. The smart-proxy is trusted and if a smart-proxy sends facts (e.g. via Puppet or Ansible) the host object is created.

Yes, this could be client software (a simple cli app) I mentioned earlier. But this app does not need to be the CA. If we make Foreman the CA, this tool could also bootstrap the CA and issue (the first) certs for the Foreman webserver. As real (human) users access Foreman, I can see that users might want to switch out the webserver certificate with a certificate that is trusted by a browser (usually some kind of internal CA). Also something to keep in mind. Does this contradict that Foreman could be the CA (for the rest of the services and the client certs use-case).

Let’s try to avoid this. I believe we have roughly touched all the important areas now. How are we going to proceed? From my point of view, the brainstorming phase is finished and we need to create a rough design.

If we get to design, I think we need an interface for a CA. Ideally it can front multiple CA’s. Reusing and integrating existing tools has always been what Foreman has been about.

You may now think about certmonger. It’s available in EL, Debian and Ubuntu. It can talk to FreeIPA but also has a local CA backend. I just learned about SCEP and it looks like Active Directory also implements that. It’s also possible to write custom backends. It can write both PEM and NSS. However, a major downside is that it uses dbus and that is usually not running in containers.

As for the CA itself, we can look at dogtag which is the CA behind FreeIPA. It is a tomcat service, so it might be quite heavy. The benefit is that we wouldn’t invent our own tools and users with an existing FreeIPA deployment could just use it without issue. I suspect it’s audited as well.

Because they’re networked and daemons, they can handle renewals automatically which removes that burden from sysadmins.

Regarding PuppetCA I think there are 2 possible ways. One is to write a certmonger backend for PuppetCA and the other is to issue an intermediate CA certificate for PuppetCA so there’s still a single CA hierarchy. https://puppet.com/docs/puppetserver/latest/intermediate_ca.html documents such a setup where you import an external CA.

An alternative is Vault’s PKI.

Lastly Candlepin is also a CA.

In short: I’d like to avoid writing Yet Another CA

We have had very good experience with this. There is even a plug-in we developed. The development of more features will start next week.

SCEP is quite limited and has known security flaws as far as I know. We should avoid it by all means. If you need a ruby client implementation, let me know. We have one. It’s not pretty.

Fair point! I wouldn’t rule it out, though.

I like the idea of using some other tool, it just needs to be something else than puppet agent. I don’t want to be forced to install Puppet to be able register a host. So it needs to be super-simple tool only dedicated to X.509 management.

On the other hand, it should not limit us or force us to implement some complex APIs. What we actually need to do is couple of openssl commands essentially. So I only agree to the point that a tool that perfectly fits our purpose exists. If not, let’s create new one.

Also the actual process should be dead simple and applicable in most operating systems or workflows. I believe a single CLI utility with minimum dependencies and a one or two HTTP calls is something that most of us expects.

IMHO individual host registration is out of scope and limit ourselves only to Foreman and all tools that talk to Foreman. I don’t want to get in the business of host registration because Puppet and Subscription Manager do that just fine. AFAIK you can also use Ansible with callbacks and use SSH as a transport.