RFC: Redesign Certificate Handling within Foreman Deployments

Development RFCs
1

For many years, there have been two dominant certificate infrastructures within the Foreman community: Puppet CA and Katello’s certificate tooling. There are on going efforts within the community that require bringing both a new design and consistency to this area. They are:

  • Optional puppet
  • Allowing Katello to install on an existing Foreman
  • Cohesive installation experience
  • Users ability to supply custom certificates to their Foreman and Katello infrastructure

The following proposal explores two options for how we can proceed. This new certificate handling will replace:

  • Default use of Puppet CA
  • Katello’s puppet-certs and katello-ssl-tool components

I’m trying out a few polls throughout the design to allow easier input on the structure of parts of the proposal.

Design Principles

To guide the design, I’ve put together the following principles to adhere to:

  • Installations should by default provide certificate infrastructure
  • Functionality must exist that can validate user provided certificates within the infrastructure
  • User must be able to supply user generated server certificates
  • User must be able to supply wildcard certificates
  • Certificates should support alternate cnames
  • Ability to generate certificates for easy Proxy installations

Proposal

Workflows

There are three overall scenarios:

  1. bootstrapping the CA and server certs to get Foreman up and running
  2. adding a foreman-proxy
  3. adding a feature that requires a service with certificates

The overall workflow would include a tool to:

  1. Generate locally or retrieve certificates remotely
  2. Place certificates in the right locations on disk for a service
  3. Create the right format when needed (e.g. PEM, keystore)
  4. Provide verification and debugging capabilities

Design

The tool would build upon lessons learned from Puppet CA usage and Katello’s puppet-certs usage to split the main functionality into two pieces: generate and deploy. The generate functionality would be used to create a set of default certificates when the user does not provide them. The deploy functionality would be used to place the certificates on the system, in the right places. The deployed certificates could come from the generated set or inputs from the user providing their custom certificates. The installation would be configured to point to the deployed certificate locations and have a common structure to set sane defaults. Summarizing the main principles (see command structure section for more details):

  • Generate certificates if user does not supply any
  • Deploy places certificates in their final locations for input to the installer
  • Hand all certificates to the installer rather than having the main puppet apply generate them
  • Store service certificates in a common pattern:
    • /etc//pki
    • Example:
      • /etc/foreman/pki/certs/client_cert.pem
      • /etc/candlepin/pki/certs/ca.cert
  • /etc/<service>/pki
  • /etc/pki/<service>
  • /etc/<service>
  • Other

0 voters

  • Stand alone tool
  • Built in to foreman-maintain
  • Built in to installer

0 voters

Command Structure

  • generate: creates all certificates needed into /var/lib/foreman-pki/
    • Examples:
      • foreman-pki generate
  • deploy: copies certificates to their deployed locations at /etc//pki
    • Examples:
      • foreman-pki deploy all
      • foreman-pki deploy apache
  • view: displays x509 metadata for a certificate
    • Examples:
      • foreman-pki view candlepin
      • foreman-pki view apache
  • verify: verifies all deployed certificates
    • Examples:
      • foreman-pki verify all
      • foreman-pk verify candlepin
  • export: creates a tarball of a hostname’s certificate directory in /var/lib/foreman-pki//export
    • Examples:
      • foreman-pki export
  • import: unbundles a tarball of certificates into /var/lib/foreman-pki/
    • Examples:
      • foreman-pki import <path_to_tarball>

Network Option

An additional design option is to provide certificate generation as a networked option. The tool would have the ability to reach out to a networked service that provides certificate signing to retrieve the certificates it needs. The tool would then translate these certificates and deploy them. For example, if certificates are needed for a Pulp on a proxy. Or, if a PKC12 keystore is required the tool would submit the signing request, retrieve the certificates, wrap them in a keystore and deploy to the location on disk.

This option could include support for interfacing with services like vault or certmonger. Or, this could be elevated to a first class citizen of Foreman to allow Proxies and their services to request certificates as services are spun up within the infrastructure.

This optional network support would be akin to the certmanager project within the Kubernetes ecosystem. I believe that this option could be a future option as it just adds a new way for the tool to fetch and deploy certificates rather than requiring an orthogonal design.

Technology Options

The tool will wrap native openssl constructs, so the language of choice should have good support for this, be familiar and easy to generate a CLI tool from without having to create too much boilerplate.

  • Ruby
  • Python
  • Ansible
  • Other (e.g. Go, Javascript)

0 voters

Background Research

I’m including some notes around how other projects handle SSL in their workflows.

How Kubernetes handles SSL

Reference: https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#kubeadm-join-phases-internal-design

For the main server:

  1. Generates a self-signed CA
    a. Or user provides one
  2. Generates a server certificate keypair

For nodes:

  1. Node registers itself to kubernetes with a token and CA hash:
kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash>
  1. Node retrieves CA certificate from server and verifies hash of the CA
  2. Node sends CSR to server
  3. Node retrieves signed certificate
  4. Node retrieves CA over secure connection and verifies one final time

How Puppet handles SSL

For main server:

  1. Generates a self-signed CA
  2. Generates a server certificate keypair

For clients

  1. Client generates CSR and sends it to Puppet server via API
  2. Puppet server autosigns or manually signs and returns keypair to client to use
2 Likes
2

Just a quick point of order: Is voting, especially at such an early stage, the correct approach to finding the best solution for the problems? I believe we need to discuss them first and then make an educated decision what’s best for the project. Do you mind removing the votes?

3

I’ll agree with that and votes should be supported by arguments.

Here I voted for /etc/<service>/pki because that feels most natural to me. Perhaps because I grew up on Debian where /etc/pki simply doesn’t exist and /etc/ssl always felt weird. By using /etc/<service> you can standardize the locations between distributions which makes operations, tooling and documentation easier.

Regarding the tool, I voted for a standalone tool. I still consider foreman-maintain as a non-existing tool because I can’t rely on it since there’s no Debian support. It also feels weird for the installer to depend on foreman-maintain and foreman-maintain on the installer. Dependency trees are better than dependency graphs.

Other reasons are support cycles. Currently foreman-maintain is used on much older versions to allow upgrading. In a standalone PKI tool you can easily make breaking changes to support development and don’t care about backporting to an older version. By not having a large versioned if/else tree in your code, it’s easier to maintain and reason about.

I’m not sure the PKI tool should have knowledge about where to deploy it and rather the installer should explicitly tell it where to place files. The installer is going to need those locations anyway and explicitness helps to prevent them going out of sync.

My main concerns is about the overlapping of two functions: there’s a CA and a client side. generate sounds like you always generate a keypair, but I don’t see anything like signing for a client. Deployment is a pure client operation.

I also don’t see anything about how you’d deal with already signed certificates from a third party CA.

Regarding implementation I voted for Ruby because by default it has OpenSSL bindings available. Inspecting certificates is something you’re going to do and wrapping the openssl binary as katello-certs-tool does is painful. However, I’m also fine with Python since I’m very comfortable with that and have experience with pyopenssl. I’m not fine with Ansible because I always start tearing my hair out after having to write it. YAML should not be used to write code is my experience.

I don’t think Puppet sends a keypair. In PKI as a client you normally generate a private key and use it in the CSR. The server can sign this CSR and send back only the certificate. This is also how tools like Letsencrypt and all commercial CA’s operate.

This is also a major flaw in the current katello-certs-tools: you generate a keypair on the server which means the server has all the private keys. You also then download a tarball with a key. Because of convenience an admin may first download it to their workstation and forget to remove it. This is a major reason why I think a networked CA is a huge benefit: by standardizing the workflows you promote secure best practices.

2 Likes
4

I’m trying to fully understand the goals of this effort. Please allow me some questions to get a better picture.

Would this effectively drop PuppetCA support for (parts of) the certificate infrastructure?

Does this mean there is the need for some kind of plugin architecture?

Why is that? I believe it is best practise to generate the key on the server and have the CA sign a CSR. Can we change this goal to “User must be able to sign the certificates by a user provided CA”?

In which cases is this useful or even required?

As mentioned before, does this mean that on the proxy a CSR is generated (more secure) and signed by a CA or would we create key & cert on a different server and copy them to the target machine?

5

I’m going to leave them for now. I don’t think they are breaking to the overall design discussion and I believe not everyone will weigh in with comments and supporting arguments. As this is about design discussion and information gathering I want to see their output. Supporting arguments would be preferred but polls are not an invalid source of information for me.

Today, the installer takes as input the location of certificates. Those certificates have to exist at those locations already. That is one of the driving factors for having deploy functionality.

They would be inputs to the tool that would know where to deploy those certificates. There are two scenarios I envision being supplied to the tool:

  • third party certificates for webserver
  • third party sub-CA that is used to generate the certificates

I will update the wording.

Correct. The use of Puppet CA would still be an option as existing deployments will be using it. For the future state where Puppet is optional at install time, we’d want to not force Puppet CA on deployments.

Not to my knowledge. This story is around ensuring that user provided certificates will work with the deployed infrastructure. We have hit issues with this before where users supply certificates that are broken or do not work with our services. Today this is captured by katello-certs-check.

I may be missing the nuance here. We typically see users go to their corporate CA and get a keypair generated for the server. They then want to use that keypair within their Foreman for the webserver as an example.

I don’t know honestly. But we see users wanting to do this.

This goes to the network option which may require more work / infrastructure to support. Today, Katello works the way you described. The server generates tarballs of certificates needed for the Foreman Proxy and the user copies that over to the Foreman Proxy and the tooling handles unwrapping and deployment of the tarball of certificates. As there tends not to be a lot of external Foreman Proxies spun up for a given infrastructure this tends to be a simpler option or at least a phase 1 option.

6

My usecase is probably an edge case, however I as a developer install Foreman many times. These are short living instances. I want to have full x509 setup for testing and I use one *.example.com certificate for all of them. With that I have valid setups without adding new self-signed CAs to my browser and system storage for hammer.

7

I think it’s pretty common to use https://letsencrypt.org/ these days. I don’t know how others use it, but I have a certbot that creates new certs and refreshes them automatically. So I have a locally available certificate and key which I’d like to use for Foreman webui. While it still follows CSR process describe, for this new tool it’s already existing signed certificate and key.

However if we want to support this, our tool should be able to avoid copying the certificate and rather symlink it. The cert are rotated very often and the rotation is done by external tooling.

2 Likes
8

What I meant was, that “the tool” probably needs some kind of plugin architecture so that candlepin can define the command “foreman-pki view candlepin” and what this means, e.g. where to store which kind of certificates and how to validate them.

The recommended workflow would be, that they generate a private key (they - not the CA), then have their corporate CA issue a certificate (via a CSR). The nuance is, that the tool should generate the private key and CSR and not import an existing private key and certificate. There may be edge cases where the key management is done by some kind of external tooling and an external tool generates the private key, so we’ll probably need some kind of import mechanism that validates if the imported cert meets all the requirements.

9

I can see that we’d want to integrate parts of this tooling in Foreman or Smart-Proxy, so I believe we should stick with Ruby.

Another thought I have is, that currently Candlepin issues certificates to Katello clients. Is that something we want to take a look at in this RFC as well?
I could see a workflow where Foreman deploys a new server which should act as a smart-proxy, the smart-proxy registers in Foreman, an administrator approves the new proxy, the proxy is issued a signed certificate and can start operating. Is this a goal we want to reach?

10

My take is that we will define and control all this logic. I’m happy to have some sort of plugin architecture if folks out there want to add their plugin/service support without contributing it to the tool directly. I want as much as possible everything to live in the tool to avoid sprawl.

If I dig into this statement more, do you forsee the network option as being something built in to Foreman or the Smart Proxy?

There is a lot to this concept that I we can explore separately, but I am not wanting to try to redefine this workflow at the moment. I’d like to stick with Candlepin needs a CA that should either be the same root CA or a subordinate CA and handles management of those certificates.

I think this captures the base level networked workflow that @ekohl has mentioned. We would need support still for being able to get signed certificates for services running on the smart proxy that need them. For example, if I deploy a Pulp there I need certificates for Apache running in front of Pulp and a certificate for that Pulp to sync from the master.

11

The way we did this at a previous employer was using Puppet CA.

Every provisioned host was provisioned with Puppet. Foreman used the Puppet CA infrastructure. We used the Foreman Puppet Modules to deploy a Smart Proxy. Our Puppetmaster had the oauth credentials to register a new smart proxy so deploying a new Smart Proxy was as simple as going to the Host form, select the right hostgroup and press provision.

Note that this is all about 7 years ago so I might miss some details but I know many users use our Puppet modules directly without the installer so I’m guessing that we have users that follow the same approach today. IMHO this should continue being a first class citizen.

1 Like
12

I believe this could open up some great possibilities. We need to be aware though, that we’re reinventing PuppetCA. :wink:

The tool (or smart-proxy) could request those, Foreman could be the CA that signs these certificates.

13

We just tested this workflow today, so 7 years later this still works well.

14

One workflow I think should be supported is the ability to provide third party certificates directly to the right location (/etc/pki/<service> for me because I grew up on RHL and not on debian :wink: ), with maybe a tool to just ensure that provided certificates are correct. We use FreeIPA+certmonger to deploy all our certificates (and automatically renew them), except for some foreman/katello components. Today it’s a pain point for our katello/smart proxies instances management.

1 Like
15

I like the idea and proposal. Whatever we do, let’s make sure that:

  • It’s explained why the change, I think the OP briefly touched this. I do understand the motivation in a sense that Foreman would like to go its own path being the tool that enables other technologies. Current state where puppet certs are reused, others are deployed next to it is confusing. But that’s my take.
  • It works fine on non-redhat systems (e.g. Debian), there might be different paths. I think the tool should not try to unify across platforms and build on what’s expected on those. I like having copies in /var and /etc that feels sane.
  • Whatever language is picked, make sure it has minimum dependencies. I’d vote for Bash or Golang but I am not going to be the one writing and maintaining this. Note that with shift to containers, distributions tend to strip down minimal installations, e.g. in RHEL8 there is no Python by default (there is a “hidden” one that might get deleted down the road).
  • Let’s encrypt is considered as the first citizen and fully integrated into this from the day one because that’s essentially the first things people in non-enterprise will be doing.
16

Whenever I think about this, I always stumble upon this question: Why does this need to be a separate tool? If this were part of Foreman core and Foreman could be the manager of the Foreman CA, we’d just need to figure out a way to initialise the CA and bootstrap the main Foreman app. All other services could then request a certificate either via built in code (smart-proxy) or via a small cli client (e.g. for candlepin, we could use hammer cli for this). A user could then approve signing requests, revoke certificates, … via Foreman’s UI, hammer-cli etc.
A nice workflow could be: A user installs a smart-proxy on some host, it signs in to Foreman, an Admin gets a UI notification, approves the request, the proxy is automatically added as a proper smart-proxy in Foreman. I believe this would be very, very useful to improve the UX of Foreman.
And in the future we could use this CA to identify Foreman managed hosts, e.g. when they check in for something like foreman_dlm.

1 Like
17

If we go this way, I hope users won’t start (ab)using Foreman as another ACME provider or their first CA. I don’t think we’d have capacity for supporting non-Foreman related use cases. Bugs are usually reported against the top layer, Foreman in this case. However, this would be super cool feature.

18

I voted for foreman_maintain as there are already a lot of tools to run foreman. Additionally, I think that functionality of foreman_mainain can maybe re-used sooner or later. If its not available for debian right now, its the best reason to get it in to debian asap - as it makes totally sense that foreman-maintain on debian / ubuntu exists, too.

19

I get your point, but e.g. in our deployment we haven’t had any success in using that as it does too many assumptions. I think it’s a good tool for the average “I-buy-software-with-support-and-have-no-clue-what-I’m-doing-and-call-support-for-everything” user and it’s good that we have it. But I don’t see any real user value in reusing the tool.

I am curious to see the result of the questions, but apparently I need to vote before I can see the results. :worried:

20

Well, have users done this with the subscription manager certificates?

I believe we need to solve these two use cases

  • Identify a host (which can be a smart-proxy) towards a Foreman service
  • Have certificates for foreman internal services (candlepin, …)

How would we do this with a standalone tool?