RFC: Redesign Certificate Handling within Foreman Deployments

[ehelms]

@TimoGoebel

If we implemented the CA as part of Foreman, the user would install the external proxy, copy the CSR into the browser, Foreman would issue a certificate that the user can download and transfer it to the smart-proxy. We could even embed some data into the CSR so that Foreman automatically creates a SmartProxy object or something. Sounds like a secure and user-friendly use case. Is this feasible?

We still need something that handles formatting for other services like Candlepin where keystores are required but I think what you noted as the base workflow and support is feasible. In the end, I forsee us spending more time discussing and hammering out the design than implementaion.

@TimoGoebel

I can’t answer this, but if the goal is to provide an agnostic tool, I guess we have to solve this ourselves. Ansible uses SSH for transport and not HTTP, so I believe it does not have this challenge. SSH can use certificates, though.

Indeed. I’m thinking more along the lines of how hosts tell Foreman they should be added to the inventory. Today that’s through a variety of tools right?

• puppet agent
• subscription-manager
• Ansible callback

Do we aim to simply support those with “common” APIs, one for each, or do we also support a generic method for a host to “register” itself for lack of a better term?

@TimoGoebel

Yep. I believe that’s what the Katello use-case currently does with the candlepin issued certificates you mentioned. An activation key basically approves a host to get a certificate. Is this secure enough?

The subscription-manager register process is what handles this. Activation keys provide a way to bring a host into the inventory without needing to know any credentials but they are not secure. If you know the name of the key you can add it the inventory.

[TimoGoebel]

@ehelms

we also support a generic method for a host to “register” itself for lack of a better term?

I’ve played with this idea in a small plugin called foreman_register (naming is hard, right? ). The workflow idea is that a user is presented with a curl http://foreman.example.com/bla/bla/bla | bash type command that is supposed to be run in order to register or enroll a host with Foreman. Part of the URL is a JWT that acts as a secret and basically makes the process secure. The JWT includes the hostname so Foreman can present a host-specific enrollment script. The script is a Foreman template.
This is might be a pattern that is worth investigating so the CA could issue a certificate. If we send some fact data along the way, maybe with the new ufacter tool, we could also create a host even if it’s unknown to Foreman.

@ehelms

Today that’s through a variety of tools right?

I believe this is mostly based on fact processing. The smart-proxy is trusted and if a smart-proxy sends facts (e.g. via Puppet or Ansible) the host object is created.

@ehelms

We still need something that handles formatting for other services like Candlepin where keystores are required

Yes, this could be client software (a simple cli app) I mentioned earlier. But this app does not need to be the CA. If we make Foreman the CA, this tool could also bootstrap the CA and issue (the first) certs for the Foreman webserver. As real (human) users access Foreman, I can see that users might want to switch out the webserver certificate with a certificate that is trusted by a browser (usually some kind of internal CA). Also something to keep in mind. Does this contradict that Foreman could be the CA (for the rest of the services and the client certs use-case).

@ehelms

In the end, I forsee us spending more time discussing and hammering out the design than implementaion.

Let’s try to avoid this. I believe we have roughly touched all the important areas now. How are we going to proceed? From my point of view, the brainstorming phase is finished and we need to create a rough design.

[ekohl]

If we get to design, I think we need an interface for a CA. Ideally it can front multiple CA’s. Reusing and integrating existing tools has always been what Foreman has been about.

You may now think about certmonger. It’s available in EL, Debian and Ubuntu. It can talk to FreeIPA but also has a local CA backend. I just learned about SCEP and it looks like Active Directory also implements that. It’s also possible to write custom backends. It can write both PEM and NSS. However, a major downside is that it uses dbus and that is usually not running in containers.

As for the CA itself, we can look at dogtag which is the CA behind FreeIPA. It is a tomcat service, so it might be quite heavy. The benefit is that we wouldn’t invent our own tools and users with an existing FreeIPA deployment could just use it without issue. I suspect it’s audited as well.

Because they’re networked and daemons, they can handle renewals automatically which removes that burden from sysadmins.

Regarding PuppetCA I think there are 2 possible ways. One is to write a certmonger backend for PuppetCA and the other is to issue an intermediate CA certificate for PuppetCA so there’s still a single CA hierarchy. https://puppet.com/docs/puppetserver/latest/intermediate_ca.html documents such a setup where you import an external CA.

An alternative is Vault’s PKI.

Lastly Candlepin is also a CA.

In short: I’d like to avoid writing Yet Another CA

[TimoGoebel]

@ekohl

An alternative is Vault’s PKI.

We have had very good experience with this. There is even a plug-in we developed. The development of more features will start next week.

@ekohl

I just learned about SCEP and it looks like Active Directory also implements that.

SCEP is quite limited and has known security flaws as far as I know. We should avoid it by all means. If you need a ruby client implementation, let me know. We have one. It’s not pretty.

@ekohl

In short: I’d like to avoid writing Yet Another CA

Fair point! I wouldn’t rule it out, though.

[lzap]

I like the idea of using some other tool, it just needs to be something else than puppet agent. I don’t want to be forced to install Puppet to be able register a host. So it needs to be super-simple tool only dedicated to X.509 management.

On the other hand, it should not limit us or force us to implement some complex APIs. What we actually need to do is couple of openssl commands essentially. So I only agree to the point that a tool that perfectly fits our purpose exists. If not, let’s create new one.

Also the actual process should be dead simple and applicable in most operating systems or workflows. I believe a single CLI utility with minimum dependencies and a one or two HTTP calls is something that most of us expects.

[ekohl]

IMHO individual host registration is out of scope and limit ourselves only to Foreman and all tools that talk to Foreman. I don’t want to get in the business of host registration because Puppet and Subscription Manager do that just fine. AFAIK you can also use Ansible with callbacks and use SSH as a transport.

[lzap]

I don’t believe this is out of scope because to register a host in Foreman today, the worfklow is actually:

1. Install puppet
2. Get a cert
3. Check in

Add existing host to foreman

Hello, I am starting to test Foreman to help with some Ubuntu Desktops (16.04) in a FreeIPA realm. Most docs around focus on installing new hosts. What is the best way for pre-existing hosts? Right now, it seems to be this steps: i. install puppet. ii. in puppet.conf, remove the [main] section and point a [agent]/server to the Foreman host. iii. restart the daemon. iv. enable the agent (puppet agent --enable). v. sign the certificate on Foreman (puppet cert sign fqhn). I wi…

[TimoGoebel]

@ekohl

IMHO individual host registration is out of scope

Can it be out if scope? If we can issue a certificate for a smart-proxy, we have basically solved “individual host registration” as well, haven’t we?

[ekohl]

@lzap

I don’t believe this is out of scope because to register a host in Foreman today, the worfklow is actually:

1. Install puppet
2. Get a cert
3. Check in

It is Puppet that creates the host, not the host itself. The authentication is bound to the Smart Proxy Puppet feature. Replacing Puppet for host creation is IMHO out of scope since you need additional authentication that the host really is allowed to upload.

@TimoGoebel

Can it be out if scope? If we can issue a certificate for a smart-proxy, we have basically solved “individual host registration” as well, haven’t we?

Currently creating a Smart Proxy happens using OAuth credentials. Replacing this makes the scope much bigger. In theory you could embed information in the certificate that it’s allowed to be a Smart Proxy but then you need to revoke certificates and deal with a CRL which Foreman currently doesn’t.

[TimoGoebel]

@ekohl

but then you need to revoke certificates and deal with a CRL which Foreman currently doesn’t.

If Foreman (or the tool we invent here) acts as a CA, we have to deal with this anyway.
I agree, that we have to thing we need to solve here. One is to verify the certificates and put them in the right place. I believe it’s a no brainer that this could very easily be done using the puppet modules (and the installer).
The harder challenge is: What tool signs these certificates and how does this tool verify that it is allowed to sign a certificate? Any suggestions?

[ehelms]

I am posting an update here with a proof of concept based off the CLI proposal. This is intended to test the simple design, identify any modifications and use it as a basis for how to proceed.

Proof of Concept

I built a clamp-based Ruby CLI named foreman_pki to generate a certificate authority and a set of certificates based on the needs of the installation.

The set of certificates desired are pre-defined bundles that define the standard set of needed certificates for an installation. In the proof of concept, this includes the bundle for Foreman, Katello and a Smart Proxy. Bundles can be enabled via a configuration file.

The implementation places all certificates under a common structure as seen below. The common structure was designed to make it easy to locate all certificates for an installation, easy to understand with a common naming structure and allow installer/puppet modules to control permissions on the certificates per the needs of a given service. There is an additional naming scheme introduced for client certificates to better understand what a certificate is used for. For example, when Katello is present and needs to communicate with Pulp a foreman-pulp certificate is generated. This is different than Foreman needing to talk to smart-proxies which get a foreman-to-smart-proxy certificate name.

root@foreman foreman_pki (master)$ tree /etc/foreman_pki/certs/
/etc/foreman_pki/certs/
├── apache
│   ├── apache.crt
│   └── apache.key
├── artemis
│   ├── password
│   └── truststore
├── ca
│   ├── ca.crt
│   └── ca.key
├── candlepin
│   ├── ca.crt
│   └── ca.key
├── foreman
│   ├── foreman-to-candlepin.crt
│   ├── foreman-to-candlepin.key
│   ├── foreman-to-pulp.crt
│   ├── foreman-to-pulp.key
│   ├── foreman-to-smart-proxy.crt
│   └── foreman-to-smart-proxy.key
├── smart-proxy
│   ├── smart-proxy.crt
│   └── smart-proxy.key
└── tomcat
    ├── keystore
    ├── password
    ├── tomcat.crt
    └── tomcat.key

Usage

Usage:
    foreman-pki [OPTIONS] SUBCOMMAND [ARG] ...

Parameters:
    SUBCOMMAND    subcommand
    [ARG] ...     subcommand arguments

Subcommands:
    generate      Generate certificates
    view          View a certificate
    list          List all certificates
    verify        Verify all certificates
    import-ca     Import a CA
    export        Export certificate bundle as a tarball
    import        Import certificate bundle

Options:
    -h, --help    print help

From the installer’s perspective, the foreman-pki generate command can be ran by the installer to create all of the certificates after laying down a config.yaml file. Let’s look at a few examples running this outside the context of the installer:

Foreman Install:

$ cat <<EOT >> /etc/foreman_pki/conifg.yaml
Bundles:
foreman
EOT
$ foreman-pki generate
$ foreman-installer --scenario foreman

Now if one were to add Katello:

$ cat <<EOT >> /etc/foreman_pki/conifg.yaml
Bundles:
Foreman
katello
EOT

$ foreman-pki generate
$ foreman-installer --scenario katello

Notes

The current proof of concept can import a certificate if pointed to a CA certificate and private key pair on disk. There is currently no integration with an HTTP based CA such a Puppet CA or Vault. The goal was to nail the simple use cases while considering how Katello installations handle certificates today. The current implementation can importa Puppet CA but using the import-ca command pointed at the Puppet CA cert and key on disk.

Thoughts and discussion are encouraged so that I can figure out next steps.

[bagasse]

Nice tool. Maybe adding a “certificate provider” abstraction layer should be nice. Your implementation may be the “embeded” provider and this abstraction will ease implementation of other certificate providers in the future (certmonger, vault…)

Regarding path name, maybe /etc/foreman-pki will be more consistent as we alreaday have /etc/foreman-proxy, /etc/foreman-installer

[ekohl]

@ehelms

$ cat <> /etc/foreman_pki/conifg.yaml

The typo looks consistent. Is this a copy paste error?

Also, the casing on bundles appears inconsistent.

Do you have an example where the hostname is set? IMHO we should not rely on getting the hostname from the system

I still think it would make sense to be able to wrap this in a puppet type and have the installer call it:

foreman_pki_bundle { 'foreman':
  ensure => present,
}

The user should not have to remember this if they just want a simple install and the installer should error out if a bundle is not present/invalid etc.

It would be useful if list provides the bundles that are present in an easy to parse format. Some tools have a --json switch and that would make it easy to parse. In particular list will benefit from this.

[ehelms]

@ekohl

The typo looks consistent. Is this a copy paste error?

Also, the casing on bundles appears inconsistent.

Formatting issues from copying this over since it was a longer draft, I’ll try to fix them.

@ekohl

Do you have an example where the hostname is set? IMHO we should not rely on getting the hostname from the system

Yea, the default is get the hostname from the system you run the command on, but there is an option to set the CN yourself:

root@foreman foreman_pki (master)$ scl enable tfm 'bundle exec ./bin/foreman-pki generate --help'
Usage:
    foreman-pki generate [OPTIONS]

Options:
    --common-name COMMON_NAME    Specify the common name (CN) for certificate (defaults to current hostname)
    --force                      Forces generation of certificates even if they already exist. Should be used if changing CA's but must be used with extreme cauation.
    -h, --help                   print help

The same is true for exports:

root@foreman foreman_pki (master)$ scl enable tfm 'bundle exec ./bin/foreman-pki export --help'
Usage:
    foreman-pki export [OPTIONS]

Options:
    --common-name COMMON_NAME    Specify the common name (CN) for exported certificates to have (required)
    --bundle BUNDLE              Specify the certificate bundle to export (required)
    --force                      Forces generation of certificates even if they already exist. Should be used if changing CA's but must be used with extreme cauation.
    -h, --help                   print help

@ekohl

The user should not have to remember this if they just want a simple install and the installer should error out if a bundle is not present/invalid etc.

Do we care that bundle is present or the certificates are present? I was thinking more the latter since that’s how the modules work today and provides the most flexibility for pure module users. The installer can generate these either within the puppet layer or the hooks layer. The hooks layer made sense to me to match a pure module use case and can be straight forward. One issue with today’s implementation of Katello’s certificate handling is there is multiple layers that make it complex to debug and change (i.e. python tool, puppet providers, puppet defines, puppet classes, puppet module).

We may find we want more granular bundles as well, e.g. breaking Katello’s (https://github.com/ehelms/foreman_pki/blob/master/bundles/katello.yaml) into Pulp and Candlepin bundles to use on say a smart proxy with Pulp.

[ekohl]

@ehelms

Do we care that bundle is present or the certificates are present?

I think we do. Mainly because I think that Apache refusing to start up is a lot harder to debug than a clear ‘Failed to ensure certificate bundle’.

We need to experiment with this. I think the current certs module doesn’t have a good API so there’s room for improvement, but I don’t know how far we can go.

[ehelms]

@bagasse

Regarding path name, maybe /etc/foreman-pki will be more consistent as we alreaday have /etc/foreman-proxy, /etc/foreman-installer

Sure!

@bagasse

Maybe adding a “certificate provider” abstraction layer should be nice. Your implementation may be the “embeded” provider and this abstraction will ease implementation of other certificate providers in the future (certmonger, vault…)

Is there one in particular you have an interest in if I were to pick one to build and test this strategy against?

[bagasse]

Yes, as we use FreeIPA to manage our internal PKI, FreeIPA/certmonger integration, (at least for user/services communications, with exception of subscription management related certificates) should be nice to have for us.

[ehelms]

Reviving this thread, after reading through it all again, I thought I would put forth a view from the users interface to ensure I capture the relevant use cases. My goal then is if we agree on the use cases and the user interfaces I can move to the next layer of design.

Workflow	Description	Interface
Default installation	Default installation. Sets up CA certificate and private key, generates server certificate and private key and inputs to puppet modules. Candlepin would use a self-signed CA to issue client certificates.	foreman-installer
Installation with user provided certificate	User has generated a server certificate, and private key from their own CA and gives it as input along with the CA certificate. Candlepin would use a self-signed CA to issue client certificates. Alternative workflow: User generates a private key and CSR on the host, user then gets a certificate issued for the host. User then inputs the certificate, private key and CA certificate to the installer.	foreman-installer --ssl-certificate <cert path> --ssl-private-key <key path> --ssl-ca-certificate <ca-cert path>
Installation with user provided certificate and intermediate CA certificate and private key for Candlepin	Similar to scenario #2. Difference here is user has also generated an intermediate CA key-pair to give to the installer along with the private key for use by Candlepin directly.	foreman-installer --ssl-certificate <cert path> --ssl-private-key <key path> --ssl-ca-certificate <ca-cert path> --candlepin-ca-certificate <ca-key path> --candlepin-ca-key <ca-key path>
User registers an existing host	User generates a private key and a certificate signing request (CSR). User registers a host with a certificate signing request and gets back a certificate as part of the global registration workflow.	curl "http://foreman.example.com/register
Provisioning a new host	User provisions a new host and uses the issue_certificate macro to fetch a certificate for the provisioned host from their CA	Issue_certificate template macro

[tbrisker]

In general looks good, only question I have is regarding the puppet usecases and how do they fit in here?

[ehelms]

I didn’t forget your question But I did need to do some further digging and thinking around designs to come back and propose some technical follow up.

Certificate Management

First, here are two options around how I have been thinking about how to approach certificates input to the installer and propagated to the puppet modules from a strategy perspective. My overarching design principle is:

Separate creation of certificates from the management of the certificates

Option 1: Copy and manage

Steps

• User provides certificates as input
• Module copies those certificates to the “correct” location and manages them (e.g. user, group, perms)

Notes

• If a user removes the certificates installation will fail next run as it won’t find the original certificate
• If users removes the certificates, services will continue to function through restarts (since copies will remain in tact)
• Any certificate changes require installer re-run

Option 2: Manage the files directly

Steps

• User provides certificates as input
• Modules manages the input certificate paths directly (e.g. user, group, perms)

Notes

• Could gate behind a manage_certificates flag for advanced users
• One downside is user has to make they put certificates in an accessible location
• One downside is the location of certificates become unpredictable on a given installation
• One downside is potentially competing services (e.g. need multiple groups, different permissions)
• One upside/downside is user has more direct control, when certificates change they need only restart services
  • However, for services that use certificate stores those stores have to be re-generated anyway
• If a user removes the certificates, services will fail on any restarts, installation will fail next run

Certificate store Management

For some services, there are advanced certificate storage mechanisms in use such as keystores and NSS databases. The design here is around moving the creation and management of those stores into the modules for the services themselves and focus on the input to the module being the needed set of certificates. This keeps the user from having to create and manage these more complex entities and focus on just certificates. Further, given these stores need properties set on them like user and group for example it is easier for the module to manage this and ensure everything is correctly set up for the service to operate correctly.

I’ve opened a proof of concept PR to show this in action: POC: Manage truststore completely within module by ehelms · Pull Request #195 · theforeman/puppet-candlepin · GitHub

How does this apply to Puppet certificates (and Katello)?

Today Foreman defaults to using certificates generated by Puppet and using the paths on disk that Puppet creates. Management of the certificate access is handled by ensuring the foreman user is part of the puppet group. In Option 1 above, we would copy these Puppet certificates to a standard location (e.g. /etc/foreman) and set user/group/perms on them accordingly. In Option 2, we would continue the model we have today but based on user input rather than the current defaults.

That gets me to some of my thinking around how do we split apart the Foreman default to Puppet certificate paths and level set Puppet certificates with user defined certificates or some other provider such as Katello’s default certificate setup or something new in the future.

I am not sure yet on that point. For Puppet, the creation of the certificates and the paths are handled within the theforeman-puppet module and thus could be used as inputs to Foreman’s parameters for SSL. This same problem applies to Katello where we hard-code the certificate paths in the answers file.

How do we orchestrate that and make it easier for the user inside the installer?

I have opened a draft POC PR with this question to help try to gather some design discussions closer to the code as I think that can be easier to think and talk about.