Maybe, or maybe they’ll do that once, twice, see how smooth it is, and just write an Ansible playbook “schedule a proxy upgrade on all available proxies” for the third time.
Security should never rely on the user catching things (if possible).
In that case it’s safe, I got a wrong impression. Should have read more carefully.
I think what @evgeni talks about is a generic chicken and egg problem. We want some unknown machine to tell us it’s actually known. We can’t trust unknown machines though. This is very similar to the proxy registration feature of the installer, an unknown machine tells Foreman, hey I’m here and I’m your new proxy. There we only rely on the knowledge of oauth credentials. Oauth credentials are stored as settings in Foreman. User pass it to the installer. We hope only people who install real proxies know the credentials.
I think that should be the base for what we use to authorize instance uuid information in facts. We should only trust these facts if the fact author knows the oauth credentials, then we should trust them. What would be the best way to combine oauth with the suggested facts approach? I looked at puppet trusted facts but that does not seem to help, at least not now with our current CA infrastructure. I’m sure there are better options.
1 Like
If we can rely on the oauth secrets being available, then we could add an additional fact which would act as a signature to the uuid fact using the oauth secret as a key. Then on Foreman’s side we would verify the signature (since we know the uuid fact and the secret) and establish the link between proxy and a host only if the signature is valid.
This would still allow us to use facts as the delivery mechanism, allow us to do it by hand if needed and most importantly add certain level of trust to the established links between hosts and proxies.
The oauth credentials are only passed into the installer. Technically they are saved to the answers file but they don’t have to be. The proxy never has these credentials. Are you suggesting to extend the registration protocol to also identify a host that it runs on and implement that in the installer?
Now that we have the ability to generate JWT tokens with limited scope to a specific action, perhaps we could actually get rid of the oauth tokens and use a JWT instead?
Does it matter? The signature would be generated by the installer when it runs and deploys the custom facts. Once that is done, the secret should not be needed any more
Not at all. All I’m suggesting is extending my original fact based proposal with a signature fact in addition to each originally proposed uuid fact.
I like this. This means the initial fact upload performed from the installer has the information signed by the same mechanism. I assume the custom fact lives only for the sake of the installation. And if someone wants to setup continous update of the mapping, they need to create such signature as a custom fact.
Even if it lived longer, I assume the custom fact deployed by the installer is not any worse than the answer file. It does not contain the credential, just the signature. If it can only be read by root, that’s good even in that scenario.
Long term, I’m all for replacing the oauth with other type of credentials. But that increases the scope of this feature that we wanted to get to 2.5, because it’s changing what we already have and rely on in the installer registration feature.
Thinking out loud here: how will we ensure the facts are uploaded? There are plans to make Puppet optional and off by default. Then how does it work? On the other hand, it would also mean that the host is not present in Foreman at all so the relation doesn’t have to be there.
I think there’s no plan to stop using puppet in the installer, so this should continue working. But using facts mechanism is easily reusable by any other fact source, be it chef, ansible, salt, subscription-manager in case one wants to use that without puppet.
Correct, but the installer does not send facts. It runs Puppet in an agentless way and we take steps to isolate it from any config so it doesn’t attempt to connect to any server. Today the installer defaults to enabling Puppet on a machine. A Puppet check in results in the Puppetserver sending facts to Foreman which ends up creating a database entry. If you disable Puppetserver, there is no foreman.example.com host in Foreman (unless you use other means).
Roughly the flow is:
- Install Foreman
- Install Foreman Proxy
- When both are done, register the Foreman Proxy (if desired)
- If registration is enabled, ensure Puppet is started after the Proxy has been registered
This piece of code takes care of the last bit.
Only after that you can see some host entry show up.
You can verify this if you follow the steps documented in Defaulting Puppet to off in the Katello Scenario.
Ah, I was unaware of the fact, the puppetserver is necessary to be running for this. I thought we’d call the enc script after the proxy is registered. Today, the proxy needs to have Puppet feature enabled for the authorization, however we’d allow the alternative authorization by the facts signature. Is it a hard change to use the script directly with isolated puppet agent run? Or could we even use the facts gathered during the installer run?
Sounds like a very good idea. Thanks.
I guess, this would also mean, that use-cases like this are possible then: Run REX Job directly on Smart Proxy
Run a REX job on a smart proxy, even if the host (= which is the smart proxy) is not in the same org/location as the currently logged in user.
Update as of 2021-04-12:
To ensure an uuid fact (be it foreman or foreman-proxy) can be trusted, it has to be accompanied by a signature fact. This signature fact is a HMAC-SHA512 of the uuid, where oauth consumer secret is used as a key. If the signature is invalid or missing, the uuid fact is ignored and no linking is performed.
Foreman, puppet-foreman and puppet-foreman_proxy PRs were updated to reflect this.
The plan for when puppet is removed is to make the installer upload the necessary facts to foreman on its own by using modified enc script. Since we still have time until puppet goes away, I’d be glad if people could start looking at the opened PRs so we could get this in for 2.5 as-is. I would then proceed with getting it ready for uploading facts ourselves for 2.6.
1 Like
FWIW, I raised concerns about this design in the past, especially about the (secure) handling of the instance_id.
With the HMAC-SHA256 signature on the fact, and the way Foreman validates/filters it, I think this is sufficiently secure for our use case.
One thing that I’d raise: we make sure that the signature fact is not leaking in Foreman, but the users might have additional Puppet reporting endpoints configured (PuppetDB etc). While we don’t deploy these, we should document very prominently that we use facts for authentication and which facts should be filtered.
I still have my doubts about using facts for this. Why can’t we have the installer explicitly create the connection? Thinking out loud here.
What if you have the following endpoints:
GET /api/smart_proxies/:smart_proxy_id/hostsPOST /api/smart_proxies/:smart_proxy_id/hostsDELETE /api/smart_proxies/:smart_proxy_id/hosts/:host_id
This would rely on the fact that we have a certname for a host that we can use to uniquely identify a host. Within Puppet (and thus the installer) this certname should be available.
It’d remove all of the concerns with the fact handling and move to a static assignment that the installer takes care of.
Implementation wise we can create a Puppet type (+ provider) smart_proxy_host that ensures the entity exists.
The only complexity is that at the time the installer runs, there is no host. We can solve that by also sending facts in the payload so that a host is created on the fly (using the regular fact parser) if needed.
So the POST payload would effectively be a set of facts:
{
"certname": "myhost.example.com",
"facts": {}
}
Security wise it would use the same API auth that’s also used elsewhere. For the installer that means the oauth credentials but other systems could also use username/password. In that case, the user permissions must be checked.
I do think this should be flexible enough to create an Ansible module for it as well.
I’m probably missing something, so please have a look.
Can you be more specific please?
What all concerns? Security concerns were IMHO resolved above.
These become very similar API endpoints as fact upload, with just one difference. It will also say the host represented by these facts is the smart proxy. Why don’t we rather extend the existing facts endpoint to do that for us? What is the benefit of explicit endpoints for this? Is that the existing oauth authorization code instead of verifying oauth secret based signature?
As @evgeni noted: facts can end up in other databases. Essentially it’s adding secrets in there which can make it a trusted channel. I can’t exactly say what could go wrong and that’s precisely why I’m cautious.
What may help to understand it is this. Foreman has capabilities to update its data from facts and you can use configuration management to make the facts fit what Foreman thinks (like with the hostname, domain, network interfaces). My general impression is that updating Foreman from facts (as the proposal is here as well) generally leads to problems. Making Foreman the source of truth is generally more reliable.
That’s why I proposed to be explicit in the installer.
I didn’t think the exact API through that much. Perhaps there’s a better way to get done.
My opinion is still that enhancing the Fact Parser with this challenging.
Looking ahead, it may also conflict with a general design that looks better to me regarding fact uploads. That is:
- Accept facts
- Store facts unaltered
- Start async fact processing
- Return job ID
If you would implement this, there is a moment when the secret data is in the raw fact upload.
I think the proposal here is to extend the existing capability of Foreman DB being updated based on incoming facts. Extension here is that some facts that would have impact on the infrastructure must be verified to be created by the real proxy. That’s why we introduce the signature that can only be created by the proxy knowing the oauth secret.
What secret data? The signature is not a secret. The hostname is not a secret either. Also I thought you suggest we use the same facts importing mechanism in these endpoints, would that be prone to the same issue then? I still don’t see the issue though.
The signed UUID is secret (which is why it’s filtered out).
Let’s say you have a group of admins who can see the hosts that actually run Foreman. Then there’s different users who do have root on their machine but are not Foreman admins. If they know the signed UUID, they can ensure facts are sent so that their machine will be registered as a host that runs a Smart Proxy.
So I disagree that it’s not secret. It very much is.
As @evgeni mentioned, Puppet uploads facts to FacterDB and there may be other places where it leaks.