@Marek_Hulan: I meant something like this (leveraging curl’s https://ec.haxx.se/usingcurl/usingcurl-tls#certificate-pinning command.
curl --pinnedpubkey "sha256//83d34tasd3rt..." https://foreman.example.com/register/host?token=123 | bash
The actual hash of of the cert would be rendered by Foreman.
Sounds great, I wasn’t aware of this curl arg. This will work well for registering to Foreman directly, we’ll need to think more about registering through smart proxy. It’s technically doable, but it may be a good idea to start caching the proxy fingerprint, when we refresh its features.
Thanks!
We are not going to create new unattended endpoint, we need to authenticate user and we want to use JWT, so the final endpoint is going to be under API scope.
Yeah, I split the work into smaller tasks so I do not end up with one big PR.
Summary of the solution:
(More details in the tasks)
Sadly, this can’t be used everywhere. From the man page:
PEM/DER support:
7.39.0: OpenSSL, GnuTLS and GSKit
Then:
# curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.36 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Another concern is that Foreman has no setting for its own CA. Apache can serve with a different CA than the client certs so the only thing it can do is call out to the URL under foreman_url and get it from there.
We have the CA certificate configured on Foreman in the ssl_ca_file setting. I wonder if that could be used somehow.
Sadly DANE never took off but that would have been a possible way to establish the trust on an infrastructure level.
Generally this is not the first time we need a “global” template. For example iPXE bootstrap and also user-data cloud init bootstrap templates are both examples when we introduced new “global” endpoint. I know it is little bit late but I am wondering if we should name the new endpoint simply “/global” and this registration would be “/global/register” so we can reuse the same controller and render stack for other global templates.
Basically what I am proposing is a slight change in naming and HTTP path.
So there’s a lot of discussion spread over a lot of PRs. It turns out that at least I misunderstood subscription-manager and the proposed workflow. Perhaps @TimoGoebel as well.
This is how I propose it should be implemented and does differ from the original design.
Roughly speaking we have 4 flows. The first 2 are both situations where the client talks directory to Foreman and there is no Smart Proxy.
I used Mermaid JS to draw some sequence diagrams with how I think it should work.
sequenceDiagram
autonumber
Participant Foreman
Participant Client
Client->>Foreman: GET /register
activate Foreman
Foreman-->>Client: Global Registration Template
deactivate Foreman
activate Client
Client->>Foreman: POST /register
activate Foreman
Foreman-->>Client: Host Registration Template
deactivate Foreman
Client->>Foreman: POST foreman_url('built')
deactivate Client
This renders to:
In step 1 the client uses curl to retrieve a global registration template. This template is rendered and returned to the client (step 2), which then executes it. Effectively the client runs curl https://foreman.example.com/register | sh.
Then execution starts. Within the template there’s another curl request (step 3). The goal of this is to create a host entry within Foreman. That’s POST /register. If this is successful, the host object is created (in the state building). With that data, a Host Registration Template can be rendered and returned (step 4). This is then executed by the client. This execution is part of the original Global Registration Template and the user doesn’t have to do anything. After everything is completed, a POST to the built URL is sent to mark the host as built (step 5).
Authentication wise the user is responsible for providing credentials for the initial GET /register, for example by passing --user to curl. In the returned Global Registration Template a token is returned that allows the POST /register to happen. This token has an expiration time; currently JWT can’t be revoked so there is a risk of replay attacks if the token is intercepted and used multiple times. If everything is retrieved over HTTPS, this should be sufficiently mitigated as long as the user doesn’t store the rendered template insecurely (in /tmp with bad permissions for example).
I believe the HRT also generates a token for the built URL update, but I’m uncertain about the details.
Subscription Manager (subman) works different because there are some additional steps required. That means the workflow is on comparable at a very high level, but implementations are very different.
This implementation is only relevant for Red Hat-based workflows, at least for now.
Again, providing the diagram:
sequenceDiagram
autonumber
Participant Foreman
Participant Client
Participant SubMan
Client->>Foreman: GET /register
activate Foreman
Foreman-->>Client: Global Registration Template
deactivate Foreman
activate Client
Client->>SubMan: subscription-manager --register
activate SubMan
SubMan->>Foreman: POST to RHSM API
activate Foreman
Foreman-->>SubMan: certificates
deactivate Foreman
deactivate SubMan
Client->>Foreman: GET /templates/hrt
activate Foreman
Foreman-->>Client: Host Registration Template
deactivate Foreman
Client->>Foreman: POST foreman_url('built')
deactivate Client
The first step is still the same: user runs curl on /register. The GRT has code to detect subman should be used and runs step 3. I’ve drawn SubMan as a separate actor, but it happens on the Client machine. Perhaps Client should be read as shell, but that’s also an implementation detail.
During step 3, subman needs to register itself. It collects facts and prepares an API request (step 4) to the RHSM API (implemented by Katello which proxies it to Candlepin). Based on the data, Foreman ends up creating the Host object. Not drawn, but Candlepin also creates client certificates which are returned to subman (step 5). It will probably not be in status building, unless some custom fact is implemented for this. I’m not too clear on the details so I’ll invite others to correct me.
Since the HRT can’t be returned via the normal way, another way must be devised. I’m suggesting a dedicated endpoint (step 6). The exact URL is not that important.
Subman authentication (step 3) happens via Activation Keys (AKs), which is already built into subman. These keys already exist today an can be reused. It should be noted that are secrets and users should treat them as such.
The HRT template endpoint (step 6) should accept client certificates and let clients identify themselves. This avoids the need for yet another token and we know exactly which host it is due to the properties on the presented certificate.
I have ideas about how this should happen, but they’re based on the previous 2 proposals. That’s why I’d suggest we first agree on those and then expand on the other case.
I’m not too sure if this is actually the case. Why can’t we use the POST /register call (#3 in your first diagram) as well? The registration via subscription-manager would just optionally happen between (2 and 3 in your diagram).
Something like this:
sequenceDiagram
autonumber
Participant Foreman
Participant Client
Participant SubMan
Client->>Foreman: GET /register
activate Foreman
Foreman-->>Client: Global Registration Template
deactivate Foreman
activate SubMan
Client->>SubMan: subscription-manager --register
SubMan->>Foreman: POST to RHSM API
deactivate SubMan
Client->>Foreman: POST /register
activate Client
activate Foreman
Foreman-->>Client: Host Registration Template
deactivate Foreman
Client->>Foreman: POST foreman_url('built')
deactivate Client
IMHO it’s odd to do POST since that’s supposed to create an entity. The entity already exists.
It would be important that you actually get the host identity correct so you don’t end up with 2 host entries that differ. For example, if hostname --fqdn returns something different than subscription-manager does. One example where that can happen is having a hostname set up in /etc/hostname and a different reverse DNS. An identified request with the right credentials avoids that.
(related to that - I didn’t check if built was POST or PUT)
Looking at case 1 for now. Can you please highlight, what are the pros of the new endpoint POST /register compared to the current design? And how does the picture change if we also want to run $facter in GRT and use that information during HRT rendering?
If I understand that correctly, this is the only difference comparing to the current implementation in PRs, that split POST /register into two requests, one is for host creation, second for rendering HRT (both existing API endpoints). I can draw the diagram tomorrow if that helps.
Rather than using the POST /hosts API call that returns JSON. The current PR “parses” JSON using grep to then perform another HTTP request. That’s fragile since JSON aren’t just strings. By using a dedicated endpoint that returns the HRT directly, you avoid one HTTP request and manual JSON “parsing”. It also avoids the need for a token on the second HTTP request (since that never happens).
Later when we add a Smart Proxy, it also avoids the need to proxy /api/hosts. That is a good thing, but something I kept out of the scope in my post.
Right now data is POSTed to /api/hosts and I imagine later on it would include facts. By making POST /register special, you could even accept a hash of facts and let the fact parser sort it out for you. I didn’t think about the exact format, but a dedicated endpoint gives you this flexibility.
Turning it around: if you want to add facts to the current method, how would you do so?
Yes, if we ignore subman and only focus on case 1 that is accurate.
That’s true. But we can’t know for sure if the entity exists or not. The host might have been created importing the VM, the OS might not have been linked to Foreman/Katello though and a user might just want to do that last step.
The call to /register should contain all fact/metadata as part of the payload imho.
That would use the existing endpoint for uploading facts and replace the POST /api/hosts. Facts upload would create the host for us. The whole point was, the host creation part is customizable. The creation could be done by pure API call, subscription-manager, facts upload or whatever else people want to use. Handling it by explicit POST /register IMHO takes the flexibility away. Also all plugins will need to extend it with additional params, POST /api/hosts is a well known and already extended endpoint.
On the flip side, what I like about this new endpoint, we could modify the parameters per need. We may not want to allow setting e.g. compute attributes of host during the registration.
@lstejska is this acceptable for the case 1, meaning vanilla Foreman, no proxy involved.
Now for the second case - Subscription Manager (Katello)
What is Red Hat specific here? This seems as generic RHEL workflow when used with Foreman+Katello.
There’s one thing to be mentioned here as well, we deploy katello-ca-consumer.rpm, which configures rhsm.conf and deployes certificates between steps 2 and 3. That is necessary for step 4 to work. The deliver mechanism may change in future, but we still need to deliver those certs and configure rhsm.conf before step 3.
Why a dedicated endpoint? This is exactly what we already have today, we can ask for a HRT for a given host since https://github.com/theforeman/foreman/pull/6813.
The existing endpoint already supports JWT authentication. I may sound as a broken record, but I’ll repeat that once again. If we rely on host certificate, we can identify and authenticate the host but not the user, who performs the registration.
The HRT may access additional resources, such as subnet, domain, various parameters. We need to make sure, the user does not use resoruces he or she does not have access to. We should not use any system certificates here. We need something that authenticates the user.
Also, while not that important given above, certifiactes makes it harder when during through-proxy registration, JWT makes it much simpler. The only difference I see comparing to x509 is, a specific JWT can’t be revoked, we could only revoke all user’s tokens or deactive his or her account completely. But that’s why we have short expiration window. All goes through SSL and we use similar mechanism for the initial GRT request. I see no problem here.
This also applies for activation keys, doesn’t it? That’s why I brought this concern up in the PR.
I don’t think this would be a problem. If the registration token has a short lifespan, I can totally imagine a button “revoke all my registration tokens” (or something better that refers to the functionality, not the technology “Revoke all global host registration links”).
Yes, but the AK grants access to explicit list of action you do. E.g. it assigns organization. By knowing the AK value, user proved they have access to it. I know, it’s weak, I’d like to replace it with the host group concept at some point and use the same token solution we’ll have for vanilla Foreman, but that’s not something we can solve here. After the subscription-manager registration, I think we still need to get the HRT though, where much more logic happens. There we need to make sure, user is properly authenticated and authorized.
I totally agree, it’s not. Just for the fair comparison, with x509 you can revoke specfic cert, with JWT we can only easily revoke all user’s registration tokens, not just the one that leaked. But again, IMHO that’s not an issue we should focus on right now.
Well, actually you could store all unique token ids that you want to revoke in the database and achieve the same result. It’s doable, but you lose the flexibility of being able to validate a token “offline”.
Fair enough!
For that we’d benefit of client cert, but given we have secured communication channel already, we can send the host unique identifier - subscription UUID. On client side, after the registration we can get it by subscription-manager identity. Then we find the host object id by searching the host with “subscription_uuid = $xyz”.
Given there is a strong pushback on forwarding GET /hosts on proxy (which I don’t understand), we may need to extend HRT API to also be able to search host by subscription_uuid, not just hostname. And that would need to be an extension done in Katello.
Then it is the same mechanism with same reliability and we don’t need a client certificate authentication for that.
AFAIK for now subscription-manager is only packaged on Red Hat-based systems. I know there are some efforts to package it for Debian, but let’s not complicate this RFC with that.
Good point. We should describe this as “configure subscription-manager” in the diagram. Today it happens via the consumer RPM but that’s an implementation detail and there are talks about changing it.
Ok, perhaps bad wording but the point is that there’s a specific URL that’s called to retrieve the HRT. I should probably have mentioned that I didn’t look up the exact URL. Perhaps I should have said something like foreman_url('hrt') in the diagram.
I think it’s actually a bad thing to scope it to a user. I view the HRT as effectively configuration management and it’s similar to the ENC: there is a host profile that must be applied and the host is the relevant thing. We already know the user has root on the target system.
I’m worried about a scenario where the senior ops person sets up a workflow. Because the senior is responsible for the Foreman instance instance, this person is also a Foreman admin.
Now the senior registers a system, checks it’s all OK and hands off the job to a junior with limited permissions. The senior forgets to set up some permissions and registration doesn’t work anymore. The senior then tries to register a system and it does work.
I’ll agree that there’s something to say for both arguments.
IMHO the main benefit of client certs is that you exactly know which system it is. It is embedded in the certificate and the server can read all the data. You are correct that it is more complex when you do proxy.
IMHO it’s about isolation. If you register via a proxy, the goal is to isolate it and only provide the minimal endpoints. Once you start to rely on the full Foreman API, that becomes something you need to ensure.
I can’t believe I’m about to say this (because I’m generally not in favor of this), but I think it’s good to be opinionated in this case.
IMHO the only goal of registration should be to register the system in Foreman and allow our existing management systems to take over. That may be Puppet, Ansible/REX or something else.
Maybe I’m nit picking here, but it’s RHEL, not Red Hat specific then.
Ok, if we can agree on this endpoin being the same as in case 1 - the POST /register, then I’m good with that. This endpoints will need to behave like “find or create” and for the host it either found or created, it will render the HRT.
The fact that user has root access to one system in the infra does not imply, he or she should have admin permissions in Foreman. That Foreman instance can manage more hosts or even more infras.
This seems to be like we try to prevent users from making mistakes by turning off the authorization. We have user impersonation feature, senior admin can easily verify, whether it works under junior’s account.
But we know this already. In the original design, we got the host ID from the host creation. In this new design, the POST /register will render the template directly for the host that’s being registered. The only problem now appears when we want to implement “find or create” logic. It’s easy in case 2, when we’ll rely on subscription UUID, which reliable identifier. For case 1, we’ll need to rely on hostname anyway, there’s no guarantee, the has any trusted certificate.
I think the design never mentioned exposing the full Foreman API. We wanted to limit this just to POST /api/hosts and GET /api/hosts/:id. In the PR there were comments about adding more. But I think if we go with POST /register, we can easily achieve that. In fact @lstejska came up with isolating also the GRT logic into the new controller. So we’ll have both actions necessary for the registration as a separate endpoints that can be forwarded.
To summarize the current status of this discussion. I think we agreed on adding new endpoint POST /register, which saves us 1 request in case 1. In case 2, we’ll use the same endpoint, since it will be able to also work with existing hosts created e.g. by subscription-manager register). The only thing where I think we didn’t reach agreement is whether we want to rely/use client certificates. I think that’s only applicable in case 2. I suggest to remain consistent and use JWT, that we already rely on in case 1. Is that acceptable to move this forward?
After that is answered and the summary is confirmed by other interested parties (@ekohl, @TimoGoebel, @lstejska), we can start talking about case 3 and 4 - through proxy registration.
Yes, I’d like that. I want to note though, that saving the processing time of the extra request is obviously not the reason why the dedicated endpoint makes sense. From my point of view we do it for usability reasons, robustness, and containment.
