Improving discovery workflow

[lzap]

We don’t need the whole chain for that

One more thing, customers tend to deploy their own intermediate CAs, I am not sure if HTTPS/TLS hands over only the server cert or whole chain up to (not including) the root CA.

[ekohl]

@lzap

store SSH keys in Foreman

I am not sure I am following here. Why we would need to store any SSH keys? In my proposal the only connection is Foreman → Discovered node and for this we only need to get Foreman’s public key to Discovered node. SSH keys on the discovered node are irrelevant.

However what is an issue is fingerprint which will always be different when a discovered node reboots. Facter should report that so ReX can have that information ready when making SSH calls to verify the identity of the remote side.

The flow I’d propose is:

sequenceDiagram
    Node->>Proxy: Get & verify certificate
    Node->>Proxy: Report facts
    Proxy->>Foreman: Create host
    Foreman->>Proxy: Start REX
    Proxy->>Foreman: Get SSH host keys
    Proxy->>Node: SSH login
    Node->>Proxy: Get SSH authorized keys
    Proxy->>Foreman: Get SSH authorized keys
    Proxy->>Node: Remote comand (repeat)

Using Online FlowChart & Diagrams Editor - Mermaid Live Editor to render this:

The reason I’m suggesting this is that currently AFAIK REX has no automated mechanism to clear SSH known hosts. That means you must solve this problem in some way.

[ekohl]

One more thing, customers tend to deploy their own intermediate CAs, I am not sure if HTTPS/TLS hands over only the server cert or whole chain up to (not including) the root CA.

If you do store the root then you can either store the intermediate on the client or present it from on the server side. It’s also sufficient to only download the server cert and ignore the intermediate + chain I think.

[lzap]

ignore the intermediate

Well I don’t know, I need to do research on that or ideally test it myself. I believe all X509 chain must be on client in order to verify it because if I can do it only with CA then what is the point of intermediate CAs?

[lzap]

Start REX

What’s that? Can you elaborate please?

[ekohl]

Start REX

What’s that? Can you elaborate please?

Should have been more explicit here. Start REX job or similar would have been more accurate.

[TimoGoebel]

I believe all X509 chain must be on client in order to verify it because if I can do it only with CA then what is the point of intermediate CAs?

It’s actually pretty straight forward. The client needs the whole certificate chain to verify a server certificate. However, you usually just want to use a single root certificate as a trust anchor. So the clients usually just knows the root certificate. If the server is configured properly, it knows the whole chain and sends it to the client. The client can than verify the whole chain for this connection. It starts with the root certificate it knows and trusts and uses that to verify the intermediate cert if got from the server. If that validates correctly, it can use the intermediate cert to verify the actual server certificate.

It’s also sufficient to only download the server cert and ignore the intermediate + chain I think.

That’s true. We either pin the server certificate (which might be relatively short-lived) or pin a root certificate.

[lzap]

It’s actually pretty straight forward.

This is exactly how I believe it works. Now, the question was if we can use fingerprint instead of SHA, therefore if this is fingerprint of the server cert then yes. It’s probably little bit more of Ruby code than doing SHA but let’s do this. Thanks.

[lzap]

Foreman->>Proxy: Start REX

At this point, discovered node don’t have Foreman’s public key yet. I don’t understand, unless there are some changes in ReX to do the bootstrap.

Proxy->>Foreman: Get SSH host keys

Is this getting the Foreman SSH public key? To do what with?

Proxy->>Node: SSH login

Again, Foreman public key hasn’t yet been deployed onto the node, this login attempt will always fail until there is something I am missing here. Did you mean to use Timo’s
AuthorizedKeysCommand idea with your workflow? Then we should probably put the HTTP request into the diagram as well.

You bring a good point however: authorized_keys. This is indeed an issue, with discovery this will be amplified as these hosts will likely have random fingerprints after each reboot.