Hello,
while I was working on some SSH key-related changes, during reviews we were unsure how SSH authentication works in this workflow. I hereby present you a summary on the topic. This post is all written by me, keeping it as short as possible while using AI to confirm how I remembered it.
Finish SSH script is a template that can be associated with a host through its operating system or host group. When the host is being provisioned (created), Foreman (Server) waits for the host to come up and tries to connect via SSH to the provision interface’s IPv6 address, then IPv4 address, and finally the given hostname if no IP address is available. It executes the provided template, typically a shell script. This is used for bootstrapping things like Ansible or Remote Execution.
In order to achieve this, the user must create an Image record for a particular Compute Resource. In this image, SSH user must be provided together with an optional password. There is also a user-data boolean flag and it must be turned off, otherwise no SSH finish script is scheduled for execution because user data is injected (typically through cloud-init) instead.
Now, here comes the tricky part. Only some Compute Resources do support SSH keys. When the Compute Resource is created, they automatically generate a fresh SSH key pair and store the private key in the Foreman database. The key pair is normally associated with the Compute Resource and reused for hosts created through it; it is not unique for each host.
Some plugins, including Azure RM and Google, also generate and store a key pair, although their capability advertising is not fully consistent with the core EC2/OpenStack implementation. Some integrations store only the private key in Foreman because the provider already stores or receives the public key; others may store both. That is a different story, and I want to keep this short. The list of supported Compute Resources:
- EC2
- OpenStack
- Azure RM
As you can see, this workflow is pretty much common for cloud, but not so much for virtualization: popular libvirt and VMWare are missing from here. The reason is that this dynamic injection of SSH keys needs dedicated infrastructure, which is provided by cloud providers (typically via cloud-init or other means).
Fun fact: the mentioned optional password field is ignored for the mentioned Compute Resources, it is only effective for other ones. It does not even work as a fallback when SSH key is wrong, in that case the SSH finish workflow just give up.
Marek made a good point that there was also a possibility to edit ~foreman/.ssh/id_rsa and provide your own key, which was accepted by the previous implementation. This has changed with the new implementation and no longer works. We should talk about that.
This feature has always been a bit shady, because all of this happens in the background. Foreman UI/API does advertise the key_pair capability where it is implemented, but the relationship between provider key-pair creation, guest key injection, and SSH finish authentication is not very visible to users. We do have this in our docs somewhere, or at least I hope that is the case. I hope at some point we can probably expose this to users more directly, or allow them to add their own SSH keys. Those would need to be handled separately from the automatically generated private keys and would only work if the corresponding public keys were also made available in the guest, for example through an image, cloud-init, or a custom provisioning template.
Hope it helps to shed some light on the matter. It definitely helped me to understand (again!) how this works.
