Removing a host from Foreman does not remove it from known_hosts on proxy

On Foreman 3.8/Katello 4.10.0 and when performing some migrations of hosts from CentOS 7 to AlmaLinux 9 I noticed that after I perform this command from within the host to remove the old host from Foreman:

subscription-manager remove --all; subscription-manager unregister

the entry for the host is still available in the /var/lib/foreman-proxy/ssh/known_hosts on the proxy. So when I now deploy the replacement host using the same IP and name and use remote execution, the proxy refuse to connect with the error:

Authentication method 'publickey' failed with:

So to correct this I have to manually purge the old key from known_hosts on the proxy.
Is this the expected behaviour?
Would be nice if when removing a host from Foreman it also removes the entry for the host in known_hosts automatically.

It should work if you really remove the host from Foreman. As far as I know, the subscription-manager sequence you used only cleans the host itself, but the host record in foreman stays.

1 Like

When I run this command the host at least disappear from Foreman so I assumed it was successfully unregistered.

Interesting, on my machine (foreman 3.9) it sticks around.

hmm, just tested again with a test machine and it for sure does not exist in Foreman after when looking for it with “hammer host list”.

I registered the host with the “curl https://:9090$proxyname/register…” command generated from the “Register Host” UI.

“subscription-manager unregister” gives me this back so I would assume all is good on the Foreman side:

Unregistering from: proxy.local:443/rhsm
System has been unregistered.

So is “subscription-manager unregister” not a supported way to unregister a host from Foreman? I mean all seems to work except the removal of the host from known_hosts on the proxy.

If there a way to remove a host from Foreman that cleans up known_hosts on the proxies?

Perhaps the difference is the host type. Registering with subscription-manager does not create a managed host if I remember correctly. Unregistering can so only remove the content facet of the host, so I would expect it to stay if managed and perhaps to be completely removed if unmanaged.

And SSH management is very likely part of a managed host, too.

1 Like

Looking with “hammer host info --id ” I see this status on all my hosts:

Managed:                  no

Not really sure what this indicate. I mean I do manage the hosts with Foreman/Katello with subscriptions/repos/errata/remote execution, ansible, scap and so on.
Does managed here means they are provisioned from Foreman?

There is a Setting, unregister_delete_host, which controls whether a host is destroyed when you unregister. (TIL)

Is enabled:

1 Like

Also tried to delete a host from the Foreman UI with the same result. The /var/lib/foreman-proxy/ssh/known_hosts file is unchanged on the proxy.

The way it should work is:

  • foreman keeps track if a rex job was execute through a particular smart proxy
  • if foreman determines the job goes through a specific proxy for the first time, the proxy should remove the entry from known hosts file just before trying to open the connection
  • when you remove the host, the record tying a host to the smart proxy gets dropped so on next execution foreman should think that it is being executed that way for the first time

So you should see the key there, until you try running the job again

OK, but since the remote execution stops working for any of my migrated hosts and Foreman complains that the key in known_hosts on the proxy no longer match the new host, something is broken in my end or something broken in Foreman?

I mean I can ad-hoc a fix for the issue, just need to add some automation that constantly erase the known_hosts files on the proxies. Pretty ugly though…

It seems to work on my machine (on foreman 3.9), although I must admit I skipped the migration step.