Foreman_remote_execution 3.2.x Cause errors when provisioning EC2 Instances

Problem: When provisioning instances to be deployed on EC2, the process fails and rolls back. This did not happen with ruby-foreman-remote-execution:all/plugins 3.0.3-1 but does starting with 3.2.0. Uninstalling the 3.2 versions and manually installing 3.0.3 allows instance creation.

Expected outcome: Instance should fully provision.

Foreman and Proxy versions:
foreman:amd64/buster 2.0.0-1 uptodate
foreman-cli:all/buster 2.0.0-1 uptodate
foreman-debug:all/buster 2.0.0-1 uptodate
foreman-ec2:all/buster 2.0.0-1 uptodate
foreman-installer:all/buster 2.0.0-1 uptodate
foreman-postgresql:all/buster 2.0.0-1 uptodate
foreman-proxy:all/buster 2.0.0-1 uptodate
ruby-foreman-deface:all/plugins 1.5.3-1 uptodate
ruby-foreman-remote-execution:all/plugins 3.0.3-1 upgradeable to 3.2.1-1
ruby-foreman-remote-execution-core:all/buster 1.3.0-1 uptodate
ruby-foreman-tasks:all/plugins 1.1.1-1 uptodate
ruby-foreman-tasks-core:all/buster 0.3.4-1 uptodate
ruby-hammer-cli-foreman:all/buster 2.0.1-1 uptodate

Distribution and version:
Debian Buster 10.3

Other relevant data:
2020-05-19T14:18:47 [W|app|c52a109a] ERF12-6886 [ProxyAPI::ProxyException]: Unable to remove host from known hosts ([RestClient::NotFound]: 404 Not Found) for proxy https://foreman:8443/ssh
2020-05-19T14:18:47 [W|app|c52a109a] Rolling back due to a problem: [#<Orchestration::Task:0x00007fee3d26a038 @name=“Remove SSH known hosts for”, @id=“ssh_remove_known_hosts_interface_3.16.67.127_1”, @status=“failed”, @priority=200, @action=[#<Nic::Managed id: 36, mac: nil, ip: “”, type: “Nic::Managed”, name: “”, host_id: 27, subnet_id: nil, domain_id: 4, attrs: {}, created_at: “2020-04-23 16:15:56”, updated_at: “2020-04-23 16:15:56”, provider: nil, username: nil, password: nil, virtual: false, link: true, identifier: “”, tag: “”, attached_to: “”, managed: true, mode: “balance-rr”, attached_devices: “”, bond_options: “”, primary: true, provision: true, compute_attributes: {}, ip6: “”, subnet6_id: nil, execution: true>, :drop_from_known_hosts, [1, “”]], @created=1589915927.0023513, @timestamp=2020-05-19 19:18:47 UTC>]

2020-05-19T14:18:47 c52a109a [I] Started DELETE /ssh/known_hosts/
2020-05-19T14:18:47 c52a109a [I] Finished DELETE /ssh/known_hosts/ with 404 (0.39 ms)

3.0.3 of the remote execution plugin does not seem to even try to remove host from known hosts.

Hi, the known hosts key removal feature was introduce in remote execution 3.2.0 and requires ruby-smart-proxy-remote-execution-ssh 0.3.0, it should be in the 2.0 repos.

When I listed my versions, I just grep’ed for packages including foreman so it excluded some of the installed packages that are relevant. Below is a list of all the packages installed from the Foreman repo, which includes ruby-smart-proxy-remote-execution-ssh 0.3.0. So having that installed doesn’t resolve the issue:
foreman 2.0.0-1
foreman-cli 2.0.0-1
foreman-debug 2.0.0-1
foreman-ec2 2.0.0-1
foreman-installer 2.0.0-1
foreman-postgresql 2.0.0-1
foreman-proxy 2.0.0-1
puppet-agent-oauth 0.5.1-2
ruby-apipie-bindings 0.3.0-1
ruby-apipie-params 0.0.5-1
ruby-bundler-ext 0.4.1-1
ruby-concurrent 1.1.6+dfsg-2
ruby-dynflow 1.4.2-1
ruby-foreman-deface 1.5.3-1
ruby-foreman-remote-execution-core 1.3.0-1
ruby-foreman-tasks 1.1.1-1
ruby-foreman-tasks-core 0.3.4-1
ruby-hammer-cli 2.0.0-1
ruby-hammer-cli-foreman 2.0.1-1
ruby-jwt 2.2.1-1
ruby-kafo 4.0.0-1
ruby-kafo-parsers 1.0.0-1
ruby-kafo-wizards 0.0.1-1
ruby-rkerberos 0.1.3-3
ruby-rubyipmi 0.10.0-1
ruby-smart-proxy-dynflow 0.2.4-1
ruby-smart-proxy-dynflow-core 0.2.4-1
ruby-smart-proxy-remote-execution-ssh 0.3.0-1

We’re currently debugging, but it seems we’re also affected by this behaviour since

May 25 14:33:12 Updated: tfm-rubygem-foreman-tasks-1.1.1-1.fm2_0.el7.noarch May 25 14:38:32 Updated: tfm-rubygem-foreman_remote_execution-3.2.1-1.fm2_0.el7.noarch May 25 14:39:01 Updated: tfm-rubygem-foreman_remote_execution-cockpit-3.2.1-1.fm2_0.el7.noarch

the following update did not fix the issue so far
May 26 02:17:56 Updated: tfm-rubygem-concurrent-ruby.noarch 1:1.1.6-1.el7 May 26 02:17:56 Updated: tfm-rubygem-concurrent-ruby-edge.noarch 1:0.6.0-1.fm2_0.el7 May 26 02:17:56 Updated: tfm-rubygem-dynflow.noarch 1.4.3-1.fm2_0.el7 May 26 02:17:56 Updated: tfm-rubygem-foreman-tasks-core.noarch 0.3.4-1.fm2_0.el7 May 26 02:17:56 Updated: tfm-rubygem-smart_proxy_remote_execution_ssh.noarch 0.3.0-1.fm2_0.el7

Have you restarted foreman-proxy after updating the package? Are you getting 404 from the smart proxy or a different error?

We’ve restarted the whole stack, not directly after upgrading but right now every component has been restarted.
Right now, we mitigated the issue of not being able to (de-)provision hosts by downgrading to
yum versionlock list Geladene Plugins: fastestmirror, versionlock 0:tfm-rubygem-foreman_remote_execution-3.0.3-2.fm2_0.el7.* 0:tfm-rubygem-foreman_remote_execution-cockpit-3.0.3-2.fm2_0.el7.*

I’m sorry, but I’m unable to give you live logs right now, since we (kind of) fixed that, but you’re right: foreman-proxy answered with a 404, which resulted at foreman in a 203 (? as i remember) instead of a 200. During the DELETE action, no matching key was present in /usr/share/foreman-proxy/.ssh/known_hosts.
We’ve manually added the key into the known_hosts, during the next DELETE the key has been successfully removed, but the foreman-proxy answered with a 404 anyway.
Just get back, if you need more information.