Foreman_remote_execution 3.2.x Cause errors when provisioning EC2 Instances

Support
,
1

Problem: When provisioning instances to be deployed on EC2, the process fails and rolls back. This did not happen with ruby-foreman-remote-execution:all/plugins 3.0.3-1 but does starting with 3.2.0. Uninstalling the 3.2 versions and manually installing 3.0.3 allows instance creation.

Expected outcome: Instance should fully provision.

Foreman and Proxy versions:
foreman:amd64/buster 2.0.0-1 uptodate
foreman-cli:all/buster 2.0.0-1 uptodate
foreman-debug:all/buster 2.0.0-1 uptodate
foreman-ec2:all/buster 2.0.0-1 uptodate
foreman-installer:all/buster 2.0.0-1 uptodate
foreman-postgresql:all/buster 2.0.0-1 uptodate
foreman-proxy:all/buster 2.0.0-1 uptodate
ruby-foreman-deface:all/plugins 1.5.3-1 uptodate
ruby-foreman-remote-execution:all/plugins 3.0.3-1 upgradeable to 3.2.1-1
ruby-foreman-remote-execution-core:all/buster 1.3.0-1 uptodate
ruby-foreman-tasks:all/plugins 1.1.1-1 uptodate
ruby-foreman-tasks-core:all/buster 0.3.4-1 uptodate
ruby-hammer-cli-foreman:all/buster 2.0.1-1 uptodate

Distribution and version:
Debian Buster 10.3

Other relevant data:
foreman/production.log:
2020-05-19T14:18:47 [W|app|c52a109a] ERF12-6886 [ProxyAPI::ProxyException]: Unable to remove host from known hosts ([RestClient::NotFound]: 404 Not Found) for proxy https://foreman:8443/ssh
2020-05-19T14:18:47 [W|app|c52a109a] Rolling back due to a problem: [#<Orchestration::Task:0x00007fee3d26a038 @name=“Remove SSH known hosts for will-pinegar.domain.com”, @id=“ssh_remove_known_hosts_interface_3.16.67.127_1”, @status=“failed”, @priority=200, @action=[#<Nic::Managed id: 36, mac: nil, ip: “3.16.67.127”, type: “Nic::Managed”, name: “will-pinegar.domain.com”, host_id: 27, subnet_id: nil, domain_id: 4, attrs: {}, created_at: “2020-04-23 16:15:56”, updated_at: “2020-04-23 16:15:56”, provider: nil, username: nil, password: nil, virtual: false, link: true, identifier: “”, tag: “”, attached_to: “”, managed: true, mode: “balance-rr”, attached_devices: “”, bond_options: “”, primary: true, provision: true, compute_attributes: {}, ip6: “”, subnet6_id: nil, execution: true>, :drop_from_known_hosts, [1, “3.16.67.127”]], @created=1589915927.0023513, @timestamp=2020-05-19 19:18:47 UTC>]

foreman-proxy/proxy.log
2020-05-19T14:18:47 c52a109a [I] Started DELETE /ssh/known_hosts/3.16.67.127
2020-05-19T14:18:47 c52a109a [I] Finished DELETE /ssh/known_hosts/3.16.67.127 with 404 (0.39 ms)

2

3.0.3 of the remote execution plugin does not seem to even try to remove host from known hosts.

3

Hi, the known hosts key removal feature was introduce in remote execution 3.2.0 and requires ruby-smart-proxy-remote-execution-ssh 0.3.0, it should be in the 2.0 repos.

4

When I listed my versions, I just grep’ed for packages including foreman so it excluded some of the installed packages that are relevant. Below is a list of all the packages installed from the Foreman repo, which includes ruby-smart-proxy-remote-execution-ssh 0.3.0. So having that installed doesn’t resolve the issue:
foreman 2.0.0-1
foreman-cli 2.0.0-1
foreman-debug 2.0.0-1
foreman-ec2 2.0.0-1
foreman-installer 2.0.0-1
foreman-postgresql 2.0.0-1
foreman-proxy 2.0.0-1
puppet-agent-oauth 0.5.1-2
ruby-apipie-bindings 0.3.0-1
ruby-apipie-params 0.0.5-1
ruby-bundler-ext 0.4.1-1
ruby-concurrent 1.1.6+dfsg-2
ruby-dynflow 1.4.2-1
ruby-foreman-deface 1.5.3-1
ruby-foreman-remote-execution-core 1.3.0-1
ruby-foreman-tasks 1.1.1-1
ruby-foreman-tasks-core 0.3.4-1
ruby-hammer-cli 2.0.0-1
ruby-hammer-cli-foreman 2.0.1-1
ruby-jwt 2.2.1-1
ruby-kafo 4.0.0-1
ruby-kafo-parsers 1.0.0-1
ruby-kafo-wizards 0.0.1-1
ruby-rkerberos 0.1.3-3
ruby-rubyipmi 0.10.0-1
ruby-smart-proxy-dynflow 0.2.4-1
ruby-smart-proxy-dynflow-core 0.2.4-1
ruby-smart-proxy-remote-execution-ssh 0.3.0-1

5

Hi!
We’re currently debugging, but it seems we’re also affected by this behaviour since

May 25 14:33:12 Updated: tfm-rubygem-foreman-tasks-1.1.1-1.fm2_0.el7.noarch May 25 14:38:32 Updated: tfm-rubygem-foreman_remote_execution-3.2.1-1.fm2_0.el7.noarch May 25 14:39:01 Updated: tfm-rubygem-foreman_remote_execution-cockpit-3.2.1-1.fm2_0.el7.noarch

the following update did not fix the issue so far
May 26 02:17:56 Updated: tfm-rubygem-concurrent-ruby.noarch 1:1.1.6-1.el7 May 26 02:17:56 Updated: tfm-rubygem-concurrent-ruby-edge.noarch 1:0.6.0-1.fm2_0.el7 May 26 02:17:56 Updated: tfm-rubygem-dynflow.noarch 1.4.3-1.fm2_0.el7 May 26 02:17:56 Updated: tfm-rubygem-foreman-tasks-core.noarch 0.3.4-1.fm2_0.el7 May 26 02:17:56 Updated: tfm-rubygem-smart_proxy_remote_execution_ssh.noarch 0.3.0-1.fm2_0.el7

6

Have you restarted foreman-proxy after updating the package? Are you getting 404 from the smart proxy or a different error?

7

We’ve restarted the whole stack, not directly after upgrading but right now every component has been restarted.
Right now, we mitigated the issue of not being able to (de-)provision hosts by downgrading to
yum versionlock list Geladene Plugins: fastestmirror, versionlock 0:tfm-rubygem-foreman_remote_execution-3.0.3-2.fm2_0.el7.* 0:tfm-rubygem-foreman_remote_execution-cockpit-3.0.3-2.fm2_0.el7.*

I’m sorry, but I’m unable to give you live logs right now, since we (kind of) fixed that, but you’re right: foreman-proxy answered with a 404, which resulted at foreman in a 203 (? as i remember) instead of a 200. During the DELETE action, no matching key was present in /usr/share/foreman-proxy/.ssh/known_hosts.
We’ve manually added the key into the known_hosts, during the next DELETE the key has been successfully removed, but the foreman-proxy answered with a 404 anyway.
Just get back, if you need more information.