After updating from foreman 3.2 to 3.3 … I’m seeing duplicate job templates on the job page when trying to execute a command… I’m also experiencing issues getting the jobs to actually complete as well – they all appear to result in “Error initializing command: RuntimeError - Failed to establish connection to remote host, exit code: 1” with no other information in the logs as to why it’s unable to establish the connection…this seems to be a intermittent result. But, let’s take care of the duplicate templates first … here’s a snapshot of my job screen
any ideas?
You get multiple inputs when there are templates with different providers within a single category. There was a migration which should have moved all templates from ssh provider to script provider, but it seems that the migration run and then the templates got re-created.
What platform are you on? How did you perform the upgrade? Anything noteworthy that happened during the upgrade?
Hi @aruzicka
I’m on EL8. The upgrade seemed to be a bit more hassle than other upgrades, I had some issues and had to rollback a few packages but seem to at least have gotten the install into a stable state (lesson learned to take a snapshot before hand in case of user error). According to foreman-rake db:migrate:status all of the displayed migration states are up. Here’s a current list of installed packages and last few output of the migrate: status command:
up 20210610134000 ********** NO FILE **********
up 20210624221630 Katello generic content
up 20210625095042 Add retain package versions count
up 20210628182553 Add generic remote options to root repository
up 20210708123832 Add parent task id to remote tasksforeman tasks
up 20210714140440 Remove repo export permission
up 20210720115251 Add purpose to recurring logic
up 20210721163730 Change gpg keys to content credentials
up 20210728130748 Create function deb version cmp
up 20210816100932 Rex setting category to dsl
up 20210831161843 Add upstream auth token to root repository
up 20210901081438 Drop view tasks permission
up 20210909140337 Add last indexed to repos
up 20210910190324 Move background settings to immediate
up 20210915132645 General setting to dsl
up 20210924103241 Remove katello id from environmentsforeman puppet
up 20210929132645 Core setting to dsl
up 20211006161617 Add filename to katello generic content units
up 20211019192121 Create cdn configurationkatello
up 20211025181315 Add additional metadata to katello generic content units
up 20211111115000 Drop fix db cache setting
up 20211111125003 Drop puppetclasses direct referencesforeman puppet
up 20211112130803 Cleanup environment from core tablesforeman puppet
up 20211115215210 Drop ostree branches
up 20211122154929 Templates settings category to dsl
up 20211123170430 Tasks settings to dsl category
up 20211129200124 Remove dependency solving algorithm setting
up 20211201154845 Add unique indexes
up 20211202172358 Add repository mirroring policy
up 20211208034230 Add content view and lifecycle environment
up 20211220185935 Clean duplicate content units
up 20220110223754 Update disconnected settings
up 20220111110149 Drop require ssl smart proxies setting
up 20220117151612 Add alternate content sources
up 20220120163252 Fix docker download policy
up 20220124191056 Add type to cdn configuration
up 20220127120843 Fix debian download policy
up 20220201205305 Migrate host class permissionforeman puppet
up 20220204171908 Rename docker tags whitelist and add exclude tags
up 20220207140355 Change deb attributes size limit
up 20220208134539 Rename cent os stream os
up 20220208135305 Migrate environment ignore typeforeman puppet
up 20220209203251 Add generated for to content views
up 20220209205137 Expand sync timeout settings
up 20220228173251 Remove drpm from ignorable content
up 20220303160220 Remove duplicate errata
up 20220321101835 Rename ssh provider to script
up 20220331112719 Add ssh user to job invocation
up 20220404190836 Delete old setting data
up 20220405220616 Update cdn configuration type
up 20220419193414 Content settings to dsl category
up 20220421204325 Drop environment from host and hgforeman puppet
up 20220428203334 Add last refreshed to katello alternate content sources
- candlepin-4.1.11-1.el8.noarch
- candlepin-selinux-4.1.11-1.el8.noarch
- foreman-3.3.0-1.el8.noarch
- foreman-cli-3.3.0-1.el8.noarch
- foreman-debug-3.3.0-1.el8.noarch
- foreman-dynflow-sidekiq-3.3.0-1.el8.noarch
- foreman-installer-3.3.0-1.el8.noarch
- foreman-installer-katello-3.3.0-1.el8.noarch
- foreman-libvirt-3.3.0-1.el8.noarch
- foreman-postgresql-3.3.0-1.el8.noarch
- foreman-proxy-3.3.0-1.el8.noarch
- foreman-release-3.3.0-1.el8.noarch
- foreman-selinux-3.3.0-1.el8.noarch
- foreman-service-3.3.0-1.el8.noarch
- katello-4.5.0-0.1.rc2.el8.noarch
- katello-certs-tools-2.9.0-1.el8.noarch
- katello-client-bootstrap-1.7.9-1.el8.noarch
- katello-common-4.5.0-0.1.rc2.el8.noarch
- katello-debug-4.5.0-0.1.rc2.el8.noarch
- katello-repos-4.5.0-0.4.rc2.el8.noarch
- katello-selinux-4.0.2-1.el8.noarch
- pulpcore-selinux-1.3.0-1.el8.x86_64
- python39-pulp-ansible-0.13.0-3.el8.noarch
- python39-pulp-certguard-1.5.2-3.el8.noarch
- python39-pulp-cli-0.14.0-2.el8.noarch
- python39-pulp-container-2.10.3-4.el8.noarch
- python39-pulp-deb-2.18.0-3.el8.noarch
- python39-pulp-file-1.10.2-2.el8.noarch
- python39-pulp-python-3.6.0-5.el8.noarch
- python39-pulp-rpm-3.17.5-3.el8.noarch
- python39-pulpcore-3.18.5-2.el8.noarch
- qpid-proton-c-0.37.0-1.el8.x86_64
- rubygem-foreman-tasks-6.0.2-1.fm3_3.el8.noarch
- rubygem-foreman_maintain-1.1.1-3.el8.noarch
- rubygem-foreman_puppet-4.0.1-1.fm3_3.el8.noarch
- rubygem-foreman_remote_execution-7.0.0-1.fm3_3.el8.noarch
- rubygem-foreman_templates-9.3.0-1.fm3_3.el8.noarch
- rubygem-hammer_cli-3.3.0-1.el8.noarch
- rubygem-hammer_cli_foreman-3.3.0-1.el8.noarch
- rubygem-hammer_cli_foreman_puppet-0.0.6-1.fm3_3.el8.noarch
- rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
- rubygem-hammer_cli_foreman_tasks-0.0.17-1.fm3_2.el8.noarch
- rubygem-hammer_cli_katello-1.5.0-0.1.pre.master.20220511142526git2ef8866.el8.noarch
- rubygem-katello-4.5.0-0.5.rc2.el8.noarch
- rubygem-pulp_ansible_client-0.12.1-1.el8.noarch
- rubygem-pulp_certguard_client-1.5.0-1.el8.noarch
- rubygem-pulp_container_client-2.10.3-1.el8.noarch
- rubygem-pulp_deb_client-2.18.0-1.el8.noarch
- rubygem-pulp_file_client-1.10.0-1.el8.noarch
- rubygem-pulp_ostree_client-2.0.0-0.1.a1.el8.noarch
- rubygem-pulp_python_client-3.6.0-1.el8.noarch
- rubygem-pulp_rpm_client-3.17.4-1.el8.noarch
- rubygem-pulpcore_client-3.18.5-1.el8.noarch
- rubygem-qpid_proton-0.37.0-1.el8.x86_64
- rubygem-smart_proxy_pulp-3.2.0-3.fm3_3.el8.noarch
That is a bit of a red flag. Do you recall which packages were those?
Thank you for the package list and output of db:migrat:status, both of those look in order
@aruzicka Yeah, so here’s my embarrassing trip through getting to my current state… originally pulpcore wasn’t even responding but I’ve since corrected that issue, now it just seems that REX is messed up… The reason for all of this was because following the update to Foreman 3.3 there was an intermittent problem with the REX plugin actually able to log into the managed hosts (as described in my original post). The gist it seems is that I was attempting to downgrade to 3.2.2, but that definitely didn’t work, so reinstalled 3.3.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------[171/9813]
226 | -d 0 -e 1 -y install rubygem-foreman_templates | 2022-06-17 12:27 | Install | 3
225 | update | 2022-06-16 11:15 | Upgrade | 2
224 | -y erase qpid-cpp-server | 2022-06-16 11:07 | Removed | 7 EE
223 | -y erase qpid-tools | 2022-06-16 11:06 | Removed | 12
222 | -d 0 -e 1 -y install qpid-cpp-server-linearstore | 2022-06-16 11:02 | Install | 2
221 | -d 0 -e 1 -y install cyrus-sasl-plain | 2022-06-16 10:55 | Install | 1
220 | -d 0 -e 1 -y install qpid-cpp-server | 2022-06-16 10:54 | Install | 1
219 | -d 0 -e 1 -y install qpid-tools | 2022-06-16 10:54 | Install | 16
218 | update --allowerasing | 2022-06-16 09:56 | E, I, O, U | 12 <
217 | -d 0 -e 1 -y install rubygem-smart_proxy_remote_execution_ssh | 2022-06-15 15:04 | Install | 2 >E
216 | -d 0 -e 1 -y install rubygem-smart_proxy_dynflow | 2022-06-15 15:04 | Install | 1 EE
215 | -d 0 -e 1 -y install rubygem-smart_proxy_dhcp_remote_isc | 2022-06-15 15:04 | Install | 1 EE
214 | -d 0 -e 1 -y install rubygem-foreman_puppet | 2022-06-15 15:02 | Install | 1
213 | -d 0 -e 1 -y install foreman-libvirt | 2022-06-15 15:02 | Install | 6
212 | update candlepin | 2022-06-15 14:37 | Upgrade | 2
211 | -d 0 -e 1 -y install rubygem-smart_proxy_pulp | 2022-06-15 13:07 | Install | 1 EE
210 | -d 0 -e 1 -y install pulpcore-plugin(certguard) | 2022-06-15 13:06 | Install | 4
209 | -d 0 -e 1 -y install pulpcore-plugin(python) | 2022-06-15 13:05 | Install | 11
208 | -d 0 -e 1 -y install pulpcore-plugin(ansible) | 2022-06-15 13:05 | Install | 32
207 | -d 0 -e 1 -y install pulpcore-plugin(deb) | 2022-06-15 13:04 | Install | 3
206 | -d 0 -e 1 -y install pulpcore-plugin(rpm) | 2022-06-15 13:04 | Install | 19
205 | -d 0 -e 1 -y install pulpcore-plugin(file) | 2022-06-15 13:04 | Install | 1
204 | -d 0 -e 1 -y install pulpcore-plugin(container) | 2022-06-15 13:01 | I, O | 104
203 | -d 0 -e 1 -y install foreman-cli | 2022-06-15 13:01 | Install | 1
202 | -d 0 -e 1 -y install katello | 2022-06-15 12:59 | Install | 31 E<
201 | -d 0 -e 1 -y install pulp-cli | 2022-06-15 12:57 | Install | 10 >
200 | -d 0 -e 1 -y install katello-client-bootstrap | 2022-06-15 12:57 | Install | 1
199 | -d 0 -e 1 -y install katello-debug | 2022-06-15 12:57 | Install | 1
198 | -d 0 -e 1 -y install foreman-dynflow-sidekiq | 2022-06-15 12:56 | Install | 5
197 | -d 0 -e 1 -y install foreman-service | 2022-06-15 12:56 | Install | 6
196 | -d 0 -e 1 -y install foreman-postgresql | 2022-06-15 12:56 | Install | 2
195 | -d 0 -e 1 -y install pulpcore-selinux | 2022-06-15 12:53 | Install | 54 EE
194 | remove python39* | 2022-06-15 12:52 | Removed | 63 EE
193 | remove python39-gunicorn | 2022-06-15 12:51 | Removed | 1
192 | remove python39-dynaconf | 2022-06-15 12:51 | Removed | 1
191 | --disableplugin=subscription-manager remove python39-django | 2022-06-15 12:50 | Removed | 20
190 | -d 0 -e 1 -y install katello-selinux | 2022-06-15 12:41 | Install | 80 EE
189 | -d 0 -e 1 -y install foreman-selinux | 2022-06-15 12:40 | Install | 1 E<
188 | install foreman foreman-installer* foreman-proxy | 2022-06-15 12:23 | Install | 155 ><
187 | --disableplugin=subscription-manager remove foreman* katello* *pulp* | 2022-06-15 12:13 | Removed | 379 ><
186 | autoremove | 2022-06-15 10:33 | Removed | 54 >
185 | downgrade katello* foreman* pulp* | 2022-06-15 10:09 | Downgrade | 38 EE
184 | -d 0 -e 1 -y install rubygem-smart_proxy_remote_execution_ssh | 2022-06-15 09:51 | Install | 2
183 | -d 0 -e 1 -y install rubygem-smart_proxy_dynflow | 2022-06-15 09:51 | Install | 1
182 | remove rubygem-smart_proxy_remote_execution_ssh | 2022-06-15 09:47 | Removed | 3 EE
181 | -d 0 -e 1 -y install rubygem-smart_proxy_pulp | 2022-06-14 15:46 | Install | 1
180 | -d 0 -e 1 -y install pulpcore-plugin(certguard) | 2022-06-14 15:44 | Install | 4
179 | -d 0 -e 1 -y install pulpcore-plugin(python) | 2022-06-14 15:44 | Install | 11
178 | -d 0 -e 1 -y install pulpcore-plugin(ansible) | 2022-06-14 15:43 | Install | 5
177 | -d 0 -e 1 -y install pulpcore-plugin(deb) | 2022-06-14 15:43 | Install | 3
176 | -d 0 -e 1 -y install pulpcore-plugin(rpm) | 2022-06-14 15:43 | Install | 18
175 | -d 0 -e 1 -y install pulpcore-plugin(file) | 2022-06-14 15:42 | Install | 1
174 | -d 0 -e 1 -y install pulpcore-plugin(container) | 2022-06-14 15:42 | Install | 5
173 | -d 0 -e 1 -y install katello | 2022-06-14 15:41 | Install | 32 EE
172 | -d 0 -e 1 -y install pulpcore-selinux | 2022-06-14 15:39 | Install | 29 EE
171 | remove *pulp* | 2022-06-14 15:34 | Removed | 115 EE
170 | update | 2022-06-14 15:20 | Upgrade | 2
169 | -d 0 -e 1 -y install rubygem-smart_proxy_container_gateway | 2022-06-14 14:12 | Install | 1
168 | remove foreman-proxy-content | 2022-06-14 13:43 | Removed | 1
167 | install foreman-proxy-content | 2022-06-14 13:43 | Install | 1
166 | -d 0 -e 1 -y install pulpcore-plugin(certguard) | 2022-06-14 13:25 | Install | 4
165 | -d 0 -e 1 -y install pulpcore-plugin(python) | 2022-06-14 13:25 | Install | 11
164 | -d 0 -e 1 -y install pulpcore-plugin(ansible) | 2022-06-14 13:25 | Install | 5
163 | -d 0 -e 1 -y install pulpcore-plugin(deb) | 2022-06-14 13:25 | Install | 2
162 | -d 0 -e 1 -y install pulpcore-plugin(rpm) | 2022-06-14 13:24 | Install | 18
161 | -d 0 -e 1 -y install pulpcore-plugin(file) | 2022-06-14 13:24 | Install | 1
160 | -d 0 -e 1 -y install pulpcore-plugin(container) | 2022-06-14 13:24 | Install | 5
159 | -d 0 -e 1 -y install pulpcore-selinux | 2022-06-14 13:22 | Install | 29 EE
158 | update --allowerasing | 2022-06-14 13:17 | E, I, O, U | 133
157 | remove python38-humanfriendly-10.0-1.el8.noarch python38-cchardet-2.1.7-1.el8.x86_64 | 2022-06-14 13:14 | Removed | 121 EE
156 | reinstall *foreman* *katello* *rubygem* *pulp* *candle* | 2022-06-14 12:41 | R | 448 EE
155 | update --allowerasing --nobest | 2022-06-14 12:15 | I, O, U | 147
154 | install katello | 2022-06-14 12:07 | I, U | 8 <
153 | update --allowerasing | 2022-06-14 11:53 | E, U | 51 ><
152 | history undo 148 | 2022-06-14 11:36 | D, E, I | 163 ><
151 | update | 2022-06-14 10:59 | Upgrade | 5 >
150 | update | 2022-06-13 20:12 | Upgrade | 11 EE
149 | install traceroute | 2022-06-10 14:20 | Install | 1
148 | update --allowerasing | 2022-06-09 12:41 | E, I, U | 163 EE
147 | history undo 146 | 2022-06-09 10:54 | Downgrade | 8
146 | update -x ansible* | 2022-06-09 10:40 | Upgrade | 8
This could be it I guess. The timeline would be something like this
- You installed 3.3 packages
- The
20220321101835 Rename ssh provider to script migration ran and moved all templates to the script provider
- You observed things don’t work as expected
- You tried to go back to 3.2.2
- During startup, Foreman re-created job templates with the old provider
- You went forward to 3.3 again
And now you have templates with both providers
The easiest way out would probably going to Hosts > Job Templates, searching for provider_type = SSH and unlocking and removing all of them.
I think I tried that… but they for some reason reappeared (and the duplicate job template field still existed)… I’ll try it again though and report back.
@aruzicka All right so good news, the duplicate templates are gone after deleting the templates, so I appear to finally be back on the stock Foreman 3.3 instance. However, I’m still having issues that I’m unable to really diagnose given the output of the REX script being extremely vague. Should I open up a new post for this one? It seems to be a intermittent issue, some of the hosts on the same network are working while others aren’t, I can verify in all certainty that the proxy is able to access the hosts it’s trying to ssh into… all I’m getting in the output is:
Error initializing command: RuntimeError - Failed to establish connection to remote host, exit code: 1
Exit status: EXCEPTION
Please do open a new one so we don’t make this hard to follow in case anything runs into a similar issue.
