Foreman 3.3 + katello 4.5 duplicate REX job templates

Support
#1

After updating from foreman 3.2 to 3.3 … I’m seeing duplicate job templates on the job page when trying to execute a command… I’m also experiencing issues getting the jobs to actually complete as well – they all appear to result in “Error initializing command: RuntimeError - Failed to establish connection to remote host, exit code: 1” with no other information in the logs as to why it’s unable to establish the connection…this seems to be a intermittent result. But, let’s take care of the duplicate templates first … here’s a snapshot of my job screen

Screenshot at 2022-06-17 13-02-52
Screenshot at 2022-06-17 13-02-521075×662 46.8 KB

any ideas?

#2

You get multiple inputs when there are templates with different providers within a single category. There was a migration which should have moved all templates from ssh provider to script provider, but it seems that the migration run and then the templates got re-created.

What platform are you on? How did you perform the upgrade? Anything noteworthy that happened during the upgrade?

#3

Hi @aruzicka

I’m on EL8. The upgrade seemed to be a bit more hassle than other upgrades, I had some issues and had to rollback a few packages but seem to at least have gotten the install into a stable state (lesson learned to take a snapshot before hand in case of user error). According to foreman-rake db:migrate:status all of the displayed migration states are up. Here’s a current list of installed packages and last few output of the migrate: status command:

   up     20210610134000  ********** NO FILE **********
   up     20210624221630  Katello generic content
   up     20210625095042  Add retain package versions count
   up     20210628182553  Add generic remote options to root repository
   up     20210708123832  Add parent task id to remote tasksforeman tasks
   up     20210714140440  Remove repo export permission
   up     20210720115251  Add purpose to recurring logic
   up     20210721163730  Change gpg keys to content credentials
   up     20210728130748  Create function deb version cmp
   up     20210816100932  Rex setting category to dsl
   up     20210831161843  Add upstream auth token to root repository
   up     20210901081438  Drop view tasks permission
   up     20210909140337  Add last indexed to repos
   up     20210910190324  Move background settings to immediate
   up     20210915132645  General setting to dsl
   up     20210924103241  Remove katello id from environmentsforeman puppet
   up     20210929132645  Core setting to dsl
   up     20211006161617  Add filename to katello generic content units
   up     20211019192121  Create cdn configurationkatello
   up     20211025181315  Add additional metadata to katello generic content units
   up     20211111115000  Drop fix db cache setting
   up     20211111125003  Drop puppetclasses direct referencesforeman puppet
   up     20211112130803  Cleanup environment from core tablesforeman puppet
   up     20211115215210  Drop ostree branches
   up     20211122154929  Templates settings category to dsl
   up     20211123170430  Tasks settings to dsl category
   up     20211129200124  Remove dependency solving algorithm setting
   up     20211201154845  Add unique indexes
   up     20211202172358  Add repository mirroring policy
   up     20211208034230  Add content view and lifecycle environment
   up     20211220185935  Clean duplicate content units
   up     20220110223754  Update disconnected settings
   up     20220111110149  Drop require ssl smart proxies setting
   up     20220117151612  Add alternate content sources
   up     20220120163252  Fix docker download policy
   up     20220124191056  Add type to cdn configuration
   up     20220127120843  Fix debian download policy
   up     20220201205305  Migrate host class permissionforeman puppet
   up     20220204171908  Rename docker tags whitelist and add exclude tags
   up     20220207140355  Change deb attributes size limit
   up     20220208134539  Rename cent os stream os
   up     20220208135305  Migrate environment ignore typeforeman puppet
   up     20220209203251  Add generated for to content views
   up     20220209205137  Expand sync timeout settings
   up     20220228173251  Remove drpm from ignorable content
   up     20220303160220  Remove duplicate errata
   up     20220321101835  Rename ssh provider to script
   up     20220331112719  Add ssh user to job invocation
   up     20220404190836  Delete old setting data
   up     20220405220616  Update cdn configuration type
   up     20220419193414  Content settings to dsl category
   up     20220421204325  Drop environment from host and hgforeman puppet
   up     20220428203334  Add last refreshed to katello alternate content sources
  • candlepin-4.1.11-1.el8.noarch
  • candlepin-selinux-4.1.11-1.el8.noarch
  • foreman-3.3.0-1.el8.noarch
  • foreman-cli-3.3.0-1.el8.noarch
  • foreman-debug-3.3.0-1.el8.noarch
  • foreman-dynflow-sidekiq-3.3.0-1.el8.noarch
  • foreman-installer-3.3.0-1.el8.noarch
  • foreman-installer-katello-3.3.0-1.el8.noarch
  • foreman-libvirt-3.3.0-1.el8.noarch
  • foreman-postgresql-3.3.0-1.el8.noarch
  • foreman-proxy-3.3.0-1.el8.noarch
  • foreman-release-3.3.0-1.el8.noarch
  • foreman-selinux-3.3.0-1.el8.noarch
  • foreman-service-3.3.0-1.el8.noarch
  • katello-4.5.0-0.1.rc2.el8.noarch
  • katello-certs-tools-2.9.0-1.el8.noarch
  • katello-client-bootstrap-1.7.9-1.el8.noarch
  • katello-common-4.5.0-0.1.rc2.el8.noarch
  • katello-debug-4.5.0-0.1.rc2.el8.noarch
  • katello-repos-4.5.0-0.4.rc2.el8.noarch
  • katello-selinux-4.0.2-1.el8.noarch
  • pulpcore-selinux-1.3.0-1.el8.x86_64
  • python39-pulp-ansible-0.13.0-3.el8.noarch
  • python39-pulp-certguard-1.5.2-3.el8.noarch
  • python39-pulp-cli-0.14.0-2.el8.noarch
  • python39-pulp-container-2.10.3-4.el8.noarch
  • python39-pulp-deb-2.18.0-3.el8.noarch
  • python39-pulp-file-1.10.2-2.el8.noarch
  • python39-pulp-python-3.6.0-5.el8.noarch
  • python39-pulp-rpm-3.17.5-3.el8.noarch
  • python39-pulpcore-3.18.5-2.el8.noarch
  • qpid-proton-c-0.37.0-1.el8.x86_64
  • rubygem-foreman-tasks-6.0.2-1.fm3_3.el8.noarch
  • rubygem-foreman_maintain-1.1.1-3.el8.noarch
  • rubygem-foreman_puppet-4.0.1-1.fm3_3.el8.noarch
  • rubygem-foreman_remote_execution-7.0.0-1.fm3_3.el8.noarch
  • rubygem-foreman_templates-9.3.0-1.fm3_3.el8.noarch
  • rubygem-hammer_cli-3.3.0-1.el8.noarch
  • rubygem-hammer_cli_foreman-3.3.0-1.el8.noarch
  • rubygem-hammer_cli_foreman_puppet-0.0.6-1.fm3_3.el8.noarch
  • rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
  • rubygem-hammer_cli_foreman_tasks-0.0.17-1.fm3_2.el8.noarch
  • rubygem-hammer_cli_katello-1.5.0-0.1.pre.master.20220511142526git2ef8866.el8.noarch
  • rubygem-katello-4.5.0-0.5.rc2.el8.noarch
  • rubygem-pulp_ansible_client-0.12.1-1.el8.noarch
  • rubygem-pulp_certguard_client-1.5.0-1.el8.noarch
  • rubygem-pulp_container_client-2.10.3-1.el8.noarch
  • rubygem-pulp_deb_client-2.18.0-1.el8.noarch
  • rubygem-pulp_file_client-1.10.0-1.el8.noarch
  • rubygem-pulp_ostree_client-2.0.0-0.1.a1.el8.noarch
  • rubygem-pulp_python_client-3.6.0-1.el8.noarch
  • rubygem-pulp_rpm_client-3.17.4-1.el8.noarch
  • rubygem-pulpcore_client-3.18.5-1.el8.noarch
  • rubygem-qpid_proton-0.37.0-1.el8.x86_64
  • rubygem-smart_proxy_pulp-3.2.0-3.fm3_3.el8.noarch
#4

That is a bit of a red flag. Do you recall which packages were those?

Thank you for the package list and output of db:migrat:status, both of those look in order

#5

@aruzicka Yeah, so here’s my embarrassing trip through getting to my current state… originally pulpcore wasn’t even responding but I’ve since corrected that issue, now it just seems that REX is messed up… The reason for all of this was because following the update to Foreman 3.3 there was an intermittent problem with the REX plugin actually able to log into the managed hosts (as described in my original post). The gist it seems is that I was attempting to downgrade to 3.2.2, but that definitely didn’t work, so reinstalled 3.3.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------[171/9813]
   226 | -d 0 -e 1 -y install rubygem-foreman_templates                                                                                                                                          | 2022-06-17 12:27 | Install        |    3   
   225 | update                                                                                                                                                                                  | 2022-06-16 11:15 | Upgrade        |    2   
   224 | -y erase qpid-cpp-server                                                                                                                                                                | 2022-06-16 11:07 | Removed        |    7 EE
   223 | -y erase qpid-tools                                                                                                                                                                     | 2022-06-16 11:06 | Removed        |   12   
   222 | -d 0 -e 1 -y install qpid-cpp-server-linearstore                                                                                                                                        | 2022-06-16 11:02 | Install        |    2   
   221 | -d 0 -e 1 -y install cyrus-sasl-plain                                                                                                                                                   | 2022-06-16 10:55 | Install        |    1   
   220 | -d 0 -e 1 -y install qpid-cpp-server                                                                                                                                                    | 2022-06-16 10:54 | Install        |    1   
   219 | -d 0 -e 1 -y install qpid-tools                                                                                                                                                         | 2022-06-16 10:54 | Install        |   16   
   218 | update --allowerasing                                                                                                                                                                   | 2022-06-16 09:56 | E, I, O, U     |   12  <
   217 | -d 0 -e 1 -y install rubygem-smart_proxy_remote_execution_ssh                                                                                                                           | 2022-06-15 15:04 | Install        |    2 >E
   216 | -d 0 -e 1 -y install rubygem-smart_proxy_dynflow                                                                                                                                        | 2022-06-15 15:04 | Install        |    1 EE
   215 | -d 0 -e 1 -y install rubygem-smart_proxy_dhcp_remote_isc                                                                                                                                | 2022-06-15 15:04 | Install        |    1 EE
   214 | -d 0 -e 1 -y install rubygem-foreman_puppet                                                                                                                                             | 2022-06-15 15:02 | Install        |    1   
   213 | -d 0 -e 1 -y install foreman-libvirt                                                                                                                                                    | 2022-06-15 15:02 | Install        |    6   
   212 | update candlepin                                                                                                                                                                        | 2022-06-15 14:37 | Upgrade        |    2   
   211 | -d 0 -e 1 -y install rubygem-smart_proxy_pulp                                                                                                                                           | 2022-06-15 13:07 | Install        |    1 EE
   210 | -d 0 -e 1 -y install pulpcore-plugin(certguard)                                                                                                                                         | 2022-06-15 13:06 | Install        |    4   
   209 | -d 0 -e 1 -y install pulpcore-plugin(python)                                                                                                                                            | 2022-06-15 13:05 | Install        |   11   
   208 | -d 0 -e 1 -y install pulpcore-plugin(ansible)                                                                                                                                           | 2022-06-15 13:05 | Install        |   32   
   207 | -d 0 -e 1 -y install pulpcore-plugin(deb)                                                                                                                                               | 2022-06-15 13:04 | Install        |    3   
   206 | -d 0 -e 1 -y install pulpcore-plugin(rpm)                                                                                                                                               | 2022-06-15 13:04 | Install        |   19   
   205 | -d 0 -e 1 -y install pulpcore-plugin(file)                                                                                                                                              | 2022-06-15 13:04 | Install        |    1   
   204 | -d 0 -e 1 -y install pulpcore-plugin(container)                                                                                                                                         | 2022-06-15 13:01 | I, O           |  104   
   203 | -d 0 -e 1 -y install foreman-cli                                                                                                                                                        | 2022-06-15 13:01 | Install        |    1   
   202 | -d 0 -e 1 -y install katello                                                                                                                                                            | 2022-06-15 12:59 | Install        |   31 E<
   201 | -d 0 -e 1 -y install pulp-cli                                                                                                                                                           | 2022-06-15 12:57 | Install        |   10 > 
   200 | -d 0 -e 1 -y install katello-client-bootstrap                                                                                                                                           | 2022-06-15 12:57 | Install        |    1   
   199 | -d 0 -e 1 -y install katello-debug                                                                                                                                                      | 2022-06-15 12:57 | Install        |    1   
   198 | -d 0 -e 1 -y install foreman-dynflow-sidekiq                                                                                                                                            | 2022-06-15 12:56 | Install        |    5   
   197 | -d 0 -e 1 -y install foreman-service                                                                                                                                                    | 2022-06-15 12:56 | Install        |    6   
   196 | -d 0 -e 1 -y install foreman-postgresql                                                                                                                                                 | 2022-06-15 12:56 | Install        |    2   
   195 | -d 0 -e 1 -y install pulpcore-selinux                                                                                                                                                   | 2022-06-15 12:53 | Install        |   54 EE
   194 | remove python39*                                                                                                                                                                        | 2022-06-15 12:52 | Removed        |   63 EE
   193 | remove python39-gunicorn                                                                                                                                                                | 2022-06-15 12:51 | Removed        |    1
   192 | remove python39-dynaconf                                                                                                                                                                | 2022-06-15 12:51 | Removed        |    1
   191 | --disableplugin=subscription-manager remove python39-django                                                                                                                             | 2022-06-15 12:50 | Removed        |   20
   190 | -d 0 -e 1 -y install katello-selinux                                                                                                                                                    | 2022-06-15 12:41 | Install        |   80 EE
   189 | -d 0 -e 1 -y install foreman-selinux                                                                                                                                                    | 2022-06-15 12:40 | Install        |    1 E<
   188 | install foreman foreman-installer* foreman-proxy                                                                                                                                        | 2022-06-15 12:23 | Install        |  155 ><
   187 | --disableplugin=subscription-manager remove foreman* katello* *pulp*                                                                                                                    | 2022-06-15 12:13 | Removed        |  379 ><
   186 | autoremove                                                                                                                                                                              | 2022-06-15 10:33 | Removed        |   54 >
   185 | downgrade katello* foreman* pulp*                                                                                                                                                       | 2022-06-15 10:09 | Downgrade      |   38 EE
   184 | -d 0 -e 1 -y install rubygem-smart_proxy_remote_execution_ssh                                                                                                                           | 2022-06-15 09:51 | Install        |    2
   183 | -d 0 -e 1 -y install rubygem-smart_proxy_dynflow                                                                                                                                        | 2022-06-15 09:51 | Install        |    1
   182 | remove rubygem-smart_proxy_remote_execution_ssh                                                                                                                                         | 2022-06-15 09:47 | Removed        |    3 EE
   181 | -d 0 -e 1 -y install rubygem-smart_proxy_pulp                                                                                                                                           | 2022-06-14 15:46 | Install        |    1
   180 | -d 0 -e 1 -y install pulpcore-plugin(certguard)                                                                                                                                         | 2022-06-14 15:44 | Install        |    4
   179 | -d 0 -e 1 -y install pulpcore-plugin(python)                                                                                                                                            | 2022-06-14 15:44 | Install        |   11
   178 | -d 0 -e 1 -y install pulpcore-plugin(ansible)                                                                                                                                           | 2022-06-14 15:43 | Install        |    5
   177 | -d 0 -e 1 -y install pulpcore-plugin(deb)                                                                                                                                               | 2022-06-14 15:43 | Install        |    3
   176 | -d 0 -e 1 -y install pulpcore-plugin(rpm)                                                                                                                                               | 2022-06-14 15:43 | Install        |   18
   175 | -d 0 -e 1 -y install pulpcore-plugin(file)                                                                                                                                              | 2022-06-14 15:42 | Install        |    1
   174 | -d 0 -e 1 -y install pulpcore-plugin(container)                                                                                                                                         | 2022-06-14 15:42 | Install        |    5
   173 | -d 0 -e 1 -y install katello                                                                                                                                                            | 2022-06-14 15:41 | Install        |   32 EE
   172 | -d 0 -e 1 -y install pulpcore-selinux                                                                                                                                                   | 2022-06-14 15:39 | Install        |   29 EE
   171 | remove *pulp*                                                                                                                                                                           | 2022-06-14 15:34 | Removed        |  115 EE
   170 | update                                                                                                                                                                                  | 2022-06-14 15:20 | Upgrade        |    2
   169 | -d 0 -e 1 -y install rubygem-smart_proxy_container_gateway                                                                                                                              | 2022-06-14 14:12 | Install        |    1
   168 | remove foreman-proxy-content                                                                                                                                                            | 2022-06-14 13:43 | Removed        |    1
   167 | install foreman-proxy-content                                                                                                                                                           | 2022-06-14 13:43 | Install        |    1
   166 | -d 0 -e 1 -y install pulpcore-plugin(certguard)                                                                                                                                         | 2022-06-14 13:25 | Install        |    4
   165 | -d 0 -e 1 -y install pulpcore-plugin(python)                                                                                                                                            | 2022-06-14 13:25 | Install        |   11
   164 | -d 0 -e 1 -y install pulpcore-plugin(ansible)                                                                                                                                           | 2022-06-14 13:25 | Install        |    5
   163 | -d 0 -e 1 -y install pulpcore-plugin(deb)                                                                                                                                               | 2022-06-14 13:25 | Install        |    2
   162 | -d 0 -e 1 -y install pulpcore-plugin(rpm)                                                                                                                                               | 2022-06-14 13:24 | Install        |   18
   161 | -d 0 -e 1 -y install pulpcore-plugin(file)                                                                                                                                              | 2022-06-14 13:24 | Install        |    1
   160 | -d 0 -e 1 -y install pulpcore-plugin(container)                                                                                                                                         | 2022-06-14 13:24 | Install        |    5
   159 | -d 0 -e 1 -y install pulpcore-selinux                                                                                                                                                   | 2022-06-14 13:22 | Install        |   29 EE
   158 | update --allowerasing                                                                                                                                                                   | 2022-06-14 13:17 | E, I, O, U     |  133
   157 | remove python38-humanfriendly-10.0-1.el8.noarch python38-cchardet-2.1.7-1.el8.x86_64                                                                                                    | 2022-06-14 13:14 | Removed        |  121 EE
   156 | reinstall *foreman* *katello* *rubygem* *pulp* *candle*                                                                                                                                 | 2022-06-14 12:41 | R              |  448 EE
   155 | update --allowerasing --nobest                                                                                                                                                          | 2022-06-14 12:15 | I, O, U        |  147
   154 | install katello                                                                                                                                                                         | 2022-06-14 12:07 | I, U           |    8  <
   153 | update --allowerasing                                                                                                                                                                   | 2022-06-14 11:53 | E, U           |   51 ><
   152 | history undo 148                                                                                                                                                                        | 2022-06-14 11:36 | D, E, I        |  163 ><
   151 | update                                                                                                                                                                                  | 2022-06-14 10:59 | Upgrade        |    5 >
   150 | update                                                                                                                                                                                  | 2022-06-13 20:12 | Upgrade        |   11 EE
   149 | install traceroute                                                                                                                                                                      | 2022-06-10 14:20 | Install        |    1
   148 | update --allowerasing                                                                                                                                                                   | 2022-06-09 12:41 | E, I, U        |  163 EE
   147 | history undo 146                                                                                                                                                                        | 2022-06-09 10:54 | Downgrade      |    8
   146 | update -x ansible*                                                                                                                                                                      | 2022-06-09 10:40 | Upgrade        |    8
#6

This could be it I guess. The timeline would be something like this

  1. You installed 3.3 packages
  2. The 20220321101835 Rename ssh provider to script migration ran and moved all templates to the script provider
  3. You observed things don’t work as expected
  4. You tried to go back to 3.2.2
  5. During startup, Foreman re-created job templates with the old provider
  6. You went forward to 3.3 again

And now you have templates with both providers

The easiest way out would probably going to Hosts > Job Templates, searching for provider_type = SSH and unlocking and removing all of them.

#7

I think I tried that… but they for some reason reappeared (and the duplicate job template field still existed)… I’ll try it again though and report back.

#8

@aruzicka All right so good news, the duplicate templates are gone after deleting the templates, so I appear to finally be back on the stock Foreman 3.3 instance. However, I’m still having issues that I’m unable to really diagnose given the output of the REX script being extremely vague. Should I open up a new post for this one? It seems to be a intermittent issue, some of the hosts on the same network are working while others aren’t, I can verify in all certainty that the proxy is able to access the hosts it’s trying to ssh into… all I’m getting in the output is:

Error initializing command: RuntimeError - Failed to establish connection to remote host, exit code: 1
Exit status: EXCEPTION
#9

Please do open a new one so we don’t make this hard to follow in case anything runs into a similar issue.