Upgrade to katello 4.4.0 from 4.3.1 (foreman-rake db:seed)

Problem:

2022-05-26 12:30:20 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: ‘/usr/sbin/foreman-rake db:seed’ returned 1 instead of one of [0]
2022-05-26 12:30:20 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: ‘/usr/sbin/foreman-rake db:seed’ returned 1 instead of one of [0]

A second run of foreman-installer completed successfully. Should this be expected? Is some data in the postgres db corrupt?

Expected outcome:
foreman-installer should complete successfully.

Foreman and Proxy versions:
Foreman 3.2.1 / katello 4.4.0.2

Foreman and Proxy plugin versions:

Distribution and version:
Centos 7.8

Other relevant data:
From foreman-installer/katello.log

2022-05-26 09:00:35 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: ActiveRecord::RecordInvalid: Validation failed: Remote execution features is invalid

This error actually appears in my current version of Katello 4.3.1 /Foreman 3.1.2
I get this error in the logs during a foreman-maintain service restart


2022-07-05T13:04:48 [I|app|] Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/db/seeds.d/75-job_templates.rb
2022-07-05T13:04:48 [E|app|] Error while attempting to seed database, please run `foreman-rake db:seed` manually!
2022-07-05T13:04:48 [E|app|] /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:80:in `raise_validation_error': Validation failed: Remote execution features is invalid (ActiveRecord::RecordInvalid)
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:53:in `save!'
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:318:in `block in save!'
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in `transaction'
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in `transaction'
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:318:in `save!'
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/suppressor.rb:48:in `save!'
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.5/app/models/job_template.rb:68:in `import_raw!'
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/db/seeds.d/75-job_templates.rb:10:in `block (3 levels) in <top (required)>'
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/db/seeds.d/75-job_templates.rb:6:in `each'
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/db/seeds.d/75-job_templates.rb:6:in `block (2 levels) in <top (required)>'
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
 |      from /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/db/seeds.d/75-job_templates.rb:3:in `block in <top (required)>'
 |      from /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'

After checking the database table remote_execution_features, I was able to find the problem. The same feature was listed twice.
I have been able to cleanly upgrade to katello 4.4 after completing these steps. I must have had a the installer fail setup the REX features a while back. I’ve only recently started to use REX.

# problem was these two rows:
su - postgres
psql 
\c foreman
foreman=# select * from backup_remote_execution_features ;
 id |              label               |                  name                   |                     description                      |      provided_inputs       | job_template_id | ho
st_action_button | notification_builder | proxy_selector_override
----+----------------------------------+-----------------------------------------+------------------------------------------------------+----------------------------+-----------------+---
-----------------+----------------------+-------------------------
 13 | katello_errata_install_by_search | Katello: Install errata by search query | Install errata using scoped search query             | Errata search query        |                 | f
                 |                      |
 12 | katello_errata_install_by_search | Katello: Install errata by search query | Install errata using scoped search query             | Errata search query        |                 | f


# ran installer to disable the plugins...
foreman-installer \
--foreman-proxy-plugin-remote-execution-ssh-enabled false \
--no-enable-foreman-cli-remote-execution \
--no-enable-foreman-plugin-remote-execution \
--no-enable-foreman-proxy-plugin-remote-execution-ssh


#cleaned out the remote execution features, hoping they would all get reseeded. 
delete from remote_execution_features;
# deleted 13 rows.

# ran installer again to see if it would repopulate the features.  (It did!)
foreman-installer \
--foreman-proxy-plugin-remote-execution-ssh-enabled true \
--enable-foreman-cli-remote-execution \
--enable-foreman-plugin-remote-execution \
--enable-foreman-proxy-plugin-remote-execution-ssh


# check the table again 
select * from remote_execution_features;
# returned 12 rows.

Not sure how you managed to get this to work. I am having the same issue you are. I also am seeing multiple entries in the database.

However, when I go to run the command above, I get:

[root@hq-1pforeman ~]# foreman-installer \

–foreman-proxy-plugin-remote-execution-ssh-enabled false
–no-enable-foreman-cli-remote-execution
–no-enable-foreman-plugin-remote-execution
–no-enable-foreman-proxy-plugin-remote-execution-ssh
2022-07-11 13:10:25 [NOTICE] [root] Loading installer configuration. This will take some time.
ERROR: Unrecognised option ‘–foreman-proxy-plugin-remote-execution-ssh-enabled’

So what I did was simply delete the duplicate record:
delete from remote_execution_features where id=‘xx’;
where xx is whatever row number one of the duplicated entries is on.

foreman-rake db:seed ran successfully, then I re-ran foreman-installer successfully.

Clearly there was a bug somewhere long the road that caused a few people to have duplicated entries.