Upgrade to Foreman 2.5 and Katello 4.1 fails

Support
1

Problem:
Upgrade from Foreman 2.4 with Katello 4.0 to Foreman 2.5 with Katello 4.1 dies with the error:

2021-06-30 08:41:08 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.example.com]: Could not evaluate: Error making GET request to Foreman at https://foreman.example.com/api/v2/smart_proxies: Response: 422 Unprocessable Entity

Expected outcome:
Upgrade successful completed.

Foreman and Proxy versions:
Foreman 2.5 with Katello 4.1
Proxies have Foreman 2.4 with Katello 4.0, because one problem at the time

Foreman and Proxy plugin versions:
Can’t check, because calling foreman now gives this webpage:

The change you wanted was rejected.
Maybe you tried to change something you didn’t have access to.
If you are the application owner check the logs for more information.

Distribution and version:
CentOS 7 up-to-date

Other relevant data:
Installer command with output:

[root@foreman ~]# foreman-installer
2021-06-30 08:39:47 [NOTICE] [root] Loading installer configuration. This will take some time.
2021-06-30 08:39:50 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2021-06-30 08:39:50 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2021-06-30 08:39:57 [NOTICE] [configure] Starting system configuration.
2021-06-30 08:40:06 [NOTICE] [configure] 250 configuration steps out of 2124 steps complete.
2021-06-30 08:40:09 [NOTICE] [configure] 500 configuration steps out of 2124 steps complete.
2021-06-30 08:40:10 [NOTICE] [configure] 750 configuration steps out of 2126 steps complete.
2021-06-30 08:40:12 [NOTICE] [configure] 1000 configuration steps out of 2131 steps complete.
2021-06-30 08:40:12 [NOTICE] [configure] 1250 configuration steps out of 2135 steps complete.
2021-06-30 08:41:00 [NOTICE] [configure] 1500 configuration steps out of 2136 steps complete.
2021-06-30 08:41:00 [NOTICE] [configure] 1750 configuration steps out of 2136 steps complete.
2021-06-30 08:41:01 [NOTICE] [configure] 2000 configuration steps out of 2136 steps complete.
2021-06-30 08:41:08 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.example.com]: Could not evaluate: Error making GET request to Foreman at https://foreman.example.com/api/v2/smart_proxies: Response: 422 Unprocessable Entity
2021-06-30 08:41:09 [NOTICE] [configure] System configuration has finished.

Content of /var/log/foreman-installer/katello.log:

2021-06-30 08:41:08 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.example.com]: Starting to evaluate the resource (2112 of 2136)
2021-06-30 08:41:08 [DEBUG ] [configure] Foreman_smartproxyforeman.example.com: Making get request to https://foreman.example.com/api/v2/smart_proxies?search=name%3D"foreman.example.com"
2021-06-30 08:41:08 [DEBUG ] [configure] Foreman_smartproxyforeman.example.com: Received response 422 from request to https://foreman.example.com/api/v2/smart_proxies?search=name%3D"foreman.example.com"
2021-06-30 08:41:08 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.example.com]: Could not evaluate: Error making GET request to Foreman at https://foreman.example.com/api/v2/smart_proxies: Response: 422 Unprocessable Entity
2021-06-30 08:41:08 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.example.com]: Evaluated in 0.17 seconds

2

Is there any backtrace in /var/log/foreman-proxy/proxy.log?

Please run this command after replacing with your values and report back the result .

curl -u <your_admin:your_password> https://<your_katello_fqdn>/api/v2/smart_proxies

Thanks.

3

Looked into the proxy.log and didn’t find any backtrace.

The result of curl:

The change you wanted was rejected (422) body { background-color: #EFEFEF; color: #2E2F30; text-align: center; font-family: arial, sans-serif; margin: 0; }

div.dialog {
width: 95%;
max-width: 33em;
margin: 4em auto 0;
}

div.dialog > div {
border: 1px solid #CCC;
border-right-color: #999;
border-left-color: #999;
border-bottom-color: #BBB;
border-top: #B00100 solid 4px;
border-top-left-radius: 9px;
border-top-right-radius: 9px;
background-color: white;
padding: 7px 12% 0;
box-shadow: 0 3px 8px rgba(50, 50, 50, 0.17);
}

h1 {
font-size: 100%;
color: #730E15;
line-height: 1.5em;
}

div.dialog > p {
margin: 0 0 1em;
padding: 1em;
background-color: #F7F7F7;
border: 1px solid #CCC;
border-right-color: #999;
border-left-color: #999;
border-bottom-color: #999;
border-bottom-left-radius: 4px;
border-bottom-right-radius: 4px;
border-top-color: #DADADA;
color: #666;
box-shadow: 0 3px 8px rgba(50, 50, 50, 0.17);
}

The change you wanted was rejected.

Maybe you tried to change something you didn't have access to.

If you are the application owner check the logs for more information.

4

Now that I had some time to dig into the logs, I found this error:

2021-07-01 11:25:20 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: ‘/usr/sbin/foreman-rake db:seed’ won’t be executed because of failed check ‘refreshonly’

Checked the log before this line, but didn’t find anything suspicious to me.

— Update

Executed “/usr/sbin/foreman-rake db:seed” manually and got this output:

ActiveRecord::RecordInvalid: Validation failed: Provisioning templates expecting provisioning templates used by hosts or inherited (check mismatches report).
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:80:in raise_validation_error' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:53:in save!’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:318:in block in save!' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:375:in block in with_transaction_returning_status’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:366:in with_transaction_returning_status' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:318:in save!’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/suppressor.rb:48:in save!' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/lib/katello/lazy_accessor.rb:67:in save!’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/models/foreman_tasks/concerns/action_triggering.rb:31:in block in save!' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/models/foreman_tasks/concerns/action_triggering.rb:127:in dynflow_task_wrap’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/models/foreman_tasks/concerns/action_triggering.rb:31:in save!' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/services/katello/organization_creator.rb:31:in block in seed!’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/services/katello/organization_creator.rb:29:in seed!' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/services/katello/organization_creator.rb:8:in block (2 levels) in seed_all_organizations!’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/services/katello/organization_creator.rb:7:in block in seed_all_organizations!' /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in as’
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in as_anonymous_admin' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/app/services/katello/organization_creator.rb:6:in seed_all_organizations!’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.0/db/seeds.d/102-organizations.rb:8:in <top (required)>' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in load’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in block in load' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in load_dependency’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in load' /usr/share/foreman/app/services/foreman_seeder.rb:51:in block (3 levels) in execute’
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in as' /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in as_anonymous_admin’
/usr/share/foreman/app/services/foreman_seeder.rb:50:in block (2 levels) in execute' /usr/share/foreman/app/services/foreman_seeder.rb:44:in each’
/usr/share/foreman/app/services/foreman_seeder.rb:44:in block in execute' /usr/share/foreman/lib/foreman/advisory_lock_manager.rb:20:in block in with_transaction_lock’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in block in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:280:in block in within_new_transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in block (2 levels) in synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in handle_interrupt’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in block in synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in handle_interrupt’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:278:in within_new_transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in transaction’
/usr/share/foreman/lib/foreman/advisory_lock_manager.rb:17:in with_transaction_lock' /usr/share/foreman/app/services/foreman_seeder.rb:38:in execute’
/usr/share/foreman/db/seeds.rb:14:in <top (required)>' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in load’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in block in load' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in load_dependency’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in load' /usr/share/foreman/config/initializers/0_print_time_spent.rb:38:in block in load’
/usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in benchmark' /usr/share/foreman/config/initializers/0_print_time_spent.rb:38:in load’
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:559:in load_seed' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/tasks/database_tasks.rb:440:in load_seed’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/databases.rake:331:in `block (2 levels) in <top (required)>’

5

Enabled the backup to check for templates.
We don’t use all features of Foreman and as expected, besides of user accounts
and synced repositories, everything has the default settings.
No Ansible roles, no custom variables, no kickstart templates, no nothing.

6

@foreman Any insight to the db seed error listed above? Thanks.

7

Uh, what do you mean?
Do you need something else, than the stack trace posted earlier?

8

Sorry, I was asking @Developers to take a look at the db:seed error you posted above.

1 Like
9

The seed error basically means that some OS is assgined some template and they don’t share the same organization/location (or something similar). Can you access the UI now? If so, the best thing is to go to Administer → Organizations and click the Mismatches Report button. The same for Locations.

1 Like
10

Recovered the system from backup and took a look.
The Mismatch Report lists 3 systems and one of them is the Foreman server itself???

Tried to reassign them to the only organisation we currently have and it doesn’t work.
I get always this error:

Cannot update Organization to myOrg because of mismatch in settings

I will check the logs to see if any error occurs, but I really don’t know which setting
should be wrong.
These systems had been installed following the official documentation with the default settings.
In between we only added LDAP authentication for central managed users, nothing else.

11

Checked all the logs and didn’t find any error.
Any ideas?

12

Looking more closely to the code, there are some resources (e.g. domain, subnet) that is not assigned to the org/loc to which some host is. And such host is also assigned to such resource.

If you go to the org/loc edit form, you can go over all the resources there and assign all of them. That could at least unblock you. Or go over all your hosts and see which could be wrong.

13

Good morning,

clicked through our locations and organisation.
Foreman found some mistakes I fixed them and still couldn’t assign
an organisation to them.
After that I clicked through all the options and enabled in all sub menus “Assign all”
I still can’t assign an organisation.

14

Let the system run for 3 days and after that I could now suddenly assign the
systems to a location and organisation.
Weird stuff.

15

That is really weird. I wonder if we have some kind of a caching for this stuff, but I’m not aware of any.

16

And now I was able to update Foreman with proxies to 2.5 with Katello 4.1.

17

this command helped me to fix this 422 get request error:

foreman-maintain service stop --exclude=postgresql ; systemctl stop pulpcore-* --all ; systemctl stop smart_proxy_dynflow_core
echo "Organization.all.each {|org| org.import_missing_ids}" | foreman-rake console
foreman-rake db:seed -t -v
foreman-installer -v