Error after upgrade to Foreman3.12/Katello 4.14

Problem:
After upgrading, anytime you run foreman installer it errors out with the following error message.

2024-10-11 12:04:56 [ERROR ] [configure] Error making POST request to Foreman at https://foreman.example.com/api/v2/hosts/facts: Response: 422 Unprocessable Content
2024-10-11 12:04:56 [ERROR ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-foreman.example.com]/ensure: change from ‘absent’ to ‘present’ failed: Error making POST request to Foreman at https://foreman.example.com/api/v2/hosts/facts: Response: 422 Unprocessable Content
Expected outcome:

Foreman and Proxy versions:
Foreman and proxy are on 3.12 / Katello 4.14

Foreman and Proxy plugin versions:
foreman-tasks 9.2.3
foreman_ansible 14.2.1
foreman_openscap 9.0.4
foreman_remote_execution 13.2.5
foreman_rh_cloud 10.0.1
foreman_templates 9.5.1
foreman_vault 2.0.0
foreman_webhooks 3.2.3
katello 4.14.0

Distribution and version:
RHEL 8 Foreman
RHEL 9 Smart Proxy

Other relevant data:

Do you happen to have the detailed logs from /var/log/foreman-installer/katello.log and /var/log/foreman/production.log that lead to this 400 error?

Logs files, sanitized.
katello.log (2.4 MB)
production.log (2.5 MB)

Okay. So the error is this:

2024-10-11T12:04:56 [W|app|31bcb6e9] ERF42-9911 [Foreman::Exception]: Host is pending for Build
2024-10-11T12:04:56 [I|app|31bcb6e9] Backtrace for 'ERF42-9911 [Foreman::Exception]: Host is pending for Build' error (Foreman::Exception): ERF42-9911 [Foreman::Exception]: Host is pending for Build

Could you go to the webinterface, find the foreman.example.com host and cancel it being built (whyever and however it got into that state)?

I would assume this should unblock the installer going forward.

And then we gotta find out how the heck the host ended up in build mode.

(The fact that this didn’t show up earlier is a side-effect of Bug #37726: After an OS version upgrade, the GUI still shows the old/wrong Operating System version - Installer - Foreman – before that change we never updated the Foreman host in the DB, now we do)

Still the same error. This time with -l DEBUG added:

2024-10-11 18:13:18 [DEBUG ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-foreman.example.com]: Starting to evaluate the resource (1438 of 1543)
2024-10-11 18:13:18 [DEBUG ] [configure] Foreman_host[foreman-foreman.example.com](provider=rest_v3): Making post request to https://foreman.example.com/api/v2/hosts/facts
2024-10-11 18:13:18 [DEBUG ] [configure] Foreman_host[foreman-foreman.example.com](provider=rest_v3): Received response 422 from request to https://foreman.example.com/api/v2/hosts/facts
2024-10-11 18:13:18 [ERROR ] [configure] Error making POST request to Foreman at https://foreman.example.com/api/v2/hosts/facts: Response: 422 Unprocessable Content
2024-10-11 18:13:18 [ERROR ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-foreman.example.com]/ensure: change from 'absent' to 'present' failed: Error making POST request to Foreman at https://foreman.example.com/api/v2/hosts/facts: Response: 422 Unprocessable Content
2024-10-11 18:13:18 [DEBUG ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-foreman.example.com]: Evaluated in 0.07 seconds

Okay. That’s super weird.
What would be setting the host into build mode?!

Can you get us another round of production.log please, whether it’s visible what moved it to “build” now?

After that, you can edit /usr/share/foreman-installer/config/foreman.hiera/common.yaml and change foreman::register::force_fact_upload: true to foreman::register::force_fact_upload: false, this should allow you to run the installer (I hope), but will be erased on the next update again.

We gotta find out what happens here.

Sorry, I was off over the weekend. I changed the settings so the installer could complete, then changed it back. As I expected I get the same error. I don’t want to leave it in a broken state, but I don’t know where else to look.

I have some hard deadlines I have to meet over the next couple of weeks and don’t know when I’ll be able to dig deeper. Anyway, temporarily I’ve added the setting to /etc/foreman-installer/custom-hiera.yaml. That should get us moving and let us launch our other project next week.