Problem:
I finally tried to upgrade from 4.4.0.2 to 4.4.1, and the upgrade didn’t went as silent as all the previous upgrades.
It basically started with this error:
2022-06-23 22:45:38 [ERROR ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-foreman.example.com]: Could not evaluate: Error making GET request to Foreman at https://foreman.fritz.box/api/v2/hosts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on foreman.fritz.box for detailed information
And there are also some more, which look pretty much the same afterwards. (long log at the end)
I then grabbed the URI from the log and tried to access it myself after the installer crashed, this gave me the following output:
{
"error": {"message":"Cannot find rabl template 'katello/api/v2/content_facet/base_with_root' within registered ([\"/usr/share/foreman/app/views\", \"/usr/share/gems/gems/katello-4.4.0.2/app/views\", \"/usr/share/gems/gems/foreman_templates-9.1.0/app/views\"]) view paths!"}
}
More interesting, after that I rolled back the system and gave the Katello 4.5 RC2 a shot!
Which finished installing without any issues!
Looks like this has been changed there somehow.
A gotcha for 4.5 is, which is not in the docs up to now, that for EL8 the 3 Module Streams have to be enabled after the release package update
dnf module enable foreman
dnf module enable katello
dnf module enable pulpcore
If not anyone is working on making the upgrade documentation ready for EL8 I can give that a try in the next days! (this would be my first for the docs, so I don’t really know how this works here, am I allowed to just send PRs there?)
And furthermore I noticed that if I tried to do a bulk-remove of CV releases, it fails with this error, beside that didn’t see any other issues so far (but as this is kind of prod I rolled back for now):
2022-06-23T23:53:03 [E|app|916c7fd5] RuntimeError: There was an issue with the backend service pulp3: Pulp redis connection issue at https://foreman.example.com/pulp/api/v3.
Expected outcome:
As 4.4 is still supported, the upgrade for 4.4.1 should work, but as 4.5 looks to not have this problem for me it’s a minor problem.
For 4.5 bulk CV removes should work
Foreman and Proxy versions:
foreman 3.2.1-1.el8 → no upgrade / foreman-3.3.0-1.el8
katello 4.4.0-1.el8 → 4.4.1-1.el8 / katello-4.5.0-0.1.rc2.el8
Foreman and Proxy plugins:
VMware provider
foreman-tasks
foreman_ansible
foreman_bootdisk
foreman_puppet
foreman_remote_execution
foreman_snapshot_management
foreman_statistics
foreman_templates
Puppet 6 is still used, not switched to 7 for now
Distribution and version:
Rocky Linux 8.6
Other relevant data:
Visible output:
2022-06-23 22:45:00 [NOTICE] [configure] Starting system configuration.
2022-06-23 22:45:09 [NOTICE] [configure] 250 configuration steps out of 2009 steps complete.
2022-06-23 22:45:11 [NOTICE] [configure] 500 configuration steps out of 2009 steps complete.
2022-06-23 22:45:11 [NOTICE] [configure] 750 configuration steps out of 2013 steps complete.
2022-06-23 22:45:13 [NOTICE] [configure] 1000 configuration steps out of 2017 steps complete.
2022-06-23 22:45:14 [NOTICE] [configure] 1250 configuration steps out of 2038 steps complete.
2022-06-23 22:45:37 [NOTICE] [configure] 1500 configuration steps out of 2038 steps complete.
2022-06-23 22:45:38 [NOTICE] [configure] 1750 configuration steps out of 2038 steps complete.
2022-06-23 22:45:38 [ERROR ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-foreman.example.com]: Could not evaluate: Error making GET request to Foreman at https://foreman.fritz.box/api/v2/hosts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on foreman.fritz.box for detailed information
2022-06-23 22:45:43 [NOTICE] [configure] 2000 configuration steps out of 2038 steps complete.
2022-06-23 22:45:43 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-foreman.example.com]: Could not evaluate: Error making GET request to Foreman at https://foreman.example.com/api/v2/hosts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on foreman.example.com for detailed information
2022-06-23 22:45:43 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.example.com]: Could not evaluate: Error making GET request to Foreman at https://foreman.example.com/api/v2/smart_proxies: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on foreman.example.com for detailed information
2022-06-23 22:45:46 [NOTICE] [configure] System configuration has finished.
And the log from the production.log:
2022-06-23 22:45:38 [DEBUG ] [configure] Class[Foreman::Service]: Starting to evaluate the resource (1911 of 2038)
2022-06-23 22:45:38 [DEBUG ] [configure] Class[Foreman::Service]: Evaluated in 0.00 seconds
2022-06-23 22:45:38 [DEBUG ] [configure] /Stage[main]/Foreman/Anchor[foreman::service]: Starting to evaluate the resource (1912 of 2038)
2022-06-23 22:45:38 [DEBUG ] [configure] /Stage[main]/Foreman/Anchor[foreman::service]: Evaluated in 0.00 seconds
2022-06-23 22:45:38 [DEBUG ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-foreman.example.com]: Starting to evaluate the resource (1913 of 2038)
2022-06-23 22:45:38 [DEBUG ] [configure] Foreman_host[foreman-foreman.example.com](provider=rest_v3): Making get request to https://foreman.fritz.box/api/v2/hosts?search=name%3D%22foreman.example.com%22
2022-06-23 22:45:38 [DEBUG ] [configure] Foreman_host[foreman-foreman.example.com](provider=rest_v3): Received response 500 from request to https://foreman.fritz.box/api/v2/hosts?search=name%3D%22foreman.example.com%22
2022-06-23 22:45:38 [ERROR ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-foreman.example.com]: Could not evaluate: Error making GET request to Foreman at https://foreman.fritz.box/api/v2/hosts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on foreman.fritz.box for detailed information
2022-06-23 22:45:38 [DEBUG ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-foreman.example.com]: Evaluated in 0.09 seconds
2022-06-23 22:45:38 [DEBUG ] [configure] /Stage[main]/Foreman::Register/Foreman_instance_host[foreman-foreman.example.com]: Starting to evaluate the resource (1914 of 2038)
2022-06-23 22:45:38 [INFO ] [configure] /Stage[main]/Foreman::Register/Foreman_instance_host[foreman-foreman.example.com]: Dependency Foreman_host[foreman-foreman.example.com] has failures: true
2022-06-23 22:45:38 [DEBUG ] [configure] /Stage[main]/Foreman::Register/Foreman_instance_host[foreman-foreman.example.com]: Skipping because of failed dependencies
2022-06-23 22:45:38 [DEBUG ] [configure] /Stage[main]/Foreman::Register/Foreman_instance_host[foreman-foreman.example.com]: Resource is being skipped, unscheduling all events
2022-06-23 22:45:38 [DEBUG ] [configure] /Stage[main]/Foreman::Register/Foreman_instance_host[foreman-foreman.example.com]: Evaluated in 0.00 seconds
2022-06-23 22:45:38 [DEBUG ] [configure] Class[Foreman::Register]: Starting to evaluate the resource (1915 of 2038)
2022-06-23 22:45:38 [DEBUG ] [configure] Class[Foreman::Register]: Resource is being skipped, unscheduling all events
2022-06-23 22:45:38 [DEBUG ] [configure] Class[Foreman::Register]: Evaluated in 0.00 seconds
...
2022-06-23 22:45:43 [DEBUG ] [configure] Class[Foreman_proxy::Service]: Starting to evaluate the resource (2005 of 2038)
2022-06-23 22:45:43 [DEBUG ] [configure] Class[Foreman_proxy::Service]: Evaluated in 0.00 seconds
2022-06-23 22:45:43 [DEBUG ] [configure] Class[Foreman_proxy::Register]: Starting to evaluate the resource (2006 of 2038)
2022-06-23 22:45:43 [DEBUG ] [configure] Class[Foreman_proxy::Register]: Evaluated in 0.00 seconds
2022-06-23 22:45:43 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-foreman.example.com]: Starting to evaluate the resource (2007 of 2038)
2022-06-23 22:45:43 [DEBUG ] [configure] Foreman_host[foreman-proxy-foreman.example.com](provider=rest_v3): Making get request to https://foreman.example.com/api/v2/hosts?search=name%3D%22foreman.example.com%22
2022-06-23 22:45:43 [DEBUG ] [configure] Foreman_host[foreman-proxy-foreman.example.com](provider=rest_v3): Received response 500 from request to https://foreman.example.com/api/v2/hosts?search=name%3D%22foreman.example.com%22
2022-06-23 22:45:43 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-foreman.example.com]: Could not evaluate: Error making GET request to Foreman at https://foreman.example.com/api/v2/hosts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on foreman.example.com for detailed information
2022-06-23 22:45:43 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-foreman.example.com]: Evaluated in 0.11 seconds
2022-06-23 22:45:43 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Datacat_collector[foreman_proxy::enabled_features]: Starting to evaluate the resource (2008 of 2038)
2022-06-23 22:45:43 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Collected {"features"=>["Puppet", "Puppet CA", "Logs", "Pulpcore", "Dynflow", "Ansible", "SSH"]}
2022-06-23 22:45:43 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Selecting source_key features
2022-06-23 22:45:43 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Now setting field :features
2022-06-23 22:45:43 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Datacat_collector[foreman_proxy::enabled_features]: Evaluated in 0.00 seconds
2022-06-23 22:45:43 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.example.com]: Starting to evaluate the resource (2009 of 2038)
2022-06-23 22:45:43 [DEBUG ] [configure] Foreman_smartproxy[foreman.example.com](provider=rest_v3): Making get request to https://foreman.example.com/api/v2/smart_proxies?search=name%3D%22foreman.example.com%22
2022-06-23 22:45:43 [DEBUG ] [configure] Foreman_smartproxy[foreman.example.com](provider=rest_v3): Received response 500 from request to https://foreman.example.com/api/v2/smart_proxies?search=name%3D%22foreman.example.com%22
2022-06-23 22:45:43 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.example.com]: Could not evaluate: Error making GET request to Foreman at https://foreman.example.com/api/v2/smart_proxies: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on foreman.example.com for detailed information
2022-06-23 22:45:43 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.example.com]: Evaluated in 0.05 seconds
2022-06-23 22:45:43 [DEBUG ] [configure] Foreman::Rake[apipie:cache:index]: Starting to evaluate the resource (2010 of 2038)
2022-06-23 22:45:43 [DEBUG ] [configure] Foreman::Rake[apipie:cache:index]: Resource is being skipped, unscheduling all events
2022-06-23 22:45:43 [DEBUG ] [configure] Foreman::Rake[apipie:cache:index]: Evaluated in 0.00 seconds