Problem:
I’m trying to upgrade Katello 3.18.4 to Katello 3.18.5 in preparation for Katello 4.0 upgrade and to fix some bugs I have encountered.
The upgrade completes successfully but running foreman-installer seems to fail with smartproxy and leaves my Foreman/Katello instance in a bad/broken state.
Foreman and Proxy versions:
- foreman-2.3.5-1.el7.noarch
- foreman-cli-2.3.5-1.el7.noarch
- foreman-proxy-2.3.5-1.el7.noarch
- katello-3.18.4-1.el7.noarch
Distribution and version:
RHEL 7.9
[root@puppetmaster-prod-02 ~]# foreman-rake katello:upgrade_check
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
This script makes no modifications and can be re-run multiple times for the most up to date results.
Checking upgradeability...
Checking for running tasks...
[SUCCESS] - There are 0 active tasks.
You may proceed with the upgrade.
[root@puppetmaster-prod-02 ~]# yum update -y
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package katello.noarch 0:3.18.4-1.el7 will be updated
---> Package katello.noarch 0:3.18.5-1.el7 will be an update
---> Package katello-common.noarch 0:3.18.4-1.el7 will be updated
---> Package katello-common.noarch 0:3.18.5-1.el7 will be an update
---> Package katello-debug.noarch 0:3.18.4-1.el7 will be updated
---> Package katello-debug.noarch 0:3.18.5-1.el7 will be an update
---> Package katello-repos.noarch 0:3.18.4-1.el7 will be updated
---> Package katello-repos.noarch 0:3.18.5-1.el7 will be an update
---> Package python2-qpid-proton.x86_64 0:0.33.0-1.el7 will be updated
---> Package python2-qpid-proton.x86_64 0:0.34.0-2.el7 will be an update
---> Package python3-pulp-2to3-migration.noarch 0:0.11.1-1.el7 will be updated
---> Package python3-pulp-2to3-migration.noarch 0:0.11.5-1.el7 will be an update
---> Package python3-pulp-rpm.noarch 0:3.10.0-1.el7 will be updated
---> Package python3-pulp-rpm.noarch 0:3.11.2-1.el7 will be an update
---> Package python3-pulpcore.noarch 0:3.7.6-1.el7 will be updated
---> Package python3-pulpcore.noarch 0:3.7.9-1.el7 will be an update
---> Package qpid-proton-c.x86_64 0:0.33.0-1.el7 will be updated
---> Package qpid-proton-c.x86_64 0:0.34.0-2.el7 will be an update
---> Package tfm-rubygem-katello.noarch 0:3.18.4-1.el7 will be updated
---> Package tfm-rubygem-katello.noarch 0:3.18.5-1.el7 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
Package Arch Version Repository Size
=============================================================================================================================================================================================
Updating:
katello noarch 3.18.5-1.el7 katello 9.7 k
katello-common noarch 3.18.5-1.el7 katello 18 k
katello-debug noarch 3.18.5-1.el7 katello 11 k
katello-repos noarch 3.18.5-1.el7 katello 12 k
python2-qpid-proton x86_64 0.34.0-2.el7 qpid 332 k
python3-pulp-2to3-migration noarch 0.11.5-1.el7 pulpcore 225 k
python3-pulp-rpm noarch 3.11.2-1.el7 pulpcore 283 k
python3-pulpcore noarch 3.7.9-1.el7 pulpcore 488 k
qpid-proton-c x86_64 0.34.0-2.el7 qpid 214 k
tfm-rubygem-katello noarch 3.18.5-1.el7 katello 9.3 M
Transaction Summary
=============================================================================================================================================================================================
Upgrade 10 Packages
Total download size: 11 M
Downloading packages:
(1/10): katello-common-3.18.5-1.el7.noarch.rpm | 18 kB 00:00:00
(2/10): katello-3.18.5-1.el7.noarch.rpm | 9.7 kB 00:00:00
(3/10): katello-debug-3.18.5-1.el7.noarch.rpm | 11 kB 00:00:00
(4/10): katello-repos-3.18.5-1.el7.noarch.rpm | 12 kB 00:00:00
warning: /var/cache/yum/x86_64/7Server/qpid/packages/python2-qpid-proton-0.34.0-2.el7.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 42366066: NOKEY
Public key for python2-qpid-proton-0.34.0-2.el7.x86_64.rpm is not installed
(5/10): python2-qpid-proton-0.34.0-2.el7.x86_64.rpm | 332 kB 00:00:00
(6/10): python3-pulp-2to3-migration-0.11.5-1.el7.noarch.rpm | 225 kB 00:00:00
(7/10): qpid-proton-c-0.34.0-2.el7.x86_64.rpm | 214 kB 00:00:00
(8/10): python3-pulpcore-3.7.9-1.el7.noarch.rpm | 488 kB 00:00:00
(9/10): python3-pulp-rpm-3.11.2-1.el7.noarch.rpm | 283 kB 00:00:00
(10/10): tfm-rubygem-katello-3.18.5-1.el7.noarch.rpm | 9.3 MB 00:00:00
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 11 MB/s | 11 MB 00:00:01
Retrieving key from https://download.copr.fedorainfracloud.org/results/@qpid/qpid/pubkey.gpg
Importing GPG key 0x42366066:
Userid : "@qpid_qpid (None) <@qpid#qpid@copr.fedorahosted.org>"
Fingerprint: 3d32 e878 3014 b232 45dd 2bd2 0503 5613 4236 6066
From : https://download.copr.fedorainfracloud.org/results/@qpid/qpid/pubkey.gpg
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : python3-pulpcore-3.7.9-1.el7.noarch 1/20
Updating : qpid-proton-c-0.34.0-2.el7.x86_64 2/20
Updating : tfm-rubygem-katello-3.18.5-1.el7.noarch 3/20
Updating : katello-debug-3.18.5-1.el7.noarch 4/20
Updating : katello-common-3.18.5-1.el7.noarch 5/20
Updating : katello-3.18.5-1.el7.noarch 6/20
Updating : python2-qpid-proton-0.34.0-2.el7.x86_64 7/20
Updating : python3-pulp-2to3-migration-0.11.5-1.el7.noarch 8/20
Updating : python3-pulp-rpm-3.11.2-1.el7.noarch 9/20
Updating : katello-repos-3.18.5-1.el7.noarch 10/20
Cleanup : katello-3.18.4-1.el7.noarch 11/20
Cleanup : katello-common-3.18.4-1.el7.noarch 12/20
Cleanup : python3-pulp-rpm-3.10.0-1.el7.noarch 13/20
Cleanup : python3-pulp-2to3-migration-0.11.1-1.el7.noarch 14/20
Cleanup : python3-pulpcore-3.7.6-1.el7.noarch 15/20
Cleanup : katello-debug-3.18.4-1.el7.noarch 16/20
Cleanup : tfm-rubygem-katello-3.18.4-1.el7.noarch 17/20
Cleanup : katello-repos-3.18.4-1.el7.noarch 18/20
Cleanup : python2-qpid-proton-0.33.0-1.el7.x86_64 19/20
Cleanup : qpid-proton-c-0.33.0-1.el7.x86_64 20/20
Verifying : katello-debug-3.18.5-1.el7.noarch 1/20
Verifying : katello-repos-3.18.5-1.el7.noarch 2/20
Verifying : tfm-rubygem-katello-3.18.5-1.el7.noarch 3/20
Verifying : katello-3.18.5-1.el7.noarch 4/20
Verifying : python3-pulpcore-3.7.9-1.el7.noarch 5/20
Verifying : katello-common-3.18.5-1.el7.noarch 6/20
Verifying : python2-qpid-proton-0.34.0-2.el7.x86_64 7/20
Verifying : qpid-proton-c-0.34.0-2.el7.x86_64 8/20
Verifying : python3-pulp-2to3-migration-0.11.5-1.el7.noarch 9/20
Verifying : python3-pulp-rpm-3.11.2-1.el7.noarch 10/20
Verifying : katello-3.18.4-1.el7.noarch 11/20
Verifying : katello-common-3.18.4-1.el7.noarch 12/20
Verifying : python3-pulp-2to3-migration-0.11.1-1.el7.noarch 13/20
Verifying : python3-pulp-rpm-3.10.0-1.el7.noarch 14/20
Verifying : qpid-proton-c-0.33.0-1.el7.x86_64 15/20
Verifying : katello-repos-3.18.4-1.el7.noarch 16/20
Verifying : python3-pulpcore-3.7.6-1.el7.noarch 17/20
Verifying : python2-qpid-proton-0.33.0-1.el7.x86_64 18/20
Verifying : tfm-rubygem-katello-3.18.4-1.el7.noarch 19/20
Verifying : katello-debug-3.18.4-1.el7.noarch 20/20
Running foreman-installer
[root@puppetmaster-prod-02 ~]# foreman-installer
2021-12-07 13:44:17 [NOTICE] [pre_migrations] Executing hooks in group pre_migrations
2021-12-07 13:44:17 [NOTICE] [pre_migrations] All hooks in group pre_migrations finished
2021-12-07 13:44:17 [NOTICE] [boot] Executing hooks in group boot
2021-12-07 13:44:17 [NOTICE] [boot] All hooks in group boot finished
2021-12-07 13:44:17 [NOTICE] [init] Executing hooks in group init
2021-12-07 13:44:17 [NOTICE] [init] All hooks in group init finished
2021-12-07 13:44:17 [NOTICE] [root] Loading default values from puppet modules…
2021-12-07 13:44:28 [NOTICE] [root] … finished
2021-12-07 13:44:28 [NOTICE] [pre_values] Executing hooks in group pre_values
2021-12-07 13:44:28 [NOTICE] [pre_values] All hooks in group pre_values finished
2021-12-07 13:44:29 [NOTICE] [pre_validations] Executing hooks in group pre_validations
2021-12-07 13:44:30 [NOTICE] [pre_validations] All hooks in group pre_validations finished
2021-12-07 13:44:30 [NOTICE] [root] Running validation checks
2021-12-07 13:44:30 [NOTICE] [pre_commit] Executing hooks in group pre_commit
2021-12-07 13:44:34 [NOTICE] [pre_commit] All hooks in group pre_commit finished
2021-12-07 13:44:34 [NOTICE] [pre] Executing hooks in group pre
2021-12-07 13:44:34 [NOTICE] [pre] All hooks in group pre finished
2021-12-07 13:44:34 [NOTICE] [configure] Starting system configuration.
The total number of configuration tasks may increase during the run.
Observe logs or specify --verbose-log-level to see individual configuration tasks.
2021-12-07 13:44:57 [NOTICE] [configure] 100 out of 2221 done.
2021-12-07 13:44:57 [NOTICE] [configure] 200 out of 2221 done.
2021-12-07 13:44:57 [NOTICE] [configure] 300 out of 2221 done.
2021-12-07 13:45:05 [NOTICE] [configure] 400 out of 2221 done.
2021-12-07 13:45:05 [NOTICE] [configure] 500 out of 2221 done.
2021-12-07 13:45:15 [NOTICE] [configure] 600 out of 2221 done.
2021-12-07 13:45:19 [NOTICE] [configure] 700 out of 2223 done.
2021-12-07 13:45:19 [NOTICE] [configure] 800 out of 2223 done.
2021-12-07 13:45:20 [NOTICE] [configure] 900 out of 2225 done.
2021-12-07 13:45:20 [NOTICE] [configure] 1000 out of 2227 done.
2021-12-07 13:45:20 [NOTICE] [configure] 1100 out of 2241 done.
2021-12-07 13:45:24 [NOTICE] [configure] 1200 out of 2243 done.
2021-12-07 13:45:24 [NOTICE] [configure] 1300 out of 2247 done.
2021-12-07 13:45:32 [NOTICE] [configure] 1400 out of 2247 done.
2021-12-07 13:45:36 [NOTICE] [configure] 1500 out of 2249 done.
2021-12-07 13:45:36 [NOTICE] [configure] 1600 out of 2249 done.
2021-12-07 13:46:09 [NOTICE] [configure] 1700 out of 2249 done.
2021-12-07 13:46:09 [NOTICE] [configure] 1800 out of 2249 done.
2021-12-07 13:46:52 [NOTICE] [configure] 1900 out of 2249 done.
2021-12-07 13:47:46 [NOTICE] [configure] 2000 out of 2249 done.
2021-12-07 13:47:48 [NOTICE] [configure] 2100 out of 2249 done.
2021-12-07 13:48:11 [NOTICE] [configure] 2200 out of 2249 done.
2021-12-07 13:48:13 [ERROR ] [configure] Proxy puppetmaster-prod-01.uit.tufts.edu cannot be updated: unknown error (response 422)
2021-12-07 13:48:13 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[puppetmaster-prod-01.uit.tufts.edu]/url: change from ‘https://puppetmaster-prod-01.uit.tufts.edu:9090’ to ‘https://puppetmaster-prod-02.uit.tufts.edu:9090’ failed: Proxy puppetmaster-prod-01.uit.tufts.edu cannot be updated: unknown error (response 422)
2021-12-07 13:48:13 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[puppetmaster-prod-01.uit.tufts.edu]: Failed to call refresh: Proxy puppetmaster-prod-01.uit.tufts.edu cannot be refreshed: unknown error (response 500)
2021-12-07 13:48:13 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[puppetmaster-prod-01.uit.tufts.edu]: Proxy puppetmaster-prod-01.uit.tufts.edu cannot be refreshed: unknown error (response 500)
2021-12-07 13:48:17 [NOTICE] [configure] System configuration has finished.
2021-12-07 13:48:17 [NOTICE] [post] Executing hooks in group post
There were errors detected during install.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.
The full log is at /var/log/foreman-installer/katello.log
2021-12-07 13:48:17 [NOTICE] [post] All hooks in group post finished
Here is the log from katello.log
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Service]: Starting to evaluate the resource (2222 of 2249)
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Service]: The container Stage[main] will propagate my refresh event
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Service]: The container Class[Foreman_proxy] will propagate my refresh event
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Service]: Scheduling refresh of Class[Foreman_proxy::Register]
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Service]: Evaluated in 0.00 seconds
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Register]: Starting to evaluate the resource (2223 of 2249)
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Register]: Scheduling refresh of Foreman_smartproxy[puppetmaster-prod-01.uit.tufts.edu]
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Register]: Evaluated in 0.00 seconds
2021-12-07 13:48:13 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Datacat_collector[foreman_proxy::enabled_features]: Starting to evaluate the resource (2224 of 2249)
2021-12-07 13:48:13 [DEBUG ] [configure] Datacat_collectorforeman_proxy::enabled_features: Collected {“features”=>[“Puppet”, “Puppet CA”, “Logs”, “Registration”, “Pulp”, “Pulpcore”, “Dynflow”, “SSH”]}
2021-12-07 13:48:13 [DEBUG ] [configure] Datacat_collectorforeman_proxy::enabled_features: Selecting source_key features
2021-12-07 13:48:13 [DEBUG ] [configure] Datacat_collectorforeman_proxy::enabled_features: Now setting field :features
2021-12-07 13:48:13 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Datacat_collector[foreman_proxy::enabled_features]: Evaluated in 0.01 seconds
2021-12-07 13:48:13 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[puppetmaster-prod-01.uit.tufts.edu]: Starting to evaluate the resource (2225 of 2249)
2021-12-07 13:48:13 [DEBUG ] [configure] Foreman_smartproxypuppetmaster-prod-01.uit.tufts.edu: Making get request to https://puppetmaster-prod-01.uit.tufts.edu/api/v2/smart_proxies?search=name=“puppetmaster-prod-01.uit.tufts.edu”
2021-12-07 13:48:13 [DEBUG ] [configure] Foreman_smartproxypuppetmaster-prod-01.uit.tufts.edu: Received response 200 from request to https://puppetmaster-prod-01.uit.tufts.edu/api/v2/smart_proxies?search=name=“puppetmaster-prod-01.uit.tufts.edu”
2021-12-07 13:48:13 [DEBUG ] [configure] Foreman_smartproxypuppetmaster-prod-01.uit.tufts.edu: Making put request to https://puppetmaster-prod-01.uit.tufts.edu/api/v2/smart_proxies/1
2021-12-07 13:48:13 [DEBUG ] [configure] Foreman_smartproxypuppetmaster-prod-01.uit.tufts.edu: Received response 422 from request to https://puppetmaster-prod-01.uit.tufts.edu/api/v2/smart_proxies/1
2021-12-07 13:48:13 [ERROR ] [configure] Proxy puppetmaster-prod-01.uit.tufts.edu cannot be updated: unknown error (response 422)
2021-12-07 13:48:13 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[puppetmaster-prod-01.uit.tufts.edu]/url: change from ‘https://puppetmaster-prod-01.uit.tufts.edu:9090’ to ‘https://puppetmaster-prod-02.uit.tufts.edu:9090’ failed: Proxy puppetmaster-prod-01.uit.tufts.edu cannot be updated: unknown error (response 422)
2021-12-07 13:48:13 [DEBUG ] [configure] Foreman_smartproxypuppetmaster-prod-01.uit.tufts.edu: Making put request to https://puppetmaster-prod-01.uit.tufts.edu/api/v2/smart_proxies/1/refresh
2021-12-07 13:48:13 [DEBUG ] [configure] Foreman_smartproxypuppetmaster-prod-01.uit.tufts.edu: Received response 500 from request to https://puppetmaster-prod-01.uit.tufts.edu/api/v2/smart_proxies/1/refresh
2021-12-07 13:48:13 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[puppetmaster-prod-01.uit.tufts.edu]: Failed to call refresh: Proxy puppetmaster-prod-01.uit.tufts.edu cannot be refreshed: unknown error (response 500)
2021-12-07 13:48:13 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[puppetmaster-prod-01.uit.tufts.edu]: Proxy puppetmaster-prod-01.uit.tufts.edu cannot be refreshed: unknown error (response 500)
2021-12-07 13:48:13 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[puppetmaster-prod-01.uit.tufts.edu]: Evaluated in 0.27 seconds
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Register]: Starting to evaluate the resource (2226 of 2249)
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Register]: Resource is being skipped, unscheduling all events
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Register]: Evaluated in 0.00 seconds
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy]: Starting to evaluate the resource (2227 of 2249)
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy]: Resource is being skipped, unscheduling all events
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy]: Unscheduling all events on Class[Foreman_proxy]
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy]: Evaluated in 0.00 seconds
2021-12-07 13:48:13 [DEBUG ] [configure] Foreman_proxy::Module[pulpcore]: Starting to evaluate the resource (2228 of 2249)
2021-12-07 13:48:13 [DEBUG ] [configure] Foreman_proxy::Module[pulpcore]: Evaluated in 0.00 seconds
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Plugin::Pulp]: Starting to evaluate the resource (2229 of 2249)
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Plugin::Pulp]: The container Stage[main] will propagate my refresh event
2021-12-07 13:48:13 [DEBUG ] [configure] Class[Foreman_proxy::Plugin::Pulp]: Evaluated in 0.00 seconds
2021-12-07 13:48:13 [DEBUG ] [configure] /Service[puppet]: Starting to evaluate the resource (2230 of 2249)
2021-12-07 13:48:13 [INFO ] [configure] /Service[puppet]: Dependency Foreman_smartproxy[puppetmaster-prod-01.uit.tufts.edu] has failures: true
2021-12-07 13:48:13 [DEBUG ] [configure] /Service[puppet]: Skipping because of failed dependencies
2021-12-07 13:48:13 [DEBUG ] [configure] /Service[puppet]: Resource is being skipped, unscheduling all events
2021-12-07 13:48:13 [DEBUG ] [configure] /Service[puppet]: Unscheduling all events on Service[puppet]
2021-12-07 13:48:13 [DEBUG ] [configure] /Service[puppet]: Evaluated in 0.00 seconds
Any ideas on how to get foreman-installer to proceed? Do I have to upgrade smartproxy separately before this? (Using this guide or does the Katello upgrade handle this? Foreman :: Plugin Manuals)