Yet another Foreman upgrade fail

chatlow · July 24, 2024, 10:43am

Problem:
Upgrading to 3.11 fails during postrges update/migration

Expected outcome:
works!

Foreman and Proxy versions:
3.10 > 3.11
Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

Attempting to upgrade from 3.10 > 3.11 with katello 4.12 > 4.13
Followed the docs outlined here exactly as they are and getting failures. Upgrading Foreman to 3.11

First issue (got around it with --nobest) was running an update AFTER running the rpm update for 3.11

Error:

Problem 1: cannot install both foreman-3.11.1-1.el8.noarch from foreman and foreman-3.10.0-1.el8.noarch from @System

package rubygem-katello-4.12.0-1.el8.noarch from @System requires foreman < 3.11, but none of the providers can be installed
cannot install the best update candidate for package foreman-3.10.0-1.el8.noarch
problem with installed package rubygem-katello-4.12.0-1.el8.noarch
Problem 2: cannot install both foreman-3.11.1-1.el8.noarch from foreman and foreman-3.10.0-1.el8.noarch from @System
package rubygem-katello-4.12.0-1.el8.noarch from @System requires foreman < 3.11, but none of the providers can be installed
package foreman-dynflow-sidekiq-3.11.1-1.el8.noarch from foreman requires foreman = 3.11.1-1.el8, but none of the providers can be installed
package katello-4.12.0-1.el8.noarch from @System requires rubygem-katello, but none of the providers can be installed
cannot install the best update candidate for package foreman-dynflow-sidekiq-3.10.0-1.el8.noarch
problem with installed package katello-4.12.0-1.el8.noarch

Anyway, I got around this with skipping broken and nobest for now - which I hate doing but just interested to see how the rest of it got on

Then during the foreman-installer, it cr@ps out during the postrgres update/migration:

2024-07-24 10:31:41 [NOTICE] [pre] Performing upgrade of PostgreSQL to 13
2024-07-24 10:31:41 [NOTICE] [pre] Upgrading PostgreSQL packages
2024-07-24 10:32:06 [NOTICE] [pre] Migrating PostgreSQL data
2024-07-24 10:32:10 [ERROR ] [root] runuser -l postgres -c ‘postgresql-setup --upgrade’ failed! Check the output for error!

ran is separaentl to find the problem and upgrade_postgresql.log shows:

Performing Consistency Checks

Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for system-defined composite types in user tables ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Creating dump of global objects ok
Creating dump of database schemas
candlepin
foreman
postgres
pulpcore
template1
ok

lc_collate values for database “postgres” do not match: old “C”, new “en_GB.UTF-8”
Failure, exiting

PLEASE can someone advise on the problems here. Really hoped for a smooth upgrade process here

aruzicka · July 24, 2024, 10:58am

Afaik Upgrading Foreman to 3.11 talks only about updating Foreman installations without katello, which would explain the issues you’re seeing. Hopefully if you stick to the foreman+katello upgrade guide, things should go more smoothly

chatlow · July 24, 2024, 1:29pm

I’m using the same upgrade how-to as I did for 3.10 and that worked mostly well. The top right of the page shows Foreman 3.11 with katello 4.13.

Do you have a different how-to URL I can try?

thanks

aruzicka · July 24, 2024, 1:41pm

Yeah, the split between pure foreman and foreman+katello guides isn’t exactly fortunate.

Oops, forgot to paste the link in the previous comment. Here you go https://docs.theforeman.org/3.11/Upgrading_Project/index-katello.html . Notice how this one says index-katello, where yours showed index-foreman-el

chatlow · July 24, 2024, 1:46pm

aha, thank you. Yeah it’s far from obvious
But this how-to contains a dnf update for katello as well as Foreman, which is promising!

Will try tomorrow

thanks

jeremylenz · July 24, 2024, 4:11pm

@maximilian Should we add a note to the Foreman-only guide to highlight that this shouldn’t be used if you have Katello?

chatlow · July 25, 2024, 9:29am

Hi All

Just thought I would mention that I followed correct guide: Upgrading Foreman to 3.11 and Im getting the exact same result

I had a conflict when running dnf module disable pulpcore. Conflicts reported with postres12/13

Problem 1: module foreman:el8:31120240718202416:34d643a2.x86_64 from foreman requires module(postgresql:13), but none of the providers can be installed

module postgresql:12:8010120191120141335:9edba152.x86_64 from ol8_appstream conflicts with module(postgresql:13) provided by postgresql:13:8040020210217220626:9f9e2e7e.x86_64 from ol8_appstream
module katello:el8:41320240724162450:37be364f.x86_64 from katello requires module(foreman:el8), but none of the providers can be installed
conflicting requests

I tried the next command to switch to 13 and try again, and it then says: missing groups or modules: pulpcore

I run the update anyway which does work, no problems, then foreman-installer but this flags the same postgres problem reported above during the migration stage I think

This is the exact same result received as the other url i followed without katello.

ekohl · July 25, 2024, 1:51pm

This may be better tracked elsewhere, but I think we should look at the usability of docs.theforeman.org. It shouldn’t be hard for someone with UI/UX knowledge to improve it. I know I wrote parts of it and know it can be better.

gvde · July 26, 2024, 6:17am

I can confirm this issue on AlmaLinux 8 upgrading from

foreman-3.10.0-1.el8.noarch
katello-4.12.1-1.el8.noarch

to 3.11/4.13.1

During the update of the repositories I already get an error:

  Cleanup          : katello-repos-4.12.1-1.el8.noarch                      3/4 
  Cleanup          : foreman-release-3.10.0-1.el8.noarch                    4/4 
  Verifying        : foreman-release-3.11.1-1.el8.noarch                    1/4 
  Verifying        : foreman-release-3.10.0-1.el8.noarch                    2/4 
  Verifying        : katello-repos-4.13.1-1.el8.noarch                      3/4 
  Verifying        : katello-repos-4.12.1-1.el8.noarch                      4/4 
Installed products updated.
Modular dependency problem:

 Problem: module postgresql:12:8090020240227082527:9edba152.x86_64 from ORG_almalinux8_AppStream_x86_64_os conflicts with module(postgresql:13) provided by postgresql:13:8090020240415140528:9edba152.x86_64 from ORG_almalinux8_AppStream_x86_64_os
  - module postgresql:13:8090020240415140528:9edba152.x86_64 from ORG_almalinux8_AppStream_x86_64_os conflicts with module(postgresql:12) provided by postgresql:12:8090020240227082527:9edba152.x86_64 from ORG_almalinux8_AppStream_x86_64_os
  - module foreman:el8:31120240718202416:34d643a2.x86_64 from foreman requires module(postgresql:13), but none of the providers can be installed
  - conflicting requests
Uploading Tracer Profile

Upgraded:
  foreman-release-3.11.1-1.el8.noarch     katello-repos-4.13.1-1.el8.noarch    

Complete!

This module conflict shows up again for the switch-to:

[root@foreman8 ~]# dnf module switch-to postgresql:13
Updating Subscription Management repositories.
subscription-manager plugin disabled 9 system repositories with respect of configuration in /etc/dnf/plugins/subscription-manager.conf
Candlepin 4.3: an open source entitlement management system. el8                                                                                                                71 kB/s | 1.5 kB     00:00
Foreman plugins 3.11 el8                                                                                                                                                        80 kB/s | 1.5 kB     00:00
AlmaLinux-8 - PowerTools x86_64 os                                                                                                                                             121 kB/s | 2.6 kB     00:00
pulpcore 3.39: Fetch, Upload, Organize, and Distribute Software Packages. el8                                                                                                   69 kB/s | 1.5 kB     00:00
ORG el8                                                                                                                                                                        68 kB/s | 1.5 kB     00:00
Extra Packages for Enterprise Linux 8 - x86_64                                                                                                                                 109 kB/s | 2.3 kB     00:00
AlmaLinux-8 - AppStream x86_64 os                                                                                                                                              128 kB/s | 2.6 kB     00:00
Katello 4.13 el8                                                                                                                                                                87 kB/s | 1.8 kB     00:00
Foreman 3.11 el8                                                                                                                                                                83 kB/s | 1.8 kB     00:00
Katello 4.12 el8                                                                                                                                                                85 kB/s | 1.8 kB     00:00
AlmaLinux-8 - extras x86_64 os                                                                                                                                                  91 kB/s | 1.8 kB     00:00
pulpcore 3.49: Fetch, Upload, Organize, and Distribute Software Packages. el8                                                                                                   76 kB/s | 1.5 kB     00:00
AlmaLinux-8 - BaseOS x86_64 os                                                                                                                                                 107 kB/s | 2.3 kB     00:00
Foreman plugins 3.10 el8                                                                                                                                                        76 kB/s | 1.5 kB     00:00
Foreman client 3.10 el8                                                                                                                                                         67 kB/s | 1.5 kB     00:00
Foreman 3.10 el8                                                                                                                                                               102 kB/s | 1.8 kB     00:00
Puppet 7 Repository el 8 - x86_64                                                                                                                                               60 kB/s | 1.3 kB     00:00
Foreman client 3.11 el8                                                                                                                                                         73 kB/s | 1.5 kB     00:00
Candlepin 4.4: an open source entitlement management system. el8                                                                                                                77 kB/s | 1.5 kB     00:00
Modular dependency problem:

 Problem: module foreman:el8:31120240718202416:34d643a2.x86_64 from ORG_foreman_3_11_el8_x86_64 requires module(postgresql:13), but none of the providers can be installed
  - module postgresql:12:8090020240227082527:9edba152.x86_64 from ORG_almalinux8_AppStream_x86_64_os conflicts with module(postgresql:13) provided by postgresql:13:8090020240415140528:9edba152.x86_64 from ORG_almalinux8_AppStream_x86_64_os
  - module postgresql:13:8090020240415140528:9edba152.x86_64 from ORG_almalinux8_AppStream_x86_64_os conflicts with module(postgresql:12) provided by postgresql:12:8090020240227082527:9edba152.x86_64 from ORG_almalinux8_AppStream_x86_64_os
  - conflicting requests
No active matches for argument 'postgresql:13' in module 'postgresql:13'
Error: Problems in request:
broken groups or modules: postgresql:13
Modular dependency problems:

 Problem: module foreman:el8:31020240618134920:b0fbc430.x86_64 from ORG_foreman_3_10_el8_x86_64 requires module(postgresql:12), but none of the providers can be installed
  - module postgresql:12:8090020240227082527:9edba152.x86_64 from ORG_almalinux8_AppStream_x86_64_os conflicts with module(postgresql:13) provided by postgresql:13:8090020240415140528:9edba152.x86_64 from ORG_almalinux8_AppStream_x86_64_os
  - module postgresql:13:8090020240415140528:9edba152.x86_64 from ORG_almalinux8_AppStream_x86_64_os conflicts with module(postgresql:12) provided by postgresql:12:8090020240227082527:9edba152.x86_64 from ORG_almalinux8_AppStream_x86_64_os
  - conflicting requests
[root@foreman8 ~]#

and I cannot do the switch.

O.K. So I’ve got into paranoia mode and went back to following the instructions by the letter. Usually, I download packages from the foreman/katello server itself. For katello upgrades, I download the packages cache-only and the enable the local repo files and disable the redhat.repo just before the foreman-installer run.

Anyway, unlike my usual procedure, I disabled redhat.repo and enabled all local repo files and ran switch-to again. It still shows the conflict but actually allows me to enable and install postgresql-13…

[root@foreman8 ~]# dnf module switch-to postgresql:13
Last metadata expiration check: 0:00:41 ago on Fri 26 Jul 2024 07:35:11 AM CEST.
Modular dependency problem:

 Problem: module foreman:el8:31120240718202416:34d643a2.x86_64 from foreman requires module(postgresql:13), but none of the providers can be installed
  - module postgresql:12:8090020240227082527:9edba152.x86_64 from appstream conflicts with module(postgresql:13) provided by postgresql:13:8090020240415140528:9edba152.x86_64 from appstream
  - module postgresql:13:8090020240415140528:9edba152.x86_64 from appstream conflicts with module(postgresql:12) provided by postgresql:12:8090020240227082527:9edba152.x86_64 from appstream
  - conflicting requests
Dependencies resolved.
===============================================================================================================================================================================================================
 Package                                          Architecture                         Version                                                                   Repository                               Size
===============================================================================================================================================================================================================
Upgrading:
 postgresql                                       x86_64                               13.14-1.module_el8.9.0+3815+0bfda440.alma.1                               appstream                               1.5 M
 postgresql-contrib                               x86_64                               13.14-1.module_el8.9.0+3815+0bfda440.alma.1                               appstream                               882 k
 postgresql-server                                x86_64                               13.14-1.module_el8.9.0+3815+0bfda440.alma.1                               appstream                               5.6 M
Switching module streams:
 postgresql                                                                            12 -> 13                                                                                                               

Transaction Summary
===============================================================================================================================================================================================================
Upgrade  3 Packages

Total download size: 8.0 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): postgresql-contrib-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64.rpm                                                                                               3.2 MB/s | 882 kB     00:00    
(2/3): postgresql-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64.rpm                                                                                                       4.9 MB/s | 1.5 MB     00:00    
(3/3): postgresql-server-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64.rpm                                                                                                 12 MB/s | 5.6 MB     00:00    
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                          9.5 MB/s | 8.0 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                       1/1 
  Running scriptlet: postgresql-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                                                         1/1 
  Upgrading        : postgresql-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                                                         1/6 
  Upgrading        : postgresql-contrib-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                                                 2/6 
  Running scriptlet: postgresql-server-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                                                  3/6 
  Upgrading        : postgresql-server-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                                                  3/6 
  Running scriptlet: postgresql-server-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                                                  3/6 
  Running scriptlet: postgresql-server-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                                                  4/6 
  Cleanup          : postgresql-server-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                                                  4/6 
  Running scriptlet: postgresql-server-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                                                  4/6 
  Cleanup          : postgresql-contrib-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                                                 5/6 
  Cleanup          : postgresql-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                                                         6/6 
  Running scriptlet: postgresql-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                                                         6/6 
  Verifying        : postgresql-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                                                         1/6 
  Verifying        : postgresql-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                                                         2/6 
  Verifying        : postgresql-contrib-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                                                 3/6 
  Verifying        : postgresql-contrib-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                                                 4/6 
  Verifying        : postgresql-server-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                                                  5/6 
  Verifying        : postgresql-server-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                                                  6/6 
Installed products updated.
Uploading Tracer Profile
Unable to upload Tracer Profile

Upgraded:
  postgresql-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64  postgresql-contrib-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64  postgresql-server-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64 

Complete!

So I continued and stopped services and run the update which didn’t complain about anything.

Thus, it looks to me as if the switch-to postgresql:13 based on repositories hosted on katello/pulp doesn’t work and just fails, while using the standard almalinux repo file and pulling rpms from alma mirrors sees the conflict but still lets you do the switch-to.

Then I ran foreman-installer which performed the upgrade to postgresql 13 at the beginning and worked just fine and until almost the end, when the last few steps took much longer than usual. It’s seems the backend isn’t responding properly

2024-07-26 07:49:58 [DEBUG ] [configure] Executing: '/bin/systemctl is-active -- foreman-proxy'
2024-07-26 07:49:58 [DEBUG ] [configure] Executing: '/bin/systemctl is-enabled -- foreman-proxy'
2024-07-26 07:49:58 [DEBUG ] [configure] Executing: '/bin/systemctl show --property=NeedDaemonReload -- foreman-proxy'
2024-07-26 07:49:58 [DEBUG ] [configure] Executing: '/bin/systemctl unmask -- foreman-proxy'
2024-07-26 07:49:59 [DEBUG ] [configure] Executing: '/bin/systemctl start -- foreman-proxy'
2024-07-26 07:50:00 [DEBUG ] [configure] Executing: '/bin/systemctl is-enabled -- foreman-proxy'
2024-07-26 07:50:00 [INFO  ] [configure] /Stage[main]/Foreman_proxy::Service/Service[foreman-proxy]/ensure: ensure changed 'stopped' to 'running'
2024-07-26 07:50:00 [DEBUG ] [configure] /Service[foreman-proxy]: The container Class[Foreman_proxy::Service] will propagate my refresh event
2024-07-26 07:50:00 [DEBUG ] [configure] /Service[foreman-proxy]: Unscheduling refresh on Service[foreman-proxy]
2024-07-26 07:50:00 [DEBUG ] [configure] /Service[foreman-proxy]: Evaluated in 1.51 seconds
2024-07-26 07:50:00 [DEBUG ] [configure] Class[Foreman_proxy::Service]: Starting to evaluate the resource (2347 of 2379)
2024-07-26 07:50:00 [DEBUG ] [configure] Class[Foreman_proxy::Service]: The container Stage[main] will propagate my refresh event
2024-07-26 07:50:00 [DEBUG ] [configure] Class[Foreman_proxy::Service]: The container Class[Foreman_proxy] will propagate my refresh event
2024-07-26 07:50:00 [DEBUG ] [configure] Class[Foreman_proxy::Service]: Scheduling refresh of Class[Foreman_proxy::Register]
2024-07-26 07:50:00 [DEBUG ] [configure] Class[Foreman_proxy::Service]: Evaluated in 0.00 seconds
2024-07-26 07:50:00 [DEBUG ] [configure] Class[Foreman_proxy::Register]: Starting to evaluate the resource (2348 of 2379)
2024-07-26 07:50:00 [DEBUG ] [configure] Class[Foreman_proxy::Register]: Scheduling refresh of Foreman_smartproxy[foreman8.example.com]
2024-07-26 07:50:00 [DEBUG ] [configure] Class[Foreman_proxy::Register]: Evaluated in 0.00 seconds
2024-07-26 07:50:00 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-foreman8.example.com]: Starting to evaluate the resource (2349 of 2379)
2024-07-26 07:50:00 [DEBUG ] [configure] Foreman_host[foreman-proxy-foreman8.example.com](provider=rest_v3): Making get request to https://foreman8.example.com/api/v2/hosts?search=name%3D%22foreman8.example.com%22
2024-07-26 07:51:00 [DEBUG ] [configure] Foreman_host[foreman-proxy-foreman8.example.com](provider=rest_v3): Received response 200 from request to https://foreman8.example.com/api/v2/hosts?search=name%3D%22foreman8.example.com%22
2024-07-26 07:51:00 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-foreman8.example.com]: Evaluated in 60.22 seconds
2024-07-26 07:51:00 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Datacat_collector[foreman_proxy::enabled_features]: Starting to evaluate the resource (2350 of 2379)
2024-07-26 07:51:00 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Collected {"features"=>["HTTPBoot", "Puppet", "Puppet CA", "Realm", "TFTP", "Templates", "Logs", "Registration", "Pulpcore", "Dynflow", "Script", "Ansible"]}
2024-07-26 07:51:00 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Selecting source_key features
2024-07-26 07:51:00 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Now setting field :features
2024-07-26 07:51:00 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Datacat_collector[foreman_proxy::enabled_features]: Evaluated in 0.01 seconds
2024-07-26 07:51:00 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman8.example.com]: Starting to evaluate the resource (2351 of 2379)
2024-07-26 07:51:00 [DEBUG ] [configure] Foreman_smartproxy[foreman8.example.com](provider=rest_v3): Making get request to https://foreman8.example.com/api/v2/smart_proxies?search=name%3D%22foreman8.example.com%22
2024-07-26 07:52:00 [DEBUG ] [configure] Foreman_smartproxy[foreman8.example.com](provider=rest_v3): Received response 200 from request to https://foreman8.example.com/api/v2/smart_proxies?search=name%3D%22foreman8.example.com%22
2024-07-26 07:52:00 [DEBUG ] [configure] Foreman_smartproxy[foreman8.example.com](provider=rest_v3): Making put request to https://foreman8.example.com/api/v2/smart_proxies/1/refresh
2024-07-26 07:53:02 [DEBUG ] [configure] Foreman_smartproxy[foreman8.example.com](provider=rest_v3): Received response 200 from request to https://foreman8.example.com/api/v2/smart_proxies/1/refresh
2024-07-26 07:53:02 [INFO  ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman8.example.com]: Triggered 'refresh' from 1 event
2024-07-26 07:53:02 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman8.example.com]: The container Class[Foreman_proxy::Register] will propagate my refresh event
2024-07-26 07:53:02 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman8.example.com]: Evaluated in 121.48 seconds
2024-07-26 07:53:02 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy_host[foreman-proxy-foreman8.example.com]: Starting to evaluate the resource (2352 of 2379)
2024-07-26 07:53:02 [DEBUG ] [configure] Foreman_smartproxy_host[foreman-proxy-foreman8.example.com](provider=rest_v3): Making get request to https://foreman8.example.com/api/v2/hosts?search=name%3D%22foreman8.example.com%22
2024-07-26 07:54:04 [DEBUG ] [configure] Foreman_smartproxy_host[foreman-proxy-foreman8.example.com](provider=rest_v3): Received response 200 from request to https://foreman8.example.com/api/v2/hosts?search=name%3D%22foreman8.example.com%22
2024-07-26 07:54:04 [DEBUG ] [configure] Foreman_smartproxy_host[foreman-proxy-foreman8.example.com](provider=rest_v3): Making get request to https://foreman8.example.com/api/v2/smart_proxies?search=name%3D%22foreman8.example.com%22
2024-07-26 07:55:04 [DEBUG ] [configure] Foreman_smartproxy_host[foreman-proxy-foreman8.example.com](provider=rest_v3): Received response 200 from request to https://foreman8.example.com/api/v2/smart_proxies?search=name%3D%22foreman8.example.com%22
2024-07-26 07:55:04 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy_host[foreman-proxy-foreman8.example.com]: Evaluated in 122.53 seconds
2024-07-26 07:55:04 [DEBUG ] [configure] Class[Foreman_proxy::Register]: Starting to evaluate the resource (2353 of 2379)
2024-07-26 07:55:04 [DEBUG ] [configure] Class[Foreman_proxy::Register]: The container Stage[main] will propagate my refresh event
2024-07-26 07:55:04 [DEBUG ] [configure] Class[Foreman_proxy::Register]: The container Class[Foreman_proxy] will propagate my refresh event
2024-07-26 07:55:04 [DEBUG ] [configure] Class[Foreman_proxy::Register]: Evaluated in 0.00 seconds
2024-07-26 07:55:04 [DEBUG ] [configure] Class[Foreman_proxy]: Starting to evaluate the resource (2354 of 2379)
2024-07-26 07:55:04 [DEBUG ] [configure] Class[Foreman_proxy]: The container Stage[main] will propagate my refresh event
2024-07-26 07:55:04 [DEBUG ] [configure] Class[Foreman_proxy]: Evaluated in 0.00 seconds

Note the one minute delay after each of the api requests. Although it says code 200 it’s actually a timeout. foreman-installer still didn’t notice this and continued successfully,=

I am seeing timeouts in production.log for requests:

2024-07-26T07:57:43 [I|app|97586085] Started GET "/notification_recipients" for 136.172.15.242 at 2024-07-26 07:57:43 +0200
2024-07-26T07:58:43 [W|app|97586085] Timed out connecting to server
 97586085 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:749:in `rescue in transmit'
 97586085 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:651:in `transmit'
 97586085 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
 97586085 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
 97586085 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get'
 97586085 | /usr/share/gems/gems/katello-4.13.1/app/models/katello/ping.rb:264:in `backend_status'
 97586085 | /usr/share/gems/gems/katello-4.13.1/app/models/katello/ping.rb:87:in `block in ping_candlepin_without_auth'
 97586085 | /usr/share/gems/gems/katello-4.13.1/app/models/katello/ping.rb:136:in `exception_watch'
 97586085 | /usr/share/gems/gems/katello-4.13.1/app/models/katello/ping.rb:86:in `ping_candlepin_without_auth'
 97586085 | /usr/share/gems/gems/katello-4.13.1/app/models/katello/ping.rb:236:in `ping_services_for_capsule'
 97586085 | /usr/share/gems/gems/katello-4.13.1/app/models/katello/ping.rb:24:in `ping!'
 97586085 | /usr/share/gems/gems/katello-4.13.1/app/services/katello/organization_creator.rb:20:in `create_all_organizations!'
 97586085 | /usr/share/gems/gems/katello-4.13.1/lib/katello/middleware/organization_created_enforcer.rb:12:in `call'
 97586085 | /usr/share/gems/gems/katello-4.13.1/lib/katello/middleware/event_daemon.rb:10:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/static.rb:24:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/static.rb:24:in `call'
 97586085 | /usr/share/gems/gems/apipie-dsl-2.6.2/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 97586085 | /usr/share/gems/gems/apipie-rails-1.3.0/lib/apipie/static_dispatcher.rb:74:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/static.rb:24:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/static.rb:24:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/static.rb:24:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/static.rb:24:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/static.rb:24:in `call'
 97586085 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 97586085 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 97586085 | /usr/share/gems/gems/apipie-rails-1.3.0/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 97586085 | /usr/share/gems/gems/rack-2.2.9/lib/rack/tempfile_reaper.rb:15:in `call'
 97586085 | /usr/share/gems/gems/rack-2.2.9/lib/rack/etag.rb:27:in `call'
 97586085 | /usr/share/gems/gems/rack-2.2.9/lib/rack/conditional_get.rb:27:in `call'
 97586085 | /usr/share/gems/gems/rack-2.2.9/lib/rack/head.rb:12:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
 97586085 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 97586085 | /usr/share/gems/gems/rack-2.2.9/lib/rack/session/abstract/id.rb:266:in `context'
 97586085 | /usr/share/gems/gems/rack-2.2.9/lib/rack/session/abstract/id.rb:260:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/cookies.rb:697:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 97586085 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:98:in `run_callbacks'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 97586085 | /usr/share/gems/gems/railties-6.1.7.8/lib/rails/rack/logger.rb:37:in `call_app'
 97586085 | /usr/share/gems/gems/railties-6.1.7.8/lib/rails/rack/logger.rb:28:in `call'
 97586085 | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 97586085 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/request_id.rb:26:in `call'
 97586085 | /usr/share/gems/gems/katello-4.13.1/lib/katello/prevent_json_parsing.rb:12:in `call'
 97586085 | /usr/share/gems/gems/rack-2.2.9/lib/rack/method_override.rb:24:in `call'
 97586085 | /usr/share/gems/gems/rack-2.2.9/lib/rack/runtime.rb:22:in `call'
 97586085 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/executor.rb:14:in `call'
 97586085 | /usr/share/gems/gems/rack-2.2.9/lib/rack/sendfile.rb:110:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/ssl.rb:77:in `call'
 97586085 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
 97586085 | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
 97586085 | /usr/share/gems/gems/railties-6.1.7.8/lib/rails/engine.rb:539:in `call'
 97586085 | /usr/share/gems/gems/railties-6.1.7.8/lib/rails/railtie.rb:207:in `public_send'
 97586085 | /usr/share/gems/gems/railties-6.1.7.8/lib/rails/railtie.rb:207:in `method_missing'
 97586085 | /usr/share/gems/gems/rack-2.2.9/lib/rack/urlmap.rb:74:in `block in call'
 97586085 | /usr/share/gems/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `each'
 97586085 | /usr/share/gems/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `call'
 97586085 | /usr/share/gems/gems/puma-6.4.2/lib/puma/configuration.rb:272:in `call'
 97586085 | /usr/share/gems/gems/puma-6.4.2/lib/puma/request.rb:100:in `block in handle_request'
 97586085 | /usr/share/gems/gems/puma-6.4.2/lib/puma/thread_pool.rb:378:in `with_force_shutdown'
 97586085 | /usr/share/gems/gems/puma-6.4.2/lib/puma/request.rb:99:in `handle_request'
 97586085 | /usr/share/gems/gems/puma-6.4.2/lib/puma/server.rb:464:in `process_client'
 97586085 | /usr/share/gems/gems/puma-6.4.2/lib/puma/server.rb:245:in `block in run'
 97586085 | /usr/share/gems/gems/puma-6.4.2/lib/puma/thread_pool.rb:155:in `block in spawn_thread'
 97586085 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-07-26T07:58:43 [I|app|97586085] Processing by NotificationRecipientsController#index as JSON
2024-07-26T07:58:43 [I|app|97586085] Completed 200 OK in 11ms (Views: 0.1ms | ActiveRecord: 0.7ms | Allocations: 1989)

After foreman-installer had finished “successfully” it was still not working. The web frontend is accessible but you cannot do anything because the backend isn’t working.

I reran foreman-installer again hoping it would fix the backend. But no change: all api calls in production log time out after a minute, but at least foreman-installed failed at some point because it got a 500 internal server error.

I did a GET https://foreman8.example.com/api/hosts/1 in my browser to verify: it looks as if the api is responding eventually but it took 1.25 minutes until it responded.

So the backend api is responding very slowly causing many timeouts of requests which have a 60s timeout. Something must be very slow in the backend.

As this is my production server I reverted back to 4.12 for now.

gvde · July 26, 2024, 6:46am

Back on 4.12 the call to GET https://foreman8.example.com/api/hosts/1 takes 491 ms, not 1.25 minutes… Something was very slow after the upgrade…

chatlow · July 26, 2024, 7:21am

thanks for testing and confirming this. I have also reverted back to 3.10/4.12 until this upgrade process is fixed and stable

gvde · July 26, 2024, 7:28am

At the moment it looks to me like two different issues:

getting the dnf update/switch-to working properly which seems to be fine when using repo files and not repos hosted by katello/pulp. Thus, technically it’s working if done as required in the docs.
terrible performance while/after running foreman-installer. That seems like a different issue.

I will try again later using only the standard repo files, disabling the katello repos. Maybe it works better then. Did you ever get the dnf update properly working so far?

evgeni · July 26, 2024, 7:33am

The first switch issue is because you have both Foreman 3.10 and 3.11 repos enabled, and they disagree on the PostgreSQL module version they want.
Drop the 3.10 ones and it should work.

gvde · July 26, 2024, 8:55am

O.K. I see. Make sense and then again not. I thought the dependencies are from the installed rpm to the module and thus it would be there whatever what and something will always break if you switch and update. Anyway, I try again using only repo files and without redhat.repo…

chatlow · July 26, 2024, 9:46am

Yeah this has also confused me, as I thought it got the content from the new Foreman/Katello deps we bring in

As for having both Foreman 3.10 and 3.11 repos enabled - they were not. I always disable repos on this machine until needed. Only my ol8repos were enabled to allow for the OS update as part of this process.

Then obviously the dnf install of both new foreman/katello rpms means their repos are enabled, but the 3.10 repo gets overwitted as an rpmsave during that process

@gvde please let us know how you get on. I tried disabling my ol8_appstream repo once Postgres13 was in place - as this contains the content, but foreman-installer moaned about it straight away during the Postgres upgrade

2024-07-26 09:36:00 [NOTICE] [pre] Upgrading PostgreSQL packages
2024-07-26 09:36:01 [ERROR ] [root] dnf module switch-to postgresql:13 -y failed! Check the output for error!

maximilian · July 26, 2024, 9:55am

I have created a PR in foreman-documentation to add a note: Add hint about upgrading Katello by maximiliankolb · Pull Request #3170 · theforeman/foreman-documentation · GitHub

gvde · July 26, 2024, 10:26am

The issue with the switch is solved for me if I make sure that not both 3.10 and 3.11 are enabled.

I have started the last test with these repos:

# dnf repolist
repo id                                                        repo name
appstream                                                      AlmaLinux 8 - AppStream
baseos                                                         AlmaLinux 8 - BaseOS
candlepin                                                      Candlepin: an open source entitlement management system.
extras                                                         AlmaLinux 8 - Extras
foreman                                                        Foreman 3.10
foreman-plugins                                                Foreman plugins 3.10
katello                                                        Katello 4.12
powertools                                                     AlmaLinux 8 - PowerTools
pulpcore                                                       pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.

And then follow the docs Upgrading Foreman to 3.11

That does work for me. The switch will install postgresql13:

[root@foreman8 ~]# dnf -y module switch-to postgresql:13
Last metadata expiration check: 0:00:05 ago on Fri 26 Jul 2024 11:46:26 AM CEST.
Modular dependency problem:

 Problem: module foreman:el8:31120240718202416:34d643a2.x86_64 from foreman requires module(postgresql:13), but none of the providers can be installed
  - module postgresql:12:8090020240227082527:9edba152.x86_64 from appstream conflicts with module(postgresql:13) provided by postgresql:13:8090020240415140528:9edba152.x86_64 from appstream
  - module postgresql:13:8090020240415140528:9edba152.x86_64 from appstream conflicts with module(postgresql:12) provided by postgresql:12:8090020240227082527:9edba152.x86_64 from appstream
  - conflicting requests
Dependencies resolved.
==================================================================================================================================================================================
 Package                                  Architecture                 Version                                                              Repository                       Size
==================================================================================================================================================================================
Upgrading:
 postgresql                               x86_64                       13.14-1.module_el8.9.0+3815+0bfda440.alma.1                          appstream                       1.5 M
 postgresql-contrib                       x86_64                       13.14-1.module_el8.9.0+3815+0bfda440.alma.1                          appstream                       882 k
 postgresql-server                        x86_64                       13.14-1.module_el8.9.0+3815+0bfda440.alma.1                          appstream                       5.6 M
Switching module streams:
 postgresql                                                            12 -> 13                                                                                                  

Transaction Summary
==================================================================================================================================================================================
Upgrade  3 Packages

Total download size: 8.0 M
Downloading Packages:
(1/3): postgresql-contrib-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64.rpm                                                                  4.6 MB/s | 882 kB     00:00    
(2/3): postgresql-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64.rpm                                                                          6.6 MB/s | 1.5 MB     00:00    
(3/3): postgresql-server-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64.rpm                                                                    13 MB/s | 5.6 MB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                             9.4 MB/s | 8.0 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                          1/1 
  Running scriptlet: postgresql-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                            1/1 
  Upgrading        : postgresql-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                            1/6 
  Upgrading        : postgresql-contrib-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                    2/6 
  Running scriptlet: postgresql-server-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                     3/6 
  Upgrading        : postgresql-server-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                     3/6 
  Running scriptlet: postgresql-server-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                     3/6 
  Running scriptlet: postgresql-server-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                     4/6 
  Cleanup          : postgresql-server-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                     4/6 
  Running scriptlet: postgresql-server-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                     4/6 
  Cleanup          : postgresql-contrib-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                    5/6 
  Cleanup          : postgresql-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                            6/6 
  Running scriptlet: postgresql-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                            6/6 
  Verifying        : postgresql-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                            1/6 
  Verifying        : postgresql-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                            2/6 
  Verifying        : postgresql-contrib-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                    3/6 
  Verifying        : postgresql-contrib-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                    4/6 
  Verifying        : postgresql-server-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                                                                                     5/6 
  Verifying        : postgresql-server-12.18-1.module_el8.9.0+3740+0e74851f.alma.1.x86_64                                                                                     6/6 
Installed products updated.
Uploading Tracer Profile
Unable to upload Tracer Profile

Upgraded:
  postgresql-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                           postgresql-contrib-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                   
  postgresql-server-13.14-1.module_el8.9.0+3815+0bfda440.alma.1.x86_64                   

Complete!

foreman-installer also works in that process. However, the issue for me at hand is the terrible performance. This is my second time, now with the working update procedure, but the API calls to foreman still take minutes instead of milliseconds…

chatlow · July 26, 2024, 10:59am

Surely the foreman repo you have at the start of this test above (v3.10) gets overwritten once you’ve run the dnf update for forman v3.11?

which repo have you disabled and at what point of the process did you do it?

the conflict is for postgres which is from appstream

gvde · July 26, 2024, 11:35am

I did not disable any repo. I started with the repos above and then followed the steps. Foreman/Katello repository files are updated in step 9 of the docs. I just continued as in the docs:

# dnf update https://yum.theforeman.org/releases/3.11/el8/x86_64/foreman-release.rpm https://yum.theforeman.org/katello/4.13/katello/el8/x86_64/katello-repos-latest.rpm
# dnf module disable pulpcore
# dnf -y module switch-to postgresql:13
# dnf module enable katello:el8
# foreman-maintain service stop
# dnf update
# foreman-installer

chatlow · July 26, 2024, 12:19pm

you said in your initial post that you disabled your redhat.repo and just used local repos - does that mean appstream got disabled?

Asking because I am able to do the switch and update to postgres13 albeit with the conflicts. But foreman-install doesn’t allows the postgres upgrade and on yours it does. We are running different OS’s but they are both supported.