Issue after yum update -> Foreman 2.5.2 -pulpcore_workers failing

This was supposed to be included prior to 2.5.2 being released:
https://github.com/theforeman/foreman-packaging/pull/6906

You must install Pulp 3.14 prior to upgrading to 2.5.2.

2 Likes

Strange - those came along with the update for me …

Ah, ok. Great.

Though, I still have this after the update:

foreman-maintain health check

Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check number of fact names in database:                               [OK]
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist:                      [OK]
--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [FAIL]
Couldn't connect to the server: undefined method `to_sym' for nil:NilClass
--------------------------------------------------------------------------------
Continue with step [Restart applicable services]?, [y(yes), n(no)] 


From
/var/log/messages

ul 16 17:34:06 foreman-server smart-proxy[12148]: `/usr/share/foreman-proxy` is not writable.
Jul 16 17:34:06 foreman-server smart-proxy[12148]: Bundler will use `/tmp/bundler20210716-12148-5qniec12148' as your home directory temporarily.
Jul 16 17:34:06 foreman-server smart-proxy[12148]: Your Gemfile lists the gem rsec (< 1) more than once.
Jul 16 17:34:06 foreman-server smart-proxy[12148]: You should probably keep only one of them.
Jul 16 17:34:06 foreman-server smart-proxy[12148]: Remove any duplicate entries and specify the gem only once.
Jul 16 17:34:06 foreman-server smart-proxy[12148]: While it's not a problem now, it could cause errors if you change the version of one of them later.
Jul 16 17:34:07 foreman-server systemd[1]: Started Foreman Proxy.
Jul 16 17:34:37 foreman-server smart-proxy[12148]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/sqlite.rb:114: warning: rb_check_safe_obj will be removed in Ruby 3.0
Jul 16 17:34:37 foreman-server smart-proxy[12148]: 192.168.116.13 - - [16/Jul/2021:17:34:37 CEST] "GET /features HTTP/1.1" 200 97
Jul 16 17:34:37 foreman-server smart-proxy[12148]: - -> /features
Jul 16 17:34:42 foreman-server foreman[11159]: warning: broker sent EOF, and connection not reliable
Jul 16 17:34:42 foreman-server foreman[11159]: #<Thread:0x000055fc96a03f50 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Jul 16 17:34:42 foreman-server foreman[11159]: /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:198:in `block (2 levels) in start_listeners': Received message is nil, and connection not reliable (Stomp::Error::NilMessageError)
Jul 16 17:34:42 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `loop'
Jul 16 17:34:42 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `block in start_listeners'
Jul 16 17:34:42 foreman-server foreman[11159]: #011from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Jul 16 17:34:43 foreman-server pulpcore-api[11072]: pulp [4653cdf7-c41e-4204-93de-b71521216a17]:  - - [16/Jul/2021:15:34:43 +0000] "GET /pulp/api/v3/status HTTP/1.1" 301 0 "-" "rest-client/2.0.2 (linux x86_64) ruby/2.7.3p183"
Jul 16 17:34:43 foreman-server pulpcore-api[11072]: pulp [4653cdf7-c41e-4204-93de-b71521216a17]:  - - [16/Jul/2021:15:34:43 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3740 "-" "rest-client/2.0.2 (linux x86_64) ruby/2.7.3p183"
Jul 16 17:34:43 foreman-server foreman[11159]: /usr/share/gems/gems/katello-4.1.1/app/lib/katello/http_resource.rb:75: warning: URI.escape is obsolete
Jul 16 17:34:58 foreman-server foreman[11159]: warning: broker sent EOF, and connection not reliable
Jul 16 17:34:58 foreman-server foreman[11159]: #<Thread:0x00007f2028ad6340 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Jul 16 17:34:58 foreman-server foreman[11159]: /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:198:in `block (2 levels) in start_listeners': Received message is nil, and connection not reliable (Stomp::Error::NilMessageError)
Jul 16 17:34:58 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `loop'
Jul 16 17:34:58 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `block in start_listeners'
Jul 16 17:34:58 foreman-server foreman[11159]: #011from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Jul 16 17:35:13 foreman-server foreman[11159]: warning: broker sent EOF, and connection not reliable
Jul 16 17:35:13 foreman-server foreman[11159]: #<Thread:0x00007f2028904e90 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Jul 16 17:35:13 foreman-server foreman[11159]: /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:198:in `block (2 levels) in start_listeners': Received message is nil, and connection not reliable (Stomp::Error::NilMessageError)
Jul 16 17:35:13 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `loop'
Jul 16 17:35:13 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `block in start_listeners'
Jul 16 17:35:13 foreman-server foreman[11159]: #011from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Jul 16 17:35:28 foreman-server foreman[11159]: warning: broker sent EOF, and connection not reliable
Jul 16 17:35:28 foreman-server foreman[11159]: #<Thread:0x00007f20285fd738 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Jul 16 17:35:28 foreman-server foreman[11159]: /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:198:in `block (2 levels) in start_listeners': Received message is nil, and connection not reliable (Stomp::Error::NilMessageError)
Jul 16 17:35:28 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `loop'
Jul 16 17:35:28 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `block in start_listeners'
Jul 16 17:35:28 foreman-server foreman[11159]: #011from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Jul 16 17:35:44 foreman-server foreman[11159]: warning: broker sent EOF, and connection not reliable
Jul 16 17:35:44 foreman-server foreman[11159]: #<Thread:0x00007f20283e12b0 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Jul 16 17:35:44 foreman-server foreman[11159]: /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:198:in `block (2 levels) in start_listeners': Received message is nil, and connection not reliable (Stomp::Error::NilMessageError)
Jul 16 17:35:44 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `loop'
Jul 16 17:35:44 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `block in start_listeners'
Jul 16 17:35:44 foreman-server foreman[11159]: #011from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Jul 16 17:35:59 foreman-server foreman[11159]: warning: broker sent EOF, and connection not reliable
Jul 16 17:35:59 foreman-server foreman[11159]: #<Thread:0x00007f20281df070 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Jul 16 17:35:59 foreman-server foreman[11159]: /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:198:in `block (2 levels) in start_listeners': Received message is nil, and connection not reliable (Stomp::Error::NilMessageError)
Jul 16 17:35:59 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `loop'
Jul 16 17:35:59 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `block in start_listeners'
Jul 16 17:35:59 foreman-server foreman[11159]: #011from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

I have not changed any file.

pulp-client-1.0-1.noarch
pulpcore-selinux-1.2.4-1.el8.x86_64
python3-pulp-ansible-0.8.0-1.el8.noarch
python3-pulp-certguard-1.4.0-1.el8.noarch
python3-pulp-container-2.7.0-1.el8.noarch
python3-pulpcore-3.14.1-1.el8.noarch
python3-pulp-deb-2.13.0-1.el8.noarch
python3-pulp-file-1.8.1-1.el8.noarch
python3-pulp-rpm-3.13.3-1.el8.noarch
rubygem-pulp_ansible_client-0.8.0-1.el8.noarch
rubygem-pulp_certguard_client-1.4.0-1.el8.noarch
rubygem-pulp_container_client-2.7.0-1.el8.noarch
rubygem-pulpcore_client-3.14.1-1.el8.noarch
rubygem-pulp_deb_client-2.13.0-1.el8.noarch
rubygem-pulp_file_client-1.8.1-1.el8.noarch
rubygem-pulp_rpm_client-3.13.3-1.el8.noarch
rubygem-smart_proxy_pulp-3.0.0-1.fm2_5.el8.noarch

OK, but that was part of the problem that it didn’t originally come with 2.5.2…

What to do now?

I did not explicitly install pulp 3.14 - I guess it came along with a yum update. Did foreman-installer run without errors. I first got errors, reverted the unit changes, daemon-reload and foreman services stop - after that foreman-installer ran smoothly and all started without problems:

> # foreman-maintain health check
> Running ForemanMaintain::Scenario::FilteredScenario
> ================================================================================
> Check number of fact names in database:                               [OK]
> --------------------------------------------------------------------------------
> Check whether all services are running:                               [OK]
> --------------------------------------------------------------------------------
> Check whether all services are running using the ping call:           [OK]
> --------------------------------------------------------------------------------
> Check for paused tasks:                                               [OK]
> --------------------------------------------------------------------------------
> Check to verify no empty CA cert requests exist:                      [OK]
> --------------------------------------------------------------------------------

I had to run yum update twice (or trice, if you count the original update)

At first, some updates came through, and that didn’t really satisfy the installer, so I ran it again and then it (supposedly) installed the rest of the updates.
Then the installer ran through.

But it’s obviously still not 100% right…

Should have waited a day :wink:

So I had hoped this would be part of the 2.5.2 release notes, but here’s the important info:

Foreman 2.5.2 is meant to be used together with Katello 4.1.1 and Pulp 3.14

This means you must ensure you upgrade them together.

Users of katello-repos

If you’re using katello-repos to get the repos, this is how you upgrade:

# Katello-repos will upgrade the Pulp repo
yum update foreman-release katello-repos
# Ensure you have correct metadata
yum clean expire-cache
# Update all packages
yum update
# Run the installer to make all service changes
foreman-installer

Note this is all from the top of my head and I haven’t tested it yet. However, given all the problems I want to highlight some crucial steps that should be taken.

Users of custom repos

Not too detailed here, but ensure you get Pulp from Index of /pulpcore/3.14 now (instead of 3.11).

2 Likes