here is explanation from systemd
systemd[85247]: Failed at step EXEC spawning /usr/libexec/pulpcore/pulpcore-worker: No such file or directory
Right - so check the ExecStart in /etc/systemd/system/pulpcore-worker@.service
if it shows:
ExecStart=/usr/libexec/pulpcore/pulpcore-worker
Change it to (after making a backup copy):
ExecStart=/usr/libexec/pulpcore/rq worker
-w pulpcore.tasking.worker.PulpWorker
-c ‘pulpcore.rqconfig’
–disable-job-desc-logging
There is no ‘/usr/libexec/pulpcore/pulpcore-worker’ - I got that ExecStart from a proxy that is not upgraded yet. After the change run systemctl daemon-reload
well, that makes the pulpcore-worker processes start again but I a service restart still results in:
…
- All services displayed [FAIL]
Some services are not running (pulpcore-worker@*)
yep. no luck. someone forgot to update systemd file in that commit probably.
I assume they should fix it over the weekend.
It looks to me as if 2.5.2 is supposed to work with pulp 3.14 which will come with katello 4.1.1 or 4.2 which isn’t available, yet…
Or it seems 4.1.1 has just arrived in the repository. Wasn’t there this morning…
Ah could be. There was a couple of versions ago some bug, also caused by foreman and katello not being synced in their releases.
OK, so we just have to wait until everything is in sync, repo-wise?
It already is - running the update as we speak (don’t forget to revert the changes I mentioned above aka restore the backup file)
Yes, you must upgrade Katello as well.
Originally we talked about adding an RPM conflicts in the installer but that was forgotten which is breaking installations.
Maybe its a race condition where this Katello package was supposed to be installed before Foreman, but after upgrade now to katello 4.1.1, no changes. pulpcore worker is still down for me with the same issue.
Hmm - that is odd - working for me now - you did revert the unit change and performed a daemon-reload?
yep. how does your systemd file look now?
The update did not touch it so I reverted manually
Do you use the original repo files or do you sync from your katello server? From Katello 4.1.0 to 4.1.1 the repository file now contains pulp 3.14 not 3.11 anymore…
And don’t forget to run foreman-installer after the update. That should create the correct pulp-worker systemd service.
My server syncs from the public internet so the original repo files and it did get pulp 3.14:
rpm -qa | grep pulp | grep 3.14
tfm-rubygem-pulpcore_client-3.14.1-1.el7.noarch
python3-pulpcore-3.14.1-1.el7.noarch
yep, I’m still missing that file /usr/libexec/pulpcore/pulpcore-worker
I ran yum update and foreman-installer afterwards, standard. But did not help.
Centos and epel repo are synced from internet daily and hosted inside network.
foreman/katello I pull directly from the internet, offical repos.
I’ll pull those, installed pulp version is still 3.11. Tnx for your help.
This was supposed to be included prior to 2.5.2 being released:
https://github.com/theforeman/foreman-packaging/pull/6906
You must install Pulp 3.14 prior to upgrading to 2.5.2.
2 Likes
Strange - those came along with the update for me …
Ah, ok. Great.
Though, I still have this after the update:
foreman-maintain health check
Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check number of fact names in database: [OK]
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist: [OK]
--------------------------------------------------------------------------------
Check whether all services are running: [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call: [FAIL]
Couldn't connect to the server: undefined method `to_sym' for nil:NilClass
--------------------------------------------------------------------------------
Continue with step [Restart applicable services]?, [y(yes), n(no)]
From
/var/log/messages
ul 16 17:34:06 foreman-server smart-proxy[12148]: `/usr/share/foreman-proxy` is not writable.
Jul 16 17:34:06 foreman-server smart-proxy[12148]: Bundler will use `/tmp/bundler20210716-12148-5qniec12148' as your home directory temporarily.
Jul 16 17:34:06 foreman-server smart-proxy[12148]: Your Gemfile lists the gem rsec (< 1) more than once.
Jul 16 17:34:06 foreman-server smart-proxy[12148]: You should probably keep only one of them.
Jul 16 17:34:06 foreman-server smart-proxy[12148]: Remove any duplicate entries and specify the gem only once.
Jul 16 17:34:06 foreman-server smart-proxy[12148]: While it's not a problem now, it could cause errors if you change the version of one of them later.
Jul 16 17:34:07 foreman-server systemd[1]: Started Foreman Proxy.
Jul 16 17:34:37 foreman-server smart-proxy[12148]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/sqlite.rb:114: warning: rb_check_safe_obj will be removed in Ruby 3.0
Jul 16 17:34:37 foreman-server smart-proxy[12148]: 192.168.116.13 - - [16/Jul/2021:17:34:37 CEST] "GET /features HTTP/1.1" 200 97
Jul 16 17:34:37 foreman-server smart-proxy[12148]: - -> /features
Jul 16 17:34:42 foreman-server foreman[11159]: warning: broker sent EOF, and connection not reliable
Jul 16 17:34:42 foreman-server foreman[11159]: #<Thread:0x000055fc96a03f50 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Jul 16 17:34:42 foreman-server foreman[11159]: /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:198:in `block (2 levels) in start_listeners': Received message is nil, and connection not reliable (Stomp::Error::NilMessageError)
Jul 16 17:34:42 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `loop'
Jul 16 17:34:42 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `block in start_listeners'
Jul 16 17:34:42 foreman-server foreman[11159]: #011from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Jul 16 17:34:43 foreman-server pulpcore-api[11072]: pulp [4653cdf7-c41e-4204-93de-b71521216a17]: - - [16/Jul/2021:15:34:43 +0000] "GET /pulp/api/v3/status HTTP/1.1" 301 0 "-" "rest-client/2.0.2 (linux x86_64) ruby/2.7.3p183"
Jul 16 17:34:43 foreman-server pulpcore-api[11072]: pulp [4653cdf7-c41e-4204-93de-b71521216a17]: - - [16/Jul/2021:15:34:43 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3740 "-" "rest-client/2.0.2 (linux x86_64) ruby/2.7.3p183"
Jul 16 17:34:43 foreman-server foreman[11159]: /usr/share/gems/gems/katello-4.1.1/app/lib/katello/http_resource.rb:75: warning: URI.escape is obsolete
Jul 16 17:34:58 foreman-server foreman[11159]: warning: broker sent EOF, and connection not reliable
Jul 16 17:34:58 foreman-server foreman[11159]: #<Thread:0x00007f2028ad6340 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Jul 16 17:34:58 foreman-server foreman[11159]: /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:198:in `block (2 levels) in start_listeners': Received message is nil, and connection not reliable (Stomp::Error::NilMessageError)
Jul 16 17:34:58 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `loop'
Jul 16 17:34:58 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `block in start_listeners'
Jul 16 17:34:58 foreman-server foreman[11159]: #011from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Jul 16 17:35:13 foreman-server foreman[11159]: warning: broker sent EOF, and connection not reliable
Jul 16 17:35:13 foreman-server foreman[11159]: #<Thread:0x00007f2028904e90 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Jul 16 17:35:13 foreman-server foreman[11159]: /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:198:in `block (2 levels) in start_listeners': Received message is nil, and connection not reliable (Stomp::Error::NilMessageError)
Jul 16 17:35:13 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `loop'
Jul 16 17:35:13 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `block in start_listeners'
Jul 16 17:35:13 foreman-server foreman[11159]: #011from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Jul 16 17:35:28 foreman-server foreman[11159]: warning: broker sent EOF, and connection not reliable
Jul 16 17:35:28 foreman-server foreman[11159]: #<Thread:0x00007f20285fd738 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Jul 16 17:35:28 foreman-server foreman[11159]: /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:198:in `block (2 levels) in start_listeners': Received message is nil, and connection not reliable (Stomp::Error::NilMessageError)
Jul 16 17:35:28 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `loop'
Jul 16 17:35:28 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `block in start_listeners'
Jul 16 17:35:28 foreman-server foreman[11159]: #011from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Jul 16 17:35:44 foreman-server foreman[11159]: warning: broker sent EOF, and connection not reliable
Jul 16 17:35:44 foreman-server foreman[11159]: #<Thread:0x00007f20283e12b0 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Jul 16 17:35:44 foreman-server foreman[11159]: /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:198:in `block (2 levels) in start_listeners': Received message is nil, and connection not reliable (Stomp::Error::NilMessageError)
Jul 16 17:35:44 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `loop'
Jul 16 17:35:44 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `block in start_listeners'
Jul 16 17:35:44 foreman-server foreman[11159]: #011from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Jul 16 17:35:59 foreman-server foreman[11159]: warning: broker sent EOF, and connection not reliable
Jul 16 17:35:59 foreman-server foreman[11159]: #<Thread:0x00007f20281df070 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Jul 16 17:35:59 foreman-server foreman[11159]: /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:198:in `block (2 levels) in start_listeners': Received message is nil, and connection not reliable (Stomp::Error::NilMessageError)
Jul 16 17:35:59 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `loop'
Jul 16 17:35:59 foreman-server foreman[11159]: #011from /usr/share/gems/gems/stomp-1.4.9/lib/client/utils.rb:194:in `block in start_listeners'
Jul 16 17:35:59 foreman-server foreman[11159]: #011from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
I have not changed any file.
pulp-client-1.0-1.noarch
pulpcore-selinux-1.2.4-1.el8.x86_64
python3-pulp-ansible-0.8.0-1.el8.noarch
python3-pulp-certguard-1.4.0-1.el8.noarch
python3-pulp-container-2.7.0-1.el8.noarch
python3-pulpcore-3.14.1-1.el8.noarch
python3-pulp-deb-2.13.0-1.el8.noarch
python3-pulp-file-1.8.1-1.el8.noarch
python3-pulp-rpm-3.13.3-1.el8.noarch
rubygem-pulp_ansible_client-0.8.0-1.el8.noarch
rubygem-pulp_certguard_client-1.4.0-1.el8.noarch
rubygem-pulp_container_client-2.7.0-1.el8.noarch
rubygem-pulpcore_client-3.14.1-1.el8.noarch
rubygem-pulp_deb_client-2.13.0-1.el8.noarch
rubygem-pulp_file_client-1.8.1-1.el8.noarch
rubygem-pulp_rpm_client-3.13.3-1.el8.noarch
rubygem-smart_proxy_pulp-3.0.0-1.fm2_5.el8.noarch
OK, but that was part of the problem that it didn’t originally come with 2.5.2…
What to do now?