Foreman / dynflow-sidekiq / http / postgresql fails to start

Problem:

I’ve got a Foreman 2.4.0 with Katello 4.0.1.1 on a CentOS 7 system, that after a reboot I would get a slew of failures following a system reboot. I was able to clear up the postgresql from starting, as for some reason the /var/tmp/postgresql directory was removed on shutdown and not created on startup. Seems like a bug to me.

However, I can’t get the other services to start, and foreman-maintain service status gives me this:

[root@repomirror dynflow]# foreman-maintain service status -b
Running Status Services
================================================================================
Get status of applicable services:

Displaying the following service(s):
rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-resource-manager, qdrouterd, qpidd, rh-redis5-redis, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
| displaying rh-redis5-redis                       [OK]
- displaying postgresql                            [OK]
- displaying pulpcore-api                          [OK]
- displaying pulpcore-content                      [OK]
- displaying pulpcore-resource-manager             [OK]
- displaying qdrouterd                             [OK]
- displaying qpidd                                 [OK]
- displaying rh-redis5-redis                       [OK]
- displaying tomcat                                [OK]
\ displaying dynflow-sidekiq@orchestrator          [FAIL]
\ displaying foreman                               [FAIL]
\ displaying httpd                                 [FAIL]
\ displaying puppetserver                          [OK]
\ displaying dynflow-sidekiq@worker                [FAIL]
\ displaying dynflow-sidekiq@worker-hosts-queue    [FAIL]
| displaying foreman-proxy                         [OK]
| All services displayed                                              [FAIL]
Some services are not running (dynflow-sidekiq@orchestrator, foreman, httpd, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue)
--------------------------------------------------------------------------------
Scenario [Status Services] failed.

The following steps ended up in failing state:

  [service-status]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="service-status"


[root@repomirror dynflow]#

So, I’m open to any suggestions on how to fix this or what I would need to provide to create a bug report.

Not sure if this helps or not… but looking at foreman-tail I see this pop up quite often:
i.e. " could not find expected ‘:’ while scanning a simple key at line 32 column 1"

Jul  6 12:00:45 repomirror dynflow-sidekiq@worker-hosts-queue: /usr/share/foreman/extras/dynflow-sidekiq.rb:6:in `<top (required)>'
Jul  6 12:00:45 repomirror dynflow-sidekiq@worker-hosts-queue: /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
Jul  6 12:00:45 repomirror dynflow-sidekiq@worker-hosts-queue: /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
Jul  6 12:00:45 repomirror dynflow-sidekiq@worker-hosts-queue: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
Jul  6 12:00:45 repomirror dynflow-sidekiq@worker-hosts-queue: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
Jul  6 12:00:45 repomirror dynflow-sidekiq@worker-hosts-queue: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
Jul  6 12:00:45 repomirror dynflow-sidekiq@worker-hosts-queue: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `load'
Jul  6 12:00:45 repomirror dynflow-sidekiq@worker-hosts-queue: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `<main>'
Jul  6 12:00:45 repomirror dynflow-sidekiq@orchestrator: (<unknown>): could not find expected ':' while scanning a simple key at line 32 column 1
Jul  6 12:00:45 repomirror dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby25/root/usr/share/ruby/psych.rb:402:in `parse'
Jul  6 12:00:45 repomirror dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby25/root/usr/share/ruby/psych.rb:402:in `parse_stream'
Jul  6 12:00:45 repomirror dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby25/root/usr/share/ruby/psych.rb:350:in `parse'
Jul  6 12:00:45 repomirror dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby25/root/usr/share/ruby/psych.rb:263:in `load'
Jul  6 12:00:45 repomirror dynflow-sidekiq@orchestrator: /usr/share/foreman/config/settings.rb:34:in `block in <top (required)>'
Jul  6 12:00:45 repomirror dynflow-sidekiq@orchestrator: /usr/share/foreman/config/settings.rb:33:in `each'
Jul  6 12:00:45 repomirror dynflow-sidekiq@orchestrator: /usr/share/foreman/config/settings.rb:33:in `<top (required)>'

I see a bunch of pulp tasks looping between stopping and starting–even after a “foreman-maintain service stop” has been executed.

Jul  6 14:53:46 repomirror systemd: Started Pulp RQ Worker.
Jul  6 14:53:46 repomirror systemd: Stopped Pulp RQ Worker.
Jul  6 14:53:46 repomirror systemd: Started Pulp RQ Worker.
Jul  6 14:53:49 repomirror pulpcore-worker-3: Error 111 connecting to localhost:6379. Connection refused.
Jul  6 14:53:50 repomirror pulpcore-worker-1: Error 111 connecting to localhost:6379. Connection refused.
Jul  6 14:53:50 repomirror pulpcore-worker-4: Error 111 connecting to localhost:6379. Connection refused.
Jul  6 14:53:50 repomirror pulpcore-worker-2: Error 111 connecting to localhost:6379. Connection refused.
Jul  6 14:53:50 repomirror systemd: pulpcore-worker@3.service: main process exited, code=exited, status=1/FAILURE
Jul  6 14:53:50 repomirror systemd: Unit pulpcore-worker@3.service entered failed state.
Jul  6 14:53:50 repomirror systemd: pulpcore-worker@3.service failed.
Jul  6 14:53:50 repomirror systemd: pulpcore-worker@1.service: main process exited, code=exited, status=1/FAILURE
Jul  6 14:53:50 repomirror systemd: Unit pulpcore-worker@1.service entered failed state.
Jul  6 14:53:50 repomirror systemd: pulpcore-worker@1.service failed.
Jul  6 14:53:50 repomirror systemd: pulpcore-worker@2.service: main process exited, code=exited, status=1/FAILURE
Jul  6 14:53:50 repomirror systemd: Unit pulpcore-worker@2.service entered failed state.
Jul  6 14:53:50 repomirror systemd: pulpcore-worker@2.service failed.
Jul  6 14:53:50 repomirror systemd: pulpcore-worker@4.service: main process exited, code=exited, status=1/FAILURE
Jul  6 14:53:50 repomirror systemd: Unit pulpcore-worker@4.service entered failed state.
Jul  6 14:53:50 repomirror systemd: pulpcore-worker@4.service failed.
Jul  6 14:53:53 repomirror systemd: pulpcore-worker@3.service holdoff time over, scheduling restart.
Jul  6 14:53:53 repomirror systemd: pulpcore-worker@1.service holdoff time over, scheduling restart.
Jul  6 14:53:53 repomirror systemd: pulpcore-worker@4.service holdoff time over, scheduling restart.
Jul  6 14:53:53 repomirror systemd: pulpcore-worker@2.service holdoff time over, scheduling restart.
Jul  6 14:53:53 repomirror systemd: Stopped Pulp RQ Worker.
Jul  6 14:53:53 repomirror systemd: Started Pulp RQ Worker.
Jul  6 14:53:53 repomirror systemd: Stopped Pulp RQ Worker.
Jul  6 14:53:53 repomirror systemd: Started Pulp RQ Worker.
Jul  6 14:53:53 repomirror systemd: Stopped Pulp RQ Worker.
Jul  6 14:53:53 repomirror systemd: Started Pulp RQ Worker.
Jul  6 14:53:53 repomirror systemd: Stopped Pulp RQ Worker.
Jul  6 14:53:53 repomirror systemd: Started Pulp RQ Worker.

That feels like a syntax error in one of the configuration files. Have you made any changes by hand?

No, only changes made was through the web interface. And it’s stuff related to adding yum repos into products.

Even I am facing the same issue…any info or any help for fixing this issue.

OS : “Red Hat Enterprise Linux Server” ( 7.9)

Yeah, the only thing I’ve managed to get to restart is postgresql. For whatever reason the directory in /var/run got removed. Re-created it, assigned the proper permissions and then ran “restorecon /var/run/postgresql”

As for pulp or dynaflow. No matter what I’ve tried, they have refused to start or show why they won’t start.

1 Like

I am facing the same issue it seems, anyone get any solutions for it??