Couldn't connect to the server: undefined method `to_sym' for nil:NilClass

Support
1

Problem:

foreman-maintain health check
Running ForemanMaintain::Scenario::FilteredScenario

Check number of fact names in database: [OK]

Check whether all services are running: [OK]

Check whether all services are running using the ping call: [FAIL]
Couldn’t connect to the server: undefined method `to_sym’ for nil:NilClass

Continue with step [Restart applicable services]?, [y(yes), n(no), q(quit)] y
Restart applicable services:

Stopping the following service(s):
rh-mongodb34-mongod, rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-resource-manager, qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, pulpcore-worker@*, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
| stopping foreman
Warning: Stopping foreman.service, but it can still be activated by:
foreman.socket
\ All services stopped

Starting the following service(s):
rh-mongodb34-mongod, rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-resource-manager, qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, pulpcore-worker@*, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
| All services started

  • Try 1/5: checking status of hammer ping
    Couldn’t connect to the server: undefined method to_sym' for nil:NilClass \ Try 2/5: checking status of hammer ping Couldn't connect to the server: undefined method to_sym’ for nil:NilClass
    \ Try 3/5: checking status of hammer ping
    Couldn’t connect to the server: undefined method `to_sym’ for nil:NilClass
  • Try 4/5: checking status of hammer ping
    Couldn’t connect to the server: undefined method `to_sym’ for nil:NilClass

tail -f /var/log/foreman/production.log

/opt/theforeman/tfm/root/usr/share/gems/gems/stomp-1.4.9/lib/stomp/client.rb:99:in initialize' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.pre.master/app/lib/katello/messaging/stomp_connection.rb:69:in new’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.pre.master/app/lib/katello/messaging/stomp_connection.rb:69:in client' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.pre.master/app/lib/katello/messaging/stomp_connection.rb:43:in subscribe’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.pre.master/app/services/katello/candlepin_event_listener.rb:37:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.pre.master/app/services/katello/event_daemon.rb:33:in block in check_services’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.pre.master/app/services/katello/event_daemon.rb:23:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.pre.master/app/services/katello/event_daemon.rb:23:in check_services’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.pre.master/app/services/katello/event_daemon.rb:16:in block (2 levels) in start' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in wrap’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.pre.master/app/services/katello/event_daemon.rb:15:in block in start' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.pre.master/app/services/katello/event_daemon.rb:14:in loop’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.pre.master/app/services/katello/event_daemon.rb:14:in start' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.pre.master/app/services/katello/event_daemon.rb:119:in block in start_monitor_thread’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’
2020-09-26T09:07:36 [I|app|f640be69] Started GET “/notification_recipients” for 127.0.0.1 at 2020-09-26 09:07:36 -0400
2020-09-26T09:07:36 [I|app|f640be69] Processing by NotificationRecipientsController#index as JSON
2020-09-26T09:07:36 [I|app|f640be69] Completed 200 OK in 9ms (Views: 0.1ms | ActiveRecord: 1.6ms | Allocations: 2172)

Expected outcome:

Should ping

Foreman and Proxy versions:

Foreman and proxy Nightly
Katello 3.18.0.pre.master

Foreman and Proxy plugin versions:

Distribution and version:

CENTOS 7.8

Other relevant data:

2

sorry I can’t read it , can someone help me in it ?

3

foreman-maintain advanced procedure run content-prepare
Running ForemanMaintain::Scenario

Prepare content for Pulp 3: [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
rake aborted!
Not all the services have been started. Check the status report above and try again.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.pre.master/lib/katello/tasks/reimport.rake:10:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
Tasks: TOP => katello:pulp3_migration => katello:check_ping
(See full trace by running task with --trace)
{:services=>
{:candlepin=>{:status=>“ok”, :duration_ms=>“43”},
:candlepin_auth=>{:status=>“ok”, :duration_ms=>“44”},
:foreman_tasks=>{:status=>“ok”, :duration_ms=>“4”},
:katello_events=>
{:status=>“ok”, :message=>“0 Processed, 0 Failed”, :duration_ms=>“0”},
:candlepin_events=>
{:status=>“FAIL”, :message=>“Not running”, :duration_ms=>“0”},
:pulp3=>{:status=>“ok”, :duration_ms=>“97”},
:pulp=>{:status=>“ok”, :duration_ms=>“141”},
:pulp_auth=>{:status=>“ok”, :duration_ms=>“94”}},
:status=>“FAIL”}

Scenario [ForemanMaintain::Scenario] failed.

The following steps ended up in failing state:

[content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-prepare”

4

https://bugzilla.redhat.com/show_bug.cgi?id=1875357

need to upgrade candlepin to candlepin-3.1.21-1.el7sat.noarch

5

I upgrade candlepin to candlepin-3.1.21-1.el7.noarch but still below error.

candlepin_events":{“status”:“FAIL”,“message”:“Not running”,“duration_ms”:“0”}

Full Error:

[2020-09-28 11:34:42-0400 #3638] DEBUG – : Response: 200, {“status”:“FAIL”,“services”:{“candlepin”:{“status”:“ok”,“duration_ms”:“41”},“candlepin_auth”:{“status”:“ok”,“duration_ms”:“44”},“foreman_tasks”:{“status”:“ok”,“duration_ms”:“6”},“katello_events”:{“status”:“ok”,“message”:“0 Processed, 0 Failed”,“duration_ms”:“0”},“candlepin_events”:{“status”:“FAIL”,“message”:“Not running”,“duration_ms”:“0”},“pulp3”:{“status”:“ok”,“duration_ms”:“71”},“pulp”:{“status”:“ok”,“duration_ms”:“154”},“pulp_auth”:{“status”:“ok”,“duration_ms”:“56”}}}

6

Hi @masadrasheed !

You’re running the ‘nightly’ version of Katello which is prone to this kind of instability, and I would not recommend it for production usage. Disclaimer said, I’m here to help!

As you discovered this issue should be resolved in candlepin-3.1.21 which you are now running. I’m curious: when did you install your Katello? That version of Candlepin has been in the nightly repos for a few weeks. Can you do a foreman-maintain service restart and see if that fixes your problem?

7

For some additional context, I just spun up a fresh nightly installation and I don’t see this problem with candlepin-3.1.21 which comes installed by default

8

you did a upgrade ?

10

2020-09-28 17:17:39,534 [thread=Thread-11 (activemq-netty-threads)] [=, org=, csid=] WARN org.apache.activemq.artemis.core.server - AMQ222208: SSL handshake failed for client from /127.0.0.1:46740: sun.security.validator.ValidatorException: Netscape cert type does not permit use for SSL client.
2020-09-28 17:17:49,481 [thread=Thread-0 (ActiveMQ-scheduled-threads)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.server - AMQ224088: Timeout (10 seconds) on acceptor “stomp” during protocol handshake with /127.0.0.1:46740 has occurred.

11

Thank you for your response and I understand this is not for production , currently testing.

I had foreman RC version but was facing some issues with web console cockpit SSO so I thought may be i find new feature in nightly build so I updated to nightly but find out it cockpit SSO only work with FQDN not IP (foreman url) otherwise give error “sorry try again” , we need to add that in manual.

Also I restart etc but this issue remain some handshake issue.

ERROR org.apache.activemq.artemis.core.server - AMQ224088: Timeout (10 seconds) on acceptor “stomp” during protocol handshake with /127.0.0.1:46740 has occurred.

https://bugzilla.redhat.com/show_bug.cgi?id=1838923

12

I did a clean install, not an upgrade. What version did you upgrade from? Can you try running the installer again? Any errors in doing so? That error leads me to believe that the installer didn’t finish properly

13

I run installer multiple times and no errors and finish properly

14

Thanks. Two questions:

  1. Which version of foreman did you upgrade from? I didn’t see it mentioned above. Asking since we only support upgrading from the last version - intermediates cannot be skipped.

  2. Do you have the following 2 files on your Katello with the same permissions as below?

[root@centos7-katello-3-18 vagrant]# ls -lahZ /etc/pki/katello/private/java-client.key 
-r--r-----. root foreman system_u:object_r:cert_t:s0      /etc/pki/katello/private/java-client.key
[root@centos7-katello-3-18 vagrant]# ls -lahZ /etc/pki/katello/certs/java-client.crt 
-r--r-----. root foreman system_u:object_r:cert_t:s0      /etc/pki/katello/certs/java-client.crt
15

Yes I have , but I have one question do these keys need to be sync with some other foreman directory because I generate new keys after upgrade and apply while checking some article online.

Also I upgrade from 2.2 rc3 to nightly

16

Thank you. Something definitely went wrong somewhere. What article did you follow? Is it possible to revert back to a time before you regenerated the certificates and upgrade again? Have you tried regenerating the certificates again after upgrading? Just throwing some ideas out there :slight_smile:

17

Its not possible to revert , I think I just update foreman proxy content package (because after upgrade it was still using old package) to nightly and then regenerate Certificates make tar of them and then reload those certificates in foreman.