Error 'There was an issue with the backend service pulp3: Not all necessary pulp workers running at https://centos8.example.com/pulp/api/v3.'

Problem:

There was an issue with the backend service pulp3: Not all necessary pulp workers running at https://centos8.example.com/pulp/api/v3.

Hi guys.
I get this error when trying to add repo into Foreman. I have searched around but there is not much related issues around.
What I have done as below:

  • Follow official guide to install. And it completed successfully without any errors.
  • Restart services: foreman-maintain service restart, systemctl restart pulpcore*
  • Health check: foreman-maintain health check
  • Change distros.

Any help is appreciated.

Expected outcome:

  • Can add repo into Foreman.

Foreman and Proxy versions:
Foreman 3.0 & Katello 4.0

Foreman and Proxy plugin versions:

Distribution and version:

  • Centos8, Redhat 7.9, Redhat 8.4
  • All are Virtualbox VMs w/: Disk: 40GB - Memory: 12GB - vCPU: 3
    Other relevant data:
[root@centos8 ~]# hammer ping
database:
    Status:          ok
    Server Response: Duration: 0ms
candlepin:
    Status:          ok
    Server Response: Duration: 63ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 37ms
candlepin_events:
    Status:          ok
    message:         1 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp3:
    Status:          FAIL
    Server Response: Message: Not all necessary pulp workers running at https://centos8.example.com/pulp/api/v3.
foreman_tasks:
    Status:          ok
    Server Response: Duration: 8ms
> [root@centos8 ~]# foreman-maintain health check
> Running ForemanMaintain::Scenario::FilteredScenario
> ================================================================================
> Check number of fact names in database:                               [OK]
> --------------------------------------------------------------------------------
> Check to verify no empty CA cert requests exist:                      [OK]
> --------------------------------------------------------------------------------
> Check whether all services are running:                               [FAIL]
> Following services are not running: puppetserver
> --------------------------------------------------------------------------------
> Continue with step [Restart applicable services]?, [y(yes), n(no)] y
> Restart applicable services:
> 
> Stopping the following service(s):
> puppetserver
> / All services stopped
> 
> Starting the following service(s):
> puppetserver
> - All services started
> | Try 1/5: checking status of hammer ping
> Couldn't connect to the server: undefined method `to_sym' for nil:NilClass
> | Try 2/5: checking status of hammer ping
> Couldn't connect to the server: undefined method `to_sym' for nil:NilClass
> - Try 3/5: checking status of hammer ping
> Couldn't connect to the server: undefined method `to_sym' for nil:NilClass
> - Try 4/5: checking status of hammer ping
> Couldn't connect to the server: undefined method `to_sym' for nil:NilClass
> | Try 5/5: checking status of hammer ping                             [FAIL]
> Server response check failed!
> --------------------------------------------------------------------------------
> Rerunning the check after fix procedure
> Check whether all services are running:                               [OK]
> --------------------------------------------------------------------------------
> Check whether all services are running using the ping call:           [FAIL]
> Couldn't connect to the server: undefined method `to_sym' for nil:NilClass