Getting error ' There was an issue with the backend service pulp: Not all necessary pulp workers running at https://theforeman.example.com/pulp/api/v2/."

enoerror · January 3, 2018, 1:12pm

Hi all,

I’m having an issue with katello and pulp. I have deployed a vagrant vm for testing katello. After installing it with foreman, when I try to create a repository, it is failing with the following error:

[root@theforeman .hammer]# hammer repository create  --name='EPEL 7 - x86_64' --organization "Default Organization"  --product='Extra Packages for Enterprise Linux'  --content-type='yum'  --publish-via-http=true  --url=http://dl.fedoraproject.org/pub/epel/7/x86_64/  --checksum-type=sha256  --gpg-key=GPG-EPEL-7    
Could not create the repository:
  There was an issue with the backend service pulp: Not all necessary pulp workers running at https://theforeman.example.com/pulp/api/v2/.
[root@theforeman .hammer]#

I have tried with other repositories as well, such as CentOS but all fail with the same error. Note that this happens irrespective of whether I try from CLI or from GUI.

What I was expecting to happen is … The repository should be created.

Here’s the debugging I’ve done so far … This is the sequence of commands if required (when trying from CLI):

[root@theforeman .hammer]# hammer settings set --name default_proxy_download_policy --value immediate
Setting [default_proxy_download_policy] updated to [immediate]

[root@theforeman .hammer]# hammer settings list | grep -i download
default_download_policy                                | on_demand                                                              | Default download policy for repositories (either 'immediate', 'on_demand', or...
default_proxy_download_policy                          | immediate                                                              | Default download policy for Smart Proxy syncs (either 'inherit', immediate', ...
[root@theforeman .hammer]# 
[root@theforeman .hammer]# 

[root@theforeman .hammer]# hammer sync-plan create --name 'Daily Sync' --description 'Daily Synchronization Plan' --organization "Default Organization" --interval daily --sync-date $(date +"%Y-%m-%d")" 00:00:00" --enabled yes
Sync plan created
[root@theforeman .hammer]# 

[root@theforeman .hammer]# hammer sync-plan list --organization "Default Organization"
---|------------|---------------------|----------|--------
ID | NAME       | START DATE          | INTERVAL | ENABLED
---|------------|---------------------|----------|--------
1  | Daily Sync | 2017/12/27 00:00:00 | daily    | yes    
---|------------|---------------------|----------|--------
[root@theforeman .hammer]# 


[root@theforeman .hammer]# wget -q https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7 -O /tmp/RPM-GPG-KEY-EPEL-7
[root@theforeman .hammer]# 
[root@theforeman .hammer]# hammer gpg create --key /tmp/RPM-GPG-KEY-EPEL-7 --name 'GPG-EPEL-7' --organization "Default Organization"
GPG Key created

[root@theforeman .hammer]# hammer gpg list --organization "Default Organization"
---|-----------
ID | NAME      
---|-----------
1  | GPG-EPEL-7
---|-----------
[root@theforeman .hammer]# 


[root@theforeman .hammer]# hammer product create --name='Extra Packages for Enterprise Linux' --organization "Default Organization"  --description 'Extra Packages for Enterprise Linux'
Product created


[root@theforeman .hammer]# hammer product list --organization "Default Organization"
---|-------------------------------------|-------------------------------------|----------------------|--------------|-----------
ID | NAME                                | DESCRIPTION                         | ORGANIZATION         | REPOSITORIES | SYNC STATE
---|-------------------------------------|-------------------------------------|----------------------|--------------|-----------
1  | centos-base                         | Base centos repository              | Default Organization | 0            |           
2  | Extra Packages for Enterprise Linux | Extra Packages for Enterprise Linux | Default Organization | 0            |           
---|-------------------------------------|-------------------------------------|----------------------|--------------|-----------
[root@theforeman .hammer]# 

[root@theforeman .hammer]# hammer repository create  --name='EPEL 7 - x86_64' --organization "Default Organization"  --product='Extra Packages for Enterprise Linux'  --content-type='yum'  --publish-via-http=true  --url=http://dl.fedoraproject.org/pub/epel/7/x86_64/  --checksum-type=sha256  --gpg-key=GPG-EPEL-7    
Could not create the repository:
  There was an issue with the backend service pulp: Not all necessary pulp workers running at https://theforeman.example.com/pulp/api/v2/.
[root@theforeman .hammer]#

Thanks!

ehelms · January 3, 2018, 2:15pm

First, you can try to work around the error by issuing a ‘katello-service restart’. Second, can you tell me which Vm you spun up? nightly? 3.4? 3.5?

enoerror · January 3, 2018, 5:54pm

Hi Eric,

I just attempted to create the repository right now and lo’ and behold, it worked:

[root@theforeman ~]# hammer product list --organization "Default Organization"
---|-------------------------------------|-------------------------------------|----------------------|--------------|-----------
ID | NAME                                | DESCRIPTION                         | ORGANIZATION         | REPOSITORIES | SYNC STATE
---|-------------------------------------|-------------------------------------|----------------------|--------------|-----------
1  | centos-base                         | Base centos repository              | Default Organization | 0            |           
2  | Extra Packages for Enterprise Linux | Extra Packages for Enterprise Linux | Default Organization | 0            |           
---|-------------------------------------|-------------------------------------|----------------------|--------------|-----------
[root@theforeman ~]# hammer repository create  --name='EPEL 7 - x86_64' --organization "Default Organization"  --product='Extra Packages for Enterprise Linux'  --content-type='yum'  --publish-via-http=true  --url=http://dl.fedoraproject.org/pub/epel/7/x86_64/  --checksum-type=sha256  --gpg-key=GPG-EPEL-7
Repository created
[root@theforeman ~]#

When I checked katello-service status I noticed there are few exited daemons. What do you think of this output:

[root@theforeman ~]# katello-service status
/usr/sbin/service-wait mongod status
● mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-03 15:13:12 UTC; 2h 36min ago
  Process: 1040 ExecStart=/usr/bin/mongod $OPTIONS run (code=exited, status=0/SUCCESS)
 Main PID: 1280 (mongod)
   CGroup: /system.slice/mongod.service
           └─1280 /usr/bin/mongod --quiet -f /etc/mongod.conf run

Jan 03 15:12:44 theforeman.example.com systemd[1]: Starting High-performance, schema-free document-oriented database...
Jan 03 15:12:58 theforeman.example.com mongod[1040]: about to fork child process, waiting until server is ready for connections.
Jan 03 15:12:58 theforeman.example.com mongod[1040]: forked process: 1280
Jan 03 15:13:12 theforeman.example.com systemd[1]: Started High-performance, schema-free document-oriented database.
/usr/sbin/service-wait postgresql status
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/etc/systemd/system/postgresql.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-03 15:13:05 UTC; 2h 36min ago
  Process: 1063 ExecStart=/usr/bin/pg_ctl start -D ${PGDATA} -s -o -p ${PGPORT} -w -t 300 (code=exited, status=0/SUCCESS)
  Process: 1029 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
 Main PID: 1085 (postgres)
   CGroup: /system.slice/postgresql.service
           ├─ 1085 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
           ├─ 1187 postgres: logger process                        
           ├─ 1284 postgres: checkpointer process                  
           ├─ 1285 postgres: writer process                        
           ├─ 1286 postgres: wal writer process                    
           ├─ 1287 postgres: autovacuum launcher process           
           ├─ 1288 postgres: stats collector process               
           ├─ 4262 postgres: candlepin candlepin 127.0.0.1(35922) idl
           ├─ 4263 postgres: candlepin candlepin 127.0.0.1(35924) idl
           ├─ 4264 postgres: candlepin candlepin 127.0.0.1(35926) idl
           ├─ 4327 postgres: foreman foreman [local] idle          
           ├─ 4339 postgres: foreman foreman [local] idle          
           ├─ 4340 postgres: foreman foreman [local] idle          
           ├─ 4361 postgres: foreman foreman [local] idle          
           ├─ 4365 postgres: foreman foreman [local] idle          
           ├─ 4533 postgres: foreman foreman [local] idle          
           ├─ 4540 postgres: foreman foreman [local] idle          
           ├─ 4541 postgres: foreman foreman [local] idle          
           ├─ 6506 postgres: foreman foreman [local] idle          
           ├─16324 postgres: candlepin candlepin 127.0.0.1(40090) idl
           ├─16325 postgres: candlepin candlepin 127.0.0.1(40092) idl
           ├─16326 postgres: candlepin candlepin 127.0.0.1(40094) idl
           ├─16327 postgres: candlepin candlepin 127.0.0.1(40096) idl
           └─16328 postgres: candlepin candlepin 127.0.0.1(40098) idl

Jan 03 15:12:43 theforeman.example.com systemd[1]: Starting PostgreSQL database server...
Jan 03 15:13:05 theforeman.example.com systemd[1]: Started PostgreSQL database server.
/usr/sbin/service-wait qpidd status
● qpidd.service - An AMQP message broker daemon.
   Loaded: loaded (/usr/lib/systemd/system/qpidd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-03 15:12:44 UTC; 2h 37min ago
     Docs: man:qpidd(1)
           http://qpid.apache.org/
 Main PID: 1044 (qpidd)
   CGroup: /system.slice/qpidd.service
           └─1044 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf

Jan 03 15:12:44 theforeman.example.com systemd[1]: Started An AMQP message broker daemon..
Jan 03 15:12:44 theforeman.example.com systemd[1]: Starting An AMQP message broker daemon....
/usr/sbin/service-wait qdrouterd status
● qdrouterd.service - Qpid Dispatch router daemon
   Loaded: loaded (/usr/lib/systemd/system/qdrouterd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-03 15:12:29 UTC; 2h 37min ago
 Main PID: 571 (qdrouterd)
   CGroup: /system.slice/qdrouterd.service
           └─571 /usr/sbin/qdrouterd -c /etc/qpid-dispatch/qdrouterd.conf

Jan 03 15:12:29 theforeman.example.com systemd[1]: Started Qpid Dispatch router daemon.
Jan 03 15:12:29 theforeman.example.com systemd[1]: Starting Qpid Dispatch router daemon...
/usr/sbin/service-wait squid status
● squid.service - Squid caching proxy
   Loaded: loaded (/usr/lib/systemd/system/squid.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-03 15:12:45 UTC; 2h 37min ago
  Process: 3127 ExecReload=/usr/sbin/squid $SQUID_OPTS -k reconfigure -f $SQUID_CONF (code=exited, status=0/SUCCESS)
  Process: 1059 ExecStart=/usr/sbin/squid $SQUID_OPTS -f $SQUID_CONF (code=exited, status=0/SUCCESS)
  Process: 1043 ExecStartPre=/usr/libexec/squid/cache_swap.sh (code=exited, status=0/SUCCESS)
 Main PID: 1067 (squid)
   CGroup: /system.slice/squid.service
           ├─1067 /usr/sbin/squid -f /etc/squid/squid.conf
           └─1069 (squid-1) -f /etc/squid/squid.conf

Jan 03 15:12:44 theforeman.example.com systemd[1]: Starting Squid caching proxy...
Jan 03 15:12:45 theforeman.example.com systemd[1]: Started Squid caching proxy.
Jan 03 15:12:45 theforeman.example.com squid[1067]: Squid Parent: will start 1 kids
Jan 03 15:12:45 theforeman.example.com squid[1067]: Squid Parent: (squid-1) process 1069 started
Jan 03 15:13:17 theforeman.example.com systemd[1]: Reloaded Squid caching proxy.
/usr/sbin/service-wait tomcat status
● tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-03 15:12:44 UTC; 2h 37min ago
 Main PID: 1041 (java)
   CGroup: /system.slice/tomcat.service
           └─1041 /usr/lib/jvm/jre/bin/java -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Jan 03 15:15:06 theforeman.example.com server[1041]: Jan 03, 2018 3:15:06 PM org.apache.catalina.startup.HostConfig deployDirectory
Jan 03 15:15:06 theforeman.example.com server[1041]: INFO: Deployment of web application directory /var/lib/tomcat/webapps/candlepin has finished in 85,510 ms
Jan 03 15:15:06 theforeman.example.com server[1041]: Jan 03, 2018 3:15:06 PM org.apache.coyote.AbstractProtocol start
Jan 03 15:15:06 theforeman.example.com server[1041]: INFO: Starting ProtocolHandler ["http-bio-8080"]
Jan 03 15:15:06 theforeman.example.com server[1041]: Jan 03, 2018 3:15:06 PM org.apache.coyote.AbstractProtocol start
Jan 03 15:15:06 theforeman.example.com server[1041]: INFO: Starting ProtocolHandler ["http-bio-8443"]
Jan 03 15:15:06 theforeman.example.com server[1041]: Jan 03, 2018 3:15:06 PM org.apache.coyote.AbstractProtocol start
Jan 03 15:15:06 theforeman.example.com server[1041]: INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Jan 03 15:15:06 theforeman.example.com server[1041]: Jan 03, 2018 3:15:06 PM org.apache.catalina.startup.Catalina start
Jan 03 15:15:06 theforeman.example.com server[1041]: INFO: Server startup in 86610 ms
/usr/sbin/service-wait pulp_workers status
● pulp_workers.service - Pulp Celery Workers
   Loaded: loaded (/usr/lib/systemd/system/pulp_workers.service; enabled; vendor preset: disabled)
   Active: active (exited) since Wed 2018-01-03 15:12:45 UTC; 2h 37min ago
  Process: 1033 ExecStart=/usr/bin/python -m pulp.server.async.manage_workers start (code=exited, status=0/SUCCESS)
 Main PID: 1033 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/pulp_workers.service

Jan 03 15:12:43 theforeman.example.com systemd[1]: Starting Pulp Celery Workers...
Jan 03 15:12:45 theforeman.example.com systemd[1]: Started Pulp Celery Workers.
/usr/sbin/service-wait pulp_resource_manager status
● pulp_resource_manager.service - Pulp Resource Manager
   Loaded: loaded (/usr/lib/systemd/system/pulp_resource_manager.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-03 15:12:43 UTC; 2h 37min ago
 Main PID: 1031 (celery)
   CGroup: /system.slice/pulp_resource_manager.service
           ├─1031 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid
           └─3818 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid

Jan 03 15:13:38 theforeman.example.com celery[1031]: - ** ---------- .> transport:   qpid://localhost:5671//
Jan 03 15:13:38 theforeman.example.com celery[1031]: - ** ---------- .> results:     disabled
Jan 03 15:13:38 theforeman.example.com celery[1031]: - *** --- * --- .> concurrency: 1 (prefork)
Jan 03 15:13:38 theforeman.example.com celery[1031]: -- ******* ----
Jan 03 15:13:38 theforeman.example.com celery[1031]: --- ***** ----- [queues]
Jan 03 15:13:38 theforeman.example.com celery[1031]: -------------- .> resource_manager exchange=resource_manager(direct) key=resource_manager
Jan 03 15:13:38 theforeman.example.com celery[1031]: .> resource_manager@theforeman.example.com.dq exchange=C.dq(direct) key=resource_manager@theforeman.example.com
Jan 03 15:13:38 theforeman.example.com pulp[1031]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jan 03 15:13:38 theforeman.example.com pulp[1031]: celery.worker.consumer:INFO: Connected to qpid://localhost:5671//
Jan 03 15:13:38 theforeman.example.com pulp[1031]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
/usr/sbin/service-wait pulp_streamer status
● pulp_streamer.service - The Pulp lazy content loading streamer
   Loaded: loaded (/usr/lib/systemd/system/pulp_streamer.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-03 15:12:43 UTC; 2h 37min ago
 Main PID: 1030 (pulp_streamer)
   CGroup: /system.slice/pulp_streamer.service
           └─1030 /usr/bin/python /usr/bin/pulp_streamer --nodaemon --syslog --prefix=pulp_streamer --pidfile= --python /usr/share/pulp/wsgi/streamer.tac

Jan 03 15:13:36 theforeman.example.com pulp_streamer[1030]: pulp.plugins.loader.manager:INFO: Loaded plugin docker_importer for types: docker_blob,docker_image,docker_manifest,docker_tag
Jan 03 15:13:36 theforeman.example.com pulp_streamer[1030]: pulp.plugins.loader.manager:INFO: Loaded plugin puppet_whole_repo_profiler for types: puppet_module
Jan 03 15:13:37 theforeman.example.com pulp_streamer[1030]: pulp.plugins.loader.manager:INFO: Loaded plugin yum_profiler for types: rpm,erratum
Jan 03 15:13:37 theforeman.example.com pulp_streamer[1030]: pulp.plugins.loader.manager:INFO: Loaded plugin yum for types: rpm
Jan 03 15:13:37 theforeman.example.com pulp_streamer[1030]: pulp.plugins.loader.manager:INFO: Loaded plugin rhui for types: rpm
Jan 03 15:13:38 theforeman.example.com pulp_streamer[1030]: [-] Log opened.
Jan 03 15:13:38 theforeman.example.com pulp_streamer[1030]: [-] twistd 12.2.0 (/usr/bin/python 2.7.5) starting up.
Jan 03 15:13:38 theforeman.example.com pulp_streamer[1030]: [-] reactor class: twisted.internet.epollreactor.EPollReactor.
Jan 03 15:13:38 theforeman.example.com pulp_streamer[1030]: [-] Site starting on 8751
Jan 03 15:13:38 theforeman.example.com pulp_streamer[1030]: [-] Starting factory <twisted.web.server.Site instance at 0x52843b0>
/usr/sbin/service-wait foreman-proxy status
● foreman-proxy.service - Foreman Proxy
   Loaded: loaded (/usr/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-03 15:13:28 UTC; 2h 36min ago
 Main PID: 1042 (ruby)
   CGroup: /system.slice/foreman-proxy.service
           └─1042 ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize

Jan 03 15:12:44 theforeman.example.com systemd[1]: Starting Foreman Proxy...
Jan 03 15:13:28 theforeman.example.com systemd[1]: Started Foreman Proxy.
/usr/sbin/service-wait pulp_celerybeat status
● pulp_celerybeat.service - Pulp's Celerybeat
   Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-03 15:12:43 UTC; 2h 37min ago
 Main PID: 1034 (celery)
   CGroup: /system.slice/pulp_celerybeat.service
           └─1034 /usr/bin/python /usr/bin/celery beat --app=pulp.server.async.celery_instance.celery --scheduler=pulp.server.async.scheduler.Scheduler

Jan 03 16:13:25 theforeman.example.com pulp[1034]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Jan 03 16:23:25 theforeman.example.com pulp[1034]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Jan 03 16:33:25 theforeman.example.com pulp[1034]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Jan 03 16:43:25 theforeman.example.com pulp[1034]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Jan 03 16:53:25 theforeman.example.com pulp[1034]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Jan 03 17:03:25 theforeman.example.com pulp[1034]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Jan 03 17:13:25 theforeman.example.com pulp[1034]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Jan 03 17:23:25 theforeman.example.com pulp[1034]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Jan 03 17:33:25 theforeman.example.com pulp[1034]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Jan 03 17:43:25 theforeman.example.com pulp[1034]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
/usr/sbin/service-wait httpd status
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-03 15:13:09 UTC; 2h 36min ago
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 1026 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─1026 /usr/sbin/httpd -DFOREGROUND
           ├─2505 (wsgi:pulp)     -DFOREGROUND
           ├─2506 (wsgi:pulp)     -DFOREGROUND
           ├─2507 (wsgi:pulp)     -DFOREGROUND
           ├─2508 (wsgi:pulp-cont -DFOREGROUND
           ├─2510 (wsgi:pulp-cont -DFOREGROUND
           ├─2511 (wsgi:pulp-cont -DFOREGROUND
           ├─2512 (wsgi:pulp_forg -DFOREGROUND
           ├─2513 PassengerWatchdog
           ├─2520 PassengerHelperAgent
           ├─2530 PassengerLoggingAgent
           ├─2540 /usr/sbin/httpd -DFOREGROUND
           ├─2541 /usr/sbin/httpd -DFOREGROUND
           ├─2542 /usr/sbin/httpd -DFOREGROUND
           ├─2543 /usr/sbin/httpd -DFOREGROUND
           ├─2544 /usr/sbin/httpd -DFOREGROUND
           ├─2545 /usr/sbin/httpd -DFOREGROUND
           ├─2546 /usr/sbin/httpd -DFOREGROUND
           ├─2547 /usr/sbin/httpd -DFOREGROUND
           └─4529 Passenger RackApp: /usr/share/foreman                     

Jan 03 15:13:40 theforeman.example.com pulp[2505]: gofer.messaging.adapter.qpid.connection:INFO: opened: qpid+ssl://localhost:5671
Jan 03 15:13:40 theforeman.example.com pulp[2505]: gofer.messaging.adapter.connect:INFO: connected: qpid+ssl://localhost:5671
Jan 03 15:13:40 theforeman.example.com pulp[2507]: gofer.messaging.adapter.qpid.connection:INFO: opened: qpid+ssl://localhost:5671
Jan 03 15:13:40 theforeman.example.com pulp[2507]: gofer.messaging.adapter.connect:INFO: connected: qpid+ssl://localhost:5671
Jan 03 16:19:15 theforeman.example.com pulp[2505]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jan 03 16:39:14 theforeman.example.com pulp[2505]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jan 03 16:39:15 theforeman.example.com pulp[2506]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jan 03 16:39:15 theforeman.example.com pulp[2506]: py.warnings:WARNING: (2506-27616) /usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py:552: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
Jan 03 16:39:15 theforeman.example.com pulp[2506]: py.warnings:WARNING: (2506-27616)   TaskStatus._get_collection().update({'task_id': task_id}, update, upsert=True)
Jan 03 16:39:15 theforeman.example.com pulp[2506]: py.warnings:WARNING: (2506-27616)
/usr/sbin/service-wait puppetserver status
● puppetserver.service - puppetserver Service
   Loaded: loaded (/usr/lib/systemd/system/puppetserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-03 15:15:30 UTC; 2h 34min ago
  Process: 1023 ExecStart=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver start (code=exited, status=0/SUCCESS)
 Main PID: 1090 (java)
   CGroup: /system.slice/puppetserver.service
           └─1090 /usr/bin/java -Xms2G -Xmx2G -XX:MaxPermSize=256m -Djava.security.egd=/dev/urandom -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/services.d/,/opt/puppetlabs/server/apps/puppetserver/config/services.d/ --restart-file /opt/puppetlabs/server/data/puppetserver/restartcounter

Jan 03 15:12:43 theforeman.example.com systemd[1]: Starting puppetserver Service...
Jan 03 15:12:52 theforeman.example.com puppetserver[1023]: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Jan 03 15:15:30 theforeman.example.com systemd[1]: Started puppetserver Service.
/usr/sbin/service-wait foreman-tasks status
● foreman-tasks.service - Foreman jobs daemon
   Loaded: loaded (/usr/lib/systemd/system/foreman-tasks.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-03 15:14:07 UTC; 2h 35min ago
     Docs: https://github.com/theforeman/foreman-tasks
  Process: 1024 ExecStart=/usr/bin/foreman-tasks start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/foreman-tasks.service
           ├─4003 dynflow_executor                 
           └─4006 dynflow_executor_monitor         

Jan 03 15:12:43 theforeman.example.com systemd[1]: Starting Foreman jobs daemon...
Jan 03 15:14:01 theforeman.example.com foreman-tasks[1024]: /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
Jan 03 15:14:01 theforeman.example.com foreman-tasks[1024]: /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
Jan 03 15:14:02 theforeman.example.com foreman-tasks[1024]: Dynflow Executor: start in progress
Jan 03 15:14:06 theforeman.example.com foreman-tasks[1024]: dynflow_executor: process with pid 4003 started.
Jan 03 15:14:07 theforeman.example.com systemd[1]: Started Foreman jobs daemon.
Success!
[root@theforeman ~]#

As for the vm, I am not using the foreman vagrant machines. I have spun up a centos7 vm myself. Katello version is 3.5

As far as the problem is concerned, what could have caused the issue? Or how do you think I can reproduce it if you want to delve deeper?

metalcated · February 14, 2018, 2:37am

I experienced this same problem when I tried to apply a package from Content Hosts using the katello-agent as the delivery system.

Mongo db service ends up crashing when attempting to apply errata from within Content Hosts and then shortly there after pulp_celerybeat service fails.

bastion 5.1.1
foreman-tasks 0.9.6
foreman_bootdisk 10.0.1
foreman_discovery 9.1.5
foreman_docker 3.2.1
foreman_openscap 0.7.11
foreman_remote_execution 1.3.7
foreman_virt_who_configure 0.1.5
katello 3.4.5

https://pastebin.com/5EvQbyMu

Thanks

metalcated · February 14, 2018, 2:38am

See my above comment. Thanks

ehelms · February 16, 2018, 1:24am

How much memory does the box have? This is odd to see MongoDB simply unavailable. If you restart services to a “green” state, and then re-run the errata install does MongoDB then crash again?

metalcated · February 16, 2018, 9:34pm

12GB RAM

Yes that is seemingly what happens and actually I went in and double checked file/directory permissions and noticed some of the files were not owned by mongodb for some odd reason. I set the permissions and restarted the service. I haven’t noticed a crash since I did that. Not too sure how the permissions got out of wack.