I made a fresh installation of Katello 2.3 on a CentOS 7 server.
I added a product called CentOS 6. I added a repository called 'CentOS 6
Base' and pointed it to http://mirror.centos.org/centos/6/os/x86_64/
I then attempted to sync.
It downloaded quickly and then got stuck at 53% saying :
New packages: 6574/6575 (5.37 GB/5.37 GB).
It has been that way for over 6 hours now.
The dynflow console list of worlds screen shows all executors as valid when checked.
If I go to the running steps tab of the task I get this :
Actions::Pulp::Repository::Sync
State: waiting for Pulp to finish the task
Input:
{"pulp_id"=>"My_Company-CentOS_6-CentOS_6_Base",
"task_id"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
Output:
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.sync.sync",
"_href"=>"/pulp/api/v2/tasks/dc079296-009e-4b65-907b-35c911569b94/",
"task_id"=>"dc079296-009e-4b65-907b-35c911569b94",
"tags"=>
["pulp:repository:My_Company-CentOS_6-CentOS_6_Base",
"pulp:action:sync"],
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>"2015-12-15T22:48:23Z",
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>
{"yum_importer"=>
{"content"=>
{"size_total"=>5770827028,
"items_left"=>1,
"items_total"=>6575,
"state"=>"IN_PROGRESS",
"size_left"=>1772712,
"details"=>
{"rpm_total"=>6575,
"rpm_done"=>6574,
"drpm_total"=>0,
"drpm_done"=>0},
"error_details"=>[]},
"comps"=>{"state"=>"NOT_STARTED"},
"distribution"=>
{"items_total"=>0,
"state"=>"NOT_STARTED",
"error_details"=>[],
"items_left"=>0},
"errata"=>{"state"=>"NOT_STARTED"},
"metadata"=>{"state"=>"FINISHED"}}},
"queue"=>
"reserved_resource_worker-2@katello1.mydomain.net.dq",
"state"=>"running",
"worker_name"=>
"reserved_resource_worker-2@katello1.mydomain.net",
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"567098b789e5bab25007e52d"},
"id"=>"567098b7d280d90ac331e303"}],
"poll_attempts"=>{"total"=>1581, "failed"=>0}}
Here is the output of katello-service status
···
------------------------------------------------[root@katello1 ~]# katello-service status
Redirecting to /bin/systemctl status mongod.service
● mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor
preset: disabled)
Active: active (running) since Tue 2015-12-15 13:04:34 PST; 8h ago
Process: 1426 ExecStart=/usr/bin/mongod $OPTIONS run (code=exited,
status=0/SUCCESS)
Main PID: 1975 (mongod)
CGroup: /system.slice/mongod.service
└─1975 /usr/bin/mongod --quiet -f /etc/mongod.conf run
Dec 15 13:04:33 katello1.mydomain.net systemd[1]: Starting
High-performance, schema-free document-oriented database…
Dec 15 13:04:34 katello1.mydomain.net mongod[1426]: about to fork child
process, waiting until server is ready for connections.
Dec 15 13:04:34 katello1.mydomain.net mongod[1426]: forked process: 1975
Dec 15 13:04:34 katello1.mydomain.net systemd[1]: Started High-performance,
schema-free document-oriented database.
Redirecting to /bin/systemctl status qpidd.service
● qpidd.service - An AMQP message broker daemon.
Loaded: loaded (/usr/lib/systemd/system/qpidd.service; enabled; vendor
preset: disabled)
Active: active (running) since Tue 2015-12-15 13:04:33 PST; 8h ago
Docs: man:qpidd(1)
http://qpid.apache.org/
Main PID: 1428 (qpidd)
CGroup: /system.slice/qpidd.service
└─1428 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf
Dec 15 13:04:33 katello1.mydomain.net systemd[1]: Started An AMQP message
broker daemon…
Dec 15 13:04:33 katello1.mydomain.net systemd[1]: Starting An AMQP message
broker daemon…
Redirecting to /bin/systemctl status qdrouterd.service
● qdrouterd.service - Qpid Dispatch router daemon
Loaded: loaded (/usr/lib/systemd/system/qdrouterd.service; enabled; vendor
preset: disabled)
Active: active (running) since Tue 2015-12-15 13:04:33 PST; 8h ago
Main PID: 1427 (qdrouterd)
CGroup: /system.slice/qdrouterd.service
└─1427 /usr/sbin/qdrouterd -c /etc/qpid-dispatch/qdrouterd.conf
Dec 15 13:04:33 katello1.mydomain.net qdrouterd[1427]: Tue Dec 15 13:04:33
2015 ROUTER (info) Configured Address: prefix=/ phase=0
fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE
Dec 15 13:04:33 katello1.mydomain.net qdrouterd[1427]: Tue Dec 15 13:04:33
2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5647 role=normal
Dec 15 13:04:33 katello1.mydomain.net qdrouterd[1427]: Tue Dec 15 13:04:33
2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5646 role=inter-router
Dec 15 13:04:33 katello1.mydomain.net qdrouterd[1427]: Tue Dec 15 13:04:33
2015 CONN_MGR (info) Configured on-demand connector:
katello1.mydomain.net:5671 name=broker
Dec 15 13:04:33 katello1.mydomain.net qdrouterd[1427]: Tue Dec 15 13:04:33
2015 ROUTER (info) Configured Link-route-pattern: prefix=pulp.
connector=broker
Dec 15 13:04:33 katello1.mydomain.net qdrouterd[1427]: Tue Dec 15 13:04:33
2015 ROUTER (info) Configured Link-route-pattern: prefix=qmf.
connector=broker
Dec 15 13:04:33 katello1.mydomain.net qdrouterd[1427]: Tue Dec 15 13:04:33
2015 SERVER (info) Operational, 4 Threads Running
Dec 15 13:04:33 katello1.mydomain.net qdrouterd[1427]: Tue Dec 15 13:04:33
2015 CONN_MGR (info) Starting on-demand connector: broker
Dec 15 13:04:39 katello1.mydomain.net qdrouterd[1427]: Tue Dec 15 13:04:39
2015 ROUTER (info) Activating Prefix ‘pulp.’ for routed links to 'broker’
Dec 15 13:04:39 katello1.mydomain.net qdrouterd[1427]: Tue Dec 15 13:04:39
2015 ROUTER (info) Activating Prefix ‘qmf.’ for routed links to 'broker’
Redirecting to /bin/systemctl status tomcat.service
● tomcat.service - Apache Tomcat Web Application Container
Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor
preset: disabled)
Active: active (running) since Tue 2015-12-15 13:04:33 PST; 8h ago
Main PID: 1440 (java)
CGroup: /system.slice/tomcat.service
└─1440 java -classpath
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
-Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat
-Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp
-Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
org.apache.catalina.startup.Bootstrap start
Dec 15 13:04:54 katello1.mydomain.net server[1440]: Dec 15, 2015 1:04:54 PM
org.apache.catalina.startup.HostConfig deployDirectory
Dec 15 13:04:54 katello1.mydomain.net server[1440]: INFO: Deployment of web
application directory /var/lib/tomcat/webapps/candlepin has finished in
8,255 ms
Dec 15 13:04:54 katello1.mydomain.net server[1440]: Dec 15, 2015 1:04:54 PM
org.apache.coyote.AbstractProtocol start
Dec 15 13:04:54 katello1.mydomain.net server[1440]: INFO: Starting
ProtocolHandler [“http-bio-8080”]
Dec 15 13:04:54 katello1.mydomain.net server[1440]: Dec 15, 2015 1:04:54 PM
org.apache.coyote.AbstractProtocol start
Dec 15 13:04:54 katello1.mydomain.net server[1440]: INFO: Starting
ProtocolHandler [“http-bio-8443”]
Dec 15 13:04:54 katello1.mydomain.net server[1440]: Dec 15, 2015 1:04:54 PM
org.apache.coyote.AbstractProtocol start
Dec 15 13:04:54 katello1.mydomain.net server[1440]: INFO: Starting
ProtocolHandler [“ajp-bio-8009”]
Dec 15 13:04:54 katello1.mydomain.net server[1440]: Dec 15, 2015 1:04:54 PM
org.apache.catalina.startup.Catalina start
Dec 15 13:04:54 katello1.mydomain.net server[1440]: INFO: Server startup in
16901 ms
● elasticsearch.service - LSB: Starts elasticsearch
Loaded: loaded (/etc/rc.d/init.d/elasticsearch)
Active: active (running) since Tue 2015-12-15 13:04:33 PST; 8h ago
Docs: man:systemd-sysv-generator(8)
Process: 1445 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited,
status=0/SUCCESS)
CGroup: /system.slice/elasticsearch.service
└─1623 java -Xms256m -Xmx256m -Xss256k -Djava.awt.headless=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch
-Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
-Des.path.home=/usr/share/java/elasticsearch -cp
:/usr/share/java/elasticsearch/lib/elasticsearch-0.90.10.jar:/usr/share/java/lucene4-core.jar:/usr/share/java/lucene4-contrib/*:/usr/share/java/sigar/sigar.jar
-Des.config=/etc/elasticsearch/elasticsearch.yml
-Des.path.conf=/etc/elasticsearch
-Des.path.home=/usr/share/java/elasticsearch
-Des.path.logs=/var/log/elasticsearch -Des.path.data=/var/lib/elasticsearch
-Des.path.work=/tmp/elasticsearch
-Des.path.plugins=/usr/share/java/elasticsearch/plugins
org.elasticsearch.bootstrap.ElasticSearch
Dec 15 13:04:33 katello1.mydomain.net systemd[1]: Starting LSB: Starts
elasticsearch…
Dec 15 13:04:33 katello1.mydomain.net runuser[1462]:
pam_unix(runuser:session): session opened for user elasticsearch by (uid=0)
Dec 15 13:04:33 katello1.mydomain.net elasticsearch[1445]: Starting
elasticsearch: [ OK ]
Dec 15 13:04:33 katello1.mydomain.net systemd[1]: Started LSB: Starts
elasticsearch.
Redirecting to /bin/systemctl status foreman-proxy.service
● foreman-proxy.service - Foreman Proxy
Loaded: loaded (/usr/lib/systemd/system/foreman-proxy.service; enabled;
vendor preset: disabled)
Active: active (running) since Tue 2015-12-15 13:04:36 PST; 8h ago
Process: 1429 ExecStart=/usr/share/foreman-proxy/bin/smart-proxy
(code=exited, status=0/SUCCESS)
Main PID: 2677 (ruby)
CGroup: /system.slice/foreman-proxy.service
└─2677 ruby /usr/share/foreman-proxy/bin/smart-proxy
Dec 15 13:04:33 katello1.mydomain.net systemd[1]: Starting Foreman Proxy…
Dec 15 13:04:35 katello1.mydomain.net systemd[1]: PID file
/run/foreman-proxy/foreman-proxy.pid not readable (yet?) after start.
Dec 15 13:04:36 katello1.mydomain.net systemd[1]: Started Foreman Proxy.
Redirecting to /bin/systemctl status pulp_celerybeat.service
● pulp_celerybeat.service - Pulp’s Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service; enabled;
vendor preset: disabled)
Active: active (running) since Tue 2015-12-15 13:04:33 PST; 8h ago
Main PID: 1423 (celery)
CGroup: /system.slice/pulp_celerybeat.service
└─1423 /usr/bin/python /usr/bin/celery beat
–scheduler=pulp.server.async.scheduler.Scheduler
Dec 15 13:04:45 katello1.mydomain.net pulp[1423]:
kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism PLAIN
Dec 15 13:04:45 katello1.mydomain.net pulp[1423]:
pulp.server.async.scheduler:INFO: Event Monitor Starting
Dec 15 13:04:45 katello1.mydomain.net pulp[1423]:
kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism PLAIN
Dec 15 13:04:45 katello1.mydomain.net pulp[1423]: kombu.mixins:INFO:
Connected to qpid://guest@katello1.mydomain.net:5671//
Dec 15 13:04:45 katello1.mydomain.net pulp[1423]:
pulp.server.async.worker_watcher:INFO: New worker
’reserved_resource_worker-2@katello1.mydomain.net’ discovered
Dec 15 13:04:45 katello1.mydomain.net pulp[1423]:
pulp.server.async.worker_watcher:INFO: New worker
’reserved_resource_worker-3@katello1.mydomain.net’ discovered
Dec 15 13:04:45 katello1.mydomain.net pulp[1423]:
pulp.server.async.worker_watcher:INFO: New worker
’resource_manager@katello1.mydomain.net’ discovered
Dec 15 13:04:45 katello1.mydomain.net pulp[1423]:
pulp.server.async.worker_watcher:INFO: New worker
’reserved_resource_worker-1@katello1.mydomain.net’ discovered
Dec 15 13:04:45 katello1.mydomain.net pulp[1423]:
pulp.server.async.worker_watcher:INFO: New worker
’reserved_resource_worker-0@katello1.mydomain.net’ discovered
Dec 15 19:04:34 katello1.mydomain.net pulp[1423]: celery.beat:INFO:
Scheduler: Sending due task reap_expired_documents
(pulp.server.db.reaper.queue_reap_expired_documents)
Redirecting to /bin/systemctl status pulp_resource_manager.service
● pulp_resource_manager.service - Pulp Resource Manager
Loaded: loaded (/usr/lib/systemd/system/pulp_resource_manager.service;
enabled; vendor preset: disabled)
Active: active (running) since Tue 2015-12-15 13:04:33 PST; 8h ago
Main PID: 1421 (celery)
CGroup: /system.slice/pulp_resource_manager.service
├─1421 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n
resource_manager@%h -Q resource_manager -c 1 --events --umask 18
└─2755 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n
resource_manager@%h -Q resource_manager -c 1 --events --umask 18
Dec 15 14:29:54 katello1.mydomain.net pulp[1421]:
celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._queue_reserved_task[17b9618c-58a5-41e9-891b-4f49dc8bca1f]
Dec 15 14:29:54 katello1.mydomain.net pulp[1421]: celery.worker.job:INFO:
Task
pulp.server.async.tasks._queue_reserved_task[17b9618c-58a5-41e9-891b-4f49dc8bca1f]
succeeded in 0.0367399759998s: None
Dec 15 14:31:09 katello1.mydomain.net pulp[1421]:
celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._queue_reserved_task[8fad56d3-32ca-4289-9271-df17eb33461e]
Dec 15 14:31:09 katello1.mydomain.net pulp[1421]: celery.worker.job:INFO:
Task
pulp.server.async.tasks._queue_reserved_task[8fad56d3-32ca-4289-9271-df17eb33461e]
succeeded in 0.0343302189995s: None
Dec 15 14:32:29 katello1.mydomain.net pulp[1421]:
celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._queue_reserved_task[bcb14a95-8b92-4eb5-8686-9f9fc0c279e5]
Dec 15 14:32:29 katello1.mydomain.net pulp[1421]: celery.worker.job:INFO:
Task
pulp.server.async.tasks._queue_reserved_task[bcb14a95-8b92-4eb5-8686-9f9fc0c279e5]
succeeded in 0.0366972069996s: None
Dec 15 14:47:17 katello1.mydomain.net pulp[1421]:
celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._queue_reserved_task[24266937-e681-4e7d-be0b-28631e152ba7]
Dec 15 14:47:17 katello1.mydomain.net pulp[1421]: celery.worker.job:INFO:
Task
pulp.server.async.tasks._queue_reserved_task[24266937-e681-4e7d-be0b-28631e152ba7]
succeeded in 0.0311211749995s: None
Dec 15 14:48:23 katello1.mydomain.net pulp[1421]:
celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._queue_reserved_task[8c7db593-8d1b-494a-a122-2fa979b75447]
Dec 15 14:48:23 katello1.mydomain.net pulp[1421]: celery.worker.job:INFO:
Task
pulp.server.async.tasks._queue_reserved_task[8c7db593-8d1b-494a-a122-2fa979b75447]
succeeded in 0.035053521s: None
Redirecting to /bin/systemctl status pulp_workers.service
● pulp_workers.service - Pulp Celery Workers
Loaded: loaded (/usr/lib/systemd/system/pulp_workers.service; enabled;
vendor preset: disabled)
Active: active (exited) since Tue 2015-12-15 13:04:34 PST; 8h ago
Process: 1432 ExecStart=/usr/libexec/pulp-manage-workers start
(code=exited, status=0/SUCCESS)
Main PID: 1432 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/pulp_workers.service
Dec 15 13:04:33 katello1.mydomain.net systemd[1]: Starting Pulp Celery
Workers…
Dec 15 13:04:34 katello1.mydomain.net systemd[1]: Started Pulp Celery
Workers.
Redirecting to /bin/systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor
preset: disabled)
Active: active (running) since Tue 2015-12-15 13:04:34 PST; 8h ago
Docs: man:httpd(8)
man:apachectl(8)
Main PID: 1420 (httpd)
Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0
B/sec"
CGroup: /system.slice/httpd.service
├─ 1420 /usr/sbin/httpd -DFOREGROUND
├─ 2137 (wsgi:pulp) -DFOREGROUND
├─ 2138 PassengerWatchdog
├─ 2142 PassengerHelperAgent
├─ 2155 PassengerLoggingAgent
├─ 2164 /usr/sbin/httpd -DFOREGROUND
├─ 2165 /usr/sbin/httpd -DFOREGROUND
├─ 2166 /usr/sbin/httpd -DFOREGROUND
├─ 2167 /usr/sbin/httpd -DFOREGROUND
├─ 2168 /usr/sbin/httpd -DFOREGROUND
├─ 2169 /usr/sbin/httpd -DFOREGROUND
├─ 2170 /usr/sbin/httpd -DFOREGROUND
├─ 2171 /usr/sbin/httpd -DFOREGROUND
├─ 2775 Passenger RackApp: /etc/puppet/rack
├─ 2807 /usr/sbin/httpd -DFOREGROUND
├─ 3040 Passenger RackApp: /usr/share/foreman
├─ 7771 /usr/sbin/httpd -DFOREGROUND
├─28489 /bin/bash /usr/bin/ruby193-ruby
/usr/share/gems/gems/passenger-4.0.18/helper-scripts/rack-preloader.rb
├─28499 scl enable ruby193 bash /tmp/tmp.CcNA0oHbKr
├─28500 /bin/bash /var/tmp/sclyQC0xQ
├─28503 bash /tmp/tmp.CcNA0oHbKr
├─28504 Passenger AppPreloader: /usr/share/foreman
├─28557 Passenger RackApp: /usr/share/foreman
├─28757 Passenger RackApp: /usr/share/foreman
├─28805 Passenger RackApp: /usr/share/foreman
└─28828 Passenger RackApp: /usr/share/foreman
Dec 15 17:04:38 katello1.mydomain.net puppet-master[2775]: Compiled catalog
for katello1.mydomain.net in environment production in 0.01 seconds
Dec 15 17:34:38 katello1.mydomain.net puppet-master[2775]: Compiled catalog
for katello1.mydomain.net in environment production in 0.01 seconds
Dec 15 18:04:38 katello1.mydomain.net puppet-master[2775]: Compiled catalog
for katello1.mydomain.net in environment production in 0.01 seconds
Dec 15 18:34:38 katello1.mydomain.net puppet-master[2775]: Compiled catalog
for katello1.mydomain.net in environment production in 0.01 seconds
Dec 15 19:04:38 katello1.mydomain.net puppet-master[2775]: Compiled catalog
for katello1.mydomain.net in environment production in 0.01 seconds
Dec 15 19:34:38 katello1.mydomain.net puppet-master[2775]: Compiled catalog
for katello1.mydomain.net in environment production in 0.01 seconds
Dec 15 20:04:38 katello1.mydomain.net puppet-master[2775]: Compiled catalog
for katello1.mydomain.net in environment production in 0.01 seconds
Dec 15 20:34:38 katello1.mydomain.net puppet-master[2775]: Compiled catalog
for katello1.mydomain.net in environment production in 0.01 seconds
Dec 15 21:04:38 katello1.mydomain.net puppet-master[2775]: Compiled catalog
for katello1.mydomain.net in environment production in 0.01 seconds
Dec 15 21:34:38 katello1.mydomain.net puppet-master[2775]: Compiled catalog
for katello1.mydomain.net in environment production in 0.01 seconds
Redirecting to /bin/systemctl status foreman-tasks.service
● foreman-tasks.service - Foreman jobs daemon
Loaded: loaded (/usr/lib/systemd/system/foreman-tasks.service; enabled;
vendor preset: disabled)
Active: active (running) since Tue 2015-12-15 13:04:48 PST; 8h ago
Docs: https://github.com/theforeman/f