[Katello] Pulp Node Content Sync

Support
1

I setup a capsule as a pulp node. Initiated a sync via CLI. Content started
to sync, it ended up doing about 30G. However the progress bar stopped at
50% and never proceeded and no additional content was synced down.

From within the GUI -> Tasks it states the task is running

Action:

Actions::Pulp::Consumer::SyncNode

Input:

{"consumer_uuid"=>"9fb60e2b-b391-4d0a-b0be-d8ebc3829275",
"repo_ids"=>nil,
"remote_user"=>"admin-8f138997",
"remote_cp_user"=>"admin",
"locale"=>"en"}

Output

{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>nil,
"_href"=>"/pulp/api/v2/tasks/ba7aed24-d30e-40c9-baee-d94d5a01fd94/",
"task_id"=>"ba7aed24-d30e-40c9-baee-d94d5a01fd94",
"tags"=>
["pulp:consumer:9fb60e2b-b391-4d0a-b0be-d8ebc3829275",
"pulp:action:unit_update"],
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>"2015-05-28T23:35:30Z",
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>
{"progress"=>
[{"state"=>"adding_units",
"repo_id"=>"8x8-CentOS_Enterprise_Linux_Server_version_6-base",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>"8x8-CentOS_Enterprise_Linux_Server_version_6-updates",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>"8x8-EPEL-4-x86_32-puppetlabs-products",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>"8x8-EPEL-5-x86_64-main",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>"8x8-EPEL-5-x86_64-puppetlabs-dependencies",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>"8x8-EPEL-5-x86_64-puppetlabs-products",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>"8x8-EPEL-6-x86_64-custom",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>"8x8-EPEL-6-x86_64-main",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>"8x8-EPEL-6-x86_64-puppetlabs-dependencies",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>"8x8-EPEL-6-x86_64-puppetlabs-products",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>
"8x8-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_Kickstart_x86_64_5_11",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>
"8x8-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5Server",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>
"8x8-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_Kickstart_x86_64_6_5",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>
"8x8-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_Kickstart_x86_64_6_6",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>
"8x8-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}},
{"state"=>"pending",
"repo_id"=>"8x8-Red_Hat_Enterprise_Linux_Server_version_4-x86_32",
"unit_add"=>{"completed"=>0, "total"=>0, "details"=>nil}}],
"state"=>"in-progress"},
"queue"=>"agent.dq",
"state"=>"running",
"worker_name"=>"agent",
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"5567a66a32679d076d97fd72"},
"id"=>"5567a66a32679d076d97fd72"}],
"poll_attempts"=>{"total"=>49, "failed"=>0}}

The following shows in /var/log/messages on the capsule

  1. May 28 16:35:31 capsule goferd: [INFO][worker-0]
    gofer.rmi.dispatcher:610 - call: Content.update()
    sn=ee0bf1c0-c2ae-4794-8b29-6f28febf93d8 data={'task_id':
    'ba7aed24-d30e-40c9-baee-d94d5a01fd94', 'consumer_id':
    '9fb60e2b-b391-4d0a-b0be-d8ebc3829275'}
  2. May 28 16:35:33 capsule pulp: kombu.transport.qpid:INFO: Connected to
    qpid with SASL mechanism PLAIN
  3. May 28 16:35:33 capsule goferd: [INFO][worker-0]
    pulp_node.handlers.model:475 - Importer:
    8x8-CentOS_Enterprise_Linux_Server_version_6-base/nodes_http_importer,
    updated
  4. May 28 16:35:33 capsule pulp: celery.worker.strategy:INFO: Received
    task:
    pulp.server.async.tasks._queue_reserved_task[ca315037-a31c-41d8-b7b7-7abb31dabde5]
  5. May 28 16:35:33 capsule pulp: kombu.transport.qpid:INFO: Connected to
    qpid with SASL mechanism PLAIN
  6. May 28 16:35:33 capsule pulp: celery.worker.strategy:INFO: Received
    task:
    pulp.server.managers.repo.importer.update_importer_config[31b890f1-5fdc-4ee4-b384-e3e3e282e83c]
  7. May 28 16:35:33 capsule pulp: celery.worker.strategy:INFO: Received
    task:
    pulp.server.async.tasks._release_resource[264ade98-08c9-48c4-9240-c3172a6c75f9]
  8. May 28 16:35:33 capsule pulp: celery.worker.strategy:INFO: Received
    task:
    pulp.server.async.tasks._queue_reserved_task[0917968b-2869-4d4e-8398-5092d1e0c85b]
  9. May 28 16:35:33 capsule pulp: celery.worker.job:INFO: Task
    pulp.server.async.tasks._queue_reserved_task[ca315037-a31c-41d8-b7b7-7abb31dabde5]
    succeeded in 0.110632666s: None
  10. May 28 16:35:33 capsule pulp: celery.worker.job:INFO: Task
    pulp.server.async.tasks._queue_reserved_task[0917968b-2869-4d4e-8398-5092d1e0c85b]
    succeeded in 0.012839345s: None
  11. May 28 16:35:33 capsule pulp: celery.worker.strategy:INFO: Received
    task:
    pulp.server.managers.repo.sync.sync[7a9a2f47-d3b6-445d-8ced-40c57fc87e81]
  12. May 28 16:35:33 capsule pulp: celery.worker.job:INFO: Task
    pulp.server.managers.repo.importer.update_importer_config[31b890f1-5fdc-4ee4-b384-e3e3e282e83c]
    succeeded in 0.0836534s: {u'repo_id':
    u'CentOS_Enterprise_Linux_Server_version_6-base', u'_ns':
    u'repo_importers', u'importer_type_id':…
  13. May 28 16:35:33 capsule pulp: celery.worker.strategy:INFO: Received
    task:
    pulp.server.async.tasks._release_resource[7141e690-8976-4242-9289-1d0d97e0e982]
  14. May 28 16:35:34 capsule pulp: celery.worker.job:INFO: Task
    pulp.server.async.tasks._release_resource[264ade98-08c9-48c4-9240-c3172a6c75f9]
    succeeded in 0.04033601s: None
  15. May 28 16:35:34 capsule pulp:
    requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
    connection (1): katello.xxxxxx.net

So it appears to have tried to kick off the sync but never proceeds.

One thing I do note is that it says for repo_id
CentOS_Enterprise_Linux_Server_version_6-base that it succeeded. But that
repository is not completely synced (I try to provision off it and packages
end up missing).

Any help in trouble shooting would be greatly appreciated.

Thanks

2

The stall at 50% is typically indicative of an unreachable Capsule.
However, since you appear to be seeing sync messages that would lend me to
think that either its grinding away or something has pegged the CPU. Can
you check the load on the Capsule? Does the Pulp output in the system log
stop or do you see activity off and on?

··· On Thu, May 28, 2015 at 8:02 PM, Matthew Ceroni wrote:

I setup a capsule as a pulp node. Initiated a sync via CLI. Content
started to sync, it ended up doing about 30G. However the progress bar
stopped at 50% and never proceeded and no additional content was synced
down.

From within the GUI -> Tasks it states the task is running

Action:

Actions::Pulp::Consumer::SyncNode

Input:

{“consumer_uuid”=>“9fb60e2b-b391-4d0a-b0be-d8ebc3829275”,
“repo_ids”=>nil,
“remote_user”=>“admin-8f138997”,
“remote_cp_user”=>“admin”,
“locale”=>“en”}

Output

{“pulp_tasks”=>
[{“exception”=>nil,
“task_type”=>nil,
"_href"=>"/pulp/api/v2/tasks/ba7aed24-d30e-40c9-baee-d94d5a01fd94/",
“task_id”=>“ba7aed24-d30e-40c9-baee-d94d5a01fd94”,
“tags”=>
[“pulp:consumer:9fb60e2b-b391-4d0a-b0be-d8ebc3829275”,
“pulp:action:unit_update”],
“finish_time”=>nil,
"_ns"=>“task_status”,
“start_time”=>“2015-05-28T23:35:30Z”,
“traceback”=>nil,
“spawned_tasks”=>[],
“progress_report”=>
{“progress”=>
[{“state”=>“adding_units”,
“repo_id”=>“8x8-CentOS_Enterprise_Linux_Server_version_6-base”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>“8x8-CentOS_Enterprise_Linux_Server_version_6-updates”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>“8x8-EPEL-4-x86_32-puppetlabs-products”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>“8x8-EPEL-5-x86_64-main”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>“8x8-EPEL-5-x86_64-puppetlabs-dependencies”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>“8x8-EPEL-5-x86_64-puppetlabs-products”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>“8x8-EPEL-6-x86_64-custom”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>“8x8-EPEL-6-x86_64-main”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>“8x8-EPEL-6-x86_64-puppetlabs-dependencies”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>“8x8-EPEL-6-x86_64-puppetlabs-products”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>
“8x8-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_Kickstart_x86_64_5_11”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>
“8x8-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5Server”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>
“8x8-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_Kickstart_x86_64_6_5”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>
“8x8-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_Kickstart_x86_64_6_6”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>
“8x8-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}},
{“state”=>“pending”,
“repo_id”=>“8x8-Red_Hat_Enterprise_Linux_Server_version_4-x86_32”,
“unit_add”=>{“completed”=>0, “total”=>0, “details”=>nil}}],
“state”=>“in-progress”},
“queue”=>“agent.dq”,
“state”=>“running”,
“worker_name”=>“agent”,
“result”=>nil,
“error”=>nil,
"_id"=>{"$oid"=>“5567a66a32679d076d97fd72”},
“id”=>“5567a66a32679d076d97fd72”}],
“poll_attempts”=>{“total”=>49, “failed”=>0}}

The following shows in /var/log/messages on the capsule

  1. May 28 16:35:31 capsule goferd: [INFO][worker-0]
    gofer.rmi.dispatcher:610 - call: Content.update()
    sn=ee0bf1c0-c2ae-4794-8b29-6f28febf93d8 data={‘task_id’:
    ‘ba7aed24-d30e-40c9-baee-d94d5a01fd94’, ‘consumer_id’:
    ‘9fb60e2b-b391-4d0a-b0be-d8ebc3829275’}
  2. May 28 16:35:33 capsule pulp: kombu.transport.qpid:INFO: Connected
    to qpid with SASL mechanism PLAIN
  3. May 28 16:35:33 capsule goferd: [INFO][worker-0]
    pulp_node.handlers.model:475 - Importer:
    8x8-CentOS_Enterprise_Linux_Server_version_6-base/nodes_http_importer,
    updated
  4. May 28 16:35:33 capsule pulp: celery.worker.strategy:INFO: Received
    task:
    pulp.server.async.tasks._queue_reserved_task[ca315037-a31c-41d8-b7b7-7abb31dabde5]
  5. May 28 16:35:33 capsule pulp: kombu.transport.qpid:INFO: Connected
    to qpid with SASL mechanism PLAIN
  6. May 28 16:35:33 capsule pulp: celery.worker.strategy:INFO: Received
    task:
    pulp.server.managers.repo.importer.update_importer_config[31b890f1-5fdc-4ee4-b384-e3e3e282e83c]
  7. May 28 16:35:33 capsule pulp: celery.worker.strategy:INFO: Received
    task:
    pulp.server.async.tasks._release_resource[264ade98-08c9-48c4-9240-c3172a6c75f9]
  8. May 28 16:35:33 capsule pulp: celery.worker.strategy:INFO: Received
    task:
    pulp.server.async.tasks._queue_reserved_task[0917968b-2869-4d4e-8398-5092d1e0c85b]
  9. May 28 16:35:33 capsule pulp: celery.worker.job:INFO: Task
    pulp.server.async.tasks._queue_reserved_task[ca315037-a31c-41d8-b7b7-7abb31dabde5]
    succeeded in 0.110632666s: None
  10. May 28 16:35:33 capsule pulp: celery.worker.job:INFO: Task
    pulp.server.async.tasks._queue_reserved_task[0917968b-2869-4d4e-8398-5092d1e0c85b]
    succeeded in 0.012839345s: None
  11. May 28 16:35:33 capsule pulp: celery.worker.strategy:INFO:
    Received task:
    pulp.server.managers.repo.sync.sync[7a9a2f47-d3b6-445d-8ced-40c57fc87e81]
  12. May 28 16:35:33 capsule pulp: celery.worker.job:INFO: Task
    pulp.server.managers.repo.importer.update_importer_config[31b890f1-5fdc-4ee4-b384-e3e3e282e83c]
    succeeded in 0.0836534s: {u’repo_id’:
    u’CentOS_Enterprise_Linux_Server_version_6-base’, u’_ns’:
    u’repo_importers’, u’importer_type_id’:…
  13. May 28 16:35:33 capsule pulp: celery.worker.strategy:INFO:
    Received task:
    pulp.server.async.tasks._release_resource[7141e690-8976-4242-9289-1d0d97e0e982]
  14. May 28 16:35:34 capsule pulp: celery.worker.job:INFO: Task
    pulp.server.async.tasks._release_resource[264ade98-08c9-48c4-9240-c3172a6c75f9]
    succeeded in 0.04033601s: None
  15. May 28 16:35:34 capsule pulp:
    requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
    connection (1): katello.xxxxxx.net

So it appears to have tried to kick off the sync but never proceeds.

One thing I do note is that it says for repo_id
CentOS_Enterprise_Linux_Server_version_6-base that it succeeded. But that
repository is not completely synced (I try to provision off it and packages
end up missing).

Any help in trouble shooting would be greatly appreciated.

Thanks

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.