Infinite repository synchronization

Support
,
#1

Problem:
synchronization of one of the repositories has been going on for several hours. There is no load on the server at the same time. There are also no errors

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/75eb5cdd-a209-4d18-9847-d90926f7a6ad/",
    "pulp_created"=>"2021-11-02T11:31:13.104+00:00",
    "state"=>"waiting",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"0a5c1a2685ba4ec5a9cef962b357453b",
    "worker"=>"/pulp/api/v3/workers/fde5ce5c-dd5d-4977-8cf2-896f98c98280/",
    "child_tasks"=>[],
    "progress_reports"=>[],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/dec5bb91-bb6a-4f10-8b9b-52ae9329caf0/",
      "/pulp/api/v3/remotes/rpm/rpm/baa6b481-222d-4776-bf42-2b98e2e600d2/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>328, "failed"=>0}}

Expected outcome:
successful completion of synchronization
Foreman and Proxy versions:
foreman 2.4.0
Foreman and Proxy plugin versions:

Distribution and version:
katello 4.0.11
pulp 3
Other relevant data:

#2

@Overlord, thanks for the report. I have a few suggestions but first I’d like to ask that you read the whole comment before making any changes in case we might be able to live-troubleshoot a rare and difficult to reproduce issue that we’ve been investigating.

First, check whether the pulpcore-worker services are indeed running (but don’t restart them).

If they aren’t running, that would be the issue, and starting them should cause the task to be picked up.

If they are running, then this looks like it may be a bug that we discovered a week or two ago. We’re already testing a fix, but since it is so difficult to reproduce we would like to double check some of our theories about what is happening. If this is the case, could you first check whether any tasks were spawned after this task, or is it the most recent one? In other words, were other tasks successfully completing while this one is stuck waiting?

Further, does spawning a new task for something completely unrelated cause it to be un-stuck? For example, changing the name of an unrelated repository?

We would really appreciate if you could test that for us. If it does not work, restarting the pulpcore-worker services would probably work. Either way let us know what happens.

Thanks!

1 Like
#3

@Overlord any update?