Unable to sync repositories with new installation of Katello 3.18

Support
,
1

Problem:
The repositories that are created after a fresh install of Katello 3.18 will not fully synchronize. Most will enter a paused state.
I get the following errors:
aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’ from the pulpcore-worker.
pulpcore-worker-8: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
I have tried the fix from the Redhat article: Repository synchronization is stuck and sync tasks are being cancelled in Red Hat Satellite 6 - Red Hat Customer Portal. This did not resolve the issue.
Expected outcome:
Repositores will synchronize without errors.
Foreman and Proxy versions:
Foreman: 2.3.5

Foreman and Proxy plugin versions:
foreman-tasks: 3.0.6
foreman_remote_execution: 4.2.2
katello: 3.18.3.1
Distribution and version:
CentOS Linux release 7.9.2009 (Core)
Other relevant data:

I have installed version 3.18 4 or 5 times with the same results.
I am trying to prepare for an upgrade from 3.18 to 4.0 but I am stuck because I cannot get the repositories to sync.

/var/log/messages:Jun 22 17:16:29 pulpcore-worker-5: pulp: rq.worker:ERROR: Traceback (most recent call last):
/var/log/messages:Jun 22 17:16:29 pulpcore-worker-5: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://mirror.centos.org/centos/7/sclo/x86_64/rh/Packages/r/rh-python38-python-test-3.8.0-15.el7.x86_64.rpm’)
/var/log/messages:Jun 22 17:16:29 pulpcore-worker-5: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://mirror.centos.org/centos/7/sclo/x86_64/rh/Packages/r/rh-python38-python-test-3.8.0-15.el7.x86_64.rpm’)
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘19418@’ has gone missing, removing from list of workers
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 19418@ is missing. Canceling the tasks in its queue.
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘19640@’ has gone missing, removing from list of workers
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 19640@ is missing. Canceling the tasks in its queue.
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘19774@’ has gone missing, removing from list of workers
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 19774@ is missing. Canceling the tasks in its queue.
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘19841@’ has gone missing, removing from list of workers
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 19841@ is missing. Canceling the tasks in its queue.
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘19909@’ has gone missing, removing from list of workers
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 19909@ is missing. Canceling the tasks in its queue.
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘19707@’ has gone missing, removing from list of workers
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 19707@ is missing. Canceling the tasks in its queue.
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘19506@’ has gone missing, removing from list of workers
/var/log/messages:Jun 22 22:44:34 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 19506@ is missing. Canceling the tasks in its queue.
/var/log/messages:Jun 22 23:00:30 dynflow-sidekiq@worker: 2021-06-22T23:00:30.877Z 1067 TID-2l8783 WARN: {“context”:“Job raised exception”,“job”:{“queue”:“default”,“args”:[{“class”:“Dynflow::Director::EventWorkItem”,“execution_plan_id”:“8ba07cbb-8ab1-4da3-80a7-31867c548cf7”,“queue”:“default”,“sender_orchestrator_id”:“51f46366-c6da-4b63-bbfc-96def9a935f8”,“step”:{“execution_plan_uuid”:“8ba07cbb-8ab1-4da3-80a7-31867c548cf7”,“id”:4,“state”:“suspended”,“class”:“Dynflow::ExecutionPlan::Steps::RunStep”,“action_class”:“Actions::Pulp3::Repository::Sync”,“action_id”:3,“error”:null,“started_at”:“2021-06-22 22:45:15 UTC”,“ended_at”:“2021-06-22 23:00:14 UTC”,“execution_time”:5.281857743999999,“real_time”:899.826988229,“progress_done”:0.1,“progress_weight”:10.0,“queue”:“default”},“request_id”:“51f46366-c6da-4b63-bbfc-96def9a935f8-89”,“event”:{“algebrick_type”:“Dynflow::Action::Polling::Poll”}}],“class”:“Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork”,“retry”:false,“backtrace”:true,“jid”:“d56031e7e93af9ba875b820f”,“created_at”:1624402830.8264909,“enqueued_at”:1624402830.8265502},“jobstr”:"{“queue”:“default”,“args”:[{“class”:“Dynflow::Director::EventWorkItem”,“execution_plan_id”:“8ba07cbb-8ab1-4da3-80a7-31867c548cf7”,“queue”:“default”,“sender_orchestrator_id”:“51f46366-c6da-4b63-bbfc-96def9a935f8”,“step”:{“execution_plan_uuid”:“8ba07cbb-8ab1-4da3-80a7-31867c548cf7”,“id”:4,“state”:“suspended”,“class”:“Dynflow::ExecutionPlan::Steps::RunStep”,“action_class”:“Actions::Pulp3::Repository::Sync”,“action_id”:3,“error”:null,“started_at”:“2021-06-22 22:45:15 UTC”,“ended_at”:“2021-06-22 23:00:14 UTC”,“execution_time”:5.281857743999999,“real_time”:899.826988229,“progress_done”:0.1,“progress_weight”:10.0,“queue”:“default”},“request_id”:“51f46366-c6da-4b63-bbfc-96def9a935f8-89”,“event”:{“algebrick_type”:“Dynflow::Action::Polling::Poll”}}],“class”:“Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork”,“retry”:false,“backtrace”:true,“jid”:“d56031e7e93af9ba875b820f”,“created_at”:1624402830.8264909,“enqueued_at”:1624402830.8265502}"}
/var/log/messages:Jun 22 23:14:46 pulpcore-worker-5: pulp: rq.worker:ERROR: Traceback (most recent call last):
/var/log/messages:Jun 22 23:14:46 pulpcore-worker-5: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://yum.puppetlabs.com/puppet6/el/7/x86_64/puppetdb-6.10.1-1.el7.noarch.rpm’)
/var/log/messages:Jun 22 23:14:46 pulpcore-worker-5: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://yum.puppetlabs.com/puppet6/el/7/x86_64/puppetdb-6.10.1-1.el7.noarch.rpm’)
/var/log/messages-20210622:Jun 22 16:24:24 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:24 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:24 pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:24 pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:24 pulpcore-worker-3: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:24 pulpcore-worker-3: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:25 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:25 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:25 pulpcore-worker-5: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:25 pulpcore-worker-5: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:26 pulpcore-worker-6: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:26 pulpcore-worker-6: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:26 pulpcore-worker-7: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:26 pulpcore-worker-7: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:26 pulpcore-worker-8: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:24:26 pulpcore-worker-8: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
/var/log/messages-20210622:Jun 22 16:36:35 pulpcore-worker-8: pulp: rq.worker:ERROR: Traceback (most recent call last):
/var/log/messages-20210622:Jun 22 16:36:35 pulpcore-worker-8: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://mirror.netcologne.de/fedora-epel/7/x86_64/Packages/f/fortune-mod-1.99.1-17.el7.x86_64.rpm’)
/var/log/messages-20210622:Jun 22 16:36:35 pulpcore-worker-8: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://mirror.netcologne.de/fedora-epel/7/x86_64/Packages/f/fortune-mod-1.99.1-17.el7.x86_64.rpm’)
/var/log/messages-20210622:Jun 22 16:36:35 pulpcore-worker-4: pulp: rq.worker:ERROR: Traceback (most recent call last):
/var/log/messages-20210622:Jun 22 16:36:35 pulpcore-worker-4: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://mirror.centos.org/centos/7/sclo/x86_64/rh/Packages/r/rh-maven35-maven-surefire-javadoc-2.19.1-6.2.el7.noarch.rpm’)
/var/log/messages-20210622:Jun 22 16:36:35 pulpcore-worker-4: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://mirror.centos.org/centos/7/sclo/x86_64/rh/Packages/r/rh-maven35-maven-surefire-javadoc-2.19.1-6.2.el7.noarch.rpm’)
/var/log/messages-20210622:Jun 22 16:36:35 pulpcore-worker-1: pulp: rq.worker:ERROR: Traceback (most recent call last):
/var/log/messages-20210622:Jun 22 16:36:35 pulpcore-worker-1: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://yum.puppetlabs.com/puppet6/el/7/x86_64/puppetdb-6.1.0-1.el7.noarch.rpm’)
/var/log/messages-20210622:Jun 22 16:36:35 pulpcore-worker-1: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://yum.puppetlabs.com/puppet6/el/7/x86_64/puppetdb-6.1.0-1.el7.noarch.rpm’)
/var/log/messages-20210622:Jun 22 16:36:42 pulpcore-worker-5: pulp: rq.worker:ERROR: Traceback (most recent call last):
/var/log/messages-20210622:Jun 22 16:36:42 pulpcore-worker-5: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://mirrors.mit.edu/centos/7/os/x86_64/Packages/kwallet-4.10.5-3.el7.x86_64.rpm’)
/var/log/messages-20210622:Jun 22 16:36:42 pulpcore-worker-5: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://mirrors.mit.edu/centos/7/os/x86_64/Packages/kwallet-4.10.5-3.el7.x86_64.rpm’)
/var/log/messages-20210622:Jun 22 16:36:59 pulpcore-worker-3: pulp: rq.worker:ERROR: Traceback (most recent call last):
/var/log/messages-20210622:Jun 22 16:36:59 pulpcore-worker-3: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://mirrors.mit.edu/centos/7/updates/x86_64/Packages/libgweather-3.28.2-4.el7_9.x86_64.rpm’)
/var/log/messages-20210622:Jun 22 16:36:59 pulpcore-worker-3: aiohttp.client_exceptions.ClientResponseError: 503, message=‘Service Unavailable’, url=URL(‘http://mirrors.mit.edu/centos/7/updates/x86_64/Packages/libgweather-3.28.2-4.el7_9.x86_64.rpm’)

2

Hi @benrick001

Have you configured a proxy to access the repositories? Can you manually download these packages from katello (curl or wget)?

3

@benrick001 are all of your Pulpcore services running? You can check with systemctl status pulpcore*

4

Hello,
Yes I have checked and all pulp processes are running. The downloads appear to start but then fail at some point. After this then cannot synchronize to that repository anymore. There are other repos that complete the synchronization and if I initiate them, they will synchronize properly.

Just to note that this is an automated install using Jenkins and I have used this process many times. There are no problems with the URLs for the repos.

I used the same build process yesterday to build a 3.15 version and the repos sync properly.

5

Hello,
The packages can be downloaded manually. I have tested this. The downloads appear to start but then fail at some point. After this then cannot synchronize to that repository anymore. There are other repos that complete the synchronization and if I initiate them, they will synchronize properly.

Just to note that this is an automated install using Jenkins and I have used this process many times. There are no problems with the URLs for the repos.

I used the same build process yesterday to build a 3.15 version and the repos sync properly.

6

Can you provide the list of repositories that work and those that fail in 503?

If you do a foreman-maintain health check, is everything ok?

It’s hard to help you because we don’t know what steps are performed in your Jenkins automated installation.

It is possible that this is the cause of your problem because Foreman 3.15 uses Pulp 2 while Foreman> = 3.18 uses Pulp 3. So customizations that work on 3.15 and before can break version 3.18

Can you do a fresh installation in 3.18 manually, only, as indicated in the documentation?

Obviously, if the Jenkins job comes down to doing a foreman-installer only, the mystery is elsewhere.

Just to confirm that it works for me in 3.18 with redhat, and others extras RPM repositories.

1 Like
7

Hello,
I performed another installation today and this time it appeared to synchronize. I will perform some additional tests tomorrow (June 29,2021) and see if I can figure out what the difference would be.

Thank you.

8

After performing several installs it appears that there is an issue with a reboot built into the installation.
In the Ansible build process I create a new organization and after this we change the SSL configuration to limit the clients from accessing the API. They can only access the 443 port to get updates. After the SSL config in “/etc/httpd/conf.d/05-foreman-ssl.conf” , I added a reboot to ensure the new SSL config is enabled. In 3.15 this did not cause any issues. However in 3.18 something breaks at this point and unable to get the repos to synchronize. I have changed this so there is a 10 minute sleep before rebooting and this seems to fix the error.

9

The issue for me was that there were still some processes that were executing when the reboot occurred. I changed from “a reboot” of the system to “foreman-maintain service restart” after re-configuring the 05-foreman-ssl.conf file. There were still some jobs that failed but the system was able to shutdown cleanly and therefore was also able to continue after the services restarted.

Closing this issue.