Pulp tasks triggered by schedulers do not run

Hi all,

Problem:
We configured multiple products with multiple repositories. Therefore we added schedulers for syncing these products daily. Unfortunately, the sync status does not change. The runner gave it to the pulp, but pulp does not sync (or does sync all the information and it doesn’t update the modify time in the webui).

Expected outcome:
Create schedulers, assign products to it. Scheduler updates these products on a configured invtervall, that can be checked in the “schedule status” tab.

Foreman and Proxy versions:
Foreman: 1.19.1

Foreman and Proxy plugin versions:
bastion: 6.1.11
foreman-tasks: 0.13.4
foreman_ansible: 2.2.9
foreman_bootdisk: 12.0.0
foreman_docker: 4.1.0
foreman_remote_execution: 1.5.6
foreman_virt_who_configure: 0.2.2
katello: 3.8.1

Other relevant data:

$ sudo journalctl -f -t pulp

Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:INFO: [98eb0c22] (102032-84704) D"}, "distribution": {"items_total": 0, "state": "FINISHED", "error_details": [], "items_left": 0}, "erra
ta": {"state": "FINISHED"}, "metadata": {"state": "FINISHED"}}, "id": "5c3f223cee146f8e918c7848", "removed_count": 0}}
Jan 16 13:23:24 katello.domain.lan pulp[11459]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[29534034-99d4-4bb3-a3ef-6521f7f9a4d7]
Jan 16 13:23:24 katello.domain.lan pulp[101957]: pulp.server.async.tasks:INFO: [e7a61984] resetting consecutive failure count for schedule 5bd5cd53ee146f1e51d11966
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704) HTTP Notification Failed
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704) Traceback (most recent call last):
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704)   File "/usr/lib/python2.7/site-packages/pulp/server/event/http.py", line 57, in _send_post
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704)     headers={'Content-Type': 'application/json'}, timeout=15)
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704)   File "/usr/lib/python2.7/site-packages/requests/api.py", line 116, in post
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704)     return request('post', url, data=data, json=json, **kwargs)
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704)   File "/usr/lib/python2.7/site-packages/requests/api.py", line 60, in request
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704)     return session.request(method=method, url=url, **kwargs)
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704)     resp = self.send(prep, **send_kwargs)
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704)     r = adapter.send(request, **kwargs)
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704)   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 514, in send
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704)     raise SSLError(e, request=request)
Jan 16 13:23:24 katello.domain.lan pulp[102032]: pulp.server.event.http:ERROR: [98eb0c22] (102032-84704) SSLError: HTTPSConnectionPool(host='katello.domain.lan', port=443): Max retries exceeded with url: /katell
o/api/v2/repositories/sync_complete?token=feoxARpri6zUSHP5P6PvRCydph994ujj (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),))
Jan 16 13:23:24 katello.domain.lan pulp[101957]: celery.app.trace:INFO: [e7a61984] Task pulp.server.managers.repo.sync.sync[e7a61984-94f5-4df4-a495-f125122c4764] succeeded in 14.388940816s: <pulp.server.async.
tasks.TaskResult object at 0x7f5c35ae8b10>

$ sudo pulp-admin list tasks

+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

No tasks found

I created new schedulers and added products to it. Dynflow console showed up a successfull assignement and a successfull creation. On the sync status pane, I can trigger a sync manually. These are working as expected.

As a dependency, I added our certificates to the server via the docmented way to deploy self-signed certificates with our own ca.

If you need further informations, please feel free to contact me.

Thanks very much,
Ludwig

Hello @balu,

This might be helpful --> Katello repo sync not working
Also, we have moved away from pulp schedules in K3.9 so you could consider an upgrade if possible.:slightly_smiling_face: