Katello 3.15 Long running publish task

Louis_Bohm · July 14, 2020, 11:55am

Tasks to publish CCVs are running for 15+ hours

Previously they were completing in minutes.

Katello 3.15 (just upgraded) on CentOS 7.8

In Dynflow I see the messages like this on all of the tasks.

6: Actions::Pulp::Repository::Create (success) [ 3.41s / 3.41s ]
7: Actions::Katello::Repository::Create (success) [ 0.01s / 0.01s ]
10: Actions::Pulp::Repository::Create (success) [ 2.18s / 2.18s ]
11: Actions::Katello::Repository::Create (success) [ 0.07s / 0.07s ]
14: Actions::Pulp::Repository::Create (success) [ 2.74s / 2.74s ]
15: Actions::Katello::Repository::Create (success) [ 0.19s / 0.19s ]
18: Actions::Pulp::Repository::Create (success) [ 1.78s / 1.78s ]
19: Actions::Katello::Repository::Create (success) [ 0.06s / 0.06s ]
22: Actions::Pulp::Repository::Create (success) [ 2.92s / 2.92s ]
23: Actions::Katello::Repository::Create (success) [ 0.02s / 0.02s ]
26: Actions::Pulp::Repository::Create (success) [ 3.32s / 3.32s ]
27: Actions::Katello::Repository::Create (success) [ 0.30s / 0.30s ]
30: Actions::Pulp::Repository::Create (success) [ 4.79s / 4.79s ]
31: Actions::Katello::Repository::Create (success) [ 0.06s / 0.06s ]
34: Actions::Pulp::Repository::Create (success) [ 4.64s / 4.64s ]
35: Actions::Katello::Repository::Create (success) [ 0.03s / 0.03s ]
38: Actions::Pulp::Repository::Create (success) [ 3.64s / 3.64s ]
39: Actions::Katello::Repository::Create (success) [ 0.02s / 0.02s ]
42: Actions::Pulp::Repository::Create (success) [ 2.99s / 2.99s ]
43: Actions::Katello::Repository::Create (success) [ 0.02s / 0.02s ]
46: Actions::Pulp::Repository::Create (success) [ 1.96s / 1.96s ]
47: Actions::Katello::Repository::Create (success) [ 0.10s / 0.10s ]
50: Actions::Pulp::Repository::Create (success) [ 1.66s / 1.66s ]
51: Actions::Katello::Repository::Create (pending)
54: Actions::Pulp::Repository::Create (success) [ 4.89s / 4.89s ]
55: Actions::Katello::Repository::Create (pending)
60: Actions::Pulp::Repository::DistributorPublish (success) [ 261.36s / 8.75s ]
62: Actions::Katello::Repository::IndexContent (success) [ 0.71s / 0.71s ]
64: Actions::Katello::Repository::PurgeEmptyContent (success) [ 140.46s / 9.56s ]
69: Actions::Pulp::Repository::Refresh (success) [ 31.13s / 7.93s ]
72: Actions::Katello::Repository::CheckMatchingContent (success) [ 2.71s / 2.71s ]
75: Actions::Pulp::Repository::DistributorPublish (success) [ 0.41s / 0.41s ]
77: Actions::Katello::Repository::IndexContent (success) [ 0.81s / 0.81s ]
82: Actions::Pulp::Repository::DistributorPublish (pending)
84: Actions::Katello::Repository::IndexContent (success) [ 0.10s / 0.10s ]
86: Actions::Katello::Repository::PurgeEmptyContent (success) [ 176.21s / 9.70s ]
91: Actions::Pulp::Repository::Refresh (success) [ 76.53s / 16.55s ]
94: Actions::Katello::Repository::CheckMatchingContent (success) [ 0.86s / 0.86s ]
97: Actions::Pulp::Repository::DistributorPublish (success) [ 0.05s / 0.05s ]
99: Actions::Katello::Repository::IndexContent (success) [ 1.94s / 1.94s ]
104: Actions::Pulp::Repository::DistributorPublish (success) [ 256.10s / 10.73s ]
106: Actions::Katello::Repository::IndexContent (success) [ 0.22s / 0.22s ]
108: Actions::Katello::Repository::PurgeEmptyContent (success) [ 137.41s / 5.84s ]
113: Actions::Pulp::Repository::Refresh (success) [ 1.24s / 1.24s ]
116: Actions::Katello::Repository::CheckMatchingContent (success) [ 0.60s / 0.60s ]
119: Actions::Pulp::Repository::DistributorPublish (success) [ 0.05s / 0.05s ]
121: Actions::Katello::Repository::IndexContent (success) [ 1.30s / 1.30s ]
126: Actions::Pulp::Repository::DistributorPublish (pending)
128: Actions::Katello::Repository::IndexContent (success) [ 0.14s / 0.14s ]
130: Actions::Katello::Repository::PurgeEmptyContent (success) [ 191.25s / 26.28s ]
135: Actions::Pulp::Repository::Refresh (success) [ 0.86s / 0.86s ]
138: Actions::Katello::Repository::CheckMatchingContent (success) [ 1.37s / 1.37s ]
141: Actions::Pulp::Repository::DistributorPublish (success) [ 0.23s / 0.23s ]
143: Actions::Katello::Repository::IndexContent (success) [ 0.38s / 0.38s ]
148: Actions::Pulp::Repository::DistributorPublish (success) [ 378.76s / 182.35s ]
150: Actions::Katello::Repository::IndexContent (pending)
152: Actions::Katello::Repository::PurgeEmptyContent (success) [ 84.03s / 4.70s ]
157: Actions::Pulp::Repository::Refresh (success) [ 17.28s / 6.06s ]
160: Actions::Katello::Repository::CheckMatchingContent (success) [ 8.40s / 8.40s ]
163: Actions::Pulp::Repository::DistributorPublish (success) [ 67.79s / 14.97s ]
165: Actions::Katello::Repository::IndexContent (success) [ 0.15s / 0.15s ]
170: Actions::Pulp::Repository::DistributorPublish (success) [ 375.14s / 177.66s ]
172: Actions::Katello::Repository::IndexContent (success) [ 15.85s / 15.85s ]
174: Actions::Katello::Repository::PurgeEmptyContent (success) [ 96.74s / 31.35s ]
179: Actions::Pulp::Repository::Refresh (success) [ 3.29s / 3.29s ]
182: Actions::Katello::Repository::CheckMatchingContent (success) [ 15.25s / 15.25s ]
185: Actions::Pulp::Repository::DistributorPublish (success) [ 0.04s / 0.04s ]
187: Actions::Katello::Repository::IndexContent (success) [ 1.35s / 1.35s ]
192: Actions::Pulp::Repository::DistributorPublish (pending)
194: Actions::Katello::Repository::IndexContent (success) [ 0.24s / 0.24s ]
196: Actions::Katello::Repository::PurgeEmptyContent (pending)
201: Actions::Pulp::Repository::Refresh (success) [ 0.88s / 0.88s ]
204: Actions::Katello::Repository::CheckMatchingContent (success) [ 3.29s / 3.29s ]
207: Actions::Pulp::Repository::DistributorPublish (success) [ 0.07s / 0.07s ]
209: Actions::Katello::Repository::IndexContent (success) [ 0.22s / 0.22s ]
214: Actions::Pulp::Repository::DistributorPublish (success) [ 370.68s / 171.86s ]
216: Actions::Katello::Repository::IndexContent (success) [ 8.19s / 8.19s ]
218: Actions::Katello::Repository::PurgeEmptyContent (success) [ 130.66s / 104.29s ]
223: Actions::Pulp::Repository::Refresh (success) [ 5.95s / 5.95s ]
226: Actions::Katello::Repository::CheckMatchingContent (success) [ 5.26s / 5.26s ]
229: Actions::Pulp::Repository::DistributorPublish (success) [ 0.17s / 0.17s ]
231: Actions::Katello::Repository::IndexContent (success) [ 1.24s / 1.24s ]
236: Actions::Pulp::Repository::DistributorPublish (pending)
238: Actions::Katello::Repository::IndexContent (success) [ 0.25s / 0.25s ]
240: Actions::Katello::Repository::PurgeEmptyContent (waiting for Pulp to start the task) [ 4.49s / 4.49s ] Cancel

Queue: default

Started at: 2020-07-13 19:26:24 UTC

Ended at: 2020-07-13 19:26:28 UTC

Real time: 4.49s

Execution time (excluding suspended state): 4.49s

Input:

— id: 6945 remote_user: admin remote_cp_user: admin current_request_id: current_timezone: UTC current_user_id: 5 current_organization_id: 3 current_location_id:

Output:

— pulp_tasks: - exception: task_type: pulp.server.managers.repo.unit_association.unassociate_by_criteria _href: “/pulp/api/v2/tasks/7118c2e5-cd4a-4e0c-9898-8b7c90c49559/” task_id: 7118c2e5-cd4a-4e0c-9898-8b7c90c49559 tags: - pulp:repository:3-TDS_OPS_Ops-Server-CCV-v21_0-cd302179-0d94-4c51-b4fd-4803a9700cec - pulp:action:unassociate finish_time: _ns: task_status start_time: traceback: spawned_tasks: progress_report: {} queue: ‘’ state: waiting worker_name: result: error: _id: “$oid”: 5f0cb56297068962ba094669 id: 5f0cb56297068962ba094669 - exception: task_type: pulp.server.managers.repo.unit_association.unassociate_by_criteria _href: “/pulp/api/v2/tasks/e3702591-9c93-491e-8903-70c8e32409fc/” task_id: e3702591-9c93-491e-8903-70c8e32409fc tags: - pulp:repository:3-TDS_OPS_Ops-Server-CCV-v21_0-cd302179-0d94-4c51-b4fd-4803a9700cec - pulp:action:unassociate finish_time: _ns: task_status start_time: traceback: spawned_tasks: progress_report: {} queue: ‘’ state: waiting worker_name: result: error: _id: “$oid”: 5f0cb56497068962ba094694 id: 5f0cb56497068962ba094694 poll_attempts: total: 0 failed: 0
245: Actions::Pulp::Repository::Refresh (pending)
248: Actions::Katello::Repository::CheckMatchingContent (pending)
251: Actions::Pulp::Repository::DistributorPublish (pending)
253: Actions::Katello::Repository::IndexContent (pending)
258: Actions::Pulp::Repository::DistributorPublish (pending)
260: Actions::Katello::Repository::IndexContent (success) [ 0.51s / 0.51s ]
262: Actions::Katello::Repository::PurgeEmptyContent (waiting for Pulp to start the task) [ 4.19s / 4.19s ] Cancel
267: Actions::Pulp::Repository::Refresh (pending)
270: Actions::Katello::Repository::CheckMatchingContent (pending)
273: Actions::Pulp::Repository::DistributorPublish (pending)
275: Actions::Katello::Repository::IndexContent (pending)
280: Actions::Pulp::Repository::DistributorPublish (pending)
282: Actions::Katello::Repository::IndexContent (success) [ 3.12s / 3.12s ]
284: Actions::Katello::Repository::PurgeEmptyContent (success) [ 279.15s / 146.97s ]
289: Actions::Pulp::Repository::Refresh (success) [ 32.39s / 6.92s ]
292: Actions::Katello::Repository::CheckMatchingContent (success) [ 4.79s / 4.79s ]
295: Actions::Pulp::Repository::DistributorPublish (success) [ 0.24s / 0.24s ]
297: Actions::Katello::Repository::IndexContent (success) [ 1.22s / 1.22s ]
302: Actions::Pulp::Repository::DistributorPublish (pending)
304: Actions::Katello::Repository::IndexContent (success) [ 0.45s / 0.45s ]
306: Actions::Katello::Repository::PurgeEmptyContent (success) [ 168.58s / 6.08s ]
311: Actions::Pulp::Repository::Refresh (pending)
314: Actions::Katello::Repository::CheckMatchingContent (success) [ 1.90s / 1.90s ]
317: Actions::Pulp::Repository::DistributorPublish (success) [ 0.06s / 0.06s ]
319: Actions::Katello::Repository::IndexContent (success) [ 1.13s / 1.13s ]
324: Actions::Pulp::Repository::DistributorPublish (pending)
326: Actions::Katello::Repository::IndexContent (success) [ 0.14s / 0.14s ]
328: Actions::Katello::Repository::PurgeEmptyContent (waiting for Pulp to start the task) [ 6.18s / 6.18s ] Cancel
333: Actions::Pulp::Repository::Refresh (pending)
336: Actions::Katello::Repository::CheckMatchingContent (pending)
339: Actions::Pulp::Repository::DistributorPublish (pending)
341: Actions::Katello::Repository::IndexContent (pending)
344: Actions::Katello::ContentViewPuppetEnvironment::Create (pending)
347: Actions::Pulp::ContentViewPuppetEnvironment::IndexContent (pending)
354: Actions::Katello::ContentView::ErrataMail (pending)
356: Actions::Katello::ContentView::Publish (pending)

sequence concurrence

Back

Googling I found this Pulp tasks stuck as Waiting, Unstarted, Incomplete. A little dated but similar. So I took at look at /etc/default/pulp_workers and it was set to 4. I dropped it down to 2 and restarted all the services.

Any one have any ideas???

Louis_Bohm · July 14, 2020, 12:04pm

One of my tasks just failed. Here is the Dynaflow for that task.

Dynflow

Execution plans

Status

Back to tasks

Id: aaf0ad36-df8e-49f7-bee5-f7bfa27ab512

Label: Actions::Katello::ContentView::Publish

Status: running Cancel

Result: error

Started at: 2020-07-13 19:08:53 UTC

Ended at:

Plan

Run

Finalize

Execution History

6: Actions::Pulp::Repository::Create (success) [ 13.41s / 13.41s ]
7: Actions::Katello::Repository::Create (success) [ 0.02s / 0.02s ]
10: Actions::Pulp::Repository::Create (success) [ 9.79s / 9.79s ]
11: Actions::Katello::Repository::Create (success) [ 0.02s / 0.02s ]
14: Actions::Pulp::Repository::Create (success) [ 12.90s / 12.90s ]
15: Actions::Katello::Repository::Create (success) [ 0.04s / 0.04s ]
18: Actions::Pulp::Repository::Create (success) [ 13.06s / 13.06s ]
19: Actions::Katello::Repository::Create (success) [ 0.02s / 0.02s ]
22: Actions::Pulp::Repository::Create (success) [ 1.79s / 1.79s ]
23: Actions::Katello::Repository::Create (success) [ 0.04s / 0.04s ]
26: Actions::Pulp::Repository::Create (success) [ 7.24s / 7.24s ]
27: Actions::Katello::Repository::Create (success) [ 0.03s / 0.03s ]
30: Actions::Pulp::Repository::Create (success) [ 6.08s / 6.08s ]
31: Actions::Katello::Repository::Create (success) [ 0.04s / 0.04s ]
34: Actions::Pulp::Repository::Create (success) [ 4.10s / 4.10s ]
35: Actions::Katello::Repository::Create (success) [ 0.03s / 0.03s ]
38: Actions::Pulp::Repository::Create (success) [ 3.00s / 3.00s ]
39: Actions::Katello::Repository::Create (success) [ 0.02s / 0.02s ]
42: Actions::Pulp::Repository::Create (success) [ 4.44s / 4.44s ]
43: Actions::Katello::Repository::Create (success) [ 0.02s / 0.02s ]
46: Actions::Pulp::Repository::Create (success) [ 6.15s / 6.15s ]
47: Actions::Katello::Repository::Create (success) [ 0.02s / 0.02s ]
50: Actions::Pulp::Repository::Create (success) [ 5.99s / 5.99s ]
51: Actions::Katello::Repository::Create (success) [ 0.15s / 0.15s ]
54: Actions::Pulp::Repository::Create (success) [ 6.19s / 6.19s ]
55: Actions::Katello::Repository::Create (success) [ 0.02s / 0.02s ]
60: Actions::Pulp::Repository::DistributorPublish (success) [ 301.05s / 10.21s ]
62: Actions::Katello::Repository::IndexContent (pending)
64: Actions::Katello::Repository::PurgeEmptyContent (success) [ 207.20s / 7.46s ]
69: Actions::Pulp::Repository::Refresh (success) [ 91.39s / 6.71s ]
72: Actions::Katello::Repository::CheckMatchingContent (success) [ 1.22s / 1.22s ]
75: Actions::Pulp::Repository::DistributorPublish (success) [ 33.29s / 2.04s ]
77: Actions::Katello::Repository::IndexContent (success) [ 1.60s / 1.60s ]
82: Actions::Pulp::Repository::DistributorPublish (waiting for Pulp to finish the task) [ 3.57s / 3.57s ] Cancel
84: Actions::Katello::Repository::IndexContent (success) [ 0.64s / 0.64s ]
86: Actions::Katello::Repository::PurgeEmptyContent (success) [ 228.01s / 10.21s ]
91: Actions::Pulp::Repository::Refresh (success) [ 171.66s / 22.67s ]
94: Actions::Katello::Repository::CheckMatchingContent (success) [ 2.48s / 2.48s ]
97: Actions::Pulp::Repository::DistributorPublish (success) [ 0.23s / 0.23s ]
99: Actions::Katello::Repository::IndexContent (success) [ 0.33s / 0.33s ]
104: Actions::Pulp::Repository::DistributorPublish (success) [ 292.70s / 5.24s ]
106: Actions::Katello::Repository::IndexContent (pending)
108: Actions::Katello::Repository::PurgeEmptyContent (success) [ 206.52s / 7.52s ]
113: Actions::Pulp::Repository::Refresh (success) [ 0.48s / 0.48s ]
116: Actions::Katello::Repository::CheckMatchingContent (success) [ 1.97s / 1.97s ]
119: Actions::Pulp::Repository::DistributorPublish (success) [ 0.03s / 0.03s ]
121: Actions::Katello::Repository::IndexContent (success) [ 0.36s / 0.36s ]
126: Actions::Pulp::Repository::DistributorPublish (success) [ 48.50s / 5.51s ]
128: Actions::Katello::Repository::IndexContent (success) [ 0.50s / 0.50s ]
130: Actions::Katello::Repository::PurgeEmptyContent (success) [ 59423.68s / 24.59s ]
135: Actions::Pulp::Repository::Refresh (success) [ 0.76s / 0.76s ]
138: Actions::Katello::Repository::CheckMatchingContent (success) [ 1.96s / 1.96s ]
141: Actions::Pulp::Repository::DistributorPublish (success) [ 0.05s / 0.05s ]
143: Actions::Katello::Repository::IndexContent (success) [ 1.61s / 1.61s ]
148: Actions::Pulp::Repository::DistributorPublish (waiting for Pulp to finish the task) [ 59924.63s / 227.77s ] Cancel
150: Actions::Katello::Repository::IndexContent (pending)
152: Actions::Katello::Repository::PurgeEmptyContent (pending)
157: Actions::Pulp::Repository::Refresh (pending)
160: Actions::Katello::Repository::CheckMatchingContent (pending)
163: Actions::Pulp::Repository::DistributorPublish (pending)
165: Actions::Katello::Repository::IndexContent (pending)
170: Actions::Pulp::Repository::DistributorPublish (waiting for Pulp to finish the task) [ 69.39s / 46.46s ] Cancel
172: Actions::Katello::Repository::IndexContent (success) [ 13.82s / 13.82s ]
174: Actions::Katello::Repository::PurgeEmptyContent (pending)
179: Actions::Pulp::Repository::Refresh (waiting for Pulp to start the task) [ 16.20s / 16.20s ] Cancel
182: Actions::Katello::Repository::CheckMatchingContent (pending)
185: Actions::Pulp::Repository::DistributorPublish (pending)
187: Actions::Katello::Repository::IndexContent (pending)
192: Actions::Pulp::Repository::DistributorPublish (success) [ 51.33s / 13.84s ]
194: Actions::Katello::Repository::IndexContent (success) [ 2.12s / 2.12s ]
196: Actions::Katello::Repository::PurgeEmptyContent (waiting for Pulp to start the task) [ 13.01s / 3.79s ] Cancel
201: Actions::Pulp::Repository::Refresh (success) [ 1.68s / 1.68s ]
204: Actions::Katello::Repository::CheckMatchingContent (success) [ 1.28s / 1.28s ]
207: Actions::Pulp::Repository::DistributorPublish (error) [ 87.16s / 2.24s ]

Queue: default

Started at: 2020-07-13 19:30:31 UTC

Ended at: 2020-07-13 19:31:58 UTC

Real time: 87.16s

Execution time (excluding suspended state): 2.24s

Input:

— smart_proxy_id: 1 options: force: false source_repository: id: 6911 name: Wazuh_-Base-_x86_64 url: http://fm.tdsops.com/pulp/repos/TDS-OPS/content_views/TDS_OPS_JIRA-Server-CCV/22.0/custom/Wazuh/Wazuh_-_Base_-_x86_64/ matching_content: false dependency: dependency: repository_id: 6522 remote_user: admin remote_cp_user: admin current_request_id: current_timezone: UTC current_user_id: 5 current_organization_id: 3 current_location_id:

Output:

— pulp_tasks: - exception: task_type: pulp.server.managers.repo.publish.publish _href: “/pulp/api/v2/tasks/bae44a64-2fb4-4905-93a0-1f0e33b6b07e/” task_id: bae44a64-2fb4-4905-93a0-1f0e33b6b07e tags: - pulp:repository:3-TDS_OPS_JIRA-Server-CCV-Library-837a301e-075b-4fc8-b428-928735f8f5f0 - pulp:action:publish finish_time: ‘2020-07-13T19:31:25Z’ _ns: task_status start_time: ‘2020-07-13T19:31:16Z’ traceback: Traceback (most recent call last): File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 367, in trace_task R = retval = fun(*args, **kwargs) File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 686, in call return super(Task, self).call(*args, **kwargs) File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 108, in call return super(PulpTask, self).call(*args, **kwargs) File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 622, in protected_call return self.run(*args, **kwargs) File “/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py”, line 1133, in publish result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config) File “/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py”, line 1235, in check_publish result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config) File “/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py”, line 1287, in do_publish publish_report = publish_repo(transfer_repo, conduit, call_config) File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 894, in wrap_f return f(*args, **kwargs) File “/usr/lib/python2.7/site-packages/pulp_katello/distributors/yum_clone_distributor/distributor.py”, line 131, in publish_repo source_working_dir = self.source_working_dir(source_repo_id) File “/usr/lib/python2.7/site-packages/pulp_katello/distributors/yum_clone_distributor/distributor.py”, line 108, in source_working_dir realpath = self.valid_symlink_dest(os.path.join(HTTPS_PUBLISH_DIR, relative_path)) File “/usr/lib/python2.7/site-packages/pulp_katello/distributors/yum_clone_distributor/distributor.py”, line 118, in valid_symlink_dest raise Exception(‘%s exists, but should be a symlink. Cannot find published directory.’ % path) Exception: /var/lib/pulp/published/yum/https/repos/TDS-OPS/content_views/TDS_OPS_JIRA-Server-CCV/22.0/custom/Wazuh/Wazuh-Base-x86_64 exists, but should be a symlink. Cannot find published directory. spawned_tasks: [] progress_report: {} queue: reserved_resource_worker-3@fm.tdsops.com.dq2 state: error worker_name: reserved_resource_worker-3@fm.tdsops.com result: error: code: PLP0000 data: {} description: "/var/lib/pulp/published/yum/https/repos/TDS-OPS/content_views/TDS_OPS_JIRA-Server-CCV/22.0/custom/Wazuh/Wazuh-Base-_x86_64 exists, but should be a symlink. Cannot find published directory." sub_errors: _id: “$oid”: 5f0cb65897068962ba0966b8 id: 5f0cb65897068962ba0966b8 poll_attempts: total: 1 failed: 1

Error:

Katello::Errors::PulpError

PLP0000: /var/lib/pulp/published/yum/https/repos/TDS-OPS/content_views/TDS_OPS_JIRA-Server-CCV/22.0/custom/Wazuh/Wazuh_-Base-_x86_64 exists, but should be a symlink. Cannot find published directory.

— - “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.2/app/lib/actions/pulp/abstract_async_task.rb:122:in block in external_task='" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.2/app/lib/actions/pulp/abstract_async_task.rb:120:in each’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.2/app/lib/actions/pulp/abstract_async_task.rb:120:in external_task='" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action/polling.rb:100:in poll_external_task_with_rescue’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action/polling.rb:22:in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action/cancellable.rb:14:in run’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.2/app/lib/actions/pulp/abstract_async_task.rb:45:in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:563:in block (3 levels) in execute_run’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in pass’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.2/app/lib/actions/middleware/skip_if_matching_content.rb:5:in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in call’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in pass’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:32:in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in call’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in pass’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.2/app/lib/actions/middleware/remote_action.rb:16:in block in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.2/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.2/app/models/katello/concerns/user_extensions.rb:21:in cp_config'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.2/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.2/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.2/app/lib/actions/middleware/remote_action.rb:16:in run’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in pass’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in wrap'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in pass’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action/progress.rb:31:in with_progress_calculation’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action/progress.rb:17:in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in call’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in pass’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in call’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in pass’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in call’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in pass’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in block in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in call’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in pass’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in call’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in pass’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:32:in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in call’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/world.rb:31:in execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:562:in block (2 levels) in execute_run’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:561:in catch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:561:in block in execute_run’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:475:in block in with_error_handling'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:475:in catch’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:475:in with_error_handling'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:556:in execute_run’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:285:in execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/director.rb:93:in execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors.rb:12:in run_user_code'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in module:Sidekiq’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog’” - “/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’”
209: Actions::Katello::Repository::IndexContent (pending)
214: Actions::Pulp::Repository::DistributorPublish (success) [ 58.22s / 43.94s ]
216: Actions::Katello::Repository::IndexContent (success) [ 9.83s / 9.83s ]
218: Actions::Katello::Repository::PurgeEmptyContent (success) [ 98.98s / 81.91s ]
223: Actions::Pulp::Repository::Refresh (success) [ 219.11s / 26.33s ]
226: Actions::Katello::Repository::CheckMatchingContent (success) [ 2.53s / 2.53s ]
229: Actions::Pulp::Repository::DistributorPublish (success) [ 58955.78s / 9.38s ]
231: Actions::Katello::Repository::IndexContent (success) [ 0.11s / 0.11s ]
236: Actions::Pulp::Repository::DistributorPublish (pending)
238: Actions::Katello::Repository::IndexContent (success) [ 0.06s / 0.06s ]
240: Actions::Katello::Repository::PurgeEmptyContent (success) [ 59398.03s / 8.66s ]
245: Actions::Pulp::Repository::Refresh (success) [ 74.38s / 15.62s ]
248: Actions::Katello::Repository::CheckMatchingContent (success) [ 2.05s / 2.05s ]
251: Actions::Pulp::Repository::DistributorPublish (success) [ 0.19s / 0.19s ]
253: Actions::Katello::Repository::IndexContent (success) [ 0.30s / 0.30s ]
258: Actions::Pulp::Repository::DistributorPublish (pending)
260: Actions::Katello::Repository::IndexContent (success) [ 0.31s / 0.31s ]
262: Actions::Katello::Repository::PurgeEmptyContent (success) [ 69.86s / 10.16s ]
267: Actions::Pulp::Repository::Refresh (success) [ 60.51s / 6.27s ]
270: Actions::Katello::Repository::CheckMatchingContent (success) [ 9.82s / 9.82s ]
273: Actions::Pulp::Repository::DistributorPublish (success) [ 0.14s / 0.14s ]
275: Actions::Katello::Repository::IndexContent (success) [ 0.57s / 0.57s ]
280: Actions::Pulp::Repository::DistributorPublish (success) [ 55.30s / 33.39s ]
282: Actions::Katello::Repository::IndexContent (success) [ 2.34s / 2.34s ]
284: Actions::Katello::Repository::PurgeEmptyContent (success) [ 351.02s / 145.64s ]
289: Actions::Pulp::Repository::Refresh (success) [ 107.17s / 21.28s ]
292: Actions::Katello::Repository::CheckMatchingContent (success) [ 2.66s / 2.66s ]
295: Actions::Pulp::Repository::DistributorPublish (success) [ 0.05s / 0.05s ]
297: Actions::Katello::Repository::IndexContent (success) [ 1.78s / 1.78s ]
302: Actions::Pulp::Repository::DistributorPublish (success) [ 272.68s / 4.92s ]
304: Actions::Katello::Repository::IndexContent (success) [ 1.37s / 1.37s ]
306: Actions::Katello::Repository::PurgeEmptyContent (success) [ 199.19s / 7.41s ]
311: Actions::Pulp::Repository::Refresh (success) [ 89.90s / 10.63s ]
314: Actions::Katello::Repository::CheckMatchingContent (success) [ 0.37s / 0.37s ]
317: Actions::Pulp::Repository::DistributorPublish (success) [ 31.20s / 1.34s ]
319: Actions::Katello::Repository::IndexContent (success) [ 1.05s / 1.05s ]
324: Actions::Pulp::Repository::DistributorPublish (success) [ 279.35s / 42.59s ]
326: Actions::Katello::Repository::IndexContent (success) [ 0.64s / 0.64s ]
328: Actions::Katello::Repository::PurgeEmptyContent (success) [ 59159.98s / 8.33s ]
333: Actions::Pulp::Repository::Refresh (success) [ 75.11s / 14.63s ]
336: Actions::Katello::Repository::CheckMatchingContent (success) [ 1.67s / 1.67s ]
339: Actions::Pulp::Repository::DistributorPublish (success) [ 0.04s / 0.04s ]
341: Actions::Katello::Repository::IndexContent (success) [ 0.18s / 0.18s ]
344: Actions::Katello::ContentViewPuppetEnvironment::Create (pending)
347: Actions::Pulp::ContentViewPuppetEnvironment::IndexContent (pending)
354: Actions::Katello::ContentView::ErrataMail (pending)
356: Actions::Katello::ContentView::Publish (pending)

sequence concurrence

Back

Louis_Bohm · July 14, 2020, 3:52pm

Forget this issue. We were having so many issues we opted to revert to a snapshot from before the upgrade.

Lets see if Katello 3.16 fixes these issues when its finally out and not in RC.