EPEL 8 Repo Sync Failure

Support
1

Problem:
When I try to sync an upstream EPEL 8 repository to Foreman, it fails with the message:
[Errno 2] No such file or directory: '/var/lib/pulp/tmp/3142565@FOREMAN_DOMAIN/tmpwmoq9ka3/tmp2maby49w'
Expected outcome:
The repository sync should succeed and create a mirror of the online repository.
Foreman and Proxy versions:
foreman 3.6.1-1
foreman-proxy 3.6.1-1
Foreman and Proxy plugin versions:
foreman-tasks 7.2.1
foreman_bootdisk 21.0.4
foreman_discovery 22.0.4
foreman_remote_execution 9.1.0
katello 4.8.1
Distribution and version:
AlmaLinux 8.8
Other relevant data:
Exception:
Katello::Errors::Pulp3Error: [Errno 2] No such file or directory: '/var/lib/pulp/tmp/3142565@FOREMAN_DOMAIN/tmpwmoq9ka3/tmp2maby49w'
Backtrace:

/usr/share/gems/gems/katello-4.8.1/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
/usr/share/gems/gems/katello-4.8.1/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
/usr/share/gems/gems/katello-4.8.1/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
/usr/share/gems/gems/katello-4.8.1/app/lib/actions/pulp3/abstract_async_task.rb:160:in `poll_external_task'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:22:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/cancellable.rb:14:in `run'
/usr/share/gems/gems/katello-4.8.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/katello-4.8.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/usr/share/gems/gems/katello-4.8.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/usr/share/gems/gems/katello-4.8.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/usr/share/gems/gems/katello-4.8.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/usr/share/gems/gems/katello-4.8.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/usr/share/gems/gems/katello-4.8.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/execution_wrapper.rb:91:in `wrap'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `catch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:575:in `execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director.rb:94:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:196:in `execute_job'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:163:in `block in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:112:in `local'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq.rb:39:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:257:in `stats'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:13:in `call'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:79:in `global'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:124:in `block in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/logger.rb:11:in `with'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:33:in `prepare'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:123:in `dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:162:in `process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:78:in `process_one'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:68:in `run'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:43:in `watchdog'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:52:in `block in safe_thread'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2

Hi @BonkTurnip

Pulp logs may be more helpful here. What do you see in the output of journalctl -u *pulp* from when the sync was attempted?

What mirroring policy is in use for the repository?

Have you tried an advanced sync?

3

Hello @jeremylenz ,

Here are the lines from journalctl -u *pulp* relevant to the error:

Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]: pulp [7148cf2d-113a-4cf9-800f-1543f383b195]: pulpcore.tasking.pulpcore_worker:INFO: Task 9aa0f651-5f6e-4791-90a8-674d686e1b16 failed ([Errno 2] No such file or directory: '/var/lib/pulp/tmp/3142565@FOREMAN_DOMAIN/tmp4x0uufy_/tmp55su83_r')
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]: pulp [7148cf2d-113a-4cf9-800f-1543f383b195]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 444, in _perform_task
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     result = func(*args, **kwargs)
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 567, in synchronize
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     repo_version = dv.create() or repo.latest_version()
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     loop.run_until_complete(pipeline)
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     return future.result()
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     await asyncio.gather(*futures)
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     await self.run()
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 250, in run
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     await sync_to_async(Artifact.objects.bulk_get_or_create)(
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 448, in __call__
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     ret = await asyncio.wait_for(future, timeout=None)
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib64/python3.9/asyncio/tasks.py", line 442, in wait_for
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     return await fut
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     result = self.fn(*self.args, **self.kwargs)
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 490, in thread_handler
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     return func(*args, **kwargs)
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/pulpcore/app/models/content.py", line 87, in bulk_get_or_create
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     return super().bulk_create(objs, batch_size=batch_size)
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     return getattr(self.get_queryset(), name)(*args, **kwargs)
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 502, in bulk_create
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     returned_columns = self._batched_insert(
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 1287, in _batched_insert
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     inserted_rows.extend(self._insert(
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     return query.get_compiler(using=using).execute_sql(returning_fields)
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1415, in execute_sql
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     for sql, params in self.as_sql():
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/django_readonly_field/compiler.py", line 31, in as_sql
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     return super(ReadonlySQLCompilerMixin, self).as_sql()
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1358, in as_sql
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     value_rows = [
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1310, in pre_save_val
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     return field.pre_save(obj, add=True)
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:   File "/usr/lib/python3.9/site-packages/pulpcore/app/models/fields.py", line 73, in pre_save
Jun 20 00:01:17 FOREMAN_DOMAIN pulpcore-worker-3[3827034]:     file._file = TemporaryDownloadedFile(open(file.name, "rb"))

I’m using the “Content Only” mirroring policy.
I tried a complete advanced sync and it failed with the same error message.

4

It could be a permissions issue.

image

If your permissions don’t look like this, re-running foreman-installer (with no arguments) should help restore them properly.

5

It looks like my permissions match those that you show in the screenshot.
I get:
drwxrwxr-x. 8 pulp pulp 94 Jun 2 13:44 /var/lib/pulp/
and
drwxr-x---. 6 pulp pulp 146 Jun 20 08:42 /var/lib/pulp/tmp/
respectively.
For the sake of completeness, I ran foreman-installer with no arguments and tried to sync the repo, but I still got the same error.

6

hmm… is it maybe running out of disk space before it creates the file? In Infrastructure > Smart Proxies > (your Foreman server) how’s the Pulp storage looking?

Is this the only repo that has the issue?

You could also maybe try creating a new repository with the same upstream URL and see if the issue persists there.

7

Pulp storage shows 38 GB out of 490 GB used, so there should be plenty of available space.
Yes, this is the only repo that has this issue. I even tried syncing EPEL 9 and that worked perfectly, but as that’s not compatible with AL8, I cannot use that one.
I’ve also recreated the repository a bunch of times to no avail.

8

What url are you syncing? I can try to see if I can reproduce it here.

I just synced Index of /pub/epel/8/Everything/x86_64 and it completed with no errors…

9

I’m syncing:
https://download-ib01.fedoraproject.org/pub/epel/8/Everything/x86_64/

10

Blockquote

I get an HTTP 403 Forbidden when I even attempt to sync this repo. I expect it’s not meant to be a stable URL

11

I have tried many different upstream repos and the one that I pasted above was just the last one I had tried. Every other EPEL 8 repo I have attempted to sync gave the same error.
I even changed to the one that @jeremylenz used and it still did not work.

12

well, 215 mirrors to choose from so I suggest pick something close:
https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/8/x86_64
Also make sure you are not blocked by a proxy/firewall or something similar.

13

I’ve tried almost a dozen of the mirrors in my region on that list and they have all failed in the same way. It can’t be a proxy/firewall issue because I have six other repos that I sync daily without any issue.
I don’t know if this means anything, but it never fails immediately; it always seems to fail on step 2010 +/- 5.

14

Just as an experiment to gather more information: Can you try changing the “Download Policy” to “On Demand” and see if that changes anything? (I realize this may not be what you ultimately want, but if the sync succeeds/fails with the changed download policy that can provide additional information where to look.)

Another question: Do you always get exactly the error:

Katello::Errors::Pulp3Error: [Errno 2] No such file or directory: '/var/lib/pulp/tmp/3142565@FOREMAN_DOMAIN/tmpwmoq9ka3/tmp2maby49w'

Down to the identical temp folder name, or is it a different temp folder each time?
If it is the exact same folder each time maybe try simply deleting that folder and then re-running the sync.

15

Also the number before the @ should be a PID. Is there a process running? Did you restart the whole foreman stack lately?

16

I changed the download policy to “On Demand” and the sync succeeded.

I do not always get exactly the same error; the temp folder names are different each time. When I was initially troubleshooting this problem, I followed the same train of thought by deleting all of the temp folders in /var/lib/pulp/tmp/ but that did not solve the problem.

Regarding your comments @gvde , there is a process running with the PID that shows up in the error.
During the troubleshooting process, I restarted the foreman stack to see if the issue would be resolved, but it was not. If I recall correctly, I tried systemctl restart foreman .

17

If you now change it back to “immediate” and sync again, does it then fail again?

I think this was a long shot idea of mine anyway I think since at least Pulp 3 it is pretty good about using new blank temp folders each time.

Just FYI: The most reliable way to restart all foreman services is foreman-maintain service restart. This will restart all related services in a sane order as designed by the devs.

1 Like
18

Yes, when I change it back to “Immediate” the sync fails again.

Thanks for the information! I’ve added this command to my notes.

19

That only restarts the foreman service not all related services. As @quba42 mentioned you need to your foreman-maintain service restart to restart everything.

You should check /var/log/foreman/production.log for errors. Also check the log of the pulp process which uses the PID you see, e.g.

# systemctl status 1084232
â—Ź pulpcore-worker@5.service - Pulp Worker
...
# journalctl -u pulpcore-worker@5.service

and check for additional errors.

20

Here’s the error I get from the pulp process with the PID I see:

Jun 27 08:52:13 FOREMAN_DOMAIN pulpcore-worker-4[1258043]: pulp [122450f8-8adf-47ee-aeec-b7993e69e723]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 525baac6-2113-4afd-b98f-ee1bcd9d9693
Jun 27 08:52:13 FOREMAN_DOMAIN pulpcore-worker-4[1258043]: pulp [122450f8-8adf-47ee-aeec-b7993e69e723]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=EPEL_8_x86_64-38628 remote=EPEL_8_x86_64-383990
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]: pulp [122450f8-8adf-47ee-aeec-b7993e69e723]: pulpcore.tasking.pulpcore_worker:INFO: Task 525baac6-2113-4afd-b98f-ee1bcd9d9693 failed ([Errno 2] No such file or directory: '/var/lib/pulp/tmp/3142395@FOREMAN_DOMAIN/tmp2ttzs5qm/tmpuy8b6w55')
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]: pulp [122450f8-8adf-47ee-aeec-b7993e69e723]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 444, in _perform_task
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     result = func(*args, **kwargs)
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 567, in synchronize
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     repo_version = dv.create() or repo.latest_version()
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     loop.run_until_complete(pipeline)
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     return future.result()
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     await asyncio.gather(*futures)
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     await self.run()
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 250, in run
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     await sync_to_async(Artifact.objects.bulk_get_or_create)(
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 448, in __call__
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     ret = await asyncio.wait_for(future, timeout=None)
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib64/python3.9/asyncio/tasks.py", line 442, in wait_for
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     return await fut
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     result = self.fn(*self.args, **self.kwargs)
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 490, in thread_handler
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     return func(*args, **kwargs)
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/pulpcore/app/models/content.py", line 87, in bulk_get_or_create
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     return super().bulk_create(objs, batch_size=batch_size)
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     return getattr(self.get_queryset(), name)(*args, **kwargs)
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 502, in bulk_create
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     returned_columns = self._batched_insert(
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 1287, in _batched_insert
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     inserted_rows.extend(self._insert(
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     return query.get_compiler(using=using).execute_sql(returning_fields)
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1415, in execute_sql
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     for sql, params in self.as_sql():
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/django_readonly_field/compiler.py", line 31, in as_sql
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     return super(ReadonlySQLCompilerMixin, self).as_sql()
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1358, in as_sql
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     value_rows = [
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1310, in pre_save_val
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     return field.pre_save(obj, add=True)
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:   File "/usr/lib/python3.9/site-packages/pulpcore/app/models/fields.py", line 73, in pre_save
Jun 27 08:52:57 FOREMAN_DOMAIN pulpcore-worker-4[1258043]:     file._file = TemporaryDownloadedFile(open(file.name, "rb"))

Here’s something else I found in the worker journal, though it may be completely unrelated:

Jun 02 14:01:51 FOREMAN_DOMAIN pulpcore-worker-4[212597]: pulp [00afb7b1-c650-451f-aead-829f25530141]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=extras_x86_64_os-43013, version=1
Jun 02 14:01:52 FOREMAN_DOMAIN pulpcore-worker-4[212597]: Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7f992798c040>
Jun 02 14:01:52 FOREMAN_DOMAIN pulpcore-worker-4[212597]: Traceback (most recent call last):
Jun 02 14:01:52 FOREMAN_DOMAIN pulpcore-worker-4[212597]:   File "/usr/lib64/python3.9/tempfile.py", line 445, in __del__
Jun 02 14:01:52 FOREMAN_DOMAIN pulpcore-worker-4[212597]:     self.close()
Jun 02 14:01:52 FOREMAN_DOMAIN pulpcore-worker-4[212597]:   File "/usr/lib64/python3.9/tempfile.py", line 441, in close
Jun 02 14:01:52 FOREMAN_DOMAIN pulpcore-worker-4[212597]:     unlink(self.name)
Jun 02 14:01:52 FOREMAN_DOMAIN pulpcore-worker-4[212597]: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/tmp/210807@FOREMAN_DOMAIN/tmpe2k11nqr/tmpr2dek1zp'

Here’s the error that shows up in /var/log/foreman/production.log :

2023-06-27T08:53:01 [E|bac|122450f8] [Errno 2] No such file or directory: '/var/lib/pulp/tmp/3142395@FOREMAN_DOMAIN/tmp2ttzs5qm/tmpuy8b6w55' (Katello::Errors::Pulp3Error)
 122450f8 | /usr/share/gems/gems/katello-4.8.1/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
 122450f8 | /usr/share/gems/gems/katello-4.8.1/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
 122450f8 | /usr/share/gems/gems/katello-4.8.1/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
 122450f8 | /usr/share/gems/gems/katello-4.8.1/app/lib/actions/pulp3/abstract_async_task.rb:160:in `poll_external_task'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:22:in `run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/cancellable.rb:14:in `run'
 122450f8 | /usr/share/gems/gems/katello-4.8.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 122450f8 | /usr/share/gems/gems/katello-4.8.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 122450f8 | /usr/share/gems/gems/katello-4.8.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 122450f8 | /usr/share/gems/gems/katello-4.8.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 122450f8 | /usr/share/gems/gems/katello-4.8.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 122450f8 | /usr/share/gems/gems/katello-4.8.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 122450f8 | /usr/share/gems/gems/katello-4.8.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 122450f8 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/execution_wrapper.rb:91:in `wrap'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:17:in `run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 122450f8 | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `catch'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `block in execute_run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `block in with_error_handling'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `catch'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `with_error_handling'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:575:in `execute_run'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:296:in `execute'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director.rb:94:in `execute'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors.rb:18:in `run_user_code'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 122450f8 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 122450f8 | [ sidekiq ]
 122450f8 | [ concurrent-ruby ]
2023-06-27T08:53:03 [I|bac|122450f8] Task {label: Actions::Katello::Repository::Sync, id: 3471df78-b2b1-4efb-b30a-49695bd8f0f1, execution_plan_id: 67060843-2df6-4610-8002-ddc2f92430d1} state changed: stopped  result: warning