On Foreman/Katello 3.3/4.5 and when trying to add the CentOS 7 Debug Info repo I get the error “Katello::Errors::Pulp3Error: Pulp task error”
Repo setup:
URL: Index of /7/x86_64
Verify SSL: No
Download Policy: On Demand
Mirroring Policy: Content Only
Anyone know what this is about?
Logs from production.log:
[root@foreman ~]# grep cd9d284c /var/log/foreman/production.log
2022-09-13T15:54:32 [I|app|cd9d284c] Started POST "/katello/api/v2/repositories/bulk/sync?organization_id=1" for 10.10.10.10 at 2022-09-13 15:54:32 +0000
2022-09-13T15:54:32 [I|app|cd9d284c] Processing by Katello::Api::V2::RepositoriesBulkActionsController#sync_repositories as HTML
2022-09-13T15:54:32 [I|app|cd9d284c] Parameters: {"ids"=>[143], "organization_id"=>"1", "api_version"=>"v2", "repositories_bulk_action"=>{"ids"=>[143]}}
2022-09-13T15:54:32 [I|bac|cd9d284c] Task {label: , execution_plan_id: 52b248b8-a1b2-4c64-9b2d-fdbfa9470f19} state changed: pending
2022-09-13T15:54:32 [I|bac|cd9d284c] Task {label: Actions::BulkAction, id: 0b54a86e-2ecb-4151-bd06-5151f06f77c5, execution_plan_id: 52b248b8-a1b2-4c64-9b2d-fdbfa9470f19} state changed: planning
2022-09-13T15:54:32 [I|bac|cd9d284c] Task {label: Actions::BulkAction, id: 0b54a86e-2ecb-4151-bd06-5151f06f77c5, execution_plan_id: 52b248b8-a1b2-4c64-9b2d-fdbfa9470f19} state changed: planned
2022-09-13T15:54:32 [I|app|cd9d284c] Rendering /usr/share/gems/gems/katello-4.5.0/app/views/katello/api/v2/common/async.json.rabl within katello/api/v2/layouts/resource
2022-09-13T15:54:32 [I|app|cd9d284c] Rendered /usr/share/gems/gems/katello-4.5.0/app/views/katello/api/v2/common/async.json.rabl within katello/api/v2/layouts/resource (Duration: 13.0ms | Allocations: 5623)
2022-09-13T15:54:32 [I|app|cd9d284c] Completed 202 Accepted in 88ms (Views: 14.5ms | ActiveRecord: 16.6ms | Allocations: 31358)
2022-09-13T15:54:32 [I|bac|cd9d284c] Task {label: Actions::BulkAction, id: 0b54a86e-2ecb-4151-bd06-5151f06f77c5, execution_plan_id: 52b248b8-a1b2-4c64-9b2d-fdbfa9470f19} state changed: running
2022-09-13T15:54:32 [I|bac|cd9d284c] Task {label: , execution_plan_id: a0fb4704-5bbe-4f8f-9fff-e887b22e36e4} state changed: pending
2022-09-13T15:54:32 [I|bac|cd9d284c] Task {label: Actions::Katello::Repository::Sync, id: 45f96414-95a4-49e4-ae90-32ed5902ce05, execution_plan_id: a0fb4704-5bbe-4f8f-9fff-e887b22e36e4} state changed: planning
2022-09-13T15:54:32 [I|bac|cd9d284c] Task {label: Actions::Katello::Repository::Sync, id: 45f96414-95a4-49e4-ae90-32ed5902ce05, execution_plan_id: a0fb4704-5bbe-4f8f-9fff-e887b22e36e4} state changed: planned
2022-09-13T15:54:33 [I|bac|cd9d284c] Task {label: Actions::Katello::Repository::Sync, id: 45f96414-95a4-49e4-ae90-32ed5902ce05, execution_plan_id: a0fb4704-5bbe-4f8f-9fff-e887b22e36e4} state changed: running
2022-09-13T15:56:24 [E|bac|cd9d284c] Pulp task error (Katello::Errors::Pulp3Error)
cd9d284c | /usr/share/gems/gems/katello-4.5.0/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
cd9d284c | /usr/share/gems/gems/katello-4.5.0/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
cd9d284c | /usr/share/gems/gems/katello-4.5.0/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
cd9d284c | /usr/share/gems/gems/katello-4.5.0/app/lib/actions/pulp3/abstract_async_task.rb:160:in `poll_external_task'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/polling.rb:22:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/cancellable.rb:14:in `run'
cd9d284c | /usr/share/gems/gems/katello-4.5.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
cd9d284c | /usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
cd9d284c | /usr/share/gems/gems/katello-4.5.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
cd9d284c | /usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
cd9d284c | /usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
cd9d284c | /usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
cd9d284c | /usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/execution_wrapper.rb:91:in `wrap'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:17:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/world.rb:31:in `execute'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `catch'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `block in execute_run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `block in with_error_handling'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `catch'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `with_error_handling'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:575:in `execute_run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:296:in `execute'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/director.rb:94:in `execute'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors.rb:18:in `run_user_code'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
cd9d284c | [ sidekiq ]
cd9d284c | [ concurrent-ruby ]
2022-09-13T15:56:26 [I|bac|cd9d284c] Task {label: Actions::Katello::Repository::Sync, id: 45f96414-95a4-49e4-ae90-32ed5902ce05, execution_plan_id: a0fb4704-5bbe-4f8f-9fff-e887b22e36e4} state changed: stopped result: warning
2022-09-13T15:56:26 [E|bac|cd9d284c] A sub task failed (RuntimeError)
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/with_sub_plans.rb:231:in `check_for_errors!'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/with_sub_plans.rb:137:in `try_to_finish'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/with_sub_plans.rb:22:in `block in run'
cd9d284c | /usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
cd9d284c | /usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
cd9d284c | /usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
cd9d284c | /usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
cd9d284c | /usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
cd9d284c | /usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
cd9d284c | /usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
cd9d284c | /usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/with_sub_plans.rb:12:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/with_bulk_sub_plans.rb:24:in `run'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/bulk_action.rb:21:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
cd9d284c | /usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/execution_wrapper.rb:91:in `wrap'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:17:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
cd9d284c | /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/world.rb:31:in `execute'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `catch'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `block in execute_run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `block in with_error_handling'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `catch'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `with_error_handling'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:575:in `execute_run'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:296:in `execute'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/director.rb:94:in `execute'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors.rb:18:in `run_user_code'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
cd9d284c | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
cd9d284c | [ sidekiq ]
cd9d284c | [ concurrent-ruby ]
2022-09-13T15:56:26 [I|bac|cd9d284c] Task {label: Actions::BulkAction, id: 0b54a86e-2ecb-4151-bd06-5151f06f77c5, execution_plan_id: 52b248b8-a1b2-4c64-9b2d-fdbfa9470f19} state changed: stopped result: warning
Can you post any of the logs for/from Pulp?
I can for sure, if I knew how. Any suggest commands ro run or logs to grab?
Hi @tedevil, you can find Pulp logs in /var/log/messages. Also, if you go into the Dynflow console on your task, you should see a stack trace from Pulp. It’ll be the same one that’s in /var/log/messages.
Looking in /var/log/messages it looks like it ran out of memory? I know this repo is pretty huge but still unexpected error I guess, Server has 16 GB RAM configured.:
Sep 16 12:53:12 foreman kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@3.service,task=pulpcore-worker,pid=491201,uid=991
Sep 16 12:53:12 foreman kernel: Out of memory: Killed process 491201 (pulpcore-worker) total-vm:3740040kB, anon-rss:3254172kB, file-rss:0kB, shmem-rss:0kB, UID:991 pgtables:7112kB oom_score_adj:0
Sep 16 12:53:12 foreman kernel: oom_reaper: reaped process 491201 (pulpcore-worker), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 16 12:53:12 foreman pulpcore-worker-3[1650]: pulp [None]: pulpcore.tasking.pulpcore_worker:WARNING: Task process for c8dffad3-a639-4f32-851f-f94e01bb66a2 exited with non zero exitcode -9.
Sep 16 12:53:12 foreman pulpcore-worker-3[1650]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Cleaning up task c8dffad3-a639-4f32-851f-f94e01bb66a2 and marking as failed. Reason: Killed by signal 9.
Do I need to increase the RAM to just sync that repo? free command right after the sync failed. ALmost maxed out swap usage looks pretty bad.
[root@foreman ~]# free
total used free shared buff/cache available
Mem: 16282784 12067364 3294140 756724 921280 3207908
Swap: 2097148 2041600 55548
Just looked at the system requirements for Foreman/Katello 3.3/4.5 and it actually says 20GB as minimum now (not 16 Gb). Increased RAM to 20 GB and ran the sync again. This time successfully.
You don’t need that much RAM just to sync the repos (usually that maxes out under 1gb), but all the services combined running together do use a fair bit of RAM. If you could dump a process listing with how much RAM each process is consuming that might be helpful in diagnosing the problem spots.
In my experience - Puma and Gunicorn are typically the biggest offenders.