Pulp3 migration failed(foreman 2.3.3/katello 3.18.2)

Problem:
With every upgrade, I’m never able to do the pulp migration. The reason changes with each attempt. It is failing now with the following error:

   foreman-maintain content prepare
   Running Prepare content for Pulp 3
        ================================================================================
    Prepare content for Pulp 3:
    Starting task.
    2021-04-07 13:37:50 -0500: Initial Migration steps complete.Migration failed, You will want to investigate: https://server.example.com
    rake aborted!
    ForemanTasks::TaskError: Task b604da6b-93eb-4257-8d28-30d8b5c5ceef: Katello::Errors::Pulp3Error: 'builtin_function_or_method' object is not subscriptable
    /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
    /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
    Tasks: TOP => katello:pulp3_migration
    (See full trace by running task with --trace)
                                                                          [FAIL]
    Failed executing foreman-rake katello:pulp3_migration, exit status 1
    --------------------------------------------------------------------------------
    Scenario [Prepare content for Pulp 3] failed.

    The following steps ended up in failing state:

      [content-prepare]

    Resolve the failed steps and rerun
    the command. In case the failures are false positives,
    use --whitelist="content-prepare"

Expected outcome:
Successful migration from Pulp2 to Pulp3

Foreman and Proxy versions:
2.3.3

Foreman and Proxy plugin versions:
foreman-tasks 3.0.5
foreman_chef 0.10.0
foreman_discovery 16.3.4
foreman_docker 5.0.0
foreman_remote_execution 4.2.2
katello 3.18.2.1

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

Other relevant data:

Hey tbrooks,

Is this the first run of the migration? Could you be able to provide logs for the task as well as the pulp logs from /var/log/messages for this migration around the error?

I have tried after every upgrade going back to at least 3.15. I made several attempts of running:

foreman-maintain content migration-stats
foreman-maintain content prepare

It always results in the same error that I posted originally. I’m going to try again to get logs related to the error.

Here is what I found in /var/log/foreman/production.log for this error:

 2021-04-08T21:28:00 [I|app|7e359d8a] Completed 200 OK in 218ms (Views: 42.6ms | ActiveRecord: 126.9ms | Allocations: 54494)
2021-04-08T21:28:00 [I|app|ea9d759a] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:00 [I|app|ea9d759a]   Parameters: {"id"=>"ec3e2939-6c04-4171-9155-82eff53b94c1"}
2021-04-08T21:28:00 [I|app|090332ad] Completed 200 OK in 244ms (Views: 0.8ms | ActiveRecord: 128.2ms | Allocations: 61365)
2021-04-08T21:28:00 [I|app|a68fa352] Completed 200 OK in 232ms (Views: 0.8ms | ActiveRecord: 157.5ms | Allocations: 49318)
2021-04-08T21:28:00 [I|app|1918c3cd] Started GET "/rhsm/consumers/07db9785-72db-44c2-8033-e43e9e8cb11e/certificates/serials" for 127.0.0.1 at 20
21-04-08 21:28:00 -0500
2021-04-08T21:28:00 [I|app|badbe65d] Started GET "/rhsm/consumers/f2495ee1-2542-4325-9a55-18b640e76a74/certificates/serials" for 127.0.0.1 at 20
21-04-08 21:28:00 -0500
2021-04-08T21:28:00 [E|bac|] 'builtin_function_or_method' object is not subscriptable (Katello::Errors::Pulp3Error)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]
2021-04-08T21:28:01 [I|app|badbe65d] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:01 [I|app|acdb7dd4] Completed 200 OK in 335ms (Views: 0.5ms | ActiveRecord: 159.5ms | Allocations: 83239)
2021-04-08T21:28:01 [I|app|badbe65d]   Parameters: {"id"=>"f2495ee1-2542-4325-9a55-18b640e76a74"}
2021-04-08T21:28:00 [I|app|dda166af] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-08T21:28:01 [I|bac|] Task {label: Actions::Katello::Host::UploadProfiles, id: 6731dbc5-09a7-4fd8-b913-92991dc46f55, execution_plan_id: 33ed321a-4296-449d-a3ac-7f8e606f949c} state changed: running
2021-04-08T21:28:00 [I|app|6cae47ad] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2021-04-08T21:28:01 [I|app|6cae47ad]   Parameters: {"root"=>{}}
2021-04-08T21:28:00 [I|app|56ec0faf] Completed 200 OK in 312ms (Views: 0.3ms | ActiveRecord: 76.5ms | Allocations: 76894)
2021-04-08T21:28:01 [I|app|1918c3cd] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:01 [I|app|1918c3cd]   Parameters: {"id"=>"07db9785-72db-44c2-8033-e43e9e8cb11e"}
2021-04-08T21:28:01 [I|app|352526ed] Completed 200 OK in 262ms (Views: 0.5ms | ActiveRecord: 143.1ms | Allocations: 64691)
2021-04-08T21:28:01 [I|app|e45f54d3] Completed 200 OK in 107ms (Views: 0.4ms | ActiveRecord: 31.9ms | Allocations: 22807)
2021-04-08T21:28:00 [I|app|65601ba1] Started GET "/rhsm/consumers/ce173772-f5a2-4154-8350-e19c4290ec01/certificates/serials" for 127.0.0.1 at 2021-04-08 21:28:00 -0500
2021-04-08T21:28:01 [I|app|68e649d3] Started GET "/rhsm/consumers/095d1500-5d53-4ff5-ada6-672dd42a5195/certificates/serials" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|f06131bc] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|e2f2fc68] Started GET "/rhsm/consumers/1cdc97fa-53e2-4c73-99ad-93e73e336cda/content_overrides" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|f06131bc] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-08T21:28:01 [I|app|68eb6242] Completed 200 OK in 132ms (Views: 0.6ms | ActiveRecord: 76.8ms | Allocations: 29540)
2021-04-08T21:28:01 [I|app|68e649d3] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:01 [I|app|68e649d3]   Parameters: {"id"=>"095d1500-5d53-4ff5-ada6-672dd42a5195"}
2021-04-08T21:28:01 [I|app|5b056e06] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|5b056e06] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-08T21:28:01 [I|app|6cae47ad]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2021-04-08T21:28:01 [I|app|6cae47ad]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (Duration: 7.5ms | Allocations: 4194)
2021-04-08T21:28:01 [I|app|415c6626] Completed 200 OK in 216ms (Views: 0.7ms | ActiveRecord: 118.3ms | Allocations: 56541)
2021-04-08T21:28:01 [I|app|ea9d759a] Completed 200 OK in 165ms (Views: 0.6ms | ActiveRecord: 67.1ms | Allocations: 36970)
2021-04-08T21:28:01 [I|app|65601ba1] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:01 [I|app|697224d8] Started GET "/rhsm/consumers/8070bed6-bff2-4618-aad6-c422203c811a/content_overrides" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|65601ba1]   Parameters: {"id"=>"ce173772-f5a2-4154-8350-e19c4290ec01"}
2021-04-08T21:28:01 [I|app|697224d8] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-08T21:28:01 [I|app|697224d8]   Parameters: {"id"=>"8070bed6-bff2-4618-aad6-c422203c811a"}
2021-04-08T21:28:01 [I|app|badbe65d] Completed 200 OK in 133ms (Views: 0.5ms | ActiveRecord: 63.2ms | Allocations: 30636)
2021-04-08T21:28:01 [I|app|f06131bc] Completed 200 OK in 87ms (Views: 0.5ms | ActiveRecord: 34.2ms | Allocations: 20794)
2021-04-08T21:28:01 [I|app|2405e89b] Completed 200 OK in 302ms (Views: 0.6ms | ActiveRecord: 154.3ms | Allocations: 91315)
2021-04-08T21:28:01 [I|app|a46ff7bd] Completed 200 OK in 414ms (Views: 0.7ms | ActiveRecord: 251.7ms | Allocations: 112643)
2021-04-08T21:28:01 [I|app|6cae47ad] Completed 200 OK in 111ms (Views: 13.2ms | ActiveRecord: 76.8ms | Allocations: 39002)
2021-04-08T21:28:01 [I|app|e2f2fc68] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-08T21:28:01 [I|app|e2f2fc68]   Parameters: {"id"=>"1cdc97fa-53e2-4c73-99ad-93e73e336cda"}
2021-04-08T21:28:01 [I|app|5b056e06] Completed 200 OK in 84ms (Views: 0.5ms | ActiveRecord: 17.5ms | Allocations: 18551)
2021-04-08T21:28:01 [I|app|20a10083] Completed 200 OK in 284ms (Views: 0.7ms | ActiveRecord: 87.7ms | Allocations: 87050)
2021-04-08T21:28:01 [I|app|153b6bc5] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|648452af] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|063da1ef] Started GET "/rhsm/consumers/837560b7-076a-4f0a-b5f9-54f6f1373e8f/certificates/serials" for 127.0.0.1 at 2021-04-08 21:28:01 -0500 
2021-04-08T21:28:01 [I|app|35f35fee] Completed 200 OK in 412ms (Views: 0.5ms | ActiveRecord: 289.0ms | Allocations: 109248)
2021-04-08T21:28:01 [I|app|648452af] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-08T21:28:01 [I|app|153b6bc5] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-08T21:28:01 [I|app|dda166af] Completed 200 OK in 203ms (Views: 0.4ms | ActiveRecord: 62.1ms | Allocations: 59826)
2021-04-08T21:28:01 [I|app|34a3ed00] Started GET "/rhsm/consumers/7b5e0023-5d63-4146-a74f-a7cf336aa39e/content_overrides" for 127.0.0.1 at 2021-
04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|34a3ed00] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-08T21:28:01 [I|app|34a3ed00]   Parameters: {"id"=>"7b5e0023-5d63-4146-a74f-a7cf336aa39e"}
2021-04-08T21:28:01 [I|app|94aa5af6] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|0e444767] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|94aa5af6] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-08T21:28:01 [I|app|7effc023] Started GET "/rhsm/consumers/941804b3-9e14-406f-a764-a5231f5dbf82/certificates/serials" for 127.0.0.1 at 20
21-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|0e444767] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-08T21:28:01 [I|app|7d83e37c] Started GET "/rhsm/consumers/582c2e29-cb4a-45e7-b51a-87ee576d432c/content_overrides" for 127.0.0.1 at 2021-
04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|1b391cda] Started GET "/rhsm/consumers/a29031ec-9a7f-4dad-9d12-9206f89d09eb/certificates/serials" for 127.0.0.1 at 20
21-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|063da1ef] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:01 [I|app|3717090c] Started GET "/rhsm/consumers/7a6ac59f-9a18-47bb-8029-f1364c3f2b7a/content_overrides" for 127.0.0.1 at 2021-
04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|063da1ef]   Parameters: {"id"=>"837560b7-076a-4f0a-b5f9-54f6f1373e8f"}
2021-04-08T21:28:01 [I|app|ed4fb96a] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|697224d8] Completed 200 OK in 144ms (Views: 0.2ms | ActiveRecord: 34.3ms | Allocations: 35940)
2021-04-08T21:28:01 [I|app|43431f9f] Started GET "/rhsm/consumers/d52a82ef-2e3e-4962-a645-0f330652d6be/certificates/serials" for 127.0.0.1 at 20
21-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|3717090c] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-08T21:28:01 [I|app|3717090c]   Parameters: {"id"=>"7a6ac59f-9a18-47bb-8029-f1364c3f2b7a"}
2021-04-08T21:28:01 [I|app|43431f9f] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:01 [I|app|1b391cda] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:01 [I|app|43431f9f]   Parameters: {"id"=>"d52a82ef-2e3e-4962-a645-0f330652d6be"}
2021-04-08T21:28:01 [I|app|1b391cda]   Parameters: {"id"=>"a29031ec-9a7f-4dad-9d12-9206f89d09eb"}
2021-04-08T21:28:01 [I|app|ed4fb96a] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-08T21:28:01 [I|app|153b6bc5] Completed 200 OK in 119ms (Views: 0.6ms | ActiveRecord: 20.2ms | Allocations: 28537)
2021-04-08T21:28:01 [I|app|68e649d3] Completed 200 OK in 256ms (Views: 0.5ms | ActiveRecord: 147.2ms | Allocations: 70408)
2021-04-08T21:28:01 [I|app|34a3ed00] Completed 200 OK in 107ms (Views: 0.2ms | ActiveRecord: 29.9ms | Allocations: 26285)
2021-04-08T21:28:01 [I|app|b81d89c4] Started GET "/rhsm/consumers/86eeb100-b3e3-4107-8440-933e094ee966/content_overrides" for 127.0.0.1 at 2021-
04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|7d83e37c] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-08T21:28:01 [I|app|648452af] Completed 200 OK in 147ms (Views: 0.7ms | ActiveRecord: 24.6ms | Allocations: 36759)
2021-04-08T21:28:01 [I|app|7d83e37c]   Parameters: {"id"=>"582c2e29-cb4a-45e7-b51a-87ee576d432c"}
2021-04-08T21:28:01 [I|app|e2f2fc68] Completed 200 OK in 175ms (Views: 0.1ms | ActiveRecord: 68.2ms | Allocations: 42770)
2021-04-08T21:28:01 [I|app|1918c3cd] Completed 200 OK in 321ms (Views: 0.4ms | ActiveRecord: 200.5ms | Allocations: 88389)
2021-04-08T21:28:01 [I|app|b81d89c4] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-08T21:28:01 [I|app|b81d89c4]   Parameters: {"id"=>"86eeb100-b3e3-4107-8440-933e094ee966"}
2021-04-08T21:28:01 [I|app|323b9746] Started GET "/rhsm/consumers/49cc1053-e409-47c1-bf92-30810cadd6fb/content_overrides" for 127.0.0.1 at 2021-
04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|6b12effb] Started GET "/rhsm/consumers/baebcccb-2f82-45b9-ba4d-ed8942161599/certificates/serials" for 127.0.0.1 at 20
21-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|65601ba1] Completed 200 OK in 221ms (Views: 0.6ms | ActiveRecord: 122.4ms | Allocations: 51733)
2021-04-08T21:28:01 [I|app|94aa5af6] Completed 200 OK in 148ms (Views: 0.6ms | ActiveRecord: 73.6ms | Allocations: 34410)
2021-04-08T21:28:01 [I|app|7effc023] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:01 [I|app|0e444767] Completed 200 OK in 146ms (Views: 0.9ms | ActiveRecord: 50.1ms | Allocations: 33485)
2021-04-08T21:28:01 [I|app|7effc023]   Parameters: {"id"=>"941804b3-9e14-406f-a764-a5231f5dbf82"}
2021-04-08T21:28:01 [I|app|323b9746] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-08T21:28:01 [I|app|323b9746]   Parameters: {"id"=>"49cc1053-e409-47c1-bf92-30810cadd6fb"}
2021-04-08T21:28:01 [I|app|6b12effb] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:01 [I|app|6b12effb]   Parameters: {"id"=>"baebcccb-2f82-45b9-ba4d-ed8942161599"}
2021-04-08T21:28:01 [I|app|9f5e0f1c] Started GET "/rhsm/consumers/f4cbe550-418b-4457-88c9-77727fcd4b61/content_overrides" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|868bbed6] Started GET "/rhsm/consumers/5fefe3d6-416c-4120-8981-bf6a56ad82d5/content_overrides" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|17c793c8] Started GET "/rhsm/consumers/57637636-c025-48d0-b7da-a41ec0e182f6/certificates/serials" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|9f5e0f1c] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-08T21:28:01 [I|app|9f5e0f1c]   Parameters: {"id"=>"f4cbe550-418b-4457-88c9-77727fcd4b61"}
2021-04-08T21:28:01 [I|app|7edb083c] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|f5f61ffd] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|868bbed6] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-08T21:28:01 [I|app|868bbed6]   Parameters: {"id"=>"5fefe3d6-416c-4120-8981-bf6a56ad82d5"}
2021-04-08T21:28:01 [I|app|17c793c8] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:01 [I|app|17c793c8]   Parameters: {"id"=>"57637636-c025-48d0-b7da-a41ec0e182f6"}
2021-04-08T21:28:01 [I|app|a9d29890] Started GET "/rhsm/consumers/f2495ee1-2542-4325-9a55-18b640e76a74/content_overrides" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|3717090c] Completed 200 OK in 204ms (Views: 0.2ms | ActiveRecord: 59.4ms | Allocations: 44595)
2021-04-08T21:28:01 [I|app|704abdbd] Started GET "/rhsm/consumers/67a3cf06-b9cf-4449-ba9c-901e256b68f0/certificates/serials" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|1b391cda] Completed 200 OK in 178ms (Views: 0.5ms | ActiveRecord: 53.9ms | Allocations: 40923)
2021-04-08T21:28:01 [I|app|2ac9bc27] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|28fc0237] Started GET "/rhsm/consumers/763e1f68-c375-4bf6-91d7-d2d8b9310995/certificates/serials" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|2ac9bc27] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-08T21:28:01 [I|app|7edb083c] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-08T21:28:01 [I|app|a9d29890] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-08T21:28:01 [I|app|a9d29890]   Parameters: {"id"=>"f2495ee1-2542-4325-9a55-18b640e76a74"}
2021-04-08T21:28:01 [I|app|704abdbd] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:01 [I|app|704abdbd]   Parameters: {"id"=>"67a3cf06-b9cf-4449-ba9c-901e256b68f0"}
2021-04-08T21:28:01 [I|app|1ea69b76] Started GET "/rhsm/consumers/ec3e2939-6c04-4171-9155-82eff53b94c1/content_overrides" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|ce32f0be] Started GET "/rhsm/consumers/9cae5771-1c1e-462a-a4c0-0c10a6118edf/content_overrides" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|b3d96ac8] Started GET "/rhsm/consumers/bc21e752-6b0a-48cf-a9eb-660d4e911ffe/certificates/serials" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|063da1ef] Completed 200 OK in 277ms (Views: 0.8ms | ActiveRecord: 162.2ms | Allocations: 66528)
2021-04-08T21:28:01 [I|app|b81d89c4] Completed 200 OK in 201ms (Views: 0.2ms | ActiveRecord: 94.2ms | Allocations: 48144)
2021-04-08T21:28:01 [I|app|7d83e37c] Completed 200 OK in 211ms (Views: 0.2ms | ActiveRecord: 50.6ms | Allocations: 49323)
2021-04-08T21:28:01 [I|app|28fc0237] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:01 [I|app|ed4fb96a] Completed 200 OK in 212ms (Views: 0.5ms | ActiveRecord: 70.7ms | Allocations: 49419)
2021-04-08T21:28:01 [I|app|28fc0237]   Parameters: {"id"=>"763e1f68-c375-4bf6-91d7-d2d8b9310995"}
2021-04-08T21:28:01 [I|app|1ea69b76] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-08T21:28:01 [I|app|1ea69b76]   Parameters: {"id"=>"ec3e2939-6c04-4171-9155-82eff53b94c1"}
2021-04-08T21:28:01 [I|app|ce32f0be] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-08T21:28:01 [I|app|9f5e0f1c] Completed 200 OK in 130ms (Views: 0.2ms | ActiveRecord: 30.6ms | Allocations: 33731)
2021-04-08T21:28:01 [I|app|ce32f0be]   Parameters: {"id"=>"9cae5771-1c1e-462a-a4c0-0c10a6118edf"}
2021-04-08T21:28:01 [I|app|6b12effb] Completed 200 OK in 167ms (Views: 0.5ms | ActiveRecord: 55.9ms | Allocations: 42909)
2021-04-08T21:28:01 [I|app|807f1c2d] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|b3d96ac8] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-08T21:28:01 [I|app|b3d96ac8]   Parameters: {"id"=>"bc21e752-6b0a-48cf-a9eb-660d4e911ffe"}
2021-04-08T21:28:01 [I|app|323b9746] Completed 200 OK in 181ms (Views: 0.3ms | ActiveRecord: 31.9ms | Allocations: 42210)
2021-04-08T21:28:01 [I|app|2ac9bc27] Completed 200 OK in 105ms (Views: 0.6ms | ActiveRecord: 28.0ms | Allocations: 24053)
2021-04-08T21:28:01 [I|app|7effc023] Completed 200 OK in 228ms (Views: 0.6ms | ActiveRecord: 138.2ms | Allocations: 52441)
2021-04-08T21:28:01 [I|app|264e6a83] Started GET "/rhsm/consumers/17b6190f-19a2-41a8-ba0d-545dce74328f/certificates/serials" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 11ff5f2e-5822-406f-bb0c-e61fe0e45788, execution_plan_id: 213a4f89-5e4a-4393-b48d-220956b5e9ea} state changed: stopped  result: warning
2021-04-08T21:28:01 [I|app|9360a8bf] Started GET "/rhsm/consumers/8070bed6-bff2-4618-aad6-c422203c811a" for 127.0.0.1 at 2021-04-08 21:28:01 -0500
2021-04-08T21:28:01 [I|app|f5f61ffd] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-08T21:28:01 [I|app|868bbed6] Completed 200 OK in 164ms (Views: 0.1ms | ActiveRecord: 34.4ms | Allocations: 39973)
2021-04-08T21:28:01 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 11ff5f2e-5822-406f-bb0c-e61fe0e45788, execution_plan_id: 213a4f89-5e4a-4393-b48d-220956b5e9ea} state changed: stopped  result: warning

Here is the pulp related logs from /var/log/messages during the time period when the migration prepare was running.
messages.pulp.gz (1.1 MB)

‘builtin_function_or_method’ object is not subscriptable (Katello::Errors::Pulp3Error)

So this error is being returned by pulp to katello and I am unable to find the corresponding error in the log.
Could you check for something like:

journalctl --since "2021-04-08 21:27" --until "2021-04-08 21:30:00"

to get the pulp logs for the period?

Sorry @sajha, I missed this and had to rerun to the output. So, the times have changed. Here is the new log entry. I’m attaching the journalctl output for this new time frame: journalctl --since “2021-04-15 13:35” --until “2021-04-15 13:45” [journalctl.gz|attachment](upload://2H1tFLVjybOUHhlUvTDQuBZ9g5Q.gz) (67.5 KB)

2021-04-15T15:38:16 [I|app|d0f55aa2] Completed 200 OK in 144ms (Views: 9.8ms | ActiveRecord: 115.2ms | Allocations: 19219)
2021-04-15T15:38:16 [I|app|1a2c1fe8] Completed 200 OK in 189ms (Views: 6.6ms | ActiveRecord: 134.5ms | Allocations: 27303)
2021-04-15T15:38:16 [I|app|c6d3bdc2] Started GET "/rhsm/consumers/94543eae-b9de-49da-ad67-91518495f25e/certificates/serials" for 127.0.
0.1 at 2021-04-15 15:38:16 -0500
2021-04-15T15:38:16 [I|app|c6d3bdc2] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-15T15:38:16 [I|app|c6d3bdc2]   Parameters: {"id"=>"94543eae-b9de-49da-ad67-91518495f25e"}
2021-04-15T15:38:16 [E|bac|] 'builtin_function_or_method' object is not subscriptable (Katello::Errors::Pulp3Error)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_fo
r_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_tas
k'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_
user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in r
un'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block
in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restor
e_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in
 run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_
curent_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run
'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_cure
nt_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block
in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restor
e_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 le
vels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculatio
n'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in ex
ecute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action
'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in p
erform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]
2021-04-15T15:38:16 [I|app|a78460c7] Started GET "/rhsm/consumers/dac8bd44-35d0-4e76-8df0-7074907474d1/certificates/serials" for 127.0.
0.1 at 2021-04-15 15:38:16 -0500
2021-04-15T15:38:16 [I|app|a78460c7] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-15T15:38:16 [I|app|a78460c7]   Parameters: {"id"=>"dac8bd44-35d0-4e76-8df0-7074907474d1"}
2021-04-15T15:38:16 [I|app|c6d3bdc2] Completed 200 OK in 53ms (Views: 0.7ms | ActiveRecord: 4.4ms | Allocations: 4496)
2021-04-15T15:38:17 [I|app|d32646e8] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|app|a78460c7] Completed 200 OK in 66ms (Views: 0.8ms | ActiveRecord: 16.9ms | Allocations: 4497)
2021-04-15T15:38:17 [I|app|d32646e8] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-15T15:38:17 [I|app|d32646e8] Completed 200 OK in 42ms (Views: 0.6ms | ActiveRecord: 12.4ms | Allocations: 2877)
2021-04-15T15:38:17 [I|app|606ebbf3] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|app|606ebbf3] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-15T15:38:17 [I|app|b9e71eaa] Started GET "/rhsm/consumers/94543eae-b9de-49da-ad67-91518495f25e/content_overrides" for 127.0.0.1
 at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|app|b9e71eaa] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-15T15:38:17 [I|app|b9e71eaa]   Parameters: {"id"=>"94543eae-b9de-49da-ad67-91518495f25e"}
2021-04-15T15:38:17 [I|app|606ebbf3] Completed 200 OK in 37ms (Views: 0.6ms | ActiveRecord: 3.9ms | Allocations: 2877)
2021-04-15T15:38:17 [I|app|f77614b0] Started GET "/rhsm/consumers/886ba3a3-6c83-457f-bda4-10b3230bef75/certificates/serials" for 127.0.
0.1 at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 56462ed8-2d77-40be-8a84-0209a9434647, execution_plan_id
: 8c0e5cb3-01dd-4139-802c-d10751d9a86e} state changed: stopped  result: warning
2021-04-15T15:38:17 [I|app|f77614b0] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-15T15:38:17 [I|app|f77614b0]   Parameters: {"id"=>"886ba3a3-6c83-457f-bda4-10b3230bef75"}
2021-04-15T15:38:17 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 56462ed8-2d77-40be-8a84-0209a9434647, execution_plan_id
: 8c0e5cb3-01dd-4139-802c-d10751d9a86e} state changed: stopped  result: warning
2021-04-15T15:38:17 [I|app|32343d04] Started GET "/rhsm/consumers/de4f9eb0-91de-4ce8-a6ca-8c1e3b20ef69/certificates/serials" for 127.0.
0.1 at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|app|32343d04] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-15T15:38:17 [I|app|32343d04]   Parameters: {"id"=>"de4f9eb0-91de-4ce8-a6ca-8c1e3b20ef69"}
2021-04-15T15:38:17 [I|app|a1b65546] Started GET "/rhsm/consumers/3cb4fe41-9394-4c2f-8f62-0ffd2148606e/certificates/serials" for 127.0.
0.1 at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|app|a1b65546] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-15T15:38:17 [I|app|a1b65546]   Parameters: {"id"=>"3cb4fe41-9394-4c2f-8f62-0ffd2148606e"}
2021-04-15T15:38:17 [I|app|050eff0e] Started GET "/rhsm/consumers/144aee3b-f686-481c-8dd6-95525562c2a8/certificates/serials" for 127.0.
0.1 at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|dyn|] start terminating throttle_limiter...
2021-04-15T15:38:17 [I|dyn|] start terminating client dispatcher...
2021-04-15T15:38:17 [I|app|050eff0e] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-15T15:38:17 [I|app|050eff0e]   Parameters: {"id"=>"144aee3b-f686-481c-8dd6-95525562c2a8"}
2021-04-15T15:38:17 [I|dyn|] stop listening for new events...
2021-04-15T15:38:17 [I|dyn|] start terminating clock...

There appear to be related logs in /var/log/messages this time, so I’m attaching that as well.messages.pulp.gz (1.5 MB)