Problem:
With every upgrade, I’m never able to do the pulp migration. The reason changes with each attempt. It is failing now with the following error:
foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Starting task.
2021-04-07 13:37:50 -0500: Initial Migration steps complete.Migration failed, You will want to investigate: https://server.example.com
rake aborted!
ForemanTasks::TaskError: Task b604da6b-93eb-4257-8d28-30d8b5c5ceef: Katello::Errors::Pulp3Error: 'builtin_function_or_method' object is not subscriptable
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
[FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.
The following steps ended up in failing state:
[content-prepare]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"
Expected outcome:
Successful migration from Pulp2 to Pulp3
Is this the first run of the migration? Could you be able to provide logs for the task as well as the pulp logs from /var/log/messages for this migration around the error?
Sorry @sajha, I missed this and had to rerun to the output. So, the times have changed. Here is the new log entry. I’m attaching the journalctl output for this new time frame: journalctl --since “2021-04-15 13:35” --until “2021-04-15 13:45” [journalctl.gz|attachment](upload://2H1tFLVjybOUHhlUvTDQuBZ9g5Q.gz) (67.5 KB)
2021-04-15T15:38:16 [I|app|d0f55aa2] Completed 200 OK in 144ms (Views: 9.8ms | ActiveRecord: 115.2ms | Allocations: 19219)
2021-04-15T15:38:16 [I|app|1a2c1fe8] Completed 200 OK in 189ms (Views: 6.6ms | ActiveRecord: 134.5ms | Allocations: 27303)
2021-04-15T15:38:16 [I|app|c6d3bdc2] Started GET "/rhsm/consumers/94543eae-b9de-49da-ad67-91518495f25e/certificates/serials" for 127.0.
0.1 at 2021-04-15 15:38:16 -0500
2021-04-15T15:38:16 [I|app|c6d3bdc2] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-15T15:38:16 [I|app|c6d3bdc2] Parameters: {"id"=>"94543eae-b9de-49da-ad67-91518495f25e"}
2021-04-15T15:38:16 [E|bac|] 'builtin_function_or_method' object is not subscriptable (Katello::Errors::Pulp3Error)
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_fo
r_errors'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_tas
k'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_
user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in r
un'
| /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block
in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restor
e_current_request_id'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in
run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_
curent_timezone'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run
'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_cure
nt_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block
in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restor
e_current_taxonomies'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 le
vels) in execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculatio
n'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in ex
ecute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action
'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in p
erform'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
| [ sidekiq ]
| [ concurrent-ruby ]
2021-04-15T15:38:16 [I|app|a78460c7] Started GET "/rhsm/consumers/dac8bd44-35d0-4e76-8df0-7074907474d1/certificates/serials" for 127.0.
0.1 at 2021-04-15 15:38:16 -0500
2021-04-15T15:38:16 [I|app|a78460c7] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-15T15:38:16 [I|app|a78460c7] Parameters: {"id"=>"dac8bd44-35d0-4e76-8df0-7074907474d1"}
2021-04-15T15:38:16 [I|app|c6d3bdc2] Completed 200 OK in 53ms (Views: 0.7ms | ActiveRecord: 4.4ms | Allocations: 4496)
2021-04-15T15:38:17 [I|app|d32646e8] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|app|a78460c7] Completed 200 OK in 66ms (Views: 0.8ms | ActiveRecord: 16.9ms | Allocations: 4497)
2021-04-15T15:38:17 [I|app|d32646e8] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-15T15:38:17 [I|app|d32646e8] Completed 200 OK in 42ms (Views: 0.6ms | ActiveRecord: 12.4ms | Allocations: 2877)
2021-04-15T15:38:17 [I|app|606ebbf3] Started GET "/rhsm/status" for 127.0.0.1 at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|app|606ebbf3] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-04-15T15:38:17 [I|app|b9e71eaa] Started GET "/rhsm/consumers/94543eae-b9de-49da-ad67-91518495f25e/content_overrides" for 127.0.0.1
at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|app|b9e71eaa] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2021-04-15T15:38:17 [I|app|b9e71eaa] Parameters: {"id"=>"94543eae-b9de-49da-ad67-91518495f25e"}
2021-04-15T15:38:17 [I|app|606ebbf3] Completed 200 OK in 37ms (Views: 0.6ms | ActiveRecord: 3.9ms | Allocations: 2877)
2021-04-15T15:38:17 [I|app|f77614b0] Started GET "/rhsm/consumers/886ba3a3-6c83-457f-bda4-10b3230bef75/certificates/serials" for 127.0.
0.1 at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 56462ed8-2d77-40be-8a84-0209a9434647, execution_plan_id
: 8c0e5cb3-01dd-4139-802c-d10751d9a86e} state changed: stopped result: warning
2021-04-15T15:38:17 [I|app|f77614b0] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-15T15:38:17 [I|app|f77614b0] Parameters: {"id"=>"886ba3a3-6c83-457f-bda4-10b3230bef75"}
2021-04-15T15:38:17 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 56462ed8-2d77-40be-8a84-0209a9434647, execution_plan_id
: 8c0e5cb3-01dd-4139-802c-d10751d9a86e} state changed: stopped result: warning
2021-04-15T15:38:17 [I|app|32343d04] Started GET "/rhsm/consumers/de4f9eb0-91de-4ce8-a6ca-8c1e3b20ef69/certificates/serials" for 127.0.
0.1 at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|app|32343d04] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-15T15:38:17 [I|app|32343d04] Parameters: {"id"=>"de4f9eb0-91de-4ce8-a6ca-8c1e3b20ef69"}
2021-04-15T15:38:17 [I|app|a1b65546] Started GET "/rhsm/consumers/3cb4fe41-9394-4c2f-8f62-0ffd2148606e/certificates/serials" for 127.0.
0.1 at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|app|a1b65546] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-15T15:38:17 [I|app|a1b65546] Parameters: {"id"=>"3cb4fe41-9394-4c2f-8f62-0ffd2148606e"}
2021-04-15T15:38:17 [I|app|050eff0e] Started GET "/rhsm/consumers/144aee3b-f686-481c-8dd6-95525562c2a8/certificates/serials" for 127.0.
0.1 at 2021-04-15 15:38:17 -0500
2021-04-15T15:38:17 [I|dyn|] start terminating throttle_limiter...
2021-04-15T15:38:17 [I|dyn|] start terminating client dispatcher...
2021-04-15T15:38:17 [I|app|050eff0e] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-04-15T15:38:17 [I|app|050eff0e] Parameters: {"id"=>"144aee3b-f686-481c-8dd6-95525562c2a8"}
2021-04-15T15:38:17 [I|dyn|] stop listening for new events...
2021-04-15T15:38:17 [I|dyn|] start terminating clock...
@Justin_Sherrill Thanks for the bug fix. I’m no longer getting that error. Unfortunately, I’m now getting a new error:
foreman-maintain content switchover
Running Switch support for certain content from Pulp 2 to Pulp 3
================================================================================
Switch support for certain content from Pulp 2 to Pulp 3:
Performing final content migration before switching content [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
Starting task.
Migration failed, You will want to investigate: https://example.com/foreman_tasks/tasks/ae51611a-c543-43e1-b601-640b854f8ffb
rake aborted!
ForemanTasks::TaskError: Task ae51611a-c543-43e1-b601-640b854f8ffb: Katello::Errors::Pulp3Error: 2 subtask(s) failed for task group /pulp/api/v3/task-groups/b4c55761-cbd6-4783-bf80-794e3ea78ceb/.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
2021-04-27 10:09:39 -0500: Distribution creation 16/81
--------------------------------------------------------------------------------
Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.
The following steps ended up in failing state:
[content-switchover]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-switchover"
You might have to check ‘journalctl -u pulpcore-worker@*’ during that time to get more information Look for tracebacks or errors. That looks fairly far along in the process (likely during repo metadata generation).
Those logs have already rolled off, so I’m currently rerunning the switchover to generate them again. You mentioned repo metadata generation, and I wonder if it’s related to another issue we’ve been having. We have an ISO repository which we are unable to sync the files to our capsule servers. We get the following error:
400 Bad RequestPLP0000: Importer indicated a failed responsePLP0000: Importer indicated a failed response404 Not Found404 Not Found404 Not Found404 Not FoundPLP0000: Importer indicated a failed responsePLP0000: Importer indicated a failed responsePLP0000: Importer indicated a failed response
Is it possible that it’s related? I’ll reply again with the results of the journalctl output once it completes. It usually takes about 3 hours to run before it fails.
