Pulp 2 to Pulp 3 migration failed

Problem:
Pulp 2 to 3 migration failed with message:

foreman-maintain content prepare

Running Prepare content for Pulp 3
Prepare content for Pulp 3:
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Starting task.
2021-05-21 09:51:33 +0200: Distribution creation 3/102Migration failed, You will want to investigate: https:tasks/66d8b056-de39-41f7-a336-ff27e7549edf
rake aborted!
ForemanTasks::TaskError: Task 66d8b056-de39-41f7-a336-ff27e7549edf: Katello::Errors::Pulp3Error: 1 subtask(sk-groups/3afa1fb4-25d9-4944-abbf-2ffb47cceaeb/.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in<top (required)>’
Tasks: TOP => katello:pulp3_migration

Expected outcome:
Pulp succesfully migrated to new version.

Foreman and Proxy versions:
Foreman 2.3.2-1.el7

Foreman and Proxy plugin versions:
Katello 3.18.2-1.el7

Distribution and version:
RHEL 7.9

Other relevant data:

[{“pulp_href”=>
“/pulp/api/v3/task-groups/3afa1fb4-25d9-4944-abbf-2ffb47cceaeb/”,
“description”=>“Migration Sub-tasks”,
“all_tasks_dispatched”=>true,
“waiting”=>189,
“skipped”=>0,
“running”=>3,
“completed”=>47,
“canceled”=>0,
“failed”=>1,
“group_progress_reports”=>
[{“message”=>“Repo version creation”,
“code”=>“create.repo_version”,
“total”=>276,
“done”=>50},
{“message”=>“Distribution creation”,
“code”=>“create.distribution”,
“total”=>1774,
“done”=>46}]}],
“poll_attempts”=>{“total”=>1697, “failed”=>1}}

Interesting lines from /var/log/foreman/production.log

2021-05-21T09:51:42 [E|bac|] 1 subtask(s) failed for task group /pulp/api/v3/task-groups/3afa1fb4-25d9-4944-abbf-2ffb47cceaeb/. (Katello::Errors::Pulp3Error)
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in block in check_for_errors' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in each’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in check_for_errors' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in poll_external_task’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in poll_external_task_with_rescue' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in block (3 levels) in execute_run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in block in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/models/katello/concerns/user_extensions.rb:21:in cp_config’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in wrap’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in with_progress_calculation' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in block in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
| [ sidekiq ]
| [ concurrent-ruby ]
2021-05-21T09:51:43 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 66d8b056-de39-41f7-a336-ff27e7549edf, execution_plan_id: 342f20b5-60fe-4ece-8737-24781b92ea44} state changed: stopped result: warning
2021-05-21T09:51:43 [I|bac|] Task {label: Actions::Pulp3::ContentMigration, id: 66d8b056-de39-41f7-a336-ff27e7549edf, execution_plan_id: 342f20b5-60fe-4ece-8737-24781b92ea44} state changed: stopped result: warning

Hi @mdedek,

Do you see any error backtraces from the Dynflow console task that failed? I’m looking for where Pulp 3 errored out. The same error backtrace should also be in /var/log/messages.

1 Like

I have only copy of /var/log/messages ans pructionlog because system has been already reverted to snapshot time.
Also take in mind that this was second run of # foreman-maintain content prepare
Initial failed run has been performed few days before 2021-05-21.

messages.log.gz (13.5 KB)

It looks like you are hitting this issue: Issue #8400: Migration fails if there is .zck metadata in pulp2 - Migration Plugin - Pulp
Which has been fixed in a newer migration plugin of pulp3. I will work on getting this built into our pulp3 repositories and let you know when its available.

This is now live in the repos for katello 3.18. I would yum update, re-run the installer, and try again. Let me know how it goes!

Im seeing the same issues, even with the fix for issue #8400. Upgraded from the repos about an hour ago, re-ran foreman-installer, verified no deb or docker content was enabled. I ran foreman-rake katello:delete_orphaned_content prior to retrying the migration.
It still fails on creating the Distributions. As this is one of our scaled down production test instances we can try out almost anything on it, except a complete re-install of foreman.

gzipped ascii-file with extracts from cli, logs and gui is attached.

Katello3.18.2-withpulpcorefix#8400-migration-failed.txt.gz (2.4 KB)

@AdamR running foreman-rake katello:pulp3_migration_reset might help. That will completely reset your migration process.

Sorry for the delay in response, it’s been a few hectic weeks at work.
This did not seem to help, I did the reset and found out that I had to restart the services afterwards(it complained about only one migration-plan allowed at a time). Once the services were restarted the migration again failed. I am re-running the reset and migration again to get fresh output, as I lost the output from the attempt after the first reset. In a few hours I can hopefully attach fresh logs here.

Fresh logs failed-migration-katello3.18.2-2021-06-01.txt.gz (3.5 KB)

Here’s the tracebacK:

 migrated_repo.pulp3_repository_version
 File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 478, in migrate_repo_distributor
 pulp2dist, repo_version)
 File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3
 publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
 File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 320, in publish
 metadata_signing_service=metadata_signing_service,
 File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 551, in create_repomd_xml
 record.fill(checksum_type)
 OSError: File icons.tar.gz doesn't exists or not a regular file

I’ve not seen this before, except for another user hit it here: Pulp3 Migration failed - #33 by jost

I’ll inquire with the pulp team about this and get back to you

The pulp team pointed me to a known issue: Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file" - Migration Plugin - Pulp

which seems tied to specific suse repositories

1 Like

Migration to Pulp 3 was finally successfull after upgdate to Katello 3.18.4

3 Likes