Problem:
foreman-maintain content prepare
Running Prepare content for Pulp 3/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in<top (required)>’
Expected outcome:
Foreman and Proxy versions:
Foreman and Proxy plugin versions:
Distribution and version:
Other relevant data:
[{“pulp_href”=>
Interesting lines from /var/log/foreman/production.log
2021-05-21T09:51:42 [E|bac|] 1 subtask(s) failed for task group /pulp/api/v3/task-groups/3afa1fb4-25d9-4944-abbf-2ffb47cceaeb/. (Katello::Errors::Pulp3Error)block in check_for_errors' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in each’check_for_errors' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in poll_external_task’poll_external_task_with_rescue' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in run’run' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in run’block (3 levels) in execute_run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run’call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in block in run’block in as_remote_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/models/katello/concerns/user_extensions.rb:21:in cp_config’as_cp_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user’run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in wrap’run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’with_progress_calculation' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run’call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run’restore_current_request_id' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in run’call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run’restore_curent_timezone' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in run’call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in block in run’restore_curent_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in run’call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run’restore_current_taxonomies' | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run’call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run’call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute’block (2 levels) in execute_run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch’block in execute_run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling’catch' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling’execute_run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute’block (2 levels) in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’block in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in execute’block (2 levels) in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code’block in perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’perform' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
Hi @mdedek ,
Do you see any error backtraces from the Dynflow console task that failed? I’m looking for where Pulp 3 errored out. The same error backtrace should also be in /var/log/messages.
1 Like
I have only copy of /var/log/messages ans pructionlog because system has been already reverted to snapshot time.
messages.log.gz (13.5 KB)
It looks like you are hitting this issue: Issue #8400: Migration fails if there is .zck metadata in pulp2 - Migration Plugin - Pulp
This is now live in the repos for katello 3.18. I would yum update, re-run the installer, and try again. Let me know how it goes!
AdamR
May 26, 2021, 7:22am
7
Im seeing the same issues, even with the fix for issue #8400 . Upgraded from the repos about an hour ago, re-ran foreman-installer, verified no deb or docker content was enabled. I ran foreman-rake katello:delete_orphaned_content prior to retrying the migration.
gzipped ascii-file with extracts from cli, logs and gui is attached.
Katello3.18.2-withpulpcorefix#8400-migration-failed.txt.gz (2.4 KB)
@AdamR running foreman-rake katello:pulp3_migration_reset might help. That will completely reset your migration process.
AdamR
June 1, 2021, 7:28am
9
Sorry for the delay in response, it’s been a few hectic weeks at work.
AdamR
June 1, 2021, 10:07am
10
Here’s the tracebacK:
migrated_repo.pulp3_repository_version
File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 478, in migrate_repo_distributor
pulp2dist, repo_version)
File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3
publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 320, in publish
metadata_signing_service=metadata_signing_service,
File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 551, in create_repomd_xml
record.fill(checksum_type)
OSError: File icons.tar.gz doesn't exists or not a regular file
I’ve not seen this before, except for another user hit it here: Pulp3 Migration failed - #33 by jost
I’ll inquire with the pulp team about this and get back to you
mdedek
July 27, 2021, 8:37am
13
Migration to Pulp 3 was finally successfull after upgdate to Katello 3.18.4
3 Likes
