Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Starting task.
2021-03-30 16:16:17 +0000: Content migration starting. Migration failed, You will want to investigate: https://foreman.isbank.is/foreman_tasks/tasks/eabe3507-91a5-451d-8c3f-9f0138870648
rake aborted!
ForemanTasks::TaskError: Task eabe3507-91a5-451d-8c3f-9f0138870648: Pulp2to3MigrationClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Tue, 30 Mar 2021 16:16:18 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"57", "via"=>"1.1 foreman.isbank.is", "connection"=>"close"}
Response body: ["Only one migration plan can run or be reset at a time"]
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
[FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.
The following steps ended up in failing state:
[content-prepare]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"
Foreman and Proxy versions:
Katello 3.18.2 - just updated
did you already start a migration previously? could it still be running? If you check Monitor > tasks and search for âlabel ~ Migrationâ you should see all the instances, check if one is still running.
I have similiar situation⊠Tasks queue is clear. I have restarted services. Nothing helps.
Somewhere are locks, but I can not find them.
foreman ~ # foreman-rake katello:pulp3_migration_reset
enabled
Starting Content Migration Reset.
Content Migration Reset failed, You will want to investigate:
ForemanTasks::TaskError: Task 305326fc-21e3-43b5-9dc0-d2b832fd144d:
Pulp2to3MigrationClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Thu, 01 Apr 2021 07:46:45 GMT", "server"=>"gunicorn/20.0.4",
"content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-
options"=>"SAMEORIGIN", "content-length"=>"57", "via"=>"1.1 ", "
"connection"=>"close"}
Response body: ["Only one migration plan can run or be reset at a time"]
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-
3.18.2.1/lib/katello/tasks/pulp3_migration_reset.rake:18:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration_reset
(See full trace by running task with --trace)
If i remember correctly, I ran the first iteration of the migration in 3.16 (or whenever it was introduced into the upgrade process), but no current migration task is running.
I had a host of other problems that i seem to have finally fixed, among them was a disk issue. Not that a disk was full, but that qdrouterd default behaviour is to stop functioning properly if disk usage is >90%, after fixing that my Foreman instance appears to finally be back to stable and this being the last thing that doesnât work.
I tested running foreman-maintain content switchover and watching logs, i found this error:
Found in another post ( [ContentMigration] foreman-maintain content prepare failed - #5 by touchardv ).
Ran command sudo su - postgres -c "psql -d pulpcore -c 'UPDATE auth_user SET is_superuser = true WHERE id =1;'"
and retried, that seems to have worked, so that seems to be the permission error at least.
Tagging @Justin_Sherrill to make sure he sees the solution, if others have the same issue.
This is probably the more elegant solution, instead of going directly into the database yourself - marking this as the better solution to the permission error.
I was so closeâŠ
Now iâm hitting another issue it seems:
OSError: Error while computing stat of compressed content of updateinfo.xml.zck:Cannot open a file updateinfo.xml.zck: createrepo_c wasnât compiled with zchunk support
I had the exact same errors as you did birkirf, both the preceding issue and the last zchunk one. That issue was resolved for me once we upgraded python3-pulp-2to3-migration to 0.11.0-1.el7 from 0.10
Unfortunately iâve not found any workarounds, but i have seen some threads that imply that finding the offending repos and removing them, finish upgrading and recreating the repos would work, but in my Production system iâm simply not going to do that, so havenât tried it.
I can wait on 3.18.x until they find a fix for a few more months if thats what it takes.
Anyone heard anything regarding the fix for zck metadata? Ticket was finished 2 months ago now, but problem still exists on latest 3.18.3 installation - Server is fully updated as of 2 minutes ago, but still getting the error
