[ContentMigration] foreman-maintain content prepare failed

bourdinx · November 19, 2020, 8:47am

Problem: When we launch the command "foreman-maintain content prepare* after migration from Katello 3.16 to 3.17 , the result is failed with a permissions error on a task

Foreman and Proxy versions: 2.2.1

Foreman and Proxy plugin versions:
foreman-tasks 3.0.1
foreman_docker 5.0.0
foreman_remote_execution 4.1.0
katello 3.17.0

Distribution and version: CentOS Linux release 7.9.2009

Other relevant data:

Running Prepare content for Pulp 3
Prepare content for Pulp 3:                                           [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
 Migration failed, You will want to investigate: https://FQDN/foreman_tasks/tasks/809b098f-218f-4ddc-8466-2b1ac599926d
rake aborted!
ForemanTasks::TaskError: Task 809b098f-218f-4ddc-8466-2b1ac599926d: PulpcoreClient::ApiError: Error message: the server returns an error
HTTP status code: 403
Response headers: {"date"=>"Thu, 19 Nov 2020 08:36:40 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"63", "via"=>"1.1 FQDN", "connection"=>"close"}
Response body: {"detail":"You do not have permission to perform this action."}
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/lib/katello/tasks/pulp3_migration.rake:17:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

bourdinx · November 20, 2020, 8:55am

Here content of /var/log/messages when the command is launched

Nov 20 09:47:44 FQDN pulpcore-api: 127.0.0.1 - - [20/Nov/2020:08:47:44 +0000] "GET /pulp/api/v3/status HTTP/1.1" 301 0 "-" "rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157"
Nov 20 09:47:44 FQDN pulpcore-api: 127.0.0.1 - - [20/Nov/2020:08:47:44 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 1684 "-" "rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157"
Nov 20 09:47:45 FQDN pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Nov 20 09:47:45 FQDN pulpcore-api: 127.0.0.1 - - [20/Nov/2020:08:47:45 +0000] "GET /pulp/api/v3/status HTTP/1.1" 301 0 "-" "rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157"
Nov 20 09:47:45 FQDN pulpcore-api: 127.0.0.1 - - [20/Nov/2020:08:47:45 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 1684 "-" "rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157"
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: 2020-11-20T08:47:45.435Z 22122 TID-2xh4q6 Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork JID-68980b506a11ef878ffd42c0 INFO: start
Nov 20 09:47:45 FQDN pulpcore-api: 127.0.0.1 - - [20/Nov/2020:08:47:45 +0000] "POST /pulp/api/v3/migration-plans/ HTTP/1.1" 201 181 "-" "OpenAPI-Generator/0.5.0/ruby"
Nov 20 09:47:45 FQDN pulpcore-api: 127.0.0.1 - - [20/Nov/2020:08:47:45 +0000] "POST /pulp/api/v3/migration-plans/a970d665-967e-439d-a918-1893a47e23db/run/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/0.5.0/ruby"
Nov 20 09:47:45 FQDN pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: ef2168f9-265a-4f07-b5cc-c3e1d04eda37
Nov 20 09:47:45 FQDN pulpcore-api: pulp: django.request:WARNING: Forbidden: /pulp/api/v3/tasks/41e6f444-d4b2-422c-ad77-f3f0ce96d731/
Nov 20 09:47:45 FQDN pulpcore-api: 127.0.0.1 - - [20/Nov/2020:08:47:45 +0000] "GET /pulp/api/v3/tasks/41e6f444-d4b2-422c-ad77-f3f0ce96d731/ HTTP/1.1" 403 63 "-" "OpenAPI-Generator/3.6.0/ruby"
Nov 20 09:47:45 FQDN pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (ef2168f9-265a-4f07-b5cc-c3e1d04eda37)
Nov 20 09:47:45 FQDN pulpcore-api: pulp: django.request:WARNING: Forbidden: /pulp/api/v3/tasks/41e6f444-d4b2-422c-ad77-f3f0ce96d731/
Nov 20 09:47:45 FQDN pulpcore-api: 127.0.0.1 - - [20/Nov/2020:08:47:45 +0000] "GET /pulp/api/v3/tasks/41e6f444-d4b2-422c-ad77-f3f0ce96d731/ HTTP/1.1" 403 63 "-" "OpenAPI-Generator/3.6.0/ruby"
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: 2020-11-20T08:47:45.687Z 22122 TID-2xh4q6 Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork JID-68980b506a11ef878ffd42c0 INFO: fail: 0.252 sec
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: 2020-11-20T08:47:45.687Z 22122 TID-2xh4q6 WARN: {"context":"Job raised exception","job":{"queue":"default","args":[{"class":"Dynflow::Director::StepWorkItem","execution_plan_id":"4c568ea3-c89b-4799-ab89-ffbb29f6bd0e","queue":"default","sender_orchestrator_id":"b7a4d688-6149-4f65-8db5-6f9994da9fa1","step":{"execution_plan_uuid":"4c568ea3-c89b-4799-ab89-ffbb29f6bd0e","id":2,"state":"pending","class":"Dynflow::ExecutionPlan::Steps::RunStep","action_class":"Actions::Pulp3::ContentMigration","action_id":1,"error":null,"started_at":null,"ended_at":null,"execution_time":0.0,"real_time":0.0,"progress_done":0,"progress_weight":1.0,"queue":"default"}}],"class":"Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork","retry":false,"backtrace":true,"jid":"68980b506a11ef878ffd42c0","created_at":1605862065.431226,"enqueued_at":1605862065.4312925},"jobstr":"{\"queue\":\"default\",\"args\":[{\"class\":\"Dynflow::Director::StepWorkItem\",\"execution_plan_id\":\"4c568ea3-c89b-4799-ab89-ffbb29f6bd0e\",\"queue\":\"default\",\"sender_orchestrator_id\":\"b7a4d688-6149-4f65-8db5-6f9994da9fa1\",\"step\":{\"execution_plan_uuid\":\"4c568ea3-c89b-4799-ab89-ffbb29f6bd0e\",\"id\":2,\"state\":\"pending\",\"class\":\"Dynflow::ExecutionPlan::Steps::RunStep\",\"action_class\":\"Actions::Pulp3::ContentMigration\",\"action_id\":1,\"error\":null,\"started_at\":null,\"ended_at\":null,\"execution_time\":0.0,\"real_time\":0.0,\"progress_done\":0,\"progress_weight\":1.0,\"queue\":\"default\"}}],\"class\":\"Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork\",\"retry\":false,\"backtrace\":true,\"jid\":\"68980b506a11ef878ffd42c0\",\"created_at\":1605862065.431226,\"enqueued_at\":1605862065.4312925}"}
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: 2020-11-20T08:47:45.687Z 22122 TID-2xh4q6 WARN: PulpcoreClient::ApiError: Error message: the server returns an error
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: HTTP status code: 403
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: Response headers: {"date"=>"Fri, 20 Nov 2020 08:47:45 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"63", "via"=>"1.1 FQDN", "connection"=>"close"}
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: Response body: {"detail":"You do not have permission to perform this action."}
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: 2020-11-20T08:47:45.687Z 22122 TID-2xh4q6 WARN: /opt/theforeman/tfm/root/usr/share/gems/gems/pulpcore_client-3.6.0/lib/pulpcore_client/api_client.rb:81:in `call_api'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/pulpcore_client-3.6.0/lib/pulpcore_client/api/tasks_api.rb:425:in `read_with_http_info'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/pulpcore_client-3.6.0/lib/pulpcore_client/api/tasks_api.rb:372:in `read'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/services/katello/pulp3/task.rb:56:in `task_data'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/services/katello/pulp3/task.rb:41:in `to_hash'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/serializable.rb:55:in `recursive_to_hash'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/serializable.rb:53:in `block in recursive_to_hash'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/serializable.rb:53:in `map'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/serializable.rb:53:in `recursive_to_hash'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/serializable.rb:51:in `block in recursive_to_hash'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/serializable.rb:51:in `each'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/serializable.rb:51:in `inject'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/serializable.rb:51:in `recursive_to_hash'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/serializable.rb:51:in `block in recursive_to_hash'
Nov 20 09:47:45 FQDN dynflow-sidekiq@orchestrator: 2020-11-20T08:47:45.688Z 20846 TID-1nq0sa Dynflow::Executors::Sidekiq::OrchestratorJobs::WorkerDone JID-eb9f0267eda038f9b83ded8e INFO: start
Nov 20 09:47:45 FQDN dynflow-sidekiq@orchestrator: 2020-11-20T08:47:45.689Z 20846 TID-1nq0sa Dynflow::Executors::Sidekiq::OrchestratorJobs::WorkerDone JID-eb9f0267eda038f9b83ded8e INFO: done: 0.001 sec
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/serializable.rb:51:in `each'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/serializable.rb:51:in `inject'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/serializable.rb:51:in `recursive_to_hash'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/serializable.rb:59:in `recursive_to_hash'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:252:in `to_hash'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/persistence.rb:46:in `save_action'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:33:in `open_action'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in `execute'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: 2020-11-20T08:47:45.711Z 22122 TID-2wxn22 Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork JID-91345c3c308f936a5b9134d4 INFO: start
Nov 20 09:47:45 FQDN pulpcore-worker-4: pulp: rq.worker:INFO: 20810@FQDN: 41e6f444-d4b2-422c-ad77-f3f0ce96d731
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: 2020-11-20T08:47:45.726Z 22122 TID-2wxn22 Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork JID-91345c3c308f936a5b9134d4 INFO: done: 0.015 sec
Nov 20 09:47:45 FQDN dynflow-sidekiq@orchestrator: 2020-11-20T08:47:45.726Z 20846 TID-1nq0sa Dynflow::Executors::Sidekiq::OrchestratorJobs::WorkerDone JID-d80b9a19bc3005a765c2289c INFO: start
Nov 20 09:47:45 FQDN dynflow-sidekiq@orchestrator: 2020-11-20T08:47:45.727Z 20846 TID-1nq0sa Dynflow::Executors::Sidekiq::OrchestratorJobs::WorkerDone JID-d80b9a19bc3005a765c2289c INFO: done: 0.001 sec
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: 2020-11-20T08:47:45.728Z 22122 TID-2xh5m2 Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork JID-ff95d70280ff1d9ef1a722a1 INFO: start
Nov 20 09:47:45 FQDN dynflow-sidekiq@worker: 2020-11-20T08:47:45.729Z 22122 TID-2xh5m2 Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork JID-ff95d70280ff1d9ef1a722a1 INFO: done: 0.001 sec
Nov 20 09:47:45 FQDN dynflow-sidekiq@orchestrator: 2020-11-20T08:47:45.730Z 20846 TID-1nq0sa Dynflow::Executors::Sidekiq::OrchestratorJobs::WorkerDone JID-0a2339c41a266f484b4ca199 INFO: start
Nov 20 09:47:45 FQDN dynflow-sidekiq@orchestrator: 2020-11-20T08:47:45.730Z 20846 TID-1nq0sa Dynflow::Executors::Sidekiq::OrchestratorJobs::WorkerDone JID-0a2339c41a266f484b4ca199 INFO: done: 0.0 sec
Nov 20 09:47:46 FQDN pulpcore-worker-4: pulp: pulp_2to3_migration.pulp2.connection:INFO: Attempting to connect to localhost:27017
Nov 20 09:47:46 FQDN mongod.27017[20649]: [conn96] received client metadata from 127.0.0.1:35216 conn96: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-1160.2.2.el7.x86_64" }, platform: "CPython 3.6.8.final.0" }
Nov 20 09:47:46 FQDN mongod.27017[20649]: [conn97] received client metadata from 127.0.0.1:35218 conn97: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-1160.2.2.el7.x86_64" }, platform: "CPython 3.6.8.final.0" }
Nov 20 09:47:48 FQDN pulpcore-worker-4: pulp: rq.worker:INFO: 20810@FQDN: Job OK (41e6f444-d4b2-422c-ad77-f3f0ce96d731)
Nov 20 09:47:48 FQDN pulpcore-worker-4: pulp: rq.worker:INFO: 20810@FQDN: 9cb7b6c1-af79-4803-96fa-49a7b35d4961
Nov 20 09:47:48 FQDN pulpcore-worker-4: pulp: rq.worker:INFO: 20810@FQDN: Job OK (9cb7b6c1-af79-4803-96fa-49a7b35d4961)

helloforeman · December 1, 2020, 11:04am

Hi,

Any news on this issue?I get the same error. Just upgraded to Katello 3.17.

Foreman and Proxy versions: 2.2.1-1

Foreman and Proxy plugin versions:
tfm-rubygem-foreman-tasks-3.0.1-1.fm2_2.el7.noarch
tfm-rubygem-foreman_docker-5.0.0-3.fm2_1.el7.noarch
tfm-rubygem-foreman_remote_execution-4.1.0-1.fm2_2.el7.noarch
katello-3.17.0-1.el7.noarch

Distribution and version: CentOS Linux release 7.9.2009 (all the latest updates as of today 12/1/2020)

Cheers,
Oliver

rob.beglinger · December 8, 2020, 9:46pm

I’m having the same issue Foreman 2.2.1 Katello 3.17.1, all packages updated today 12/8/2020.

Output from foreman-rake katello:pulp3_migration COMMIT=true --trace

** Invoke katello:pulp3_migration (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke katello:disable_dynflow (first_time)
** Execute katello:disable_dynflow
** Invoke katello:check_ping (first_time)
** Invoke environment
** Execute katello:check_ping
** Execute katello:pulp3_migration
Migration failed, You will want to investigate: https:///foreman_tasks/tasks/a50e0d8e-66ee-4777-aa0f-82e383b0e98a
rake aborted!
ForemanTasks::TaskError: Task a50e0d8e-66ee-4777-aa0f-82e383b0e98a: PulpcoreClient::ApiError: Error message: the server returns an error
HTTP status code: 403
Response headers: {“date”=>“Tue, 08 Dec 2020 21:38:54 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“GET, PATCH, DELETE, HEAD, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“63”, “via”=>"1.1 ", “connection”=>“close”}
Response body: {“detail”:“You do not have permission to perform this action.”}
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.1/lib/katello/tasks/pulp3_migration.rake:17:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in block in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in mon_synchronize’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in invoke’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in invoke_task' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block (2 levels) in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/

touchardv · December 16, 2020, 12:26pm

Having the same issue (“You do not have permission to perform this action.” when Pulp tasks are involved), so far a workaround I could find is:

Connect to the Pulp postgres database
Execute the following statement: UPDATE auth_user SET is_superuser = true WHERE id =1;

Without this, a curl HTTP call to the Pulp API, as admin, with the certs, would even fail like:
curl -v -u admin:$pulpAdminPassword https://my-foreman/pulp/api/v3/tasks/4324570c-549b-4796-aaf1-f585b7077077/ --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key | python -m json.tool

=> HTTP/1.1 403 Forbidden
{
“detail”: “You do not have permission to perform this action.”
}

sasskinn12 · January 7, 2021, 9:44am

Thanks a lot. This workaround helped me out!
Now i got next error, same as them here:

diw · March 27, 2021, 6:42pm

Just wanted to add I needed change the user permission on my deployment as well 3.18.2.
I don’t even have any content, so I’m not sure what caused it to go wrong.

I’m not running into:

ehelms · April 9, 2021, 4:05pm

@iballou or @Justin_Sherrill do either of y’all have any guidance here for this issue? anything we need to update in our code?

iballou · April 12, 2021, 7:05pm

I haven’t seen this issue before, but I’m guessing the Pulp 3 DB needs a reset and a reset-admin-password. I’ll see if the Pulp team has any idea, or maybe @ekohl ? Makes me wonder if the issue could relate to puppet-pulpcore.