Katello 3.17 Upgrade issues

Support
,
1

Upgraded Katello 3.16.1.2 to 3.17.

First issue:

Package upgrade successful. Running foreman-installer fails with the following message:
Could not prefetch yumrepo provider ‘inifile’: Section “pulp” is already defined, cannot redefine (file: /etc/yum.repos.d/pulp.repo)
Failed to apply catalog: Section “pulp” is already defined, cannot redefine (file: /etc/yum.repos.d/pulp.repo)

Manually deleting /etc/yum.repos.d/pulp.repo and running foreman-installer successfully finishes.

Second Issue:

Running “foreman-maintain advanced procedure run content-prepare” produces the following error

Running ForemanMaintain::Scenario
================================================================================
Prepare content for Pulp 3:                                           [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
 Rubocop not loaded.
Migration failed, You will want to investigate: https://FQDN/foreman_tasks/tasks/b57d7a70-bbe1-46b2-beaf-fa627bcf02cd
rake aborted!
ForemanTasks::TaskError: Task b57d7a70-bbe1-46b2-beaf-fa627bcf02cd: Katello::Errors::Pulp3Error: null value in column "stream" violates not-null constraint
DETAIL:  Failing row contains (ff6ac58a-cc9e-48f4-bfde-05a6e04fdb29, 2020-11-12 22:57:58.425104+00, 2020-11-12 22:57:58.425113+00, 389-directory-server, null, {"stable": ["default"], "testing": ["default"]}, 015ae06403a7d0318bb78bdf560d3616bb11369161ffaa7bd2eb658aae5c2764, 1-Red_Hat_Enterprise_Linux_8_x86_64-v5_0-18c3ae48-ca68-4b4e-b396..., 7746a071-6587-44a3-8082-b75641a9aa64).
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/lib/katello/tasks/pulp3_migration.rake:17:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
--------------------------------------------------------------------------------
Scenario [ForemanMaintain::Scenario] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

Running the command a second time produces the following message:

Running ForemanMaintain::Scenario
================================================================================
Prepare content for Pulp 3:                                           [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
 Rubocop not loaded.
Migration failed, You will want to investigate: https://FQDN/foreman_tasks/tasks/b3afcbc8-db69-4580-97aa-148572c11851
rake aborted!
ForemanTasks::TaskError: Task b3afcbc8-db69-4580-97aa-148572c11851: Katello::Errors::Pulp3Error: __init__() got an unexpected keyword argument 'box_settings'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/lib/katello/tasks/pulp3_migration.rake:17:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
--------------------------------------------------------------------------------
Scenario [ForemanMaintain::Scenario] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"
2

Output of first task:

Action:

Actions::Pulp3::ContentMigration

Input:

{"services_checked"=>["pulp3"],
 "smart_proxy_id"=>1,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_user_id"=>1,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}

Output:

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/30074796-a0ea-491c-8b3b-bcf6b1a59373/",
    "pulp_created"=>"2020-11-12T22:15:04.749+00:00",
    "state"=>"failed",
    "name"=>"pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
    "started_at"=>"2020-11-12T22:15:05.006+00:00",
    "finished_at"=>"2020-11-12T22:57:58.561+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n" +
       "    rv = job.perform()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n" +
       "    self._result = self._execute()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n" +
       "    return self.func(*self.args, **self.kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 135, in migrate_from_pulp2\n" +
       "    pre_migrate_all_content(plan)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py\", line 68, in pre_migrate_all_content\n" +
       "    pre_migrate_content_type(content_model, mutable_type, lazy_type, premigrate_hook)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py\", line 212, in pre_migrate_content_type\n" +
       "    content_model.pulp_2to3_detail.pre_migrate_content_detail(pulp2content_batch)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/pulp_2to3_models.py\", line 672, in pre_migrate_content_detail\n" +
       "    batch_size=DEFAULT_BATCH_SIZE)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method\n" +
       "    return getattr(self.get_queryset(), name)(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 468, in bulk_create\n" +
       "    self._batched_insert(objs_with_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 1211, in _batched_insert\n" +
       "    self._insert(item, fields=fields, using=self.db, ignore_conflicts=ignore_conflicts)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 1186, in _insert\n" +
       "    return query.get_compiler(using=using).execute_sql(return_id)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py\", line 1377, in execute_sql\n" +
       "    cursor.execute(sql, params)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/utils.py\", line 67, in execute\n" +
       "    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers\n" +
       "    return executor(sql, params, many, context)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n" +
       "    return self.cursor.execute(sql, params)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__\n" +
       "    raise dj_exc_value.with_traceback(traceback) from exc_value\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n" +
       "    return self.cursor.execute(sql, params)\n",
      "description"=>
       "null value in column \"stream\" violates not-null constraint\n" +
       "DETAIL:  Failing row contains (ff6ac58a-cc9e-48f4-bfde-05a6e04fdb29, 2020-11-12 22:57:58.425104+00, 2020-11-12 22:57:58.425113+00, 389-directory-server, null, {\"stable\": [\"default\"], \"testing\": [\"default\"]}, 015ae06403a7d0318bb78bdf560d3616bb11369161ffaa7bd2eb658aae5c2764, 1-Red_Hat_Enterprise_Linux_8_x86_64-v5_0-18c3ae48-ca68-4b4e-b396..., 7746a071-6587-44a3-8082-b75641a9aa64).\n"},
    "worker"=>"/pulp/api/v3/workers/63daed6a-e6fe-4e97-8d59-d44bfa49484b/",
    "child_tasks"=>[],
    "task_group"=>
     "/pulp/api/v3/task-groups/ced53e64-1825-4e3c-b917-626b12069383/",
    "progress_reports"=>
     [{"message"=>"Pre-migrating Pulp 2 SRPM content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 SRPM content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 ERRATUM content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>129211,
       "done"=>129211},
      {"message"=>"Pre-migrating Pulp 2 ERRATUM content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>129211,
       "done"=>129211},
      {"message"=>"Processing Pulp 2 repositories, importers, distributors",
       "code"=>"processing.repositories",
       "state"=>"completed",
       "total"=>754,
       "done"=>754},
      {"message"=>"Pre-migrating Pulp 2 DISTRIBUTION content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>1,
       "done"=>1},
      {"message"=>"Pre-migrating Pulp 2 DISTRIBUTION content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>1,
       "done"=>1},
      {"message"=>"Pre-migrating Pulp 2 MODULEMD content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>312,
       "done"=>312},
      {"message"=>"Pre-migrating Pulp 2 MODULEMD content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>312,
       "done"=>312},
      {"message"=>
        "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"running",
       "total"=>229,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"running",
       "total"=>229,
       "done"=>229},
      {"message"=>"Pre-migrating Pulp 2 RPM content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>215132,
       "done"=>215132},
      {"message"=>"Pre-migrating Pulp 2 RPM content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>215132,
       "done"=>215132}],
    "created_resources"=>
     ["/pulp/api/v3/task-groups/ced53e64-1825-4e3c-b917-626b12069383/"]}],
 "task_groups"=>
  [{"pulp_href"=>
     "/pulp/api/v3/task-groups/ced53e64-1825-4e3c-b917-626b12069383/",
    "description"=>"Migration Sub-tasks",
    "all_tasks_dispatched"=>false,
    "waiting"=>0,
    "skipped"=>0,
    "running"=>0,
    "completed"=>0,
    "canceled"=>0,
    "failed"=>1,
    "group_progress_reports"=>
     [{"message"=>"Repo version creation",
       "code"=>"create.repo_version",
       "total"=>0,
       "done"=>0},
      {"message"=>"Distribution creation",
       "code"=>"create.distribution",
       "total"=>0,
       "done"=>0}]}],
 "poll_attempts"=>{"total"=>180, "failed"=>1}}

Exception:

Katello::Errors::Pulp3Error: null value in column "stream" violates not-null constraint
DETAIL:  Failing row contains (ff6ac58a-cc9e-48f4-bfde-05a6e04fdb29, 2020-11-12 22:57:58.425104+00, 2020-11-12 22:57:58.425113+00, 389-directory-server, null, {"stable": ["default"], "testing": ["default"]}, 015ae06403a7d0318bb78bdf560d3616bb11369161ffaa7bd2eb658aae5c2764, 1-Red_Hat_Enterprise_Linux_8_x86_64-v5_0-18c3ae48-ca68-4b4e-b396..., 7746a071-6587-44a3-8082-b75641a9aa64).

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Action:

Actions::Pulp3::ImportMigration

Input:

{"reimport_all"=>nil,
 "dependency"=>Step(2).output,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_user_id"=>1,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}

Output:

{}
3

Output of second task:

Action:

Actions::Pulp3::ContentMigration

Input:

{"services_checked"=>["pulp3"],
 "smart_proxy_id"=>1,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_user_id"=>1,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}

Output:

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/496d138e-84e7-430f-9ec8-2acb0c446152/",
    "pulp_created"=>"2020-11-12T23:12:06.522+00:00",
    "state"=>"failed",
    "name"=>"pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
    "started_at"=>"2020-11-12T23:12:07.719+00:00",
    "finished_at"=>"2020-11-12T23:24:43.749+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n" +
       "    rv = job.perform()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n" +
       "    self._result = self._execute()\n" +
       "  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n" +
       "    return self.func(*self.args, **self.kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 139, in migrate_from_pulp2\n" +
       "    migrate_content(plan)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 36, in migrate_content\n" +
       "    plugin.migrator.migrate_content_to_pulp3()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py\", line 140, in migrate_content_to_pulp3\n" +
       "    loop.run_until_complete(dm.create())\n" +
       "  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
       "    return future.result()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 88, in create\n" +
       "    await pipeline\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n" +
       "    await asyncio.gather(*futures)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n" +
       "    await self.run()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 169, in run\n" +
       "    self.migrate_to_pulp3(cmodel, ctype)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 264, in migrate_to_pulp3\n" +
       "    pulp3content, extra_info = pulp_2to3_detail_content.create_pulp3_content()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/pulp_2to3_models.py\", line 150, in create_pulp3_content\n" +
       "    cr_package = get_cr_obj(self)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/xml_utils.py\", line 185, in get_cr_obj\n" +
       "    primary_xml = render_metadata(pkg, 'primary')\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/xml_utils.py\", line 165, in render_metadata\n" +
       "    return render_primary(xml_template, pkg.checksum, pkg.checksumtype)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/xml_utils.py\", line 110, in render_primary\n" +
       "    return Template(template).render(context)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/template/base.py\", line 149, in __init__\n" +
       "    engine = Engine.get_default()\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/template/engine.py\", line 76, in get_default\n" +
       "    for engine in engines.all():\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/template/utils.py\", line 90, in all\n" +
       "    return [self[alias] for alias in self]\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/template/utils.py\", line 90, in <listcomp>\n" +
       "    return [self[alias] for alias in self]\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/template/utils.py\", line 81, in __getitem__\n" +
       "    engine = engine_cls(params)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/template/backends/django.py\", line 27, in __init__\n" +
       "    self.engine = Engine(self.dirs, self.app_dirs, **options)\n",
      "description"=>
       "__init__() got an unexpected keyword argument 'box_settings'"},
    "worker"=>"/pulp/api/v3/workers/5ee21008-9dd9-4ed2-8665-5aa6aa798069/",
    "child_tasks"=>[],
    "task_group"=>
     "/pulp/api/v3/task-groups/f2fca2dd-b5fd-4b2a-8493-ffd480d974ad/",
    "progress_reports"=>
     [{"message"=>"Pre-migrating Pulp 2 ERRATUM content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>58458,
       "done"=>58458},
      {"message"=>"Processing Pulp 2 repositories, importers, distributors",
       "code"=>"processing.repositories",
       "state"=>"completed",
       "total"=>754,
       "done"=>754},
      {"message"=>"Pre-migrating Pulp 2 ERRATUM content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>58458,
       "done"=>58458},
      {"message"=>"Pre-migrating Pulp 2 DISTRIBUTION content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 RPM content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 RPM content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 DISTRIBUTION content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 SRPM content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 SRPM content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 MODULEMD content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 MODULEMD content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>
        "Pre-migrating Pulp 2 YUM_REPO_METADATA_FILE content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>164,
       "done"=>164},
      {"message"=>
        "Pre-migrating Pulp 2 YUM_REPO_METADATA_FILE content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>164,
       "done"=>164},
      {"message"=>
        "Pre-migrating Pulp 2 PACKAGE_LANGPACKS content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>11,
       "done"=>11},
      {"message"=>
        "Pre-migrating Pulp 2 PACKAGE_LANGPACKS content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>11,
       "done"=>11},
      {"message"=>"Pre-migrating Pulp 2 PACKAGE_GROUP content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>7946,
       "done"=>7946},
      {"message"=>"Migrating rpm content to Pulp 3 rpm",
       "code"=>"migrating.rpm.content",
       "state"=>"failed",
       "total"=>215132,
       "done"=>0},
      {"message"=>"Migrating content to Pulp 3",
       "code"=>"migrating.content",
       "state"=>"failed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Pre-migrating Pulp 2 PACKAGE_GROUP content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>7946,
       "done"=>7946},
      {"message"=>
        "Pre-migrating Pulp 2 PACKAGE_CATEGORY content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>74,
       "done"=>74},
      {"message"=>
        "Pre-migrating Pulp 2 PACKAGE_CATEGORY content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>74,
       "done"=>74},
      {"message"=>
        "Pre-migrating Pulp 2 PACKAGE_ENVIRONMENT content (general info)",
       "code"=>"premigrating.content.general",
       "state"=>"completed",
       "total"=>312,
       "done"=>312},
      {"message"=>
        "Pre-migrating Pulp 2 PACKAGE_ENVIRONMENT content (detail info)",
       "code"=>"premigrating.content.detail",
       "state"=>"completed",
       "total"=>312,
       "done"=>312},
      {"message"=>"Creating repositories in Pulp 3",
       "code"=>"creating.repositories",
       "state"=>"completed",
       "total"=>253,
       "done"=>253},
      {"message"=>"Migrating importers to Pulp 3",
       "code"=>"migrating.importers",
       "state"=>"completed",
       "total"=>115,
       "done"=>115}],
    "created_resources"=>
     ["/pulp/api/v3/task-groups/f2fca2dd-b5fd-4b2a-8493-ffd480d974ad/"]}],
 "task_groups"=>
  [{"pulp_href"=>
     "/pulp/api/v3/task-groups/f2fca2dd-b5fd-4b2a-8493-ffd480d974ad/",
    "description"=>"Migration Sub-tasks",
    "all_tasks_dispatched"=>false,
    "waiting"=>0,
    "skipped"=>0,
    "running"=>0,
    "completed"=>0,
    "canceled"=>0,
    "failed"=>1,
    "group_progress_reports"=>
     [{"message"=>"Repo version creation",
       "code"=>"create.repo_version",
       "total"=>0,
       "done"=>0},
      {"message"=>"Distribution creation",
       "code"=>"create.distribution",
       "total"=>0,
       "done"=>0}]}],
 "poll_attempts"=>{"total"=>67, "failed"=>1}}

Exception:

Katello::Errors::Pulp3Error: __init__() got an unexpected keyword argument 'box_settings'

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Action:

Actions::Pulp3::ImportMigration

Input:

{"reimport_all"=>nil,
 "dependency"=>Step(2).output,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_user_id"=>1,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}

Output:

{}
4

Hey @Antarctic_Guy,

Sorry to hear you are having issues with the migration. @iballou have you seen anything like this?

5

I have also found a problem with remote execution (SSH and Ansible) and am currently digging into it. It appears to be a problem with the sudo password. Manually connecting and sudo works fine but running a command through remote execution connects but fails password on sudo. Once I know more I’ll make another post to track it.

6

@Antarctic_Guy,

I’ve made a Pulp 3 bug that reflects the issues you saw during the migration: https://pulp.plan.io/issues/7846

Could you please find your Pulpcore versions with the following command:

sudo pip3 list | grep pulp

and paste them here and/or in the Pulp 3 bug above?

7

pulp-2to3-migration (0.5.0)
pulp-certguard (1.0.2)
pulp-container (2.0.1)
pulp-file (1.2.0)
pulp-rpm (3.6.2)
pulpcore (3.6.3)

1 Like
8

I upgraded foreman 2.2.0 to 2.2.1 today and get a new error message

Running ForemanMaintain::Scenario

Prepare content for Pulp 3: [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
Rubocop not loaded.
Migration failed, You will want to investigate: https://FQDN/foreman_tasks/tasks/c4dd18e4-1aae-48c5-8c1e-7854cadbae4d
rake aborted!
ForemanTasks::TaskError: Task c4dd18e4-1aae-48c5-8c1e-7854cadbae4d: Katello::Errors::Pulp3Error: ‘NoneType’ object has no attribute ‘cast’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/lib/katello/tasks/pulp3_migration.rake:17:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)

Scenario [ForemanMaintain::Scenario] failed.

The following steps ended up in failing state:

[content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-prepare”

9

[root@SERVER ~]# foreman-rake katello:pulp3_migration --trace
Rubocop not loaded.
** Invoke katello:pulp3_migration (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke katello:disable_dynflow (first_time)
** Execute katello:disable_dynflow
** Invoke katello:check_ping (first_time)
** Invoke environment
** Execute katello:check_ping
** Execute katello:pulp3_migration
Migration failed, You will want to investigate: https://FQDN/foreman_tasks/tasks/946b9d39-30e2-4e07-8dce-a78cf4aee477
rake aborted!
ForemanTasks::TaskError: Task 946b9d39-30e2-4e07-8dce-a78cf4aee477: Katello::Errors::Pulp3Error: ‘NoneType’ object has no attribute ‘cast’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/lib/katello/tasks/pulp3_migration.rake:17:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in block in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in mon_synchronize’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in invoke’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in invoke_task' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block (2 levels) in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in run_with_threads' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in block in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in standard_exception_handling’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in load' /opt/rh/rh-ruby25/root/usr/bin/rake:23:in ’
Tasks: TOP => katello:pulp3_migration

10

@Antarctic_Guy was your 3.16.2 server a fresh install or did you upgrade from an older version? I’m curious which content types are using Pulp 3 for you. Are docker, file, and yum under Pulpcore as overridden to Pulp 2 under Smart Proxy -> Services ?

The Pulp 3 migration is more of a “tech preview” for 3.17, so I might recommend you wait until 3.18 or 3.17.1 (which shouldn’t be out in too long) to try the migration. 3.17.0 currently has older versions of Pulp 3 (3.6 instead of 3.7) installed which might be causing your troubles. We’re fixing that in 3.17.1.

11

It was an upgrade. I think it started on 3.13 or 3.14 and has been upgraded to each release as they come out.

At this time we do not have any docker or file content. We only have YUM repositories at this time. We are in no hurry to do the fully Pulp 3 migration. I just wanted to pass along the errors I was finding.

12

Glad you’re not stuck on it, thanks for the bug report.

As for the installer issue you first mentioned, I’ve created a redmine for it here: Bug #31335: `Section “pulp” is already defined` error when upgrading from Katello 3.16.1.2 to 3.17.0 - Installer - Foreman

1 Like
13

This is very weird. By default we don’t manage the repo in the installer. Did you set a parameter to do so in custom-hiera.yaml?

14

I have The same issue when trying to upgrade katello from 3.17 to 3.18.1.

[root@katello]#  foreman-rake katello:pulp3_migration --trace

** Invoke katello:pulp3_migration (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke katello:disable_dynflow (first_time)
** Execute katello:disable_dynflow
** Invoke katello:check_ping (first_time)
** Invoke environment
** Execute katello:check_ping
enabled
** Execute katello:pulp3_migration
Starting task.
2021-01-07 11:50:22 +0200: Processing Pulp 2 repositories, importers, distributors 1170/4030Migration failed, You will want to investigate: https://katelloserver.com/foreman_tasks/tasks/2a031cc0-e7d5-40c6-9634-5589f08e9baf
rake aborted!
ForemanTasks::TaskError: Task 2a031cc0-e7d5-40c6-9634-5589f08e9baf: Katello::Errors::Pulp3Error: ‘NoneType’ object has no attribute ‘cast’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/lib/katello/tasks/pulp3_migration.rake:31:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in block in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in mon_synchronize’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in invoke’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in invoke_task' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block (2 levels) in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in run_with_threads' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in block in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in standard_exception_handling’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in load' /opt/rh/rh-ruby25/root/usr/bin/rake:23:in ’
Tasks: TOP => katello:pulp3_migration

[root@katello]# sudo pip3 list | grep pulp
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pulp-2to3-migration (0.6.0)
pulp-certguard (1.0.3)
pulp-container (2.1.0)
pulp-file (1.3.0)
pulp-rpm (3.7.0)
pulpcore (3.7.3)

15

same here. Any clue?

16

This error was brought up here as well: [ContentMigration] migration fails with error "Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'cast'" - it could be a valid pulp3 bug, please follow the thread for updates