Paused tasks after upgrading to katello 3.14

Support
1

Problem:

I see 3 tasks that are paused.
When I look at the error message I see this:

Katello::Errors::PulpError: PLP0000: Configuration errors:
The configuration parameter <basic_auth_username> requires the <basic_auth_password> parameter to also be set.

This is a very simple setup with 1 katello server only.

Expected outcome:

I’d think there’s no reason for paused tasks like this.

Foreman and Proxy versions:

Foreman 1.24.2

Foreman and Proxy plugin versions:

Katello 3.14.1

Distribution and version:

CentOS 7.7.

Other relevant data:

Update http proxy details

Action:

Actions::Pulp::Repository::Refresh

Input:

{"capsule_id"=>1,
 "pulp_id"=>"0db3e572-7b95-49e3-a7d6-079eadeb1f6c",
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "locale"=>"en",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_user_id"=>1,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}

Output:

{"pulp_tasks"=>
  [{"exception"=>nil,
    "task_type"=>"pulp.server.managers.repo.importer.update_importer_config",
    "_href"=>"/pulp/api/v2/tasks/398d24c3-3257-4444-b8b0-bae9542b1040/",
    "task_id"=>"398d24c3-3257-4444-b8b0-bae9542b1040",
    "tags"=>
     ["pulp:repository:0db3e572-7b95-49e3-a7d6-079eadeb1f6c",
      "pulp:repository_importer:yum_importer",
      "pulp:action:update_importer"],
    "finish_time"=>"2020-03-22T14:32:03Z",
    "_ns"=>"task_status",
    "start_time"=>"2020-03-22T14:32:03Z",
    "traceback"=>nil,
    "spawned_tasks"=>[],
    "progress_report"=>{},
    "queue"=>"reserved_resource_worker-1@satellite4.example.com.dq2",
    "state"=>"finished",
    "worker_name"=>"reserved_resource_worker-1@satellite4.example.com",
    "result"=>
     {"repo_id"=>"0db3e572-7b95-49e3-a7d6-079eadeb1f6c",
      "last_updated"=>"2020-03-22T14:32:03Z",
      "_href"=>
       "/v2/repositories/0db3e572-7b95-49e3-a7d6-079eadeb1f6c/importers/yum_importer/",
      "_ns"=>"repo_importers",
      "importer_type_id"=>"yum_importer",
      "last_override_config"=>{},
      "last_sync"=>"2020-03-21T19:01:40Z",
      "scratchpad"=>{"repomd_revision"=>1582709193},
      "_id"=>{"$oid"=>"5b9a88c474a3940553e18689"},
      "config"=>
       {"feed"=>"http://mirror.centos.org/centos/7/sclo/x86_64/sclo/",
        "basic_auth_password"=>"*****",
        "remove_missing"=>true,
        "ssl_validation"=>true,
        "basic_auth_username"=>"",
        "download_policy"=>"on_demand"},
      "id"=>"yum_importer"},
    "error"=>nil,
    "_id"=>{"$oid"=>"5e7776e346386b2b422ac1dd"},
    "id"=>"5e7776e346386b2b422ac1dd"},
   {"exception"=>nil,
    "task_type"=>"pulp.server.managers.repo.importer.update_importer_config",
    "_href"=>"/pulp/api/v2/tasks/d7f1d2c0-b72e-4a9a-bca1-3dc564d1fe97/",
    "task_id"=>"d7f1d2c0-b72e-4a9a-bca1-3dc564d1fe97",
    "tags"=>
     ["pulp:repository:0db3e572-7b95-49e3-a7d6-079eadeb1f6c",
      "pulp:repository_importer:yum_importer",
      "pulp:action:update_importer"],
    "finish_time"=>"2020-03-22T14:32:03Z",
    "_ns"=>"task_status",
    "start_time"=>"2020-03-22T14:32:03Z",
    "traceback"=>
     "Traceback (most recent call last):\n" +
     "  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n" +
     "    R = retval = fun(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 686, in __call__\n" +
     "    return super(Task, self).__call__(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 108, in __call__\n" +
     "    return super(PulpTask, self).__call__(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n" +
     "    return self.run(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/importer.py\", line 273, in update_importer_config\n" +
     "    validate_importer_config(repo_obj, repo_importer.importer_type_id, repo_importer.config)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/importer.py\", line 177, in validate_importer_config\n" +
     "    raise exceptions.PulpDataException(message)\n" +
     "PulpDataException: Configuration errors:\n" +
     "The configuration parameter <basic_auth_username> requires the <basic_auth_password> parameter to also be set.\n",
    "spawned_tasks"=>[],
    "progress_report"=>{},
    "queue"=>"reserved_resource_worker-1@satellite4.example.com.dq2",
    "state"=>"error",
    "worker_name"=>"reserved_resource_worker-1@satellite4.example.com",
    "result"=>nil,
    "error"=>
     {"code"=>"PLP0000",
      "data"=>{},
      "description"=>
       "Configuration errors:\n" +
       "The configuration parameter <basic_auth_username> requires the <basic_auth_password> parameter to also be set.",
      "sub_errors"=>[]},
    "_id"=>{"$oid"=>"5e7776e346386b2b422ac1ee"},
    "id"=>"5e7776e346386b2b422ac1ee"},
   {"exception"=>nil,
    "task_type"=>"pulp.server.tasks.repository.distributor_update",
    "_href"=>"/pulp/api/v2/tasks/b2db6fe3-24c2-476f-abab-e0c14ca47691/",
    "task_id"=>"b2db6fe3-24c2-476f-abab-e0c14ca47691",
    "tags"=>
     ["pulp:repository:0db3e572-7b95-49e3-a7d6-079eadeb1f6c",
      "pulp:repository_distributor:0db3e572-7b95-49e3-a7d6-079eadeb1f6c",
      "pulp:action:update_distributor"],
    "finish_time"=>nil,
    "_ns"=>"task_status",
    "start_time"=>"2020-03-22T14:32:03Z",
    "traceback"=>nil,
    "spawned_tasks"=>[],
    "progress_report"=>{},
    "queue"=>"reserved_resource_worker-1@satellite4.example.com.dq2",
    "state"=>"running",
    "worker_name"=>"reserved_resource_worker-1@satellite4.example.com",
    "result"=>nil,
    "error"=>nil,
    "_id"=>{"$oid"=>"5e7776e346386b2b422ac1ff"},
    "id"=>"5e7776e346386b2b422ac1ff"},
   {"exception"=>nil,
    "task_type"=>"pulp.server.tasks.repository.distributor_update",
    "_href"=>"/pulp/api/v2/tasks/398ec5de-a6d3-43a5-a74b-413521d649d8/",
    "task_id"=>"398ec5de-a6d3-43a5-a74b-413521d649d8",
    "tags"=>
     ["pulp:repository:0db3e572-7b95-49e3-a7d6-079eadeb1f6c",
      "pulp:repository_distributor:0db3e572-7b95-49e3-a7d6-079eadeb1f6c_clone",
      "pulp:action:update_distributor"],
    "finish_time"=>nil,
    "_ns"=>"task_status",
    "start_time"=>nil,
    "traceback"=>nil,
    "spawned_tasks"=>[],
    "progress_report"=>{},
    "queue"=>"reserved_resource_worker-1@satellite4.example.com.dq2",
    "state"=>"waiting",
    "worker_name"=>"reserved_resource_worker-1@satellite4.example.com",
    "result"=>nil,
    "error"=>nil,
    "_id"=>{"$oid"=>"5e7776e346386b2b422ac211"},
    "id"=>"5e7776e346386b2b422ac211"},
   {"exception"=>nil,
    "task_type"=>"pulp.server.tasks.repository.distributor_update",
    "_href"=>"/pulp/api/v2/tasks/18e02a8e-4dec-4efe-84c3-02b31ca3d941/",
    "task_id"=>"18e02a8e-4dec-4efe-84c3-02b31ca3d941",
    "tags"=>
     ["pulp:repository:0db3e572-7b95-49e3-a7d6-079eadeb1f6c",
      "pulp:repository_distributor:export_distributor",
      "pulp:action:update_distributor"],
    "finish_time"=>nil,
    "_ns"=>"task_status",
    "start_time"=>nil,
    "traceback"=>nil,
    "spawned_tasks"=>[],
    "progress_report"=>{},
    "queue"=>"reserved_resource_worker-1@satellite4.example.com.dq2",
    "state"=>"waiting",
    "worker_name"=>"reserved_resource_worker-1@satellite4.example.com",
    "result"=>nil,
    "error"=>nil,
    "_id"=>{"$oid"=>"5e7776e346386b2b422ac225"},
    "id"=>"5e7776e346386b2b422ac225"}],
 "poll_attempts"=>{"total"=>1, "failed"=>2}}

Exception:

Katello::Errors::PulpError: PLP0000: Configuration errors:
The configuration parameter <basic_auth_username> requires the <basic_auth_password> parameter to also be set.

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/abstract_async_task.rb:122:in `block in external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/abstract_async_task.rb:120:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/abstract_async_task.rb:120:in `external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/polling.rb:90:in `rescue in resume_external_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/polling.rb:87:in `resume_external_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/polling.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:563:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/keep_locale.rb:11:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/keep_locale.rb:11:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:562:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:561:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:561:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:475:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:475:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:475:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:556:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/director.rb:68:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors.rb:12:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:14:in `on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/context.rb:46:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/actor.rb:106:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/actor.rb:47:in `block in on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/actor.rb:59:in `with_backtrace'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/actor.rb:47:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/core.rb:162:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/synchronization/mutex_lockable_object.rb:41:in `block in synchronize'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/serialized_execution.rb:18:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/serialized_execution.rb:96:in `work'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2

Hey @PenguinsAreKewl,

Can you tell me which Katello version you upgraded from? Often things will get a bit weird if you have skipped versions while upgrading ie going from 3.11 -> 3.14 would not be recommended at all.

Something you can try in the mean time is foreman-rake katello:upgrades:3.10:clear_invalid_repo_credentials and let me know what the output of that is.

3

I did an upgrade from a 3.7 :slight_smile: I went through all versions (3.8, 3.9…). Perhaps it was already there before I upgraded.
I’ve followed your suggestion and resumed the tasks. They seem to be gone now.

Thank you sir for your help, greatly appreciated!

EDIT: this was the output:

Clearing invalid credentials for SCLo-RH (7)
Refreshing repository SCLo-RH (822)
Refreshing repository SCLo-RH (48)
Refreshing repository SCLo-RH (824)
Refreshing repository SCLo-RH (44)
Refreshing repository SCLo-RH (812)
Refreshing repository SCLo-RH (820)
Refreshing repository SCLo-RH (66)
Clearing invalid credentials for SCLo (8)
Refreshing repository SCLo (823)
Refreshing repository SCLo (46)
Refreshing repository SCLo (825)
Refreshing repository SCLo (43)
Refreshing repository SCLo (813)
Refreshing repository SCLo (821)
Refreshing repository SCLo (64)