Foreman 3.11/Katello4.13 - Unable to delete product/repository

Support
41

@pulp might have more context about why a remote delete would hang. They would probably need some logs from /var/log/messages for the action.

We can cancel that task using
pulp task cancel --href="" with the href /pulp/api/v3/tasks/… from output above and try to delete the repository from foreman-rake above. Hopefully remote is the only record acting weird here.

42

I tried canceling it, but geting the following output:

pulp task cancel --href=“/pulp/api/v3/remotes/rpm/rpm/01905067-3b9a-7567-a5a6-5cef25d1221f/”
Traceback (most recent call last):
File “/usr/bin/pulp”, line 8, in
sys.exit(main())
^^^^^^
File “/usr/lib/python3.11/site-packages/pulp_cli/init.py”, line 38, in main
return _main()
^^^^^^^
File “/usr/lib/python3.11/site-packages/click/core.py”, line 1130, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/click/core.py”, line 1055, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/pulpcore/cli/common/generic.py”, line 211, in invoke
return super().invoke(ctx)
^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/click/core.py”, line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/pulpcore/cli/common/generic.py”, line 211, in invoke
return super().invoke(ctx)
^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/click/core.py”, line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/pulpcore/cli/common/generic.py”, line 211, in invoke
return super().invoke(ctx)
^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/click/core.py”, line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/click/core.py”, line 760, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/click/decorators.py”, line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/click/core.py”, line 760, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/click/decorators.py”, line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/click/core.py”, line 760, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/pulpcore/cli/core/task.py”, line 163, in cancel
if entity[“state”] not in [“waiting”, “running”]:
~~~~~~^^^^^^^^^
KeyError: ‘state’

43

Oh…you’d need to use this href:
Screenshot from 2024-06-27 13-21-33

44

pulp task cancel --href=“/pulp/api/v3/tasks/01905a88-3ff9-7cb7-ba8b-2fa17eeb8fa9/”

45

I tried that onen as well, this was the output:

pulp task cancel --href=“/pulp/api/v3/tasks/01905a88-3ff9-7cb7-ba8b-2fa17eeb8fa9/”
Error: ‘“/pulp/api/v3/tasks/01905a88-3ff9-7cb7-ba8b-2fa17eeb8fa9/”’ is not a valid HREF value for a task

46

I did a show on the task ID you provided as is coming back as “not found”

pulp show --href=“/pulp/api/v3/tasks/01905a88-3ff9-7cb7-ba8b-2fa17eeb8fa9/”
Error: {“status”:404,“error”:“Not Found”}

47

I was trying to type out the task href from the image you uploaded earlier. Might be a typo in the href. Can you copy the task id from the output of the pulp remote destroy task earlier .
Screenshot from 2024-06-27 13-21-33

48

you mean this one:

/pulp/api/v3/tasks/01905a88-3ff9-7cb7-ba8b-2fa17eeb8fa9/

1 Like
49

Looks like this time it got cancel, output as “Done”

pulp task cancel --href=“/pulp/api/v3/tasks/01905a88-3ff9-7cb7-ba8b-2fa17eeb8fa9/”
Error: Call aborted due to safe mode
[root@sjglcsappprdg16 ~]# pulp --force task cancel --href=“/pulp/api/v3/tasks/01905a88-3ff9-7cb7-ba8b-2fa17eeb8fa9/”
Waiting to cancel task /pulp/api/v3/tasks/01905a88-3ff9-7cb7-ba8b-2fa17eeb8fa9/
.Done.
Done.

50

Should I proceed with the above?

51

Ya…Is it not the same? Not sure why pulp isn’t fiding the task and returning 404…:smiley:

In any case, if it’s gone, it’s gone…We don’t need to worry about it anmore…You can try this in the foreman-rake console

ForemanTasks.sync_task(::Actions::Pulp3::Repository::Delete,
1620,
SmartProxy.pulp_primary)

This skips the remote delete part because that seems to be hanging for some reason in pulp.

52

I executed the above, got the following output:

foreman-rake console
Loading production environment (Rails 6.1.7.8)
irb(main):001:0> F1620,
irb(main):001:1* ForemanTasks.sync_task(::Actions::Pulp3::Repository::Delete,
irb(main):002:1* 1620,
irb(main):003:0> SmartProxy.pulp_primary)
Traceback (most recent call last):
5: from lib/tasks/console.rake:5:in block in <top (required)>' 4: from (irb):1 3: from foreman-tasks (9.1.1) lib/foreman_tasks.rb:55:in sync_task’
2: from foreman-tasks (9.1.1) lib/foreman_tasks.rb:55:in tap' 1: from foreman-tasks (9.1.1) lib/foreman_tasks.rb:56:in block in sync_task’
ForemanTasks::TaskError (Task a165c0f9-03f4-4c5f-878b-f380af011a02: ActiveRecord::RecordNotFound: Couldn’t find Katello::Repository with ‘id’=1620)

53

@sajha , I checked the product and the repo appears to be gone, but now I have to deal with deleting the empty product which the GUI is not gettign rid of…should we try something similar for the empty product?

54

The product should get deleted from the UI now since what was blocked was the repository.

55

I tried deleting it via the UI, but it remains there

56

Can you refresh the UI? The products page is one of the older pages we have and might not refresh upon deletion automatically.

If it’s still there, I’d check tasks tab and see if there’s a failed task there upon product deletion.

57

When I deleted via the UI, it shows as “success” see below…let me check the task list and see if anything is showing there.

Screenshot from 2024-06-27 10-45-50
Screenshot from 2024-06-27 10-45-50841×236 15.2 KB

58

The error in the task shows the following:

Unable to find a product with the ID “974555559626” for owner “XXXXXXXX”

59

A complete log of the errors in the task shows the following:

Action:
Actions::Candlepin::Product::Destroy
Input:
{“cp_id”=>“974555559626”,
“owner”=>“XXXXXXXX”,
“session_id”=>“faa3a46c-14f2-47a2-adf7-ed6abd088b1f”,
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“current_request_id”=>“faa3a46c-14f2-47a2-adf7-ed6abd088b1f”,
“current_timezone”=>“America/Los_Angeles”,
“current_organization_id”=>1,
“current_location_id”=>nil,
“current_user_id”=>5}
Output:
{}
Exception:
Katello::Errors::CandlepinError: Unable to find a product with the ID “974555559626” for owner “XXXXXXX”
Backtrace:
/usr/share/gems/gems/katello-4.13.0.rc1/app/lib/actions/middleware/propagate_candlepin_errors.rb:26:in rescue in propagate_candlepin_errors' /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/actions/middleware/propagate_candlepin_errors.rb:18:in propagate_candlepin_errors’
/usr/share/gems/gems/katello-4.13.0.rc1/app/lib/actions/middleware/propagate_candlepin_errors.rb:9:in run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’
/usr/share/gems/gems/katello-4.13.0.rc1/app/lib/actions/middleware/remote_action.rb:16:in block in run' /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user’
/usr/share/gems/gems/katello-4.13.0.rc1/app/models/katello/concerns/user_extensions.rb:21:in cp_config' /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user’
/usr/share/gems/gems/katello-4.13.0.rc1/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user' /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/actions/middleware/remote_action.rb:16:in run’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’
/usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:91:in wrap' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/progress.rb:29:in with_progress_calculation’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/progress.rb:15:in run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’
/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’
/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in restore_current_request_id’
/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’
/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’
/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:33:in run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’
/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:54:in restore_curent_user’
/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/world.rb:31:in execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:589:in block (2 levels) in execute_run’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:588:in catch' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:588:in block in execute_run’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in block in with_error_handling' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in catch’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in with_error_handling' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:583:in execute_run’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:304:in execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract.rb:168:in with_meta_calculation' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/director.rb:70:in execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in block (2 levels) in perform’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors.rb:18:in run_user_code' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in block in perform’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in with_telemetry' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in perform’
/usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/serialization.rb:28:in perform' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in execute_job’
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in block (2 levels) in process' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:172:in invoke’
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in block in process' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in block (6 levels) in dispatch’
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in local' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in block (5 levels) in dispatch’
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in block in <module:Sidekiq>' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in block (4 levels) in dispatch’
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in stats' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in block (3 levels) in dispatch’
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in call' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in block (2 levels) in dispatch’
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in global' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in block in dispatch’
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in prepare' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in dispatch’
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in process' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in process_one’
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in run' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in watchdog’
/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in block in safe_thread' /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’

60

That is a candlepin backend ‘error’ saying the product has been deleted from candlepin.

We’ll do it from foreman-rake console with:

Katello::Product.find(id_from_url).destroy!