Problem:
Expected outcome:
Foreman and Proxy versions:
Foreman and Proxy plugin versions:
Distribution and version:
Other relevant data:
Hi @jeremylenz , please let me know if you may have some tips on how to deal with this zombi product I am tyring to get rid of.
If you click “Running steps” what step is it stuck on? Any “Errors”?
This is what I see in running step, this morning I stop it since it was running for over 12 hours…I though a simple delete operation would be quick specially in products that are less than 1GB in size.
It’s possible the Pulp task is stuck. Can you try foreman-maintain service restart? (dnf install foreman-maintain if you need to)
Hi @jeremylenz , the following is the output of my restart:
foreman-maintain service restart
Restart applicable services:
Ok, now see if the task can be resumed.
it goes back into zombie land. Maybe you or someone from the foreman team can help me wipe out that product directly from the posgresql DB? I know that low level, but I think this is a one off issue that we need to deal with at a lower level
What do you see in the Pulp logs? journalctl -u *pulp*
I execute a few sync and they are working fine, I even create a new product and delete it and it works fine…but this one product doesnt want to go away no matter what I do to it, we may need brain surgery for this one.
BTW on 3.11 when we run a product sync and check the sync status it is not showing the active jobs (perhaps a code bug), in order for me to see the running sync I need to “exoand all”. I though I mentioned that in case someone else is facing the same on 3.11
@jeremylenz , do you know anyone from the foreman crew that can provide me the steps to wipe out that product directly from psql DB? Thank you in advance.
sajha
June 26, 2024, 7:39pm
13
Does the product have multiple repositories? Can you do the repo deletions individually and see which repo is causing the hung task for you?
In the worst case, we could run some tasks to delete the katello products from the DB and you’d stop seeing them anywhere on the UI/API, but want to make sure you have as little orphaned content in the pulp back-end as possible.
Hi @sajha , the product only has a single repo in it which I tried deleting and same thing, task goes into zombie mode. If I create a new product from scratch and add new repos it works fine, but the one I am trying to delete is an old product with an old repo. Looks like the GUI is no doing justice in this special case. It seems to me this one needs brain surgery
BTW, I try not to mess around with the foreman DB ever since you guys do a fantastic job and it hasnt give me any issues in ages, though I just mentioned that.
sajha
June 26, 2024, 8:04pm
15
Hi @cucu007 ,
Is this repository part of any content views? in 4.13, you should see a confirmation modal on UI when deleting if the repo were part of any published versions.
The stuck pulp task you provided in the image earlier specifically deals with the remote that points to whatever url that repo is pointing to. You also mentioned the repo is around 1G so not properly dealing with that might waste that space for you. I’d try the following: Update the repository upstream URL to some smaller repo url for ex: Fedora People if you need a dummy repo and then syncing it. That will disconnect the old remote and also all the old packages from the repo and the next orphan cleanup task will clear out the space for you.
You can try another deletion after updating repo URL and syncing and see if that helps. If not we can go into manually cleaning up the data.
1 Like
Hi @sajha , when I tried adjusting the repo I am getting the following error upon saving:
An error occurred saving the Repository: undefined method `label’ for nil:NilClass
This product/repo is not part of any content view nor it has anyt content published. PLease advise.
sajha
June 27, 2024, 1:44pm
17
Do you have a stacktrace in the logs for the error above?
Hi @sajha , I am not sure what stacktrace is nor I have use it in the past…I can executed if you provide me with some guidance on how to do it.
Hi @sajha , I was able to get some logs on the running task, please let me know if anything below makes sense.
2024-06-27T07:09:04 [I|bac|b68f63e6] Task {label: , execution_plan_id: dd6a81a9-aefe-4fc6-b159-94694332bff1} state changed: pendinghttp://foreman.local/internal-product/epel-rpm-internal-repo/ , Index of /fedora-epel/testing/7/x86_64 label' for nil:NilClass (NoMethodError) b68f63e6 | /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/actions/katello/repository/update.rb:31:in plan’block (3 levels) in execute_plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:37:in plan’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:37:in plan’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/load_setting_values.rb:15:in plan’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:10:in block in plan’with_current_request_id' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:9:in plan’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:10:in block in plan’with_current_timezone' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:9:in plan’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in block in plan’with_current_taxonomies' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in plan’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:37:in plan’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:10:in block in plan’with_current_user' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:9:in plan’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/world.rb:31:in execute’block (2 levels) in execute_plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan.rb:387:in switch_flow’concurrence' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:532:in block in execute_plan’block in with_error_handling' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in catch’with_error_handling' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:531:in execute_plan’execute' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/plan_step.rb:56:in block in execute’with_meta_calculation' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/plan_step.rb:55:in execute’block (2 levels) in plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan.rb:387:in switch_flow’with_planning_scope' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan.rb:288:in block in plan’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’plan_phase' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’plan_phase' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’plan_phase' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’plan_phase' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’plan_phase' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’plan_phase' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’block in rollback_on_error' b68f63e6 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in block in transaction’block in within_new_transaction' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in block (2 levels) in synchronize’handle_interrupt' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in block in synchronize’handle_interrupt' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in synchronize’within_new_transaction' b68f63e6 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in transaction’transaction' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/transaction_adapters/active_record.rb:7:in transaction’rollback_on_error' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/common/transaction.rb:7:in plan_phase’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:45:in plan_phase’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/world.rb:31:in execute’plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/world.rb:220:in block (2 levels) in plan_with_options’acquire' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/world.rb:218:in block in plan_with_options’tap' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/world.rb:217:in plan_with_options’plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/world.rb:181:in trigger’trigger' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/lib/foreman_tasks.rb:26:in block in trigger_task’block in rails_safe_trigger_task' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:48:in block in permit_concurrent_loads’yield_shares' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies/interlock.rb:47:in permit_concurrent_loads’rails_safe_trigger_task' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/lib/foreman_tasks.rb:24:in trigger_task’sync_task' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/lib/foreman_tasks/triggers.rb:22:in sync_task’update' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_controller/metal/basic_implicit_render.rb:6:in send_action’process_action' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_controller/metal/rendering.rb:30:in process_action’block in process_action' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:117:in block in run_callbacks’set_timezone' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:126:in block in run_callbacks’clear_thread' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:126:in block in run_callbacks’set_topbar_sweeper_controller' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:126:in block in run_callbacks’around' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:126:in block in run_callbacks’around' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:126:in block in run_callbacks’run_callbacks' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/abstract_controller/callbacks.rb:41:in process_action’process_action' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_controller/metal/instrumentation.rb:34:in block in process_action’block in instrument' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/notifications/instrumenter.rb:24:in instrument’instrument' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_controller/metal/instrumentation.rb:33:in process_action’process_action' b68f63e6 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/railties/controller_runtime.rb:27:in process_action’process_action' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/abstract_controller/base.rb:165:in process’process' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_controller/metal.rb:190:in dispatch’dispatch' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/routing/route_set.rb:50:in dispatch’serve' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/routing/mapper.rb:19:in block in class:Constraints ’serve' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/journey/router.rb:50:in block in serve’each' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/journey/router.rb:32:in serve’call' b68f63e6 | /usr/share/gems/gems/railties-6.1.7.8/lib/rails/engine.rb:539:in call’public_send' b68f63e6 | /usr/share/gems/gems/railties-6.1.7.8/lib/rails/railtie.rb:207:in method_missing’block in <class:Constraints>' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/routing/mapper.rb:49:in serve’block in serve' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/journey/router.rb:32:in each’serve' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/routing/route_set.rb:842:in call’call' b68f63e6 | /usr/share/gems/gems/katello-4.13.0.rc1/lib/katello/middleware/event_daemon.rb:10:in call’call' b68f63e6 | /usr/share/gems/gems/apipie-dsl-2.6.2/lib/apipie_dsl/static_dispatcher.rb:67:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/static.rb:24:in call’call' b68f63e6 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in call’call' b68f63e6 | /usr/share/gems/gems/rack-2.2.9/lib/rack/tempfile_reaper.rb:15:in call’call' b68f63e6 | /usr/share/gems/gems/rack-2.2.9/lib/rack/conditional_get.rb:40:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/http/permissions_policy.rb:22:in call’call' b68f63e6 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in call’context' b68f63e6 | /usr/share/gems/gems/rack-2.2.9/lib/rack/session/abstract/id.rb:260:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/callbacks.rb:27:in block in call’run_callbacks' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/callbacks.rb:26:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/debug_exceptions.rb:29:in call’call' b68f63e6 | /usr/share/gems/gems/railties-6.1.7.8/lib/rails/rack/logger.rb:37:in call_app’call' b68f63e6 | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/remote_ip.rb:81:in call’call' b68f63e6 | /usr/share/gems/gems/katello-4.13.0.rc1/lib/katello/prevent_json_parsing.rb:12:in call’call' b68f63e6 | /usr/share/gems/gems/rack-2.2.9/lib/rack/runtime.rb:22:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/executor.rb:14:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/ssl.rb:77:in call’call' b68f63e6 | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in call’call' b68f63e6 | /usr/share/gems/gems/railties-6.1.7.8/lib/rails/railtie.rb:207:in public_send’method_missing' b68f63e6 | /usr/share/gems/gems/rack-2.2.9/lib/rack/urlmap.rb:74:in block in call’each' b68f63e6 | /usr/share/gems/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in call’call' b68f63e6 | /usr/share/gems/gems/puma-6.4.2/lib/puma/request.rb:100:in block in handle_request’with_force_shutdown' b68f63e6 | /usr/share/gems/gems/puma-6.4.2/lib/puma/request.rb:99:in handle_request’process_client' b68f63e6 | /usr/share/gems/gems/puma-6.4.2/lib/puma/server.rb:245:in block in run’block in spawn_thread' b68f63e6 | [ concurrent-ruby ] 2024-06-27T07:09:04 [I|bac|b68f63e6] Task {label: Actions::Katello::Repository::Update, id: da972c2e-8493-420c-92c5-c8df415bec19, execution_plan_id: dd6a81a9-aefe-4fc6-b159-94694332bff1} state changed: stopped result: error 2024-06-27T07:09:04 [E|app|b68f63e6] NoMethodError: undefined method label’ for nil:NilClassplan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:534:in block (3 levels) in execute_plan’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’block in plan' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:34:in with_current_request_id’plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’block in plan' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:31:in with_current_timezone’plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’block in plan' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in with_current_taxonomies’plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’block in plan' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:41:in with_current_user’plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’execute' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:533:in block (2 levels) in execute_plan’switch_flow' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:436:in concurrence’block in execute_plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in block in with_error_handling’catch' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in with_error_handling’execute_plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:304:in execute’block in execute' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract.rb:168:in with_meta_calculation’execute' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan.rb:289:in block (2 levels) in plan’switch_flow' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan.rb:377:in with_planning_scope’block in plan' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:45:in plan_phase’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:45:in plan_phase’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:45:in plan_phase’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:45:in plan_phase’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:45:in plan_phase’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:45:in plan_phase’call' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in pass’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/common/transaction.rb:18:in block in rollback_on_error’block in transaction' b68f63e6 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:319:in block in within_new_transaction’block (2 levels) in synchronize' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in handle_interrupt’block in synchronize' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in handle_interrupt’synchronize' b68f63e6 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:317:in within_new_transaction’transaction' b68f63e6 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/transactions.rb:209:in transaction’transaction' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/common/transaction.rb:17:in rollback_on_error’plan_phase' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’pass' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in pass’plan_phase' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in call’execute' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan.rb:287:in plan’block (2 levels) in plan_with_options' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/coordinator.rb:334:in acquire’block in plan_with_options' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/world.rb:217:in tap’plan_with_options' b68f63e6 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/world.rb:213:in plan’trigger' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/lib/foreman_tasks.rb:20:in trigger’block in trigger_task' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/lib/foreman_tasks.rb:46:in block in rails_safe_trigger_task’block in permit_concurrent_loads' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/share_lock.rb:187:in yield_shares’permit_concurrent_loads' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/lib/foreman_tasks.rb:45:in rails_safe_trigger_task’trigger_task' b68f63e6 | /usr/share/gems/gems/foreman-tasks-9.1.1/lib/foreman_tasks.rb:55:in sync_task’sync_task' b68f63e6 | /usr/share/gems/gems/katello-4.13.0.rc1/app/controllers/katello/api/v2/repositories_controller.rb:407:in update’send_action' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/abstract_controller/base.rb:228:in process_action’process_action' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/abstract_controller/callbacks.rb:42:in block in process_action’block in run_callbacks' b68f63e6 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in set_timezone’block in run_callbacks' b68f63e6 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in clear_thread’block in run_callbacks' b68f63e6 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in set_topbar_sweeper_controller’block in run_callbacks' b68f63e6 | /usr/share/gems/gems/audited-5.6.0/lib/audited/sweeper.rb:16:in around’block in run_callbacks' b68f63e6 | /usr/share/gems/gems/audited-5.6.0/lib/audited/sweeper.rb:16:in around’block in run_callbacks' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:137:in run_callbacks’process_action' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_controller/metal/rescue.rb:22:in process_action’block in process_action' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/notifications.rb:203:in block in instrument’instrument' b68f63e6 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/notifications.rb:203:in instrument’process_action' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_controller/metal/params_wrapper.rb:249:in process_action’process_action' b68f63e6 | /usr/share/gems/gems/katello-4.13.0.rc1/app/controllers/katello/concerns/api/api_controller.rb:46:in process_action’process' b68f63e6 | /usr/share/gems/gems/actionview-6.1.7.8/lib/action_view/rendering.rb:39:in process’dispatch' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_controller/metal.rb:254:in dispatch’dispatch' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/routing/route_set.rb:33:in serve’block in <class:Constraints>' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/routing/mapper.rb:49:in serve’block in serve' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/journey/router.rb:32:in each’serve' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/routing/route_set.rb:842:in call’call' b68f63e6 | /usr/share/gems/gems/railties-6.1.7.8/lib/rails/railtie.rb:207:in public_send’method_missing' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/routing/mapper.rb:20:in block in class:Constraints ’serve' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/journey/router.rb:50:in block in serve’each' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/journey/router.rb:32:in serve’call' b68f63e6 | /usr/share/gems/gems/katello-4.13.0.rc1/lib/katello/middleware/organization_created_enforcer.rb:18:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/static.rb:24:in call’call' b68f63e6 | /usr/share/gems/gems/apipie-dsl-2.6.2/lib/apipie_dsl/static_dispatcher.rb:67:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/static.rb:24:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/static.rb:24:in call’call' b68f63e6 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in call’call' b68f63e6 | /usr/share/gems/gems/rack-2.2.9/lib/rack/tempfile_reaper.rb:15:in call’call' b68f63e6 | /usr/share/gems/gems/rack-2.2.9/lib/rack/conditional_get.rb:40:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/http/permissions_policy.rb:22:in call’call' b68f63e6 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in call’context' b68f63e6 | /usr/share/gems/gems/rack-2.2.9/lib/rack/session/abstract/id.rb:260:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/callbacks.rb:27:in block in call’run_callbacks' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/callbacks.rb:26:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/debug_exceptions.rb:29:in call’call' b68f63e6 | /usr/share/gems/gems/railties-6.1.7.8/lib/rails/rack/logger.rb:37:in call_app’call' b68f63e6 | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/remote_ip.rb:81:in call’call' b68f63e6 | /usr/share/gems/gems/katello-4.13.0.rc1/lib/katello/prevent_json_parsing.rb:12:in call’call' b68f63e6 | /usr/share/gems/gems/rack-2.2.9/lib/rack/runtime.rb:22:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/executor.rb:14:in call’call' b68f63e6 | /usr/share/gems/gems/actionpack-6.1.7.8/lib/action_dispatch/middleware/ssl.rb:77:in call’call' b68f63e6 | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in call’call' b68f63e6 | /usr/share/gems/gems/railties-6.1.7.8/lib/rails/railtie.rb:207:in public_send’method_missing' b68f63e6 | /usr/share/gems/gems/rack-2.2.9/lib/rack/urlmap.rb:74:in block in call’each' b68f63e6 | /usr/share/gems/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in call’call' b68f63e6 | /usr/share/gems/gems/puma-6.4.2/lib/puma/request.rb:100:in block in handle_request’with_force_shutdown' b68f63e6 | /usr/share/gems/gems/puma-6.4.2/lib/puma/request.rb:99:in handle_request’process_client' b68f63e6 | /usr/share/gems/gems/puma-6.4.2/lib/puma/server.rb:245:in block in run’block in spawn_thread' b68f63e6 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’
sajha
June 27, 2024, 3:23pm
20
That log suggests the repository has been removed from Candlepin. We currently don’t have any params to skip candlepin deletes when removing repo but there is an open PR to do that: Fixes #37600 - Add params to skip Candlepin content on repository remove by chris1984 · Pull Request #11052 · Katello/katello · GitHub
In the meantime , we can try this in foreman-rake console:id of the repository from the URL of the server where the last ID in the URL for repo page is what you want. i.e https://foreman_server/products/product_id/repositories/id
Then you can run foreman-rake console on your server and try this:
repository = Katello::Repository.find(_id_)
ForemanTasks.sync_task(::Actions::Pulp3::Orchestration::Repository::Delete,
repository,
SmartProxy.pulp_primary)
repository.destroy!
