Using Katello API to sync a repo trigger sync tasks on all content proxies that fail

Support
,
1

On Foreman 4.5.1/Katello 4.7.
Using the this API to trigger the start of the sync of a repository.
https://theforeman.org/plugins/katello/3.17/api/apidoc/v2/repositories/sync.html

Been using this successfully for a year or so but since Katello 4.7 I now see that each time I trigger a sync using the API, the repo sync successfully as before but sync tasks are also created for each of my content proxies that fails.
Makes no sense for those tasks to be created since I assume the Foreman/Katello server is the one doing the sync, not the proxies.
Can it be related to the “Alternate Content Sources” that was added to 4.7? I have not used or have not configured “Alternate Content Sources” though.
Some logs:

Actions::Katello::CapsuleContent::Sync
Input:
{"smart_proxy"=>{"id"=>10, "name"=>"proxy.internal"},
 "services_checked"=>["pulp3"],
 "smart_proxy_id"=>10,
 "current_request_id"=>"4b0f5044-899a-4c04-b57c-464f10c7d697",
 "current_timezone"=>"UTC",
 "current_organization_id"=>nil,
 "current_location_id"=>nil,
 "current_user_id"=>6}
Output:
{}
Exception:
ActiveRecord::RecordNotFound: Couldn't find SmartProxy with 'id'=10 [WHERE (1=0)]
Backtrace:
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation/finder_methods.rb:357:in `raise_record_not_found_exception!'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation/finder_methods.rb:476:in `find_one'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation/finder_methods.rb:458:in `find_with_ids'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation/finder_methods.rb:69:in `find'
/usr/share/gems/gems/katello-4.7.0/app/lib/actions/katello/capsule_content/sync.rb:51:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:604:in `block (2 levels) in execute_finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/rails_executor_wrap.rb:20:in `block in finalize'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/execution_wrapper.rb:91:in `wrap'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/rails_executor_wrap.rb:19:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:23:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/load_setting_values.rb:25:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_request_id.rb:19:in `block in finalize'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_request_id.rb:19:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_timezone.rb:19:in `block in finalize'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_timezone.rb:19:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in `block in finalize'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:40:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:25:in `block in finalize'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:25:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:603:in `block in execute_finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:602:in `execute_finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:78:in `run_step'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:63:in `dispatch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:70:in `block in run_in_sequence'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:70:in `all?'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:70:in `run_in_sequence'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:59:in `dispatch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:28:in `block in finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:48:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:48:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:48:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:48:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:48:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:48:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/common/transaction.rb:17:in `block in rollback_on_error'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/usr/share/gems/gems/activerecord-6.1.7/lib/active_record/transactions.rb:209:in `transaction'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/transaction_adapters/active_record.rb:6:in `transaction'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/common/transaction.rb:16:in `rollback_on_error'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/common/transaction.rb:10:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:29:in `block in finalize_phase'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:29:in `finalize_phase'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director/sequential_manager.rb:27:in `finalize'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director.rb:143:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:196:in `execute_job'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:163:in `block in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:112:in `local'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq.rb:39:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:257:in `stats'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:13:in `call'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:79:in `global'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:124:in `block in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/logger.rb:11:in `with'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:33:in `prepare'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:123:in `dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:162:in `process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:78:in `process_one'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:68:in `run'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:43:in `watchdog'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:52:in `block in safe_thread'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2

Having the smart-proxies sync after a repo sync is somewhat normal, in my experience, at least if your smart-proxies have the Library lifecycle environment synced. What you are seeing is not that the smart-proxies are trying to sync the repos from the internet, but that Foreman informs the smart-proxy “there were changes in one your lifecycle-environments, go grab them” and the proxy syncs the updated content from Foreman.
You can disable this behaviour via the “Sync Smart Proxies after Content View promotion” option in the setting, but that will also disable automatic syncs after promotions to any other lifecycle environments.
Yet, the source of your problem seems to be a different one. Foreman tries to do the above for smart-proxy with the id 10, but cannot find it. Do you have a smart-proxy with that id? Does it have content enabled? Is it actually “proxy.local” that is mentioned in the error?

3

I have 10 smart proxies using ID 1-10. All proxies are configured the same.
Download polixy: On demand
Lifecycle Environments: Library

The names are “modified” in the log so they are not showing the “real” host names.
Since I get one task for each smart proxy that fails, each task tell me it can not find the smart proxy with the id for that smart proxy.
So 10 tasks with these failures:
Couldn’t find SmartProxy with ‘id’=1 [WHERE (1=0)]
Couldn’t find SmartProxy with ‘id’=2 [WHERE (1=0)]
Couldn’t find SmartProxy with ‘id’=3 [WHERE (1=0)]
Couldn’t find SmartProxy with ‘id’=4 [WHERE (1=0)]
Couldn’t find SmartProxy with ‘id’=5 [WHERE (1=0)]
Couldn’t find SmartProxy with ‘id’=6 [WHERE (1=0)]
Couldn’t find SmartProxy with ‘id’=7 [WHERE (1=0)]
Couldn’t find SmartProxy with ‘id’=8 [WHERE (1=0)]
Couldn’t find SmartProxy with ‘id’=9 [WHERE (1=0)]
Couldn’t find SmartProxy with ‘id’=10 [WHERE (1=0)]

4

Went back in the task list before I upgraded to 3.5.1/4.7 and then these sync tasks completed successfully for all proxies triggered by the API. Comparing the “raw input” for the tasks, they are the same, before the upgrade and after.

The API call is using a seperate user assigned with a costum role configured:

Resource Type: Product and Repositories
Permission: view_products, sync_products
Search: name = <repo name1> or name = <repo nam2> or name = <repo name3>
5

Proxy ids confirmed with hammer:

# hammer proxy list --fields id
--
ID
--
1
2
3
4
5
6
7
8
9
10
6

Well, the line where fails has been added in 4.7

https://github.com/Katello/katello/blame/17ce3e44cf8ea6fdde6c6f4ca636bcec319bb41d/app/lib/actions/katello/capsule_content/sync.rb#L51

It seems to pass/generate a where condition 1=0 which obviously fails. That’s why it doesn’t find anything. One of the developers should have a look on how this is possible…

On a sidenote: syncing the library environment to smart proxies is not recommended because as you have noticed, any upstream sync of any repository will cause all smart proxies to sync immediately after. If you have 100 repositories in your foreman server and sync all 100 once a day, this causes 100 syncs to your content proxies…

7

I have seen the “Where 1=0” pattern a few times on projects.theforeman.org for queries related to authorization, so maybe it is a problem with permissions.
@tedevil does you user have the “view_capsule_content” permission? (in roles → filters under “smart proxy”)
From the line @gvde linked, it looks like it’s checking against that permission of the user when triggering the syncs.

8

I am using “Download policy: On demand” so I assume it will only sync meta data right?
Also considering that I currently not using content views (all hosts use “Default Organization View”), I do not really see how it can be configured in any other way. What else then Library do I want to/should I sync?

9

The roles I created to allow these API calls does not have the “view_capsule_content” permission. Only “view_products, sync_products”.
image

So you think I need to add “view_capsule_content” to the role?

10

At least that’s what I would assume, yes. Maybe also add “view_smart_proxies” if it does not work with only view_capsule_content.

1 Like
11

I have opened Bug #35951: Show proper error on sync when user does not have “view_capsule_content” permission? - Katello - Foreman based on the discussion here. The permissions error should be properly reported instead of the current Couldn’t find SmartProxy with ‘id’=1 [WHERE (1=0)] errors.

1 Like
12

I can confirm that adding this filter to the role solved the problem:
Resource: Smart proxy
Permission: view_capsule_content

Thanks!