Problem:
Hi all,
thanks for the help in advance. I`ve been experiencing warnings with the daily reposync plan, for some concrete repositories.
I’ve tried to delete and add again the offending repository with no success. There is no apparent error on the UI:
(I will also add the messages log of the moment)
When trying to run an advanced full sync, I get the error “too many open files”. So that is a new issue that I found out due to the first error.
Expected outcome:
Clean of warnings, green sync
Foreman and Proxy versions:
2.3.3
Foreman and Proxy plugin versions:
3.18.21 Katello
Distribution and version:
RHEL 7
Other relevant data:
The system messages:
2021-06-07T17:53:29 [I|app|ba6f775e] Started POST "/katello/api/v2/repositories/81/sync?organization_id=4" for 127.0.0.1 at 2021-06-07 17:53:29 +0200
2021-06-07T17:53:29 [I|app|ba6f775e] Processing by Katello::Api::V2::RepositoriesController#sync as HTML
2021-06-07T17:53:29 [I|app|ba6f775e] Parameters: {"id"=>"81", "organization_id"=>"4", "api_version"=>"v2", "repository"=>{"id"=>"81"}}
2021-06-07T17:53:29 [I|bac|ba6f775e] Task {label: Actions::Katello::Repository::Sync, id: 360438e8-3879-43fb-849c-dafadd825696, execution_plan_id: 8ac739bd-7ec7-42d6-bd83-1674003fac57} state changed: planning
2021-06-07T17:53:30 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 360438e8-3879-43fb-849c-dafadd825696, execution_plan_id: 8ac739bd-7ec7-42d6-bd83-1674003fac57} state changed: planned
2021-06-07T17:53:30 [I|app|] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource
2021-06-07T17:53:30 [I|app|] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource (Duration: 48.5ms | Allocations: 36682)
2021-06-07T17:53:30 [I|app|] Completed 202 Accepted in 559ms (Views: 50.2ms | ActiveRecord: 41.7ms | Allocations: 157292)
2021-06-07T17:53:30 [I|app|477bc283] Started GET "/tasks/views/task-details.html" for 127.0.0.1 at 2021-06-07 17:53:30 +0200
2021-06-07T17:53:30 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 360438e8-3879-43fb-849c-dafadd825696, execution_plan_id: 8ac739bd-7ec7-42d6-bd83-1674003fac57} state changed: running
2021-06-07T17:53:30 [I|app|668d49a5] Started POST "/foreman_tasks/api/tasks/bulk_search" for 127.0.0.1 at 2021-06-07 17:53:30 +0200
2021-06-07T17:53:30 [I|app|668d49a5] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2021-06-07T17:53:30 [I|app|668d49a5] Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"360438e8-3879-43fb-849c-dafadd825696", "search_id"=>"1"}], "task"=>{}}
2021-06-07T17:53:30 [I|app|668d49a5] Completed 200 OK in 92ms (Views: 1.0ms | ActiveRecord: 5.6ms | Allocations: 39495)
2021-06-07T17:53:31 [I|app|8cf6b9ec] Started POST "/foreman_tasks/api/tasks/bulk_search" for 127.0.0.1 at 2021-06-07 17:53:31 +0200
2021-06-07T17:53:31 [I|app|8cf6b9ec] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2021-06-07T17:53:31 [I|app|8cf6b9ec] Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"360438e8-3879-43fb-849c-dafadd825696", "search_id"=>"1"}], "task"=>{}}
2021-06-07T17:53:31 [I|app|8cf6b9ec] Completed 200 OK in 49ms (Views: 0.9ms | ActiveRecord: 3.6ms | Allocations: 40015)
2021-06-07T17:53:33 [I|app|5e125c4b] Started POST "/foreman_tasks/api/tasks/bulk_search" for 127.0.0.1 at 2021-06-07 17:53:33 +0200
2021-06-07T17:53:33 [I|app|5e125c4b] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2021-06-07T17:53:33 [I|app|5e125c4b] Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"360438e8-3879-43fb-849c-dafadd825696", "search_id"=>"1"}], "task"=>{}}
2021-06-07T17:53:33 [I|app|5e125c4b] Completed 200 OK in 53ms (Views: 0.9ms | ActiveRecord: 3.7ms | Allocations: 40843)
2021-06-07T17:53:37 [I|app|a264cb5f] Started POST "/foreman_tasks/api/tasks/bulk_search" for 127.0.0.1 at 2021-06-07 17:53:37 +0200
2021-06-07T17:53:37 [I|app|a264cb5f] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2021-06-07T17:53:37 [I|app|a264cb5f] Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"360438e8-3879-43fb-849c-dafadd825696", "search_id"=>"1"}], "task"=>{}}
2021-06-07T17:53:37 [I|app|a264cb5f] Completed 200 OK in 55ms (Views: 0.8ms | ActiveRecord: 8.7ms | Allocations: 40851)
2021-06-07T17:53:42 [I|app|ac8c639a] Started POST "/foreman_tasks/api/tasks/bulk_search" for 127.0.0.1 at 2021-06-07 17:53:42 +0200
2021-06-07T17:53:42 [I|app|ac8c639a] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2021-06-07T17:53:42 [I|app|ac8c639a] Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"360438e8-3879-43fb-849c-dafadd825696", "search_id"=>"1"}], "task"=>{}}
2021-06-07T17:53:42 [I|app|ac8c639a] Completed 200 OK in 47ms (Views: 0.9ms | ActiveRecord: 3.5ms | Allocations: 40754)
2021-06-07T17:53:48 [I|app|808ee031] Started POST "/foreman_tasks/api/tasks/bulk_search" for 127.0.0.1 at 2021-06-07 17:53:48 +0200
2021-06-07T17:53:48 [I|app|808ee031] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2021-06-07T17:53:48 [I|app|808ee031] Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"360438e8-3879-43fb-849c-dafadd825696", "search_id"=>"1"}], "task"=>{}}
2021-06-07T17:53:48 [I|app|808ee031] Completed 200 OK in 46ms (Views: 0.9ms | ActiveRecord: 3.4ms | Allocations: 40789)
2021-06-07T17:53:55 [I|app|20214e7e] Started POST "/foreman_tasks/api/tasks/bulk_search" for 127.0.0.1 at 2021-06-07 17:53:55 +0200
2021-06-07T17:53:55 [I|app|20214e7e] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2021-06-07T17:53:55 [I|app|20214e7e] Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"360438e8-3879-43fb-849c-dafadd825696", "search_id"=>"1"}], "task"=>{}}
2021-06-07T17:53:55 [I|app|20214e7e] Completed 200 OK in 50ms (Views: 0.8ms | ActiveRecord: 3.6ms | Allocations: 40754)
2021-06-07T17:54:03 [I|app|cda817ce] Started POST "/foreman_tasks/api/tasks/bulk_search" for 127.0.0.1 at 2021-06-07 17:54:03 +0200
2021-06-07T17:54:03 [I|app|cda817ce] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2021-06-07T17:54:03 [I|app|cda817ce] Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"360438e8-3879-43fb-849c-dafadd825696", "search_id"=>"1"}], "task"=>{}}
2021-06-07T17:54:03 [I|app|cda817ce] Completed 200 OK in 52ms (Views: 0.9ms | ActiveRecord: 4.7ms | Allocations: 40754)
2021-06-07T17:54:12 [I|app|fbe28f3c] Started POST "/foreman_tasks/api/tasks/bulk_search" for 127.0.0.1 at 2021-06-07 17:54:12 +0200
2021-06-07T17:54:13 [I|app|fbe28f3c] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2021-06-07T17:54:13 [I|app|fbe28f3c] Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"360438e8-3879-43fb-849c-dafadd825696", "search_id"=>"1"}], "task"=>{}}
2021-06-07T17:54:13 [I|app|fbe28f3c] Completed 200 OK in 188ms (Views: 1.0ms | ActiveRecord: 7.3ms | Allocations: 40915)
2021-06-07T17:54:17 [E|bac|] None (Katello::Errors::Pulp3Error)
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
| [ sidekiq ]
| [ concurrent-ruby ]
2021-06-07T17:54:19 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 360438e8-3879-43fb-849c-dafadd825696, execution_plan_id: 8ac739bd-7ec7-42d6-bd83-1674003fac57} state changed: stopped result: warning
2021-06-07T17:54:19 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: 360438e8-3879-43fb-849c-dafadd825696, execution_plan_id: 8ac739bd-7ec7-42d6-bd83-1674003fac57} state changed: stopped result: warning
2021-06-07T17:54:22 [I|app|ba42565b] Started POST "/foreman_tasks/api/tasks/bulk_search" for 127.0.0.1 at 2021-06-07 17:54:22 +0200
2021-06-07T17:54:22 [I|app|ba42565b] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2021-06-07T17:54:22 [I|app|ba42565b] Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"360438e8-3879-43fb-849c-dafadd825696", "search_id"=>"1"}], "task"=>{}}
2021-06-07T17:54:22 [I|app|ba42565b] Completed 200 OK in 56ms (Views: 1.1ms | ActiveRecord: 4.4ms | Allocations: 41266)
2021-06-07T17:58:32 [I|app|e7dde560] Started GET "/notification_recipients" for 127.0.0.1 at 2021-06-07 17:58:32 +0200
2021-06-07T17:58:32 [I|app|e7dde560] Processing by NotificationRecipientsController#index as JSON
Thanks for the help 