RH Cloud "Sync inventory status" fails

Problem:
Clicking “RH Cloud” > “Inventory Upload” > “Sync inventory status” creates a task that quickly goes to 75% complete, stays there for approx. 14 minutes, then fails with this message:

undefined method `encode' for URI:Module

  • When I go to the RH Hybrid Cloud Console, I can see new Foreman RHEL clients and my ESXi hypervisors.

  • Within the Cloud Console, I can also see info under the “Advisor” tab for systems that are Foreman RHEL clients. So it would appear that insights info is making its way to the cloud.

  • In Foreman, the same RHEL client systems do not have any info under the “Insights” tab (i.e. “There are no recommendations for your hosts”). Clicking “RH Cloud” > “Insights” also shows “There are no recommendations for your hosts

  • It looks like insights data is making its way up to the cloud, but none of the advisories/recommendations are making it back down.

Expected outcome:
Successfully complete the 'Inventory full sync" task. Advisory/Insights data makes it way from RH Cloud Console to Foreman.

Foreman and Proxy versions:

  • OS: Red Hat Enterprise Linux release 9.4 (Plow)
  • Foreman and Proxies are v 3.11.1
  • rubygem-foreman_rh_cloud-9.0.56-1.fm3_11.el9.noarch
  • rubygem-hammer_cli_foreman_rh_cloud-1.0.2-1.fm3_10.el9.noarch

Other relevant data:

Below is from /var/log/foreman/production.log, with NoMethodError

2024-08-29T18:15:21 [E|bac|c3d47dd5] undefined method `encode' for URI:Module (NoMethodError)
 c3d47dd5 | /usr/share/gems/gems/foreman_rh_cloud-9.0.56/lib/foreman_inventory_upload.rb:78:in `inventory_export_url'
 c3d47dd5 | /usr/share/gems/gems/foreman_rh_cloud-9.0.56/lib/inventory_sync/async/query_inventory_job.rb:74:in `request_url'
 c3d47dd5 | /usr/share/gems/gems/foreman_rh_cloud-9.0.56/lib/inventory_sync/async/query_inventory_job.rb:57:in `query_inventory'
 c3d47dd5 | /usr/share/gems/gems/foreman_rh_cloud-9.0.56/lib/inventory_sync/async/query_inventory_job.rb:34:in `block (3 levels) in try_execute'
 c3d47dd5 | /usr/share/gems/gems/foreman_rh_cloud-9.0.56/lib/inventory_sync/async/query_inventory_job.rb:33:in `loop'
 c3d47dd5 | /usr/share/gems/gems/foreman_rh_cloud-9.0.56/lib/inventory_sync/async/query_inventory_job.rb:33:in `block (2 levels) in try_execute'
 c3d47dd5 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/delegation.rb:88:in `each'
 c3d47dd5 | /usr/share/gems/gems/foreman_rh_cloud-9.0.56/lib/inventory_sync/async/query_inventory_job.rb:24:in `block in try_execute'
 c3d47dd5 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 c3d47dd5 | /usr/share/gems/gems/foreman_rh_cloud-9.0.56/lib/inventory_sync/async/inventory_full_sync.rb:21:in `block in setup_statuses'
 c3d47dd5 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:318:in `transaction'
 c3d47dd5 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/transactions.rb:209:in `transaction'
 c3d47dd5 | /usr/share/gems/gems/foreman_rh_cloud-9.0.56/lib/inventory_sync/async/inventory_full_sync.rb:19:in `setup_statuses'
 c3d47dd5 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 c3d47dd5 | /usr/share/gems/gems/foreman_rh_cloud-9.0.56/lib/inventory_sync/async/inventory_hosts_sync.rb:18:in `block in setup_facet_transaction'
 c3d47dd5 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
 c3d47dd5 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
 c3d47dd5 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
 c3d47dd5 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
 c3d47dd5 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
 c3d47dd5 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
 c3d47dd5 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
 c3d47dd5 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
 c3d47dd5 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
 c3d47dd5 | /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/transactions.rb:209:in `transaction'
 c3d47dd5 | /usr/share/gems/gems/foreman_rh_cloud-9.0.56/lib/inventory_sync/async/inventory_hosts_sync.rb:17:in `setup_facet_transaction'
 c3d47dd5 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 c3d47dd5 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:137:in `run_callbacks'
 c3d47dd5 | /usr/share/gems/gems/foreman_rh_cloud-9.0.56/lib/inventory_sync/async/query_inventory_job.rb:23:in `try_execute'
 c3d47dd5 | /usr/share/gems/gems/foreman_rh_cloud-9.0.56/lib/foreman_rh_cloud/async/exponential_backoff.rb:37:in `poll_external_task'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/polling.rb:22:in `run'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 c3d47dd5 | /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:91:in `wrap'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/progress.rb:15:in `run'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:33:in `run'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 c3d47dd5 | /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/world.rb:31:in `execute'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:588:in `catch'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:588:in `block in execute_run'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `block in with_error_handling'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `catch'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `with_error_handling'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:583:in `execute_run'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:304:in `execute'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/director.rb:95:in `execute'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in `block (2 levels) in perform'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors.rb:18:in `run_user_code'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in `block in perform'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in `with_telemetry'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `perform'
 c3d47dd5 | /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/serialization.rb:28:in `perform'
 c3d47dd5 | [ sidekiq ]
 c3d47dd5 | [ concurrent-ruby ]
2024-08-29T18:15:21 [I|bac|c3d47dd5] Task {label: InventorySync::Async::InventoryFullSync, id: d49c3a82-e5c9-4c66-b4c5-7f730dde9b00, execution_plan_id: 4dc78871-54d3-40e0-9e20-ea6ff717a6ea} state changed: stopped  result: error
2024-08-29T18:15:27 [I|app|a8b24426] Started GET "/notification_recipients" for 10.137.1.170 at 2024-08-29 18:15:27 +0000
2024-08-29T18:15:27 [I|app|a8b24426] Processing by NotificationRecipientsController#index as JSON
2024-08-29T18:15:27 [I|app|a8b24426] Completed 200 OK in 5ms (Views: 0.1ms | ActiveRecord: 0.5ms | Allocations: 1844)
2024-08-29T18:15:27 [I|app|b9548715] Started GET "/foreman_tasks/api/tasks/d49c3a82-e5c9-4c66-b4c5-7f730dde9b00/details?include_permissions" for 10.137.1.170 at 2024-08-29 18:15:27 +0000
2024-08-29T18:15:27 [I|app|b9548715] Processing by ForemanTasks::Api::TasksController#details as JSON
2024-08-29T18:15:27 [I|app|b9548715]   Parameters: {"include_permissions"=>nil, "id"=>"d49c3a82-e5c9-4c66-b4c5-7f730dde9b00"}
2024-08-29T18:15:27 [I|app|b9548715]   Rendered /usr/share/gems/gems/foreman-tasks-9.1.1/app/views/foreman_tasks/api/tasks/details.json.rabl (Duration: 15.1ms | Allocations: 7712)
2024-08-29T18:15:27 [I|app|b9548715] Completed 200 OK in 22ms (Views: 13.8ms | ActiveRecord: 2.6ms | Allocations: 10285)
2024-08-29T18:15:30 [I|app|77b5dee6] Started GET "/foreman_inventory_upload/accounts" for 10.137.1.170 at 2024-08-29 18:15:30 +0000
2024-08-29T18:15:30 [I|app|77b5dee6] Processing by ForemanInventoryUpload::AccountsController#index as JSON
2024-08-29T18:15:30 [I|app|3ba8f79c] Started GET "/foreman_inventory_upload/1/reports/last" for 10.137.1.170 at 2024-08-29 18:15:30 +0000
2024-08-29T18:15:30 [I|app|3ba8f79c] Processing by ForemanInventoryUpload::ReportsController#last as JSON
2024-08-29T18:15:30 [I|app|3ba8f79c]   Parameters: {"organization_id"=>"1"}
2024-08-29T18:15:30 [I|app|77b5dee6] Completed 200 OK in 15ms (Views: 4.7ms | ActiveRecord: 1.8ms | Allocations: 7422)
2024-08-29T18:15:30 [I|app|3ba8f79c] Completed 200 OK in 11ms (Views: 2.0ms | ActiveRecord: 1.7ms | Allocations: 5131)
2024-08-29T18:15:30 [I|app|7cfb1432] Started GET "/foreman_inventory_upload/accounts" for 10.137.1.170 at 2024-08-29 18:15:30 +0000
2024-08-29T18:15:30 [I|app|7cfb1432] Processing by ForemanInventoryUpload::AccountsController#index as JSON
2024-08-29T18:15:30 [I|app|ab25d562] Started GET "/foreman_inventory_upload/1/reports/last" for 10.137.1.170 at 2024-08-29 18:15:30 +0000
2024-08-29T18:15:30 [I|app|ab25d562] Processing by ForemanInventoryUpload::ReportsController#last as JSON
2024-08-29T18:15:30 [I|app|ab25d562]   Parameters: {"organization_id"=>"1"}
2024-08-29T18:15:30 [I|app|7cfb1432] Completed 200 OK in 14ms (Views: 4.4ms | ActiveRecord: 1.6ms | Allocations: 7405)
2024-08-29T18:15:30 [I|app|ab25d562] Completed 200 OK in 10ms (Views: 1.9ms | ActiveRecord: 1.6ms | Allocations: 5131)
2024-08-29T18:15:37 [I|app|d06924a0] Started GET "/notification_recipients" for 10.137.1.170 at 2024-08-29 18:15:37 +0000
2024-08-29T18:15:37 [I|app|d06924a0] Processing by NotificationRecipientsController#index as JSON
2024-08-29T18:15:37 [I|app|d06924a0] Completed 200 OK in 6ms (Views: 0.1ms | ActiveRecord: 0.6ms | Allocations: 1844)
2024-08-29T18:15:47 [I|app|23cb8f2a] Started GET "/notification_recipients" for 10.137.1.170 at 2024-08-29 18:15:47 +0000
2024-08-29T18:15:47 [I|app|23cb8f2a] Processing by NotificationRecipientsController#index as JSON
2024-08-29T18:15:47 [I|app|23cb8f2a] Completed 200 OK in 5ms (Views: 0.1ms | ActiveRecord: 0.6ms | Allocations: 1842)

Please let me know if there is more info I can provide that might help. Thank you.