Unique Hosts in Foreman (salt)

Support
21

Thanks for your detailed description and and the code diff! I’ll look into this and check why the validation was not enabled in the first place.

When upgrading to Salt 3006, you might wanna keep an eye on the permissions of the salt-master service. We ran into some issues that Salt changed it from root to a new salt user and I’m currently working on integrating this into Foreman (see here for a first hint).

I’m currently evaluating adding a new foreman-salt user instead. However, such user would also need access to the salt states to import them via the API. I’ll soon create an RFC where your input might be helpful! I’ll link it here.

22

Thanks for the warning. We are doing a lot of testing with 3006, first. And foreman/plugin testing will be part of that. We are already forcing all of our salt-master services to run as root, because PAM doesnt work with non root users. Even salt says that running as root doesnt have that big of a security impact, because if someone has access to salt, under any user, they have access to all the minions. But I think enough people have told them they dont want it running as root.

Thanks again!

1 Like
23

Oh!! And I meant validate from false to true, but I think you got that.

1 Like
24

Well. Something still isnt right with this. Out of some 60,000 hosts, Im seeing the below error on about 20,000 of them. Im not really sure whats going on. The error and stacktrace isnt seemingly very helpful.

Foreman::Exception

ERF42-6331 [Foreman::Exception]: Failed to create host


---
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:87:in
  `find_or_create_host'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:32:in
  `initialize'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:17:in
  `new'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:17:in
  `block in import'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:15:in
  `each'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:15:in
  `map'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:15:in
  `import'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/lib/actions/foreman_salt/report_import.rb:14:in
  `block in run'"
- "/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'"
- "/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/lib/actions/foreman_salt/report_import.rb:13:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:589:in `block (3 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
  `block in run'"
- "/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/execution_wrapper.rb:91:in
  `wrap'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action/progress.rb:17:in `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in
  `restore_current_request_id'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in
  `restore_curent_timezone'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
  `restore_current_taxonomies'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:32:in `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:54:in
  `restore_curent_user'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/world.rb:31:in `execute'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:588:in `block (2 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:587:in `catch'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:587:in `block in execute_run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:490:in `block in with_error_handling'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:490:in `catch'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:490:in `with_error_handling'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:582:in `execute_run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:303:in `execute'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
  `block (2 levels) in execute'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract.rb:167:in
  `with_meta_calculation'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
  `block in execute'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
  `open_action'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
  `execute'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/director.rb:69:in `execute'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in
  `block (2 levels) in perform'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors.rb:18:in `run_user_code'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
  `block in perform'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in
  `with_telemetry'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in
  `perform'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors/sidekiq/serialization.rb:27:in
  `perform'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in `execute_job'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in `block (2 levels)
  in process'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:172:in `invoke'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in `block in process'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in `block (6 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in `local'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in `block (5 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in `block in <module:Sidekiq>'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread'"
- "/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block
  in create_with_logging_context'"
25

Production logs show a bit more. I wonder if it has something to do with upper/lowercase somewhere along the stack from salt to foreman:

2024-10-23T12:34:05 [D|app|69dcf8f3] Initializing ReportImporter with host: ASPALPSTNRXA.xxx.com, proxy_id:
2024-10-23T12:34:05 [D|app|69dcf8f3] Finding or creating host: ASPALPSTNRXA.xxx.com
2024-10-23T12:34:05 [I|app|69dcf8f3] Host not found; creating new host with name: ASPALPSTNRXA.xxx.com
2024-10-23T12:34:05 [W|app|69dcf8f3] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:34:05 [W|app|69dcf8f3] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:34:05 [W|app|69dcf8f3] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:34:05 [E|app|69dcf8f3] Failed to create host ASPALPSTNRXA.xxx.com: Name has already been taken
2024-10-23T12:34:05 [E|app|69dcf8f3] Error in find_or_create_host: ERF42-6331 [Foreman::Exception]: Failed to create host
2024-10-23T12:34:05 [E|app|69dcf8f3] Import failed: ERF42-6331 [Foreman::Exception]: Failed to create host
2024-10-23T12:34:05 [D|tax|69dcf8f3] Current organization set to none
2024-10-23T12:34:05 [D|tax|69dcf8f3] Current location set to none
2024-10-23T12:34:05 [E|bac|69dcf8f3] ERF42-6331 [Foreman::Exception]: Failed to create host (Foreman::Exception)
26

Well, not ALL failures are uppercase hostnames:

--
2024-10-23T12:31:10 [W|app|445e6a7d] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:31:10 [W|app|445e6a7d] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:31:10 [W|app|445e6a7d] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:31:10 [E|app|445e6a7d] Failed to create host 10-136-57-119.redacted: Name has already been taken
2024-10-23T12:31:10 [E|app|445e6a7d] Error in find_or_create_host: ERF42-6331 [Foreman::Exception]: Failed to create host
2024-10-23T12:31:10 [E|app|445e6a7d] Import failed: ERF42-6331 [Foreman::Exception]: Failed to create host
2024-10-23T12:31:10 [D|tax|445e6a7d] Current organization set to none
2024-10-23T12:31:10 [D|tax|445e6a7d] Current location set to none
2024-10-23T12:31:10 [E|bac|445e6a7d] ERF42-6331 [Foreman::Exception]: Failed to create host (Foreman::Exception)

This seems like an issue:

irb(main):001:0> Host::Base.where('name LIKE ?', '%10-136-57-119%')
=>
[#<Host::Base:0x000072045ef03500
  id: 28562,
  name: "10-136-57-119.redacted",
  last_compile: nil,
  last_report: "[FILTERED]",
  updated_at: Thu, 17 Oct 2024 17:12:49.174780000 UTC +00:00,
  created_at: Thu, 17 Oct 2024 17:12:48.215720000 UTC +00:00,
  root_pass: nil,
  architecture_id: nil,
  operatingsystem_id: nil,
  ptable_id: nil,
  medium_id: nil,
  build: false,
  comment: nil,
  disk: nil,
  installed_at: nil,
  model_id: nil,
  hostgroup_id: nil,
  owner_id: 22,
  owner_type: "User",
  enabled: true,
  puppet_ca_proxy_id: nil,
  managed: false,
  use_image: nil,
  image_file: nil,
  uuid: nil,
  compute_resource_id: nil,
  puppet_proxy_id: nil,
  certname: nil,
  image_id: nil,
  organization_id: nil,
  location_id: nil,
  type: nil,
  otp: nil,
  realm_id: nil,
  compute_profile_id: nil,
  provision_method: nil,
  salt_proxy_id: nil,
  grub_pass: nil,
  salt_environment_id: nil,
  global_status: 0,
  lookup_value_matcher: "[FILTERED]",
  pxe_loader: nil,
  initiated_at: nil,
  build_errors: nil,
  salt_autosign_key: nil,
  salt_status: nil,
  creator_id: 1>]

Host does exist but it is not managed. Its like it got soft deleted at some point and cannot be recreated.

27

Im not sure if this is a lowercase issue, or just another managed: False issue.

2024-10-23T12:30:00 [W|app|e23da0dc] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:30:00 [W|app|e23da0dc] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:30:00 [W|app|e23da0dc] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:30:00 [E|app|e23da0dc] Failed to create host TVAWBD0FPP02: Name has already been taken
2024-10-23T12:30:00 [E|app|e23da0dc] Error in find_or_create_host: ERF42-6331 [Foreman::Exception]: Failed to create host
2024-10-23T12:30:00 [E|app|e23da0dc] Import failed: ERF42-6331 [Foreman::Exception]: Failed to create host
2024-10-23T12:30:00 [D|tax|e23da0dc] Current organization set to none
2024-10-23T12:30:00 [D|tax|e23da0dc] Current location set to none
2024-10-23T12:30:00 [E|bac|e23da0dc] ERF42-6331 [Foreman::Exception]: Failed to create host (Foreman::Exception)

[#<Host::Managed:0x000073fb6cf7b8a0
  id: 33186,
  name: "tvawbd0fpp02",
  last_compile: Wed, 23 Oct 2024 12:28:25.818802000 UTC +00:00,
  last_report: "[FILTERED]",
  updated_at: Wed, 23 Oct 2024 12:28:26.821676000 UTC +00:00,
  created_at: Thu, 17 Oct 2024 18:40:38.225524000 UTC +00:00,
  root_pass: nil,
  architecture_id: nil,
  operatingsystem_id: 7,
  ptable_id: nil,
  medium_id: nil,
  build: false,
  comment: nil,
  disk: nil,
  installed_at: nil,
  model_id: 1,
  hostgroup_id: nil,
  owner_id: 22,
  owner_type: "User",
  enabled: true,
  puppet_ca_proxy_id: nil,
  managed: false,
  use_image: nil,
  image_file: nil,
  uuid: nil,
  compute_resource_id: nil,
  puppet_proxy_id: nil,
  certname: nil,
  image_id: nil,
  organization_id: 1,
  location_id: 10,
  type: "Host::Managed",
  otp: nil,
  realm_id: nil,
  compute_profile_id: nil,
  provision_method: nil,
  salt_proxy_id: nil,
  grub_pass: nil,
  salt_environment_id: nil,
  global_status: 2,
  lookup_value_matcher: "[FILTERED]",
  pxe_loader: nil,
  initiated_at: nil,
  build_errors: nil,
  salt_autosign_key: nil,
  salt_status: nil,
  creator_id: 1>]

Umm WOAH what in the world is going on here?

irb(main):007:0> Host::Base.where(managed: false).count
=> 26360

Oh, disregard, its not an issue with them being managed/unmanaged. I have no idea what is going on here with these host creation failures:

irb(main):007:0> Host::Base.where(managed: false).count
=> 26360
irb(main):008:0> Host::Base.where(managed: true).count
=> 0
28

So @bastian-src it does appear that even with validate: true, I still get duplicate hosts.

image
image1620×478 86.4 KB

I think Ill have to turn off the functionality that auto creates hosts on facts/reports upload, and use the api to create the host first, unfortunately. I just cannot figure this out.

irb(main):002:0> duplicates = Host::Base.group(:name).having('COUNT(*) > 1').count
=>
{"wdc-dssrv01devb.xxxxx"=>13,
...
irb(main):003:0> duplicates.size
=> 193
29

After some 30-40 hours of combined work on this, I cannot figure it out. Our plan is to completely disable the options to create hosts from reports/facts and somehow write automation that uses the foreman api to create a host, with multiple guard rails to first check that the host doesnt exist.

30

Still back at this because its driving me crazy. In just 12 hours, we had this many hosts duplicated, its literally creating a new host for every single report:

=>
{"wdc-aixdevdb-87.redacted.net"=>50,
 "wdc-aixdevdb-02.redacted.net"=>50,
 "wdc-aixdb-129.redacted.net"=>50,
 "wdc-aixdb-81.redacted.net"=>50,
 "wdc-aixdb-65.redacted.net"=>50,
 "wdc-aixdevdb-88.redacted.net"=>49,
 "wdc-aixbetadb-78.redacted.net"=>49,
 "wdc-aixdb-73b.redacted.net"=>49,
 "wdc-aixdb-112.redacted.net"=>49,
 "wdc-aixdb-149.redacted.net"=>49,
 "wdc-aixdb-78.redacted.net"=>49,
 "wdc-aixdb-137.redacted.net"=>49,
 "wdc-aixbetadb-65.redacted.net"=>49,
 "wdc-aixdb-68.redacted.net"=>49,
 "wdc-aixdb-82.redacted.net"=>49,
 "wdc-aixbetadb-07.redacted.net"=>49,
 "wdc-aixdevdb-109.redacted.net"=>49,
 "wdc-aixdb-106.redacted.net"=>49,
 "wdc-aixdb-73a.redacted.net"=>49,
 "wdc-aixbetadb-44.redacted.net"=>49,
 "wdc-aixdb-136.redacted.net"=>49,
 "wdc-aixdb-44.redacted.net"=>49,
 "wdc-aixdevdb-43.redacted.net"=>48,
 "wdc-aixbetadb-35.redacted.net"=>48,
 "wdc-aixbetadb-112.redacted.net"=>48,
 "wdc-aixbetadb-36.redacted.net"=>48,
 "wdc-aixbetadb-136.redacted.net"=>48,
 "uxtapesswcnim01.m.redacted.net"=>48,
 "wdc-aixbetadb-129.redacted.net"=>48,
 "wdc-aixtstdb-94.redacted.net"=>47,
 "wdc-aixdevdb-150.redacted.net"=>47,
 "wdc-secapptst02.redacted.net"=>47,
 "wdc-aixrp2db-53.redacted.net"=>47,
 "wdc-aixbetadb-109.redacted.net"=>47,
 "wdc-aixdb-80.redacted.net"=>46,
 "wdc-aixdb-64.redacted.net"=>46,
 "uxddbhssmhcdv01.redacted.net"=>46,
 "wdc-aixbetadb-99.redacted.net"=>46,
 "wdc-aixdb-75.redacted.net"=>46,
 "wdc-aixdb-63.redacted.net"=>43,
 "wdc-aixbetadb-30.redacted.net"=>43,
 "wdc-aixdb-13.redacted.net"=>39,
 "wdc-aixdevdb-13.redacted.net"=>35,
 "wdc-aixbetadb-04.redacted.net"=>35,
 "wdc-aixbetadb-06.redacted.net"=>34,
 "wdc-aixbetadb-26.redacted.net"=>33,
 "wdc-aixbetadb-20.ec.redacted.net"=>33,
 "wdc-aixbetadb-43.redacted.net"=>32,
 "wdc-aixdb-29.redacted.net"=>32,
 "wisley-hisupp.redacted"=>30,
 "wisley-hipsupp.redacted"=>30,
 "wdc-aixbetadb-26.m.redacted.net"=>25,

I’ve added some additional logging to the report importer, and I can see it doing exactly that right here. It even finds the host, but for some reason just creates another host object for it.

2024-11-07T09:26:10 [I|app|fd723c3e] Importing report for host: wdc-aixdevdb-87
2024-11-07T09:26:10 [I|app|fd723c3e] Initializing ReportImporter for host: wdc-aixdevdb-87 with proxy_id:
2024-11-07T09:26:10 [I|app|fd723c3e] Finding or creating host: wdc-aixdevdb-87
2024-11-07T09:26:10 [I|app|fd723c3e] Found existing host: wdc-aixdevdb-87 with ID: 75107
2024-11-07T09:26:10 [I|app|fd723c3e] Processing report for wdc-aixdevdb-87
2024-11-07T09:26:10 [I|app|fd723c3e] Processing report normally for host: wdc-aixdevdb-87
2024-11-07T09:26:10 [I|app|fd723c3e] Processing normal report for host: wdc-aixdevdb-87
2024-11-07T09:26:10 [I|app|fd723c3e] Saved ConfigReport for host: wdc-aixdevdb-87 with ID: 908590
2024-11-07T09:26:10 [I|app|fd723c3e] Saving host wdc-aixdevdb-87 after processing report
2024-11-07T09:26:10 [I|app|fd723c3e] Imported report for wdc-aixdevdb-87 in 0.05 seconds
2024-11-07T09:26:10 [I|app|fd723c3e] Successfully imported report for host: wdc-aixdevdb-87
2024-11-07T09:29:42 [I|app|0214ca8b] Importing report for host: wdc-aixdevdb-87
2024-11-07T09:29:42 [I|app|0214ca8b] Initializing ReportImporter for host: wdc-aixdevdb-87 with proxy_id:
2024-11-07T09:29:42 [I|app|0214ca8b] Finding or creating host: wdc-aixdevdb-87
2024-11-07T09:29:42 [I|app|0214ca8b] Found existing host: wdc-aixdevdb-87 with ID: 75107
2024-11-07T09:29:42 [I|app|0214ca8b] Processing report for wdc-aixdevdb-87
2024-11-07T09:29:42 [I|app|0214ca8b] Processing report normally for host: wdc-aixdevdb-87
2024-11-07T09:29:42 [I|app|0214ca8b] Processing normal report for host: wdc-aixdevdb-87
2024-11-07T09:29:42 [I|app|0214ca8b] Saved ConfigReport for host: wdc-aixdevdb-87 with ID: 908763
2024-11-07T09:29:42 [I|app|0214ca8b] Saving host wdc-aixdevdb-87 after processing report
2024-11-07T09:29:42 [I|app|0214ca8b] Imported report for wdc-aixdevdb-87 in 0.05 seconds
2024-11-07T09:29:42 [I|app|0214ca8b] Successfully imported report for host: wdc-aixdevdb-87
2024-11-07T10:50:59 [I|app|136a72eb] Importing report for host: wdc-aixdevdb-87
2024-11-07T10:50:59 [I|app|136a72eb] Initializing ReportImporter for host: wdc-aixdevdb-87 with proxy_id:
2024-11-07T10:50:59 [I|app|136a72eb] Finding or creating host: wdc-aixdevdb-87
2024-11-07T10:50:59 [I|app|136a72eb] Found existing host: wdc-aixdevdb-87 with ID: 75107
2024-11-07T10:50:59 [I|app|136a72eb] Processing report for wdc-aixdevdb-87
2024-11-07T10:50:59 [I|app|136a72eb] Processing report normally for host: wdc-aixdevdb-87
2024-11-07T10:50:59 [I|app|136a72eb] Processing normal report for host: wdc-aixdevdb-87
2024-11-07T10:50:59 [I|app|136a72eb] Saved ConfigReport for host: wdc-aixdevdb-87 with ID: 911443
2024-11-07T10:50:59 [I|app|136a72eb] Saving host wdc-aixdevdb-87 after processing report
2024-11-07T10:50:59 [I|app|136a72eb] Imported report for wdc-aixdevdb-87 in 0.05 seconds
2024-11-07T10:50:59 [I|app|136a72eb] Successfully imported report for host: wdc-aixdevdb-87
[root@10-222-206-158 foreman]# grep 0214ca8b production.log
2024-11-07T09:29:42 [I|app|0214ca8b] Starting import of reports with proxy_id:
2024-11-07T09:29:42 [I|app|0214ca8b] Importing report for host: wdc-aixdevdb-87
2024-11-07T09:29:42 [I|app|0214ca8b] Initializing ReportImporter for host: wdc-aixdevdb-87 with proxy_id:
2024-11-07T09:29:42 [I|app|0214ca8b] Finding or creating host: wdc-aixdevdb-87
2024-11-07T09:29:42 [I|app|0214ca8b] Found existing host: wdc-aixdevdb-87 with ID: 75107
2024-11-07T09:29:42 [I|app|0214ca8b] Processing report for wdc-aixdevdb-87
2024-11-07T09:29:42 [I|app|0214ca8b] Processing report normally for host: wdc-aixdevdb-87
2024-11-07T09:29:42 [I|app|0214ca8b] Processing normal report for host: wdc-aixdevdb-87
2024-11-07T09:29:42 [I|app|0214ca8b] Calculating metrics for report ID:
2024-11-07T09:29:42 [I|app|0214ca8b] Saved ConfigReport for host: wdc-aixdevdb-87 with ID: 908763
2024-11-07T09:29:42 [I|app|0214ca8b] Importing log messages for report ID: 908763
2024-11-07T09:29:42 [I|app|0214ca8b] Completed importing log messages for report ID: 908763
2024-11-07T09:29:42 [I|app|0214ca8b] Saving host wdc-aixdevdb-87 after processing report
2024-11-07T09:29:42 [I|app|0214ca8b] Imported report for wdc-aixdevdb-87 in 0.05 seconds
2024-11-07T09:29:42 [I|app|0214ca8b] Successfully imported report for host: wdc-aixdevdb-87

Whats interesting is when it finds the host, its the same host id each time. So it seems like it should know where to put the report, but instead it just seems to create an entirely new host record.

Surely someone has to have some idea where this issue might be occurring?

31

Im starting to think this is a SQL DB issue. I need some help with this as the backend part of foreman isnt my strong suite.

Here is what Im seeing below:

Get the duplicates

irb(main):001:0> duplicates = Host::Managed
irb(main):002:0>   .group(:name)
irb(main):003:0>   .having('COUNT(*) > 1')
irb(main):004:0>   .order(Arel.sql('COUNT(*) DESC'))
irb(main):005:0>   .count
=>
{"10-222-52-104.redacted1.redacted2.global"=>2,
...
irb(main):006:0>
irb(main):007:1* duplicates.each do |name, count|
irb(main):008:1*   puts "#{name} appears #{count} times"
irb(main):009:0> end
10-222-52-104.redacted1.redacted2.global appears 2 times

There are hundreds more, but lets take this first one.

Foreman Rake Inspect It

host2 = Host::Managed.find_by('LOWER(name) = ?', "10-222-52-104.redacted1.redacted2.global")
=>
#<Host::Managed:0x000079d8761c3860

### No other existing record:
irb(main):002:0> host = Host::Managed.find_by('LOWER(name) = ?', "10-222-52-104")
=> nil

Ok so we have what foreman thinks is one instance of this host.

However, SQL see’s:

foreman=# SELECT id, name, last_compile, updated_at, created_at FROM hosts WHERE name = '10-222-52-104.redacted1.redacted2.global';
  id   |                   name                   |        last_compile        |         updated_at         |         created_at
-------+------------------------------------------+----------------------------+----------------------------+----------------------------
 65492 | 10-222-52-104.redacted1.redacted2.global | 2024-10-29 18:00:11.174354 | 2024-10-29 18:00:11.925573 | 2024-10-29 00:36:05.240964
 73105 | 10-222-52-104.redacted1.redacted2.global | 2024-11-07 19:00:11.566837 | 2024-11-07 19:00:12.353696 | 2024-11-07 00:17:45.039598
(2 rows)

2 records!? For the same host…
And thus, we see duplicate hosts in the UI.

Can anyone possible shed some light on what is happening here?

32

Hi Guys, I installed salt and foreman in one server and installed salt-plugins . I’m getting connectivity issue please can some help to resolve this.

I Installed foreman 3.12 and salt 3006 LTS versions.

Do I need to install any other plugins

33

I think you should start a new thread for your problem as its not related to this issue. Thank you.

34

Latest. Im just not sure how or why Foreman is creating records this many times. Look at the time stamps. Anyone that understand the SQL side of foreman around? Im really trying to be patient and provide as much data as I can until someone from Foreman can weigh in.

foreman=# SELECT id, name, created_at FROM hosts WHERE name LIKE '%10-222-50-4.cloudad.ssncad.global%';
  id   |               name                |         created_at
-------+-----------------------------------+----------------------------
 96011 | 10-222-50-4.cloudad.ssncad.global | 2024-11-21 19:00:20.984907
 96009 | 10-222-50-4.cloudad.ssncad.global | 2024-11-21 19:00:20.950697
 96008 | 10-222-50-4.cloudad.ssncad.global | 2024-11-21 19:00:20.911751
 96016 | 10-222-50-4.cloudad.ssncad.global | 2024-11-21 19:00:21.278237
 96010 | 10-222-50-4.cloudad.ssncad.global | 2024-11-21 19:00:20.967898
 96012 | 10-222-50-4.cloudad.ssncad.global | 2024-11-21 19:00:20.987898
 96014 | 10-222-50-4.cloudad.ssncad.global | 2024-11-21 19:00:21.236892
 96017 | 10-222-50-4.cloudad.ssncad.global | 2024-11-21 19:00:21.352687
 96013 | 10-222-50-4.cloudad.ssncad.global | 2024-11-21 19:00:21.225736
 96015 | 10-222-50-4.cloudad.ssncad.global | 2024-11-21 19:00:21.240235