Unique Hosts in Foreman (salt)

Thanks for your detailed description and and the code diff! I’ll look into this and check why the validation was not enabled in the first place.

When upgrading to Salt 3006, you might wanna keep an eye on the permissions of the salt-master service. We ran into some issues that Salt changed it from root to a new salt user and I’m currently working on integrating this into Foreman (see here for a first hint).

I’m currently evaluating adding a new foreman-salt user instead. However, such user would also need access to the salt states to import them via the API. I’ll soon create an RFC where your input might be helpful! I’ll link it here.

Thanks for the warning. We are doing a lot of testing with 3006, first. And foreman/plugin testing will be part of that. We are already forcing all of our salt-master services to run as root, because PAM doesnt work with non root users. Even salt says that running as root doesnt have that big of a security impact, because if someone has access to salt, under any user, they have access to all the minions. But I think enough people have told them they dont want it running as root.

Thanks again!

1 Like

Oh!! And I meant validate from false to true, but I think you got that.

1 Like

Well. Something still isnt right with this. Out of some 60,000 hosts, Im seeing the below error on about 20,000 of them. Im not really sure whats going on. The error and stacktrace isnt seemingly very helpful.

Foreman::Exception

ERF42-6331 [Foreman::Exception]: Failed to create host


---
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:87:in
  `find_or_create_host'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:32:in
  `initialize'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:17:in
  `new'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:17:in
  `block in import'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:15:in
  `each'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:15:in
  `map'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/services/foreman_salt/report_importer.rb:15:in
  `import'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/lib/actions/foreman_salt/report_import.rb:14:in
  `block in run'"
- "/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'"
- "/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'"
- "/usr/share/gems/gems/foreman_salt-16.0.2/app/lib/actions/foreman_salt/report_import.rb:13:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:589:in `block (3 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
  `block in run'"
- "/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/execution_wrapper.rb:91:in
  `wrap'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action/progress.rb:17:in `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in
  `restore_current_request_id'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in
  `restore_curent_timezone'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
  `restore_current_taxonomies'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:32:in `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:54:in
  `restore_curent_user'"
- "/usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/world.rb:31:in `execute'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:588:in `block (2 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:587:in `catch'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:587:in `block in execute_run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:490:in `block in with_error_handling'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:490:in `catch'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:490:in `with_error_handling'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:582:in `execute_run'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:303:in `execute'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
  `block (2 levels) in execute'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract.rb:167:in
  `with_meta_calculation'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
  `block in execute'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
  `open_action'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
  `execute'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/director.rb:69:in `execute'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in
  `block (2 levels) in perform'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors.rb:18:in `run_user_code'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
  `block in perform'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in
  `with_telemetry'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in
  `perform'"
- "/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors/sidekiq/serialization.rb:27:in
  `perform'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in `execute_job'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in `block (2 levels)
  in process'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:172:in `invoke'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in `block in process'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in `block (6 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in `local'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in `block (5 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in `block in <module:Sidekiq>'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog'"
- "/usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread'"
- "/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block
  in create_with_logging_context'"

Production logs show a bit more. I wonder if it has something to do with upper/lowercase somewhere along the stack from salt to foreman:

2024-10-23T12:34:05 [D|app|69dcf8f3] Initializing ReportImporter with host: ASPALPSTNRXA.xxx.com, proxy_id:
2024-10-23T12:34:05 [D|app|69dcf8f3] Finding or creating host: ASPALPSTNRXA.xxx.com
2024-10-23T12:34:05 [I|app|69dcf8f3] Host not found; creating new host with name: ASPALPSTNRXA.xxx.com
2024-10-23T12:34:05 [W|app|69dcf8f3] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:34:05 [W|app|69dcf8f3] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:34:05 [W|app|69dcf8f3] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:34:05 [E|app|69dcf8f3] Failed to create host ASPALPSTNRXA.xxx.com: Name has already been taken
2024-10-23T12:34:05 [E|app|69dcf8f3] Error in find_or_create_host: ERF42-6331 [Foreman::Exception]: Failed to create host
2024-10-23T12:34:05 [E|app|69dcf8f3] Import failed: ERF42-6331 [Foreman::Exception]: Failed to create host
2024-10-23T12:34:05 [D|tax|69dcf8f3] Current organization set to none
2024-10-23T12:34:05 [D|tax|69dcf8f3] Current location set to none
2024-10-23T12:34:05 [E|bac|69dcf8f3] ERF42-6331 [Foreman::Exception]: Failed to create host (Foreman::Exception)

Well, not ALL failures are uppercase hostnames:

--
2024-10-23T12:31:10 [W|app|445e6a7d] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:31:10 [W|app|445e6a7d] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:31:10 [W|app|445e6a7d] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:31:10 [E|app|445e6a7d] Failed to create host 10-136-57-119.redacted: Name has already been taken
2024-10-23T12:31:10 [E|app|445e6a7d] Error in find_or_create_host: ERF42-6331 [Foreman::Exception]: Failed to create host
2024-10-23T12:31:10 [E|app|445e6a7d] Import failed: ERF42-6331 [Foreman::Exception]: Failed to create host
2024-10-23T12:31:10 [D|tax|445e6a7d] Current organization set to none
2024-10-23T12:31:10 [D|tax|445e6a7d] Current location set to none
2024-10-23T12:31:10 [E|bac|445e6a7d] ERF42-6331 [Foreman::Exception]: Failed to create host (Foreman::Exception)

This seems like an issue:

irb(main):001:0> Host::Base.where('name LIKE ?', '%10-136-57-119%')
=>
[#<Host::Base:0x000072045ef03500
  id: 28562,
  name: "10-136-57-119.redacted",
  last_compile: nil,
  last_report: "[FILTERED]",
  updated_at: Thu, 17 Oct 2024 17:12:49.174780000 UTC +00:00,
  created_at: Thu, 17 Oct 2024 17:12:48.215720000 UTC +00:00,
  root_pass: nil,
  architecture_id: nil,
  operatingsystem_id: nil,
  ptable_id: nil,
  medium_id: nil,
  build: false,
  comment: nil,
  disk: nil,
  installed_at: nil,
  model_id: nil,
  hostgroup_id: nil,
  owner_id: 22,
  owner_type: "User",
  enabled: true,
  puppet_ca_proxy_id: nil,
  managed: false,
  use_image: nil,
  image_file: nil,
  uuid: nil,
  compute_resource_id: nil,
  puppet_proxy_id: nil,
  certname: nil,
  image_id: nil,
  organization_id: nil,
  location_id: nil,
  type: nil,
  otp: nil,
  realm_id: nil,
  compute_profile_id: nil,
  provision_method: nil,
  salt_proxy_id: nil,
  grub_pass: nil,
  salt_environment_id: nil,
  global_status: 0,
  lookup_value_matcher: "[FILTERED]",
  pxe_loader: nil,
  initiated_at: nil,
  build_errors: nil,
  salt_autosign_key: nil,
  salt_status: nil,
  creator_id: 1>]

Host does exist but it is not managed. Its like it got soft deleted at some point and cannot be recreated.

Im not sure if this is a lowercase issue, or just another managed: False issue.

2024-10-23T12:30:00 [W|app|e23da0dc] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:30:00 [W|app|e23da0dc] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:30:00 [W|app|e23da0dc] Not queueing Host::Managed: ["Name has already been taken"]
2024-10-23T12:30:00 [E|app|e23da0dc] Failed to create host TVAWBD0FPP02: Name has already been taken
2024-10-23T12:30:00 [E|app|e23da0dc] Error in find_or_create_host: ERF42-6331 [Foreman::Exception]: Failed to create host
2024-10-23T12:30:00 [E|app|e23da0dc] Import failed: ERF42-6331 [Foreman::Exception]: Failed to create host
2024-10-23T12:30:00 [D|tax|e23da0dc] Current organization set to none
2024-10-23T12:30:00 [D|tax|e23da0dc] Current location set to none
2024-10-23T12:30:00 [E|bac|e23da0dc] ERF42-6331 [Foreman::Exception]: Failed to create host (Foreman::Exception)
[#<Host::Managed:0x000073fb6cf7b8a0
  id: 33186,
  name: "tvawbd0fpp02",
  last_compile: Wed, 23 Oct 2024 12:28:25.818802000 UTC +00:00,
  last_report: "[FILTERED]",
  updated_at: Wed, 23 Oct 2024 12:28:26.821676000 UTC +00:00,
  created_at: Thu, 17 Oct 2024 18:40:38.225524000 UTC +00:00,
  root_pass: nil,
  architecture_id: nil,
  operatingsystem_id: 7,
  ptable_id: nil,
  medium_id: nil,
  build: false,
  comment: nil,
  disk: nil,
  installed_at: nil,
  model_id: 1,
  hostgroup_id: nil,
  owner_id: 22,
  owner_type: "User",
  enabled: true,
  puppet_ca_proxy_id: nil,
  managed: false,
  use_image: nil,
  image_file: nil,
  uuid: nil,
  compute_resource_id: nil,
  puppet_proxy_id: nil,
  certname: nil,
  image_id: nil,
  organization_id: 1,
  location_id: 10,
  type: "Host::Managed",
  otp: nil,
  realm_id: nil,
  compute_profile_id: nil,
  provision_method: nil,
  salt_proxy_id: nil,
  grub_pass: nil,
  salt_environment_id: nil,
  global_status: 2,
  lookup_value_matcher: "[FILTERED]",
  pxe_loader: nil,
  initiated_at: nil,
  build_errors: nil,
  salt_autosign_key: nil,
  salt_status: nil,
  creator_id: 1>]

Umm WOAH what in the world is going on here?

irb(main):007:0> Host::Base.where(managed: false).count
=> 26360

Oh, disregard, its not an issue with them being managed/unmanaged. I have no idea what is going on here with these host creation failures:

irb(main):007:0> Host::Base.where(managed: false).count
=> 26360
irb(main):008:0> Host::Base.where(managed: true).count
=> 0

So @bastian-src it does appear that even with validate: true, I still get duplicate hosts.

I think Ill have to turn off the functionality that auto creates hosts on facts/reports upload, and use the api to create the host first, unfortunately. I just cannot figure this out.

irb(main):002:0> duplicates = Host::Base.group(:name).having('COUNT(*) > 1').count
=>
{"wdc-dssrv01devb.xxxxx"=>13,
...
irb(main):003:0> duplicates.size
=> 193

After some 30-40 hours of combined work on this, I cannot figure it out. Our plan is to completely disable the options to create hosts from reports/facts and somehow write automation that uses the foreman api to create a host, with multiple guard rails to first check that the host doesnt exist.