Provisioning servers in Multiple REALMs

Support
1

I’m looking for a better/best practice for provisioning servers in multiple REALMs.
We currently provision servers in two different REALMs - say ABC.COM and XYZ.COM.

The way I set this up a few years ago was to have

   /etc/foreman-proxy/settings.d/realm_freeipa.yml.ABC and 
   /etc/foreman-proxy/settings.d/realm_freeipa.yml.XYZ

each with their own principal:

   :principal: realm-proxy@ABC.COM or
   :principal: realm-proxy@XYZ.COM

I also have a completely separate small proxy server whose only purpose in life is to handle the REALM duties for XYZ, configured as the “Realm proxy” in Foreman.
ABC realm is, of course, handled by the main Foreman server.

When I need to provision a server in XYZ, I simply

cp /etc/foreman-proxy/settings.d/realm_freeipa.yml.XYZ /etc/foreman-proxy/settings.d/realm_freeipa.yml

and restart foreman services.
It works just fine, except when you forget to make the change and get “Realm Error: ERF12-5287” during provisioning. :face_with_raised_eyebrow:

I was never able to find any method within Foreman/Katello to handle that automatically, though I can have both REALMs configured in the UI.
I’m sure there has to be a better way.

We are on Foreman 3.3 and Katello 4.5 on CentOS7, and obviously have to upgrade. Since we also have to get off CentOS7, we are going to completely rebuild all servers and proxies…a somewhat daunting task not recommended for the faint of heart.
In doing so, we are using lessons learned and looking for best practices for some things we may have just done in the past and let slide.

Anyone know of a more correct way of doing this?

2

From what I can tell, your explanation seems to be correct in how it should be setup. I had a similar setup previously (though I didn’t have two freeipa realms setup, but instead one freeipa realm and one ad domain).
In my situation, I had the freeipa realm setup on the primary server and the ad domain setup on the proxy. I don’t have this setup now, but I believe the only thing I had to do was change the realm on the Host tab when creating a new host.

Are you certain the realm capsule is set properly for each of the two realms you have configured in Foreman? Realm XYZ should be set to use the proxy server.

3

Yes, when we create a new host, we choose which of the two Realm options we have in the “Realm” dropdown.
In Infrastructure->Realms, we have XYZ Realm setup with the proper Proxy assigned, and ABC Realm setup with the main server.

The problem seems to lie in the fact that “principal” in /etc/foreman-proxy/settings.d/realm_freeipa.yml is the ultimate setting when the build begins.
For an ABC server, if I have :principal: realm-proxy@ABC.COM in that file, all works fine.
And vice-versa with an XYZ server.

For example, if I have the file setup for XYZ and I try to build a server in ABC, it immediately fails with:

2023-04-18T17:50:37 [I|app|49914a6f] Add realm entry for new host test1.abc.com
2023-04-18T17:50:37 [W|app|49914a6f] Failed to create test1.abc.com's realm entry: ERF12-5287 [ProxyAPI::ProxyException]: Unable to create realm entry ([RestClient::BadRequest]: 400 Bad Request) for proxy https://foremanserver.abc.com:9090/realm/ABC.COM
2023-04-18T17:50:37 [I|app|49914a6f] Backtrace for 'Failed to create test1.abc.com's realm entry: ERF12-5287 [ProxyAPI::ProxyException]: Unable to create realm entry ([RestClient::BadRequest]: 400 Bad Request) for proxy https://foremanserver.abc.com:9090/realm/ABC.COM' error (ProxyAPI::ProxyException): ERF12-5287 [ProxyAPI::ProxyException]: Unable to create realm entry ([RestClient::BadRequest]: 400 Bad Request) for proxy https://foremanserver.abc.com:9090/realm/ABC.COM
 49914a6f | /usr/share/foreman/app/services/proxy_api/realm.rb:14:in `rescue in create'
 49914a6f | /usr/share/foreman/app/services/proxy_api/realm.rb:11:in `create'
 49914a6f | /usr/share/foreman/app/models/concerns/orchestration/realm.rb:34:in `set_realm'
 49914a6f | /usr/share/foreman/app/models/concerns/orchestration.rb:227:in `execute'
 49914a6f | /usr/share/foreman/app/models/concerns/orchestration.rb:152:in `block in process'
 49914a6f | /usr/share/foreman/app/models/concerns/orchestration.rb:144:in `each'
 49914a6f | /usr/share/foreman/app/models/concerns/orchestration.rb:144:in `process'
 49914a6f | /usr/share/foreman/app/models/concerns/orchestration.rb:44:in `around_save_orchestration'
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:139:in `run_callbacks'
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:825:in `_run_save_callbacks'
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/callbacks.rb:327:in `create_or_update'
... 
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.6.2/lib/puma/server.rb:441:in `process_client'
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.6.2/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-04-18T17:50:37 [W|app|49914a6f] Rolling back due to a problem: [#<Orchestration::Task:0x0000000017f6a880 @name="Create realm entry for test1.abc.com", @id="Create realm entry for test1.abc.com", @status="failed", @priority=1, @action=[#<Host::Managed id: nil, name: "test1.abc.com", last_compile: nil, last_report: nil, updated_at: nil, created_at: nil, root_pass: nil, architecture_id: 1, operatingsystem_id: 2, ptable_id: 159, medium_id: 10, build: true, comment: "", disk: "", installed_at: nil, model_id: nil, hostgroup_id: 18, owner_id: 4, owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: 2, puppet_proxy_id: 1, certname: nil, image_id: nil, organization_id: 3, location_id: 16, type: "Host::Managed", otp: nil, realm_id: 1, compute_profile_id: 9, provision_method: "build", grub_pass: nil, global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: "PXELinux BIOS", openscap_proxy_id: nil, initiated_at: nil, build_errors: nil, discovery_rule_id: nil>, :set_realm], @created=1681854636.9786983, @timestamp=2023-04-18 21:50:37.162110838 UTC>]
2023-04-18T17:50:37 [I|app|49914a6f] Processed 1 tasks from queue 'Host::Managed Main', completed 0/10
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Create realm entry for test1.abc.com' *failed*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Set up compute instance test1.abc.com' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Query instance details for test1.abc.com' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Create DHCP Settings for test1-mgmt.abc.com-mgmt.abc.com' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Deploy TFTP PXELinux config for test1-mgmt.abc.com-mgmt.abc.com' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Deploy TFTP PXEGrub2 config for test1-mgmt.abc.com-mgmt.abc.com' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Deploy TFTP PXEGrub config for test1-mgmt.abc.com-mgmt.abc.com' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Deploy TFTP iPXE config for test1-mgmt.abc.com-mgmt.abc.com' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Fetch TFTP boot files for test1-mgmt.abc.com-mgmt.abc.com' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Power up compute instance test1.abc.com' *canceled*

As soon as I change realm_freeipa.yml to use ABC and restart foreman, it works fine.
It all works essentially as it should, other than having to change that file for any servers built in a different Realm.