Provisioning servers in Multiple REALMs

I’m looking for a better/best practice for provisioning servers in multiple REALMs.
We currently provision servers in two different REALMs - say ABC.COM and XYZ.COM.

The way I set this up a few years ago was to have

   /etc/foreman-proxy/settings.d/realm_freeipa.yml.ABC and 

each with their own principal:

   :principal: realm-proxy@ABC.COM or
   :principal: realm-proxy@XYZ.COM

I also have a completely separate small proxy server whose only purpose in life is to handle the REALM duties for XYZ, configured as the “Realm proxy” in Foreman.
ABC realm is, of course, handled by the main Foreman server.

When I need to provision a server in XYZ, I simply

cp /etc/foreman-proxy/settings.d/realm_freeipa.yml.XYZ /etc/foreman-proxy/settings.d/realm_freeipa.yml

and restart foreman services.
It works just fine, except when you forget to make the change and get “Realm Error: ERF12-5287” during provisioning. :face_with_raised_eyebrow:

I was never able to find any method within Foreman/Katello to handle that automatically, though I can have both REALMs configured in the UI.
I’m sure there has to be a better way.

We are on Foreman 3.3 and Katello 4.5 on CentOS7, and obviously have to upgrade. Since we also have to get off CentOS7, we are going to completely rebuild all servers and proxies…a somewhat daunting task not recommended for the faint of heart.
In doing so, we are using lessons learned and looking for best practices for some things we may have just done in the past and let slide.

Anyone know of a more correct way of doing this?

From what I can tell, your explanation seems to be correct in how it should be setup. I had a similar setup previously (though I didn’t have two freeipa realms setup, but instead one freeipa realm and one ad domain).
In my situation, I had the freeipa realm setup on the primary server and the ad domain setup on the proxy. I don’t have this setup now, but I believe the only thing I had to do was change the realm on the Host tab when creating a new host.

Are you certain the realm capsule is set properly for each of the two realms you have configured in Foreman? Realm XYZ should be set to use the proxy server.

Yes, when we create a new host, we choose which of the two Realm options we have in the “Realm” dropdown.
In Infrastructure->Realms, we have XYZ Realm setup with the proper Proxy assigned, and ABC Realm setup with the main server.

The problem seems to lie in the fact that “principal” in /etc/foreman-proxy/settings.d/realm_freeipa.yml is the ultimate setting when the build begins.
For an ABC server, if I have :principal: realm-proxy@ABC.COM in that file, all works fine.
And vice-versa with an XYZ server.

For example, if I have the file setup for XYZ and I try to build a server in ABC, it immediately fails with:

2023-04-18T17:50:37 [I|app|49914a6f] Add realm entry for new host
2023-04-18T17:50:37 [W|app|49914a6f] Failed to create's realm entry: ERF12-5287 [ProxyAPI::ProxyException]: Unable to create realm entry ([RestClient::BadRequest]: 400 Bad Request) for proxy
2023-04-18T17:50:37 [I|app|49914a6f] Backtrace for 'Failed to create's realm entry: ERF12-5287 [ProxyAPI::ProxyException]: Unable to create realm entry ([RestClient::BadRequest]: 400 Bad Request) for proxy' error (ProxyAPI::ProxyException): ERF12-5287 [ProxyAPI::ProxyException]: Unable to create realm entry ([RestClient::BadRequest]: 400 Bad Request) for proxy
 49914a6f | /usr/share/foreman/app/services/proxy_api/realm.rb:14:in `rescue in create'
 49914a6f | /usr/share/foreman/app/services/proxy_api/realm.rb:11:in `create'
 49914a6f | /usr/share/foreman/app/models/concerns/orchestration/realm.rb:34:in `set_realm'
 49914a6f | /usr/share/foreman/app/models/concerns/orchestration.rb:227:in `execute'
 49914a6f | /usr/share/foreman/app/models/concerns/orchestration.rb:152:in `block in process'
 49914a6f | /usr/share/foreman/app/models/concerns/orchestration.rb:144:in `each'
 49914a6f | /usr/share/foreman/app/models/concerns/orchestration.rb:144:in `process'
 49914a6f | /usr/share/foreman/app/models/concerns/orchestration.rb:44:in `around_save_orchestration'
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport- `block in run_callbacks'
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport- `run_callbacks'
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport- `_run_save_callbacks'
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord- `create_or_update'
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.6.2/lib/puma/server.rb:441:in `process_client'
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.6.2/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 49914a6f | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-04-18T17:50:37 [W|app|49914a6f] Rolling back due to a problem: [#<Orchestration::Task:0x0000000017f6a880 @name="Create realm entry for", @id="Create realm entry for", @status="failed", @priority=1, @action=[#<Host::Managed id: nil, name: "", last_compile: nil, last_report: nil, updated_at: nil, created_at: nil, root_pass: nil, architecture_id: 1, operatingsystem_id: 2, ptable_id: 159, medium_id: 10, build: true, comment: "", disk: "", installed_at: nil, model_id: nil, hostgroup_id: 18, owner_id: 4, owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: 2, puppet_proxy_id: 1, certname: nil, image_id: nil, organization_id: 3, location_id: 16, type: "Host::Managed", otp: nil, realm_id: 1, compute_profile_id: 9, provision_method: "build", grub_pass: nil, global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: "PXELinux BIOS", openscap_proxy_id: nil, initiated_at: nil, build_errors: nil, discovery_rule_id: nil>, :set_realm], @created=1681854636.9786983, @timestamp=2023-04-18 21:50:37.162110838 UTC>]
2023-04-18T17:50:37 [I|app|49914a6f] Processed 1 tasks from queue 'Host::Managed Main', completed 0/10
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Create realm entry for' *failed*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Set up compute instance' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Query instance details for' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Create DHCP Settings for' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Deploy TFTP PXELinux config for' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Deploy TFTP PXEGrub2 config for' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Deploy TFTP PXEGrub config for' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Deploy TFTP iPXE config for' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Fetch TFTP boot files for' *canceled*
2023-04-18T17:50:37 [E|app|49914a6f] Task 'Power up compute instance' *canceled*

As soon as I change realm_freeipa.yml to use ABC and restart foreman, it works fine.
It all works essentially as it should, other than having to change that file for any servers built in a different Realm.