Creating a Foreman HA cluster

Foreman HA Cluster
I am looking Forword to build a Foreman HA cluster. Right now, we have single nodes workable Foreman environment like this:
Single Main Foreman / Katello Server - Ricky Linux
Single Memcache Server - Oracle Linux
Single Database (PostgreSQL) Server - Oracle Linux
Single Smart Proxy (Ansible) Server - Ricky Linux

We would like to promote this environment to HA cluster, that spread across two Data Centers.
Basically, like this:
Two node cluster in DC1 and DC2, Main Foreman / Katello Server - Ricky Linux
Two node cluster in DC1 and DC2, Memcache Server - Oracle Linux
Two node cluster in DC1 and DC2, Database (PostgreSQL) Server - Oracle Linux
Two node cluster in DC1 and DC2, Smart Proxy (Ansible) Server - Ricky Linux

Is this possible?
If possible, how can it be done?
Obviously, we would like to promote our existing environment rather than rebuilding it as we already have lot of configurations are done and it has been heavily used.
Could someone help me with some experience and knowledge?

Thanks, in advance.

I have looked into this same topic in the past and afaik, HA setups are not possible right now, at least not with Katello.
IIRC, it might be possible without Katello, but with Katello there are some components involved that do not not support HA configurations.

With Katello, you can have load-balanced smart proxies. See the documentation

Note, however, the restrictions:

The main Foreman server cannot be set up for HA or load balancing.

Might be this could be a topic for foreman professional services?
https://theforeman.org/professional-services.html

NB: I am working for ATIX.

In principle you can setup any service on an HA cluster, but you have to make the whole stack HA. Means storage needs to be duplicated/replicated/HA available. Same for each component. Network layer may be an issue, but you can work with a virtual IP moving from one node to the other. Instead of multiple 2 node physical clusters, you can setup virtual clusters with failover and rules avoiding to have the same type of worker on the same physical host. This can be handled on the virtual layer or a cluster software manages the virtual hosts.
NB: I was responsible for SUSE HA for some time.

Thank you to all three of you. I am reading your messages and discussing them with my team. I will update the outcome, problems, and any further questions in this chat. I greatly appreciate your inputs.

Hello

I know that OP wanted to create separate 2 clusters, but is it possible to create following setup
SERVER_1: LOAD BALANCER
SERVER_2: DATABASE + REDIS CACHE & DYNFLOW
SERVER_3: FOREMAN + FOREMAN UI + SMART PROXY
SERVER_4: FOREMAN + FOREMAN UI + SMART PROXY
+
later on
SERVER_5: FOREMAN_SMART_PROXY
SERVER_6: FOREMAN_SMART_PROXY

So, is it possible to have 2 foreman servers and 1 database? From the different articles available I thought it is possible, however, right now I am not so sure.
If yes, how it should be configured from the certs perspective? I guess puppet should be installed only on one host and be used to generate certs for HOST_B?