Katello does not honor host.port for Pulp calls

[singularity001]

This is a AI summary of 6 days of debugging an issue with Katello.
To be fair, we have a very unique Katello setup, although its I dont think it would be unheard of for others out there to set up something similar to us.

Here is our environment. We created our own High Availability and expandable Infrastructure. Yes, this is far off what Satellite recommends, I assume because of the complexities, but its all working.

Environment and context

• Foreman with Katello 4.18.1

• Pulp RPM client gem: pulp_rpm_client-3.29.5

• Pulpcore feature enabled on smart proxies

• Pulp API is listening on port 8443 on the capsule

  • Example Pulp URL: https://10-221-143-136.redacted:8443

• Smart Proxy URLs (the Foreman side) use port 9090

  • Example: https://10-221-143-136.redacted:9090

Goal: create an external AlmaLinux RPM repository through the Katello UI, which should create a Pulp RPM remote and then sync.

Observed: doing it in the Rails console works, doing it through the UI fails with an SSL error.

---

Initial symptom

From the Katello UI, when creating a new AlmaLinux repository, the “create remote” step failed with:

Faraday::SSLError
SSL_connect returned=1 errno=0 state=error: record layer failure

The same operation done by hand in foreman-rake console against the Pulp RPM client worked and successfully created a remote.

This immediately told us:

• Pulp itself responds fine.

• TLS and CA look OK when we drive the client manually.

• The problem is specific to how Katello constructs and uses the Pulp client from the UI path.

---

Step 1. Exploring the Katello Pulp3 API classes

We first tried to follow some older advice that used a Base API class:

rpm_client = ::Katello::Pulp3::Api::Base.new(:rpm).client

This failed with:

NameError: uninitialized constant Katello::Pulp3::Api::Base

We then introspected Katello::Pulp3:

Katello::Pulp3.constants.sort
# => [:AlternateContentSource, :AnsibleCollection, :Api, :Content, ... :Rpm, :ServiceCommon, :Task, :TaskGroup]

Katello::Pulp3::Api.constants.sort
# => [:AnsibleCollection, :Apt, :ContentGuard, :Core, :Docker, :File, :Generic, :Yum]

So there is no Base, but there is Core and Yum.

We inspected methods:

y = Katello::Pulp3::Api::Yum
c = Katello::Pulp3::Api::Core

y.singleton_methods(false)
# => [:add_remove_content_class, :remote_uln_class, :copy_class, :rpm_package_group_class, :alternate_content_source_class]

y.instance_methods(false)
# => [:content_package_environments_api, :alternate_content_source_api, :content_modulemd_defaults_api, ... :content_package_groups_api]

c.instance_methods(false)
# => [:distributions_list_all, :remotes_list, :remotes_list_all, :repository_type=,
#     :repair_class, :repository_versions_api, :get_remotes_api, ...]

We also checked the constructors:

Katello::Pulp3::Api::Core.instance_method(:initialize).parameters
# => [[:req, :smart_proxy], [:opt, :repository_type]]

Katello::Pulp3::Api::Yum.instance_method(:initialize).parameters
# => [[:req, :smart_proxy], [:opt, :repository_type]]

So both Core and Yum expect a SmartProxy (or a Core instance) and can build the Pulp clients.

---

Step 2. Proving that the Pulp RPM client works from the console

We wired things manually in foreman-rake console:

sp   = SmartProxy.find_by(name: 'katello-na-02')
core = Katello::Pulp3::Api::Core.new(sp)
yum  = Katello::Pulp3::Api::Yum.new(sp)

Then we used yum.get_remotes_api correctly by passing a URL:

rpm_remotes = yum.get_remotes_api(
  url: 'https://repo.almalinux.org/almalinux/8/AppStream/x86_64/kickstart/'
)
rpm_client  = rpm_remotes.api_client
cfg         = rpm_client.config

cfg.base_url    # => "https://10-221-143-136.redacted"
cfg.scheme      # => "https"
cfg.host        # => "10-221-143-136.redacted"
cfg.base_path   # => ""
cfg.ssl_ca_file # was nil at first until we changed defaults
cfg.ssl_verify  # => true

We then built a remote body:

body = {
  name:                 "x86_64_kickstart-DEBUG",
  url:                  "https://repo.almalinux.org/almalinux/8/AppStream/x86_64/kickstart/",
  ca_cert:              nil,
  client_cert:          nil,
  client_key:           nil,
  tls_validation:       true,
  proxy_url:            nil,
  proxy_username:       nil,
  proxy_password:       nil,
  username:             nil,
  password:             nil,
  policy:               "immediate",
  total_timeout:        3600,
  connect_timeout:      60,
  sock_connect_timeout: 60,
  sock_read_timeout:    3600,
  rate_limit:           0
}

Our early attempts failed with SSL errors until we fixed the Pulp client config:

• We set the correct Pulp port and base path:

  cfg.host      = '10-221-143-136.redacted:8443'
  cfg.base_path = ''             # let the client add /pulp/api/v3
  
  

• Enabled debugging and attached Rails logger:

  cfg.debugging = true
  cfg.logger    = Rails.logger
  
  

Once the host included :8443, the call succeeded:

remote = rpm_remotes.create(body)
remote.pulp_href
# => "/pulp/api/v3/remotes/rpm/rpm/019afef5-.../"
remote.url
# => "https://repo.almalinux.org/almalinux/8/AppStream/x86_64/kickstart/"

We also verified via:

rpm_remotes.list.results.each do |r|
  puts "#{r.name}  ->  #{r.pulp_href}"
end

# x86_64_kickstart-DEBUG-1765219237  ->  /pulp/api/v3/remotes/rpm/rpm/019aff44-.../
# x86_64_kickstart-DEBUG             ->  /pulp/api/v3/remotes/rpm/rpm/019afef5-.../

So:

• The Pulp RPM client and Pulp API work fine when we explicitly target host:8443.

• The SSL issue is not a generic TLS trust problem but a wrong endpoint issue.

---

Step 3. Trying to fix things via pulp_rpm_client Configuration defaults

We inspected the OpenAPI generated configuration class:

# /usr/share/gems/gems/pulp_rpm_client-3.29.5/lib/pulp_rpm_client/configuration.rb
module PulpRpmClient
  class Configuration
    attr_accessor :scheme, :host, :base_path, :ssl_verify, :ssl_ca_file, ...
    def initialize
      @scheme = 'https'
      @host   = '10-221-143-136.redacted:8443'
      @base_path = ''
      @ssl_verify    = true
      @ssl_ca_file   = '/etc/pki/ca-trust/source/anchors/foreman-ha.pem'
      @debugging     = true
      @logger        = defined?(Rails) ? Rails.logger : Logger.new(STDOUT)
      ...
    end

    def host=(host)
      # remove http(s):// and anything after a slash
      @host = host.sub(/https?:\/\//, '').split('/').first
    end

    def base_path=(base_path)
      @base_path = "/#{base_path}".gsub(/\/+/, '/')
      @base_path = '' if @base_path == '/'
    end
    ...
  end
end

We confirmed that the defaults were applied:

cfg_new = PulpRpmClient::Configuration.new
cfg_new.ssl_ca_file
# => "/etc/pki/ca-trust/source/anchors/foreman-ha.pem"

PulpRpmClient::Configuration.default.ssl_ca_file
# => "/etc/pki/ca-trust/source/anchors/foreman-ha.pem"

In the console, this worked nicely. But when the UI path ran, we still got Faraday::SSLError, even though Configuration.default looked correct.

This suggested that something in Katello was overriding our host during pulp3_configuration, not that the defaults were wrong.

---

Step 4. Adding heavy logging to PulpRpmClient::Configuration

To see what was actually happening during a UI call, we added logging into configuration.rb, for example:

def initialize
  @scheme   = 'https'
  @host     = '10-221-143-136.redacted:8443'
  @base_path = ''
  @ssl_verify = true
  @ssl_ca_file = '/etc/pki/ca-trust/source/anchors/foreman-ha.pem'
  @debugging = true
  @logger    = defined?(Rails) ? Rails.logger : Logger.new(STDOUT)

  if defined?(Rails)
    Rails.logger.warn(
      "[PulpRpmClient::Configuration] initialize(before yield): " \
      "object_id=#{object_id} scheme=#{@scheme} host=#{@host} base_path=#{@base_path.inspect} " \
      "ssl_verify=#{@ssl_verify} ssl_ca_file=#{@ssl_ca_file.inspect}"
    )
  end

  yield(self) if block_given?

  if defined?(Rails)
    Rails.logger.warn(
      "[PulpRpmClient::Configuration] initialize(after  yield): " \
      "object_id=#{object_id} scheme=#{@scheme} host=#{@host} base_path=#{@base_path.inspect} " \
      "ssl_verify=#{@ssl_verify} ssl_ca_file=#{@ssl_ca_file.inspect}"
    )
  end
end

def host=(host)
  cleaned = host.sub(/https?:\/\//, '').split('/').first
  Rails.logger.warn(
    "[PulpRpmClient::Configuration] host= called with #{host.inspect}, " \
    "cleaned=#{cleaned.inspect} (object_id=#{object_id})"
  ) if defined?(Rails)
  @host = cleaned
  Rails.logger.warn(
    "[PulpRpmClient::Configuration] host set to #{@host.inspect} (object_id=#{object_id})"
  ) if defined?(Rails)
end

def configure_faraday_connection(&block)
  @configure_connection_blocks << block
end

def configure_connection(conn)
  Rails.logger.warn(
    "[PulpRpmClient::Configuration] configure_connection: before applying blocks " \
    "url=#{base_url} scheme=#{scheme} host=#{host} base_path=#{base_path.inspect} " \
    "ssl_verify=#{ssl_verify} ssl_ca_file=#{ssl_ca_file.inspect} (object_id=#{object_id})"
  ) if defined?(Rails)
  Rails.logger.warn(
    "[PulpRpmClient::Configuration] configure_connection: conn.class=#{conn.class}"
  ) if defined?(Rails)
  Rails.logger.warn(
    "[PulpRpmClient::Configuration] configure_connection: conn.builder.handlers(before)=#{conn.builder.handlers.inspect}"
  ) if defined?(Rails)

  @configure_connection_blocks.each { |b| b.call(conn) }

  Rails.logger.warn(
    "[PulpRpmClient::Configuration] configure_connection: conn.builder.handlers(after)=#{conn.builder.handlers.inspect}"
  ) if defined?(Rails)
  Rails.logger.warn(
    "[PulpRpmClient::Configuration] configure_connection: finished for object_id=#{object_id}"
  ) if defined?(Rails)
end

Then we triggered a repo create in the UI and looked at /var/log/foreman/production.log:

2025-12-09T16:29:56 [W|app|1bdbb1ea] [PulpRpmClient::Configuration]
  initialize(before yield): object_id=507060 scheme=https host=10-221-143-136.redacted:8443 base_path="" ssl_verify=true ssl_ca_file="/etc/pki/ca-trust/source/anchors/foreman-ha.pem"

2025-12-09T16:29:56 [W|app|1bdbb1ea] [PulpRpmClient::Configuration]
  host= called with "10-221-151-100.redacted", cleaned="10-221-151-100.redacted" (object_id=507060)

2025-12-09T16:29:56 [W|app|1bdbb1ea] [PulpRpmClient::Configuration]
  host set to "10-221-151-100.redacted" (object_id=507060)

2025-12-09T16:29:56 [W|app|1bdbb1ea] [PulpRpmClient::Configuration]
  ssl_ca_file= called with "/etc/pki/ca-trust/source/anchors/foreman-ha.pem" (object_id=507060)

2025-12-09T16:29:56 [W|app|1bdbb1ea] [PulpRpmClient::Configuration]
  initialize(after  yield): object_id=507060 scheme=https host=10-221-151-100.redacted base_path="" ssl_verify=true ssl_ca_file="/etc/pki/ca-trust/source/anchors/foreman-ha.pem"

Key observations:

• The Configuration starts life with host=10-221-143-136.redacted:8443 from our defaults.

• Then something calls config.host = "10-221-151-100.redacted" without the port.

• After the block, host is the bare hostname. The port information is gone.

• When the Pulp client then builds the URL, it uses https://10-221-151-100.redacted which effectively targets port 443 instead of 8443.

• That explains the SSL record layer failure: the client is talking TLS to the wrong service.

So the problem is not in the Configuration class itself, but in the Katello code that calls PulpRpmClient::Configuration.new and then sets host.

---

Step 5. Following the call stack into Katello

The log also included the Ruby call stack at the moment Configuration was created:

[PulpRpmClient::Configuration] initialize caller:
  /usr/share/gems/gems/katello-4.18.1/app/models/katello/concerns/smart_proxy_extensions.rb:332:in `new'
  /usr/share/gems/gems/katello-4.18.1/app/models/katello/concerns/smart_proxy_extensions.rb:332:in `pulp3_configuration'
  /usr/share/gems/gems/katello-4.18.1/app/services/katello/pulp3/api/core.rb:33:in `api_client'
  /usr/share/gems/gems/katello-4.18.1/app/services/katello/pulp3/api/core.rb:43:in `remotes_api'
  /usr/share/gems/gems/katello-4.18.1/app/services/katello/pulp3/api/yum.rb:42:in `get_remotes_api'
  /usr/share/gems/gems/katello-4.18.1/app/services/katello/pulp3/service_common.rb:12:in `block in create_remote'
  ...

We opened /usr/share/gems/gems/katello-4.18.1/app/models/katello/concerns/smart_proxy_extensions.rb and found:

def pulp3_configuration(config_class)
  config_class.new do |config|
    uri = pulp3_uri!
    config.host = uri.host
    config.scheme = uri.scheme
    pulp3_ssl_configuration(config)
    config.debugging = ::Foreman::Logging.logger('katello/pulp_rest').debug?
    config.timeout = SETTINGS[:katello][:rest_client_timeout]
    config.logger = ::Foreman::Logging.logger('katello/pulp_rest')
    config.username = self.setting(PULP3_FEATURE, 'username')
    config.password = self.setting(PULP3_FEATURE, 'password')
  end
end

And:

def pulp3_uri!
  url = self.setting(PULP3_FEATURE, 'pulp_url')
  fail "Cannot determine pulp3 url, check smart proxy configuration" unless url
  URI.parse(url)
end

We verified that pulp3_uri! returns the correct URL with port 8443 for each proxy:

SmartProxy.
  joins(:features).
  where(features: { name: 'Pulpcore' }).
  find_each do |sp|
  uri = sp.send(:pulp3_uri!)
  puts "SmartProxy ##{sp.id} #{sp.name}"
  puts "  url:       #{sp.url.inspect}"
  puts "  pulp3_uri!: #{uri.to_s.inspect}"
  puts "    host=#{uri.host.inspect} port=#{uri.port.inspect} path=#{uri.path.inspect}"
end

Output:

SmartProxy #2 katello-na-01
  url:        "https://10-221-151-100.redacted:9090"
  pulp3_uri!: "https://10-221-151-100.redacted:8443"
    host="10-221-151-100.redacted" port=8443 path=""

SmartProxy #5 katello-na-02
  url:        "https://10-221-143-136.redacted:9090"
  pulp3_uri!: "https://10-221-143-136.redacted:8443"
    host="10-221-143-136.redacted" port=8443 path=""

SmartProxy #6 katello-proxy-kc-01
  url:        "https://10-221-150-61.redacted:9090"
  pulp3_uri!: "https://10-221-150-61.redacted"
    host="10-221-150-61.redacted" port=443 path=""

SmartProxy #7 katello-proxy-kc-02
  url:        "https://10-221-137-125.redacted:9090"
  pulp3_uri!: "https://10-221-137-125.redacted"
    host="10-221-137-125.redacted" port=443 path=""

So:

• The Pulp URL stored in the smart proxy settings (pulp_url) already includes :8443 where appropriate.

• However, pulp3_configuration uses uri.host, which discards the port.

• The PulpRpmClient::Configuration then ends up with host set to the bare hostname and no port, so the client targets port 443.

That is the core bug.

Our earlier attempts to fix this inside PulpRpmClient::Configuration#host= did not help, because the loss of port happens before that, when config.host = uri.host is called.

---

Step 6. The actual fix that solved everything

We fixed the problem by patching pulp3_configuration to preserve the port when it is non standard.

Original:

def pulp3_configuration(config_class)
  config_class.new do |config|
    uri = pulp3_uri!
    config.host = uri.host
    config.scheme = uri.scheme
    pulp3_ssl_configuration(config)
    ...
  end
end

Patched:

def pulp3_configuration(config_class)
  config_class.new do |config|
    uri = pulp3_uri!

    # Preserve port if it is non default
    host_with_port =
      if uri.port && ![80, 443].include?(uri.port)
        "#{uri.host}:#{uri.port}"
      else
        uri.host
      end

    config.host   = host_with_port
    config.scheme = uri.scheme

    # If pulp_url ever contains a path, also honor it
    if uri.path && !uri.path.empty? && uri.path != '/'
      config.base_path = uri.path
    end

    pulp3_ssl_configuration(config)
    config.debugging = ::Foreman::Logging.logger('katello/pulp_rest').debug?
    config.timeout   = SETTINGS[:katello][:rest_client_timeout]
    config.logger    = ::Foreman::Logging.logger('katello/pulp_rest')
    config.username  = self.setting(PULP3_FEATURE, 'username')
    config.password  = self.setting(PULP3_FEATURE, 'password')
  end
end

After restarting Foreman and Dynflow services, we repeated the test and saw in the log:

[PulpRpmClient::Configuration] initialize(before yield): object_id=505800 scheme=https host=10-221-143-136.redacted:8443 ...

[PulpRpmClient::Configuration] host= called with "10-221-143-136.redacted:8443", cleaned="10-221-143-136.redacted:8443"

[PulpRpmClient::Configuration] initialize(after  yield): object_id=505800 scheme=https host=10-221-143-136.redacted:8443 ...

Now the host keeps the port 8443 all the way through. Creating a repository from the UI no longer produces the Faraday SSL error and the Pulp remote is created successfully, just as it is when we drive it from the console.

---

What we tried that did not work

For completeness, here are some of the things we tried that turned out to be red herrings or only partly helpful:

1. Editing /etc/foreman/plugins/katello.yaml pulp_url
   We tried adding a pulp_url there, but that is not where Katello is reading the Pulp URL for Pulpcore. The source of truth is the Smart Proxy settings for the Pulpcore feature (setting(PULP3_FEATURE, 'pulp_url')).

2. Tweaking pulpcode.yaml on the proxy
   We changed fields like :pulp_url and :content_app_url to include variations of the host, including dummy suffixes like cloudsdf:8443, then searched logs. These URLs did not show up in the Pulp RPM client path that was failing. That file controls the proxy side and content serving, not the client library used by Katello to talk to Pulpcore.

3. Overriding PulpRpmClient::Configuration#host= to force :8443
   We tried:

   def host=(host)
     cleaned = host.sub(/https?:\/\//, '').split('/').first
     if cleaned == '10-221-143-136.redacted'
       cleaned = '10-221-143-136.redacted:8443'
     end
     @host = cleaned
   end
   
   

   This did not solve the UI path because the real problem was that Katello was feeding uri.host which already lost the port. On top of that, this approach is brittle and infrastructure specific.

4. Assuming a TLS CA problem
   We checked and set ssl_ca_file and ssl_verify in configuration.rb and verified that console calls worked, which made it very tempting to blame CA trust. The logging later made clear that the real failure was at the TCP/TLS endpoint level, not CA trust.

---

Proposed change for Katello upstream

From the perspective of the Katello codebase, the bug is that pulp3_configuration ignores the port that is already present in the pulp_url smart proxy setting.

pulp3_uri! already correctly returns a URI object that includes a port. For example:

pulp_url = 'https://capsule.example.com:8443'
uri      = URI.parse(pulp_url)

uri.host # => "capsule.example.com"
uri.port # => 8443

Current code:

config.host   = uri.host      # loses port information
config.scheme = uri.scheme

Expected behavior:

• If pulp_url contains an explicit non default port, the Pulp client should connect to that port.

• If pulp_url does not specify a port, the client should default to 443 for https and 80 for http.

• If pulp_url contains a path, the Pulp client configuration should set base_path accordingly.

Suggested patch:

def pulp3_configuration(config_class)
  config_class.new do |config|
    uri = pulp3_uri!

    host_with_port =
      if uri.port && ![80, 443].include?(uri.port)
        "#{uri.host}:#{uri.port}"
      else
        uri.host
      end

    config.host   = host_with_port
    config.scheme = uri.scheme

    if uri.path && !uri.path.empty? && uri.path != '/'
      config.base_path = uri.path
    end

    pulp3_ssl_configuration(config)
    config.debugging = ::Foreman::Logging.logger('katello/pulp_rest').debug?
    config.timeout   = SETTINGS[:katello][:rest_client_timeout]
    config.logger    = ::Foreman::Logging.logger('katello/pulp_rest')
    config.username  = self.setting(PULP3_FEATURE, 'username')
    config.password  = self.setting(PULP3_FEATURE, 'password')
  end
end

This is backward compatible and respects the administrator defined pulp_url setting, including non default ports.

---

Why this matters

Many environments terminate TLS for Pulpcore on a port that is not 443. For example:

• Reverse proxy in front of Pulp on 8443

• Custom firewall or load balancer rules that keep 443 for Foreman UI and 8443 for back end services

In such setups:

• Pulp content app and Pulp API can sit on any port, not just 8443.

• Smart Proxy pulp_url is correctly defined with :8443.

• Katello currently discards the port via uri.host, and the OpenAPI client then talks TLS to the wrong endpoint on port 443.

• This manifests as a generic Faraday::SSLError with “record layer failure”, which is misleading without deep logging.

Honoring the port from pulp_url avoids that failure, aligns UI behavior with what we can already do in the console, and makes Katello more robust for varied infrastructure designs.

[iballou]

Hi @singularity001 ,

Thanks for the detailed explanation. The proposed patch looks fair to me. It could be simplified further to always respect the URI’s port always rather than skip if it’s is 80/443.

What do you think about creating a redmine + pull request (example: Fixes #38922 - temporarily pin Pulp clients by ianballou · Pull Request #11580 · Katello/katello · GitHub) and going through the review process? If necessary, we could collaborate on your branch in case there is back and forth about unit testing, etc and you don’t have time to get back to it.

---

As an aside, I would be curious to hear how you keep Node 1 and Node 2 in sync with each other. Is one a source of truth and the two content via import/export, or perhaps it’s a network sync situation? We’re interested in collecting data about different HA setups.

[singularity001]

Hey. Yes, I can work on a PR, although it may be a few days/weeks as I am far behind my own work and need to get this environment up before end of year.

Thanks for asking. I’ve been using Foreman (without Katello) for about 8 years. Always at very large private cloud companies. Dell for 4 years with ~ 50,000 hosts using Puppet, now a new company for the last year with ~ 80,000 hosts using Salt (huge huge shout out to @Bernhard_Suttner and anyone else that worked on the integration). We used it as a starting point and have since completely customized it for our needs.

Because of my experience setting up Foreman in HA, in the past, I was asked if I could set up a new Katello infrastructure for the company. So cue my first time ever really having to deal with Katello/Pulp, etc. Ive worked with them minimally in the past, but nothing with installs, configs, services, etc.

We essentially have 2 Foreman servers behind a load balancer/vip. We generated certs from 1, copied them to 2, then create a bundled cert/certstore for 1 and 2, then copied that bundle back over to 1. Now both servers services trust each other. We rewrote some apache settings to deal with our load balancer killing SSL when it gets to the servers (as SSL is handled at the LB). This is the reason we needed to run Pulp on 8443, because any requests going out from one service on one server to the other, would get dropped once it left the node, and went back through the load balancer. So 8443 became our SSL connections internally to each server. There was also quite a bit of custom rewrites we had to do with Tomcat/Candlepin to allow them to talk to each other (including copying and sharing of key/trust stores and their credentials). Below it all is a single NFS share on Pure storage that is mounted to both servers. Only 1 server is allowed to sync from upstream repositories down to the NFS share, but now both servers know of the content. Even if both servers were allowed to pull content we have NFS4 file locking in place, so there wouldnt be corrupt data. So, when 1 of our 80+ proxies around the world (2 in each pair, and also all behind load balancers, also with shared NFS storage) sync content, they can sync from either server, depending on where the LB in front of the main servers directed the requests.

Both servers share a postgresql database(s) that are in a k8s HA environment with triple redundancy. Both servers share 2 redis backends for syncs. 1 for rails, 1 for dynflow/jobs. That keeps everything in sync.

It looks roughly like this. Parts of it might be sort of incorrect, as there are some things we did try to limit going out the VIP. Like if a request comes in to Node 2, to Katello/Pulp, it should forward its API calls within itself, and not back out to the VIP and then over to Node 1. Even though I believe if it did, it would handle them correctly.

[ekohl]

It’s interesting that PulpCoreClient, which is a potential config_class passed into pulp3_configuration(), doesn’t have a port property. Looking at the template for that file we can see it has this for the base_url:

templates/ruby/v4.3.1/configuration.mustache

So your solution does work, but it feels a bit like a hack. I wonder if the bindings themselves should be enhanced to support it.

Looking at the original for it (openapi-generator/modules/openapi-generator/src/main/resources/ruby-client/configuration.mustache at 6d7e8c69a06c882bb16432cd43a29c8c80c321da · OpenAPITools/openapi-generator · GitHub) then we do see a mention of a port. I think this is because often the OpenAPI service runs only on a single place where it won’t change. But Pulp is different because you can self host it anywhere. Specifying the port was probably just an oversight.

[iballou]

@ekohl

pulp-openapi-generator/templates/ruby/v4.3.1/configuration.mustache at 9e60631924467752836b3af4e4344f7163fef600 · pulp/pulp-openapi-generator · GitHub

So your solution does work, but it feels a bit like a hack. I wonder if the bindings themselves should be enhanced to support it.

I agree here, it would be best if PulpcoreClient could support a port rather than sticking it in the host. That’s not to say I wouldn’t accept this change, but in the future we could tidy it up if Pulp accepts the proposal. I created Pulpcore API client bindings should support configuring a port separately from the host · Issue #7164 · pulp/pulpcore · GitHub.

---

Thanks for the architectural layout, @singularity001 . I didn’t realize it was a ‘true’ load balancing set up with shared data - that takes out a lot of the complexity of keeping the servers in sync. I suppose the power of Foreman being flexible means you can create architectures like this.
Seeing this makes me wonder if we could expand the different architectures that we suggest for HA cases, but it’s a difficult question since there are just so many possibilities.

[singularity001]

Thanks for the architectural layout, @singularity001 . I didn’t realize it was a ‘true’ load balancing set up with shared data - that takes out a lot of the complexity of keeping the servers in sync. I suppose the power of Foreman being flexible means you can create architectures like this.
Seeing this makes me wonder if we could expand the different architectures that we suggest for HA cases, but it’s a difficult question since there are just so many possibilities.

The current Katello HA suggestions, by Satellite, seem like they are based on 10-15 year old technology. Where the suggestion is to simply use different VCenters and fail-over. Thats an ancient way of doing HA, especially these days. If I had the time, I would have put this all into K8s and containerized all the services, with backup/fail-overs. That is something Ill be doing for our current Foreman/Salt (non Katello) infrastructure, next year. Currently it is handling around 70,000 hosts, running a Salt highstate 1-2 times a day, with around 50-250 facts per host, as well as around 40-300 jobs a day against 500-40,000 hosts at a time. We’ve super customized it and rewritten a lot of it to do what we want. The power of open source!

It looks like this at the moment. I do have an on-going issue where the 2 orchestrators can never seem to stay active/passive, so I plan on adding another redis jobs server and then having two orchestrators.

For this new Katello instance, I think the way Ive set up this current HA architecture is pretty straight forward, albeit a real PITA when it comes to dealing with all the certs and ports. Especially when you need both servers services to trust each other. It took me about 6 full days to set it all up, and another 6-8 full days just dealing with SSL/Cert madness everywhere, and Apache proxy settings. It does provide full HA though (unless obviously the load balancer does down, or the sql database has an issue) - and also allows us to add in more servers if we ever need to, in order to handle the stress. In addition, we have all our proxies around the world behind load balancers as well, with a shared NFS share on each of them. Somewhere around 80+ proxies, handling patching for around 45,000 linux hosts and increasing daily.

[iballou]

This is really awesome, the host count there is impressive. For the smart proxies you mentioned having globally, do you use any parts from the documented load balancing method from Configuring Smart Proxies with a load balancer ? Or is it all custom?

Since you mentioned containers too, I’ll pass along RFC: Foreman Production Installation via Containers and Podman Quadlets just in case you haven’t seen it yet. The containerized installer project is called foremanctl now.

[singularity001]

Yes, we more or less use the documented process.

Thanks, no, I had not seen that RFC. Amazing. That really should be the direction Foreman goes.

Thanks for sharing.

[ekohl]

@singularity001 I agree the whole HA model that Katello is using today is outdated and I never liked it. People will have heard me rant about it in various places.

The future I’d like to to see is that it can run independently and scale separately from Foreman. This was always part of the new Pulp 3 integration design: Foreman should never read Pulp files directly from disk (unlike in Pulp 2).

So the HA model I’d see is one where you run Foreman on some HA setup. There is 0 reason you need to run a Foreman Proxy on the same host as Foreman itself and you can attach a Pulp somewhere else. We’re working on GitHub - theforeman/pulp_smart_proxy which makes Pulp behave as a Smart Proxy that you can register directly. With Fixes #38364 - Support empty rhsm_url setting on proxies (#11328) · Katello/katello@46abdb2 · GitHub it can present itself as purely content.

So you’d have a service foreman.example.com comprising of a few components:

• Application server (Puma)
• DynFlow orchestrator
• DynFlow workers
• PostgreSQL database
• Redis queue & cache

Plain Foreman doesn’t really use the filesystem, but we’ve found that there are cases where plugins do. Especially foreman_rh_cloud is a notable offender, but Katello is another (see Share /var/run/foreman between Foreman and Dynflow containers so that Manifest can be imported by evgeni · Pull Request #248 · theforeman/foremanctl · GitHub). The proper solution is to adopt Rails’ Active Storage.

Next up is a Pulp instance. Pulp is set up very well because it’s already deployed HA in various places. With pulp_smart_proxy you’d have a pulp.example.com with the following services:

• API application server (gunicorn)
• Content application server (aiohttp)
• Workers
• PostgreSQL database
• Shared storage (NFS or s3 compatible storage)

Then you have Candlepin, which I haven’t looked into that much but know it can be done. Just that our default deployment doesn’t work because Artemis runs in the same process instead of standalone. It should be noted clients never talk to Candlepin directly and it’s only Foreman itself. So you could have a candlepin.example.com that runs:

• Application server (Tomcat)
• Message bus (Artemis)
• PostgreSQL database

And you may also want a Foreman Proxy. It depends on your environment, but REX is likely something you want.

The way we’re designing the containerize deployment is that we’re building a container for each:

• foreman-oci-images for both Foreman and Foreman Proxy
• pulp-oci-images for Pulp
• candlepin-oci-images for Candlepin

And we’re combining these in foremanctl to deploy on a single host, but I’d love it if someone comes up with other deployments. For example, Kubernetes or OpenShift.