RFC: Reuse persistent smart-proxy -> Satellite RHSM connections on Capsules

Development RFCs
1

RFC: Reuse persistent smart-proxy → Foreman RHSM connections on smart proxies

Decision Due Date: TBD

Context and Problem Statement

What is the problem being solved?

On the smart-proxy path, hosts typically talk to Apache on the smart-proxy host, which forwards RHSM-related traffic through smart-proxy and onward to Foreman.

The gap today is on the smart-proxy -> Foreman leg: forwarded RHSM requests can pay repeated TCP/TLS setup cost upstream instead of reusing an already-established connection.

This proposal is specifically about that upstream hop. It is not about client keep-alive between the host and Apache on the smart-proxy host, and it is not about the local Apache → smart-proxy hop.

The concern is that this RHSM forwarding pattern generates enough upstream requests that repeated connection setup becomes avoidable overhead.

How does this affect the user or developer?

For users, the expected effect is improved efficiency on smart-proxy-mediated RHSM traffic, especially under concurrency, by reducing repeated TCP/TLS handshake overhead between smart proxies and Foreman.

Expected improvements include:

  • reduced per-request latency for forwarded RHSM traffic
  • lower TCP/TLS handshake overhead
  • lower connection churn between smart proxies and Foreman
  • better scalability under concurrency

This may become more important in future PQC-oriented TLS deployments, where connection establishment could become more expensive and connection reuse would amortize that cost more effectively.

For developers, this introduces a transport refactor in smart-proxy and raises packaging/installability considerations because the proposed implementation targets Faraday 2.

Background information that helps understand the current state

Today, smart-proxy uses a per-request outbound client path for this traffic. That means forwarded RHSM requests can repeatedly create new upstream connections instead of reusing an existing one.

There is already an implementation draft exploring this direction:

Technical objectives

  • reuse persistent upstream connections on the smart-proxy -> Foreman leg for forwarded /rhsm/* traffic
  • keep the higher-level request-building API stable for existing callers
  • preserve current SSL client certificate / CA handling
  • preserve historical timeout defaults unless an explicit proxy setting is provided
  • make the change reviewable in a way that separates transport migration from pooling behavior

Non-goals

This proposal is not intended to:

  • redesign the registration flow
  • change host-facing registration gateway behavior
  • change Apache-level keep-alive behavior
  • introduce broader application-managed crypto-policy logic
  • optimize unrelated traffic paths in smart-proxy

Proposal

The proposal is to refactor the outbound Foreman request path in smart-proxy to:

  • move the request transport onto Faraday
  • use faraday-net_http_persistent for persistent upstream connections
  • preserve existing SSL material handling
  • preserve historical timeout defaults when no explicit proxy setting is configured

The implementation is split into two reviewable steps:

  1. Introduce Faraday for outbound Foreman requests
  2. Add shared upstream connection reuse on top of that

The intent of that split is to make the transport migration understandable separately from the pooling behavior.

Technical and project-level impacts

  • introduces faraday and faraday-net_http_persistent as dependencies
  • targets Faraday 2
  • has packaging implications across the Foreman/Katello ecosystem
  • the ecosystem has historically been aligned around Faraday 1.x, and plugins such as foreman_google and foreman_azure_rm are part of the reason Faraday compatibility has needed explicit coordination
  • therefore, this proposal would require the Faraday 2 packaging/version-clash problem to be addressed before the change can be considered installable across supported environments
  • keeps timeout behavior aligned with historical Net::HTTP defaults unless an explicit proxy setting overrides them
  • should continue relying on system/OpenSSL defaults rather than introducing application-managed crypto policy behavior

Alternative Designs

1. Keep the current per-request outbound client path

This keeps the implementation simpler, but leaves the repeated upstream connection setup cost in place.

2. Add pooling directly on top of the existing Net::HTTP path

This is possible in theory, but Faraday provides a cleaner seam for transport behavior and adapter choice while keeping the higher-level request interface stable.

3. Solve this elsewhere in the stack

Client-side keep-alive and Apache behavior help on other hops, but they do not address the specific smart-proxy -> Foreman upstream connection reuse problem.

Decision Outcome

TBD

The intended decision to discuss in this RFC is whether smart-proxy should adopt a Faraday-based outbound transport for this path and use persistent upstream connection reuse for forwarded RHSM traffic on smart proxies.

Impacts

If accepted, this decision would have the following impacts:

  • smart-proxy would gain a clearer outbound transport abstraction for this path
  • smart-proxy-mediated RHSM forwarding would be able to reuse upstream connections instead of repeatedly paying connection setup cost
  • installability would depend not only on the smart-proxy code change but also on resolving the Faraday 2 packaging/version-alignment issue in supported environments
  • the change would provide a cleaner base for future discussion of transport behavior in more expensive TLS scenarios, including PQC-oriented ones
  • the proposal does not change the external registration flow semantics, but it does change the internal transport implementation used to reach Foreman
2

Please use upstream terminology: Satellite and Capsules do not exist within upstream.

I think this also helps to be split into smaller chunks.

  • Update from Faraday 1.x to 2.x
  • Align Foreman on Faraday, replacing net-http, rest-client (which depends on net-http) and excon
  • Introduce Faraday in smart-proxy as the preferred HTTP client, replacing net-http

A consideration I haven’t seen is HTTP proxies. Users can configure their HTTP proxy and if that changes, today is takes immediate effect because there are no persistent connections. After this you probably need to force a reconnection everywhere. How is that implemented?

I see GitHub - drbrain/net-http-persistent: Thread-safe persistent connections with Net::HTTP · GitHub exists. If we implement this today, don’t we get a lot for free already?