RFC: First-class Kubernetes deployment for Foreman, Katello, Candlepin and Pulp

[jakduch]

RFC: First-class Kubernetes deployment for Foreman, Katello, Candlepin and Pulp

I am actively working on a Kubernetes deployment of Foreman together with Katello, Candlepin and Pulp. The implementation is already running in a real test environment rather than existing only as a collection of manifests or an architectural proposal.

Development and testing are still ongoing. The deployment is being iteratively improved based on behaviour observed in the running environment, while automated tests verify Helm rendering, configuration schemas, security boundaries, runtime contracts, migration ordering and compatibility metadata. End-to-end coverage is being expanded as the required capabilities become available in upstream images.

The project is not yet being presented as production-ready or officially supported. Before publishing the integration repository and the individual upstream pull requests, I would like to get feedback on the overall direction and on the boundaries between the projects.

I am also preparing every generally useful application change as a focused issue and pull request in its respective upstream project rather than maintaining Kubernetes-specific application patches in the integration repository.

The main principle is:

The applications should not need a “Kubernetes mode”.

Users should not have to tell Foreman, Katello, Candlepin or Pulp whether they are running on a traditional host, under systemd, in a container or in Kubernetes.

The same upstream code and published artifacts should work correctly in all supported environments. Existing standalone behaviour should remain the default. Kubernetes should only supply configuration, process arguments, storage and service endpoints through normal application interfaces.

Request for feedback

Before publishing the integration repository, I would like to ask the Foreman team two broader questions:

1. Is a first-class Kubernetes deployment a direction the Foreman project would be interested in pursuing?

2. If the implementation reaches an acceptable level of quality, would there be interest in hosting it as a project repository under the theforeman GitHub organization?

An official project repository would make it easier to coordinate compatibility across Foreman, Katello and their dependencies, review architectural decisions and avoid creating another unofficial deployment that gradually diverges from upstream.

The main motivation is to make larger Foreman and Katello installations easier to operate. Kubernetes would allow request-serving components and background workers to be scaled independently, provide controlled rollouts and recovery, and make application dependencies and availability contracts explicit.

It would not remove the need to understand Foreman’s architecture, but it could provide a reproducible and testable deployment model for larger environments.

Goals

The proposed integration should provide:

• official upstream applications and images without downstream forks;

• independently deployable Foreman/Katello, Candlepin and Pulp components;

• horizontal scaling where the application architecture allows it;

• explicit singleton workloads where concurrency is unsafe;

• support for external PostgreSQL, Redis/Valkey, Artemis and object storage;

• separate and observable database migrations;

• graceful startup, readiness, endpoint draining and shutdown;

• non-root workloads and restricted Kubernetes permissions;

• upgrade, backup and recovery workflows;

• compatibility tracking between application and Kubernetes releases;

• support for both central in-cluster services and external Smart Proxies.

The Kubernetes integration would remain a separate project. Its responsibility would be orchestration, not modification of application code.

High-level architecture

A simplified deployment looks like this:

                         +----------------------+
Users and clients -----> | Ingress / mTLS edge |
                         +----------+-----------+
                                    |
                 +------------------+------------------+
                 |                                     |
          +------v------+                      +-------v-------+
          | Foreman web |                      | Pulp content  |
          | + Katello   |                      | and registry  |
          +------+------+                      +-------+-------+
                 |                                     |
       +---------+----------+                          |
       |         |          |                          |
+------v--+ +----v-----+ +--v-------------+     +------v------+
| Dynflow | | Katello  | | Candlepin      |     | Pulp API    |
| workers | | event    | |                |     | and workers |
+---------+ | daemon   | +----------------+     +-------------+
            +----------+

       PostgreSQL, Valkey, Artemis and object storage are
       external or independently managed services.

Katello remains a Foreman plugin and therefore runs with Foreman. Candlepin and Pulp remain separate applications with their own images, services and release lifecycles.

No platform selector

This proposal does not add settings such as:

RUNNING_IN_KUBERNETES=true

It also does not make applications query the Kubernetes API or attempt to detect the platform.

Instead, upstream applications expose generic runtime capabilities, for example:

• a configurable writable runtime directory;

• an optional readiness signal tied to the actual process lifecycle;

• a foreground process entry point suitable for any supervisor;

• an explicit migration command;

• configurable TLS trust;

• correct handling of externally supplied service URLs;

• graceful termination.

A package installed on a normal server continues using its existing defaults. A container supervisor or Kubernetes chart supplies the relevant generic options automatically.

The model is therefore:

Same application, same code paths, different orchestration.

There should be no step in Foreman, Katello, Candlepin or Pulp where the user has to select whether the application runs standalone or in Kubernetes.

Upstream changes

Every generally useful application change will be proposed to its owning project as a separate issue and pull request. Kubernetes-specific application patches will not be stored in the integration repository.

The initial set currently being prepared includes the following changes.

Foreman

• Allow the Puma runtime directory to be configured while retaining the current Rails.root/tmp default.

• Expose an optional Dynflow readiness file following the real Sidekiq lifecycle.

• Accept a verified URL-escaped client certificate supplied by a reverse proxy while retaining the existing PEM and DER formats.

• Allow a custom CA to be used when Dynflow connects to Redis/Valkey over TLS.

These are generic capabilities useful with containers, systemd hardening and external managed services. None of them requires Foreman to know that Kubernetes exists.

Katello

• Preserve an explicitly configured non-default Pulp API port when constructing Pulp clients.

• Provide a foreground Katello event-daemon runner with a configurable runtime directory and health heartbeat.

The existing standalone daemon behaviour remains available and remains the default.

Candlepin image

• Keep the existing Tomcat server command as the default.

• Provide an explicit database migration entry point.

• Make the container runtime identity suitable for a non-root orchestrated environment.

Standalone Candlepin startup should not require a different workflow.

Pulpcore packaging and Pulp image

• Correct the EL10 dependency relationship required by the object-storage Python packages.

• Package django-storages for Pulpcore without enabling it automatically.

• Include the supported S3 runtime dependencies in the Pulp image.

Filesystem storage remains the default. S3-compatible storage is selected through normal Pulp configuration, not by detecting Kubernetes.

Optional KubeVirt provider

The optional KubeVirt integration also revealed generally applicable correctness issues:

• discover and use the supported grouped KubeVirt API version;

• keep namespaced resources scoped to the selected namespace;

• validate configuration before partially creating resources;

• clean up partially created volumes safely;

• avoid deleting storage before VM deletion has succeeded.

These changes belong in the KubeVirt provider and fog adapter, not in the Kubernetes deployment project.

Kubernetes integration repository

The separate integration repository would contain:

• Helm charts and values schemas;

• workload definitions and service discovery;

• Secrets and certificate wiring;

• migration and registration Jobs;

• probes based on upstream runtime contracts;

• persistent and temporary storage definitions;

• NetworkPolicies and security contexts;

• disruption budgets and topology rules;

• autoscaling policies;

• backup and restore workflows;

• compatibility and release metadata;

• integration, upgrade and recovery tests.

It must not contain or inject:

• replacement Rails initializers;

• patched Ruby or Python libraries;

• JVM agents modifying application behaviour;

• application preload hooks;

• application source overlays;

• private forks of Foreman, Katello, Candlepin or Pulp.

The integration will only consume a new runtime capability after it is available in a published upstream release or image.

Kubernetes-specific changes in this repository will be kept as logically separated commits. Application behaviour will remain owned and tested by the appropriate upstream project.

Relationship to foremanctl

This proposal is intended to complement foremanctl, not replace it.

Foremanctl is not limited to Smart Proxy deployment. It is the containerized deployment and lifecycle tool for running the Foreman and Katello stack on conventional hosts using Podman Quadlets, systemd and Ansible. It can also deploy and configure separate Smart Proxy or Capsule systems.

The Kubernetes project would provide a different orchestration layer for the same upstream applications:

Deployment	Primary orchestration
Traditional package installation	foreman-installer and systemd
Host-based container deployment	foremanctl, Podman Quadlets and systemd
Cluster deployment	Kubernetes and Helm

All three deployment models should consume the same generic upstream runtime contracts. A foreground process entry point, configurable runtime directory, explicit migration command or external TLS configuration should not be specific to Kubernetes and may also simplify or strengthen foremanctl deployments.

Foremanctl is also a useful existing reference for compatible Foreman, Katello, Candlepin and Pulp image versions, health checks, migrations, certificate handling and upgrade ordering. Where possible, the Kubernetes project should align with those established contracts rather than inventing an incompatible parallel model.

For distributed deployments, foremanctl could continue to manage Smart Proxy nodes outside the Kubernetes cluster. The Kubernetes-hosted Foreman control plane would connect to those proxies in the same way as any other Foreman installation.

                 Kubernetes control plane
          Foreman / Katello / Candlepin / Pulp
                           |
            +--------------+--------------+
            |                             |
  foremanctl-managed proxy      foremanctl-managed proxy
  DHCP / DNS / TFTP / REX       DHCP / DNS / TFTP / REX

The initial proposal does not require adding a Kubernetes backend to foremanctl. Kubernetes orchestration can remain in its own project while both projects share upstream application contracts, compatible image sets and operational knowledge.

A deeper integration could be discussed later if the Foreman team considers a common deployment frontend useful.

Decoupling Katello from the Kubernetes node operating system

Another important benefit is reducing the direct dependency between Katello and the operating system installed on the application host.

The current Foreman with Katello documentation supports Enterprise Linux on a specific architecture. This applies to both the package-based installation and the current foremanctl container deployment, because the deployment tooling and host integration still target Enterprise Linux.

In Kubernetes, the application dependencies are delivered inside the upstream container images. Kubernetes nodes do not need Katello, Candlepin, Pulp or Foreman RPM packages installed on the host. The same application release can therefore run on clusters whose nodes use different supported Linux distributions.

This does not mean that every operating system, architecture or Kubernetes distribution would automatically be supported. The project would still need an explicit and tested matrix covering:

• Kubernetes distributions and versions;

• Linux node operating systems;

• CPU architectures for which upstream images are published;

• container runtimes;

• ingress implementations;

• storage classes and access modes;

• SELinux and other host security integrations.

The important distinction is that application compatibility would primarily depend on the published container images and Kubernetes interfaces rather than on the availability of Katello packages for the node operating system.

This could make Foreman with Katello practical in environments where the organization already operates a supported Kubernetes platform but does not use Enterprise Linux for every cluster node.

Scaling model

Not every process should scale in the same way.

Foreman and Katello

Foreman web replicas can scale horizontally after their database, cache, certificates, runtime configuration and shared Katello hand-off storage are externalized.

Katello continues to run as a Foreman plugin rather than becoming an artificial standalone service.

Dynflow

Dynflow has several distinct roles:

• exactly one orchestrator;

• horizontally scalable general workers;

• independently scalable hosts-queue workers.

Scaling all Sidekiq processes as one Deployment would violate the existing orchestration contract.

Katello event daemon

The event daemon remains a singleton process. Events remain durable outside the pod, while the process receives an explicit supervised foreground lifecycle.

Candlepin

Candlepin can only run with multiple replicas when:

• embedded messaging is disabled;

• an external Artemis broker is used;

• Quartz uses a clustered JDBC job store;

• every instance receives a unique identifier;

• schema migration ownership is explicit.

The initial deployment should therefore remain single-replica unless the complete HA contract is enabled.

Pulp

Pulp already separates its API, content and worker processes:

• API and content replicas can scale independently;

• workers share the same database, cache and content storage;

• filesystem storage requires a shared filesystem;

• object storage removes that shared-filesystem requirement;

• worker autoscaling should eventually use queue depth rather than CPU alone.

Smart Proxy and network-local services

The Kubernetes deployment would not attempt to move every infrastructure service into the cluster.

DHCP, DNS, TFTP and other network-local provisioning services would continue to be provided by Smart Proxies. These proxies could remain on dedicated machines in the relevant sites, networks or broadcast domains and register with the central Foreman instance running in Kubernetes.

This preserves one of Foreman’s important architectural strengths: the central management plane does not need direct access to every managed network.

                    Kubernetes cluster
          +----------------------------------+
          | Foreman, Katello, Candlepin, Pulp|
          | Dynflow workers and central APIs |
          +----------------+-----------------+
                           |
                 Smart Proxy connections
                           |
             +-------------+-------------+
             |                           |
      Site A Smart Proxy          Site B Smart Proxy
      DHCP / DNS / TFTP           DHCP / DNS / TFTP
      Remote Execution            Remote Execution

Some central and container-friendly Smart Proxy functions may also run in Kubernetes, but only where their persistence, networking and job-lifecycle requirements are properly supported.

The deployment must not assume that DHCP, DNS or TFTP belong inside the cluster.

I would therefore also appreciate feedback on whether this separation—central application services in Kubernetes and network-local provisioning through existing Smart Proxies—matches how the Foreman community would expect larger installations to evolve.

Compatibility and releases

The integration repository will maintain an explicit registry of required upstream contracts.

A Kubernetes release will only be declared supported when:

1. every required upstream change has been merged;

2. the change is present in an official release or image;

3. the exact image digest has passed integration testing;

4. migration, upgrade and rollback behaviour has been verified;

5. the compatible versions are recorded in release metadata.

A successful Helm render alone is not sufficient evidence that an application release is supported.

This also avoids assuming that the newest independently published Foreman, Katello, Candlepin and Pulp images are automatically compatible with each other.

Proposed contribution model

The work would be split as follows:

1. Create an issue in the owning upstream for every generic runtime or correctness change.

2. Submit a focused upstream pull request with standalone defaults preserved.

3. Link dependent changes across Foreman, Katello, Candlepin and Pulp.

4. Merge application capabilities before relying on them in a supported Kubernetes release.

5. Keep Kubernetes-only orchestration changes as separate, logically scoped commits in the integration repository.

6. Publish compatibility evidence for exact image versions and digests.

This should make every upstream pull request independently useful and reviewable rather than presenting maintainers with one large Kubernetes-specific patch set.

Non-goals

This proposal is not intended to:

• replace foreman-installer;

• replace foremanctl;

• deprecate standalone installations;

• merge Foreman, Katello, Candlepin and Pulp into one application or image;

• introduce Kubernetes-specific branches throughout application code;

• require DHCP, DNS or TFTP to run inside Kubernetes;

• claim high availability for singleton processes that cannot safely overlap;

• maintain downstream application forks.

Feedback requested

I would especially appreciate feedback on:

1. Whether a first-class Kubernetes deployment is a direction the Foreman project would like to pursue.

2. Whether the boundary between generic upstream runtime contracts and Kubernetes orchestration looks appropriate.

3. Whether the proposed capabilities belong in the identified upstream projects.

4. Whether there would be interest in eventually hosting the integration under the theforeman GitHub organization.

5. How the Kubernetes project should relate to foremanctl and whether sharing compatibility metadata or operational tests would be useful.

6. What level of integration and upgrade testing would be required before describing a release as supported.

7. Whether an initial experimental release should target Foreman with Katello first or also include a Foreman-only profile from the beginning.

8. Whether the proposed separation between the central Kubernetes deployment and external network-local Smart Proxies is appropriate.

9. Which Kubernetes distributions, node operating systems and deployment environments should be included in the initial test matrix.

The immediate next step is to publish the individual upstream issues and pull requests, followed by the integration repository and its compatibility matrix.

AI assistance disclosure:

• Assisted by Codex 5.6 Sol High.

[ekohl]

Architecturally and plan/roadmap wise this is very much what @ehelms, @evgeni, @akumari and myself had in mind when we sat down back in 2024. Then we talked about the need for highly available (HA) deployments and quickly realized that the existing deployment model would be very hard to maintain or easily break down. We believed that relying on a platform like Kubernetes (or in RH’s case, OpenShift) could be feasible. For that we need to containerize the whole platform and a new installer makes that easier. This is where we are today: we haven’t containerized all features yet, but all the foundations exist.

I think your timing is pretty good: we should start building the next level of foundations while we get feature parity with containerized installations.

Like I said before, your overall plan is what we had in mind so instead I’m going to point out implementation details.

Biggest easy win: we got rid of the Katello event deamon and that also removes the need for Artemis in Candlepin.

Then IMHO the first thing to focus on is supporting S3 compatible object storage everywhere. You already identified Pulp and I’d love for someone to finish up what I started (Support s3 storage by ekohl · Pull Request #379 · theforeman/puppet-pulpcore · GitHub / Add pulpcore-storage-s3 subpackage to pulpcore by ekohl · Pull Request #1760 · theforeman/pulpcore-packaging · GitHub) but then for a lack of time didn’t finish.

The next step is IMHO a gap in your analysis: Foreman should also be enhanced to support a similar model as Pulp. Using activestorage we can achieve this. We tried to start this a long time ago (Fixes #35330 - Add support for ActiveStorage and PostgreSQL adapter by ShimShtein · Pull Request #9339 · theforeman/foreman · GitHub), but never committed to finish it. Making sure Foreman and all its plugins consistently use this in a secure way should simplify the shared storage problem that Foreman has. Note Foreman itself actually doesn’t really store files, but it’s mostly in plugins like Katello and foreman_rh_cloud.

Those 2 are changes that already make a lot of sense in each respective upstream project. For Pulp outside of Foreman there are already known large deployments using S3-compatible object storage so I foresee little changes there. The use of activestorage may simplify some things in single node Foreman deployments (backups, simpler code in foremanctl).

On the certificate side I also don’t see many problems, but must also admit I don’t know Kubernetes well enough to say much useful.

@jakduch

I would especially appreciate feedback on:

1. Whether a first-class Kubernetes deployment is a direction the Foreman project would like to pursue.

2. Whether the boundary between generic upstream runtime contracts and Kubernetes orchestration looks appropriate.

3. Whether the proposed capabilities belong in the identified upstream projects.

4. Whether there would be interest in eventually hosting the integration under the theforeman GitHub organization.

5. How the Kubernetes project should relate to foremanctl and whether sharing compatibility metadata or operational tests would be useful.

Yes to all of the above

@jakduch

6. What level of integration and upgrade testing would be required before describing a release as supported.

This is the hardest. With foremanctl we’ve been looking at how to test things. It has a test directory. Ideally those should only test functionality that really is specific to foremanctl. We also have smoker which is intended as a more generic testing of the application. That is also used for the classic installer-based pipelines. In Rely on smoker for the content tests by ekohl · Pull Request #266 · theforeman/foremanctl · GitHub I attempted to move it more into that generic direction: have smoker as a full platform validation testsuite. With proper coverage, it could also be used to verify a Kubernetes-based deployment.

Of course, there is a big gap between “it installs and works” vs operating it at scale. Usually we start with saying that it is experimental and then judge the feedback.

@jakduch

8. Whether an initial experimental release should target Foreman with Katello first or also include a Foreman-only profile from the beginning.

A plain Foreman release could be even easier, but I wonder how many people would run that vs an installation with Katello. It’s something I’m asking myself already and would love to dig in at cfgmgmtcamp and other places.

@jakduch

9. Whether the proposed separation between the central Kubernetes deployment and external network-local Smart Proxies is appropriate.

At first glance, it looks correct.

@jakduch

10. Which Kubernetes distributions, node operating systems and deployment environments should be included in the initial test matrix.

This is a hard question that I don’t have a specific answer to. Traditionally the rule was “whatever the person who’s building it needs” so you have a big say in that.

[jakduch]

@ekohl Thanks, this is exactly the kind of feedback I was hoping for.

I was working from an outdated assumption about the Katello event daemon, so I will remove both that component and the related Artemis requirement from the design.

I will also move S3-compatible storage much higher in the initial roadmap. I’ll review what remains in puppet-pulpcore PR #379 and pulpcore-packaging PR #1760, and I’ll look at reviving the Active Storage direction from Foreman PR #9339. Treating shared storage as an upstream application concern instead of trying to solve it only in the Helm charts makes much more sense.

Using smoker as the common functional validation layer also sounds right. The Kubernetes-specific tests can then focus on orchestration, upgrades, recovery, scaling and failure behaviour instead of duplicating the application tests.

Would you be willing to create a repository for this under the theforeman organization now, even if it initially remains private and explicitly experimental?

I already have a working implementation and a growing test suite, but I think it would be much more useful to continue the work there instead of developing it privately in my own namespace. It would let you and the other maintainers follow the progress early, comment on individual decisions and redirect me before larger parts settle in the wrong direction.

I do not expect any support commitment or public release at this stage. A private experimental repository would be completely sufficient. I can push the existing work as a cleaned-up initial history, split the remaining work into issues and draft PRs, and keep the status and unsupported parts clearly documented.

[ekohl]

@jakduch

Would you be willing to create a repository for this under the theforeman organization now, even if it initially remains private and explicitly experimental?

AFAIK we don’t have any private repos under theforeman organization. I’d prefer to keep it public and be very explicit that it is experimental. That’s what we did with foremanctl (back then the repo was called foreman-quadlet). Dating back even further, there’s grapple which never even got passed merging the first PRs.

[jakduch]

That works for me. Public and very explicitly experimental is probably the better approach.

My main concern was not keeping the code private, but avoiding developing much more of it in a personal repository without giving the Foreman maintainers an opportunity to follow the work and correct its direction early.

The history of foremanctl and grapple sounds like a good precedent: hosting the repository under theforeman would make collaboration possible without implying that the project is supported, production-ready or guaranteed to graduate.

Would you be able to create a foreman-kubernetes repository under the organization? Once it exists, I can prepare the initial contribution immediately. The README and repository description will prominently mark it as experimental and unsupported, with the current limitations, unverified areas and upstream dependencies listed clearly.

I can either preserve the existing development history or prepare a cleaner initial import as a pull request, whichever you would prefer.

[ehelms]

I have some inline comments for quoted places as it was easier to read such a long post and comment as I went. Here is a general summary of the highlights:

• This is 100% on our desired direction, and not the first time we’ve explored it. Now I think we can do it in earnest.
• I am more than happy to see it progressing from the community. We can do our best to help support it, but keep in mind we still have quite a bit to do with foremanctl and the migration path from rpm-based to container-based.
• I would highly prefer transparency and seeing the work in the open, so I’ll gladly make you a repository today. While foreman-kubernetes is kinda boring, and I love a good name, we can always have bikeshed naming fun later

foreman-kubernetes – this has an associated team

@jakduch

1. Is a first-class Kubernetes deployment a direction the Foreman project would be interested in pursuing?

2. If the implementation reaches an acceptable level of quality, would there be interest in hosting it as a project repository under the theforeman GitHub organization?

Yes. One of your goals it to make the application available in more places, including Kubernetes deployments. And the sooner the better making what you are working on transparent and available.

I’d consider, since this is requiring multiple changes a github Project for this effort that will allow linking up all the efforts.

@jakduch

• official upstream applications and images without downstream forks;

Does this imply a separate set of images from what we are producing today?

@jakduch

• independently deployable Foreman/Katello, Candlepin and Pulp components;

Why do I want this if I need them to work together?

@jakduch

• support for external PostgreSQL, Redis/Valkey, Artemis and object storage;

I agree this can be a nice to have, but is that strictly a goal of or necessary to run in Kubernetes? We can easily over scope this effort and muddy the initial intent. External support can also apply to our podman-based installation method. I would ask that we separate architectural features that benefit both from being tied directly into Kubernetes.

@jakduch

• separate and observable database migrations;

What does this mean?

@jakduch

High-level architecture

This is missing what we think of as foreman-proxy, and is missing Foreman talking to Pulp.

@jakduch

• Helm charts and values schemas;

Why Helm? I’m not questioning it per say, but this is an example of a choice that would benefit from some discussion and vetting over other directions (e.g. operators).

@jakduch

A deeper integration could be discussed later if the Foreman team considers a common deployment frontend useful.

I think your reasoning is right on track – this should be considered once we know enough about the structure.

@jakduch

The important distinction is that application compatibility would primarily depend on the published container images and Kubernetes interfaces rather than on the availability of Katello packages for the node operating system.

I can’t tell if you are saying this is novel to the Kubernetes approach or just re-stating the intent of foremanctl with container images. Can you clarify what this is trying to say?

@jakduch

I would therefore also appreciate feedback on whether this separation—central application services in Kubernetes and network-local provisioning through existing Smart Proxies—matches how the Foreman community would expect larger installations to evolve.

Basically yes, I would expect that we will have on-cluster smart-proxies and off-cluster smart-proxies.

@jakduch

1. Create an issue in the owning upstream for every generic runtime or correctness change.

We do not require this today in our repositories, as a pull request can serve this purpose, and opening an issue followed immediately by a pull request can feel repetitive and overkill. Can we relax this a bit?

@jakduch

Proposed contribution model

I think this generally reads as a good list, I’d incorporate what’s necessary in a contributing markdown file, with what is specific to the repository keeping the Foreman standard model as the baseline.

[jakduch]

@ehelms Thanks for the detailed review and for creating theforeman/foreman-kubernetes. The direction and the distinction between generic container capabilities and Kubernetes-specific orchestration make sense to me.

I have gone through the comments and reworked the implementation and documentation accordingly. The current version is explicitly experimental and unsupported; the goal of publishing it now is transparency and collaboration, not claiming that it is already production-ready.

official upstream applications and images without downstream forks;

Does this imply a separate set of images from what we are producing today?

No. The intention is to consume the existing, separately released upstream Foreman/Katello, Candlepin, Pulp and Smart Proxy images.

The Kubernetes repository should not publish replacement application images or carry downstream application patches. During development, local candidate images may be used to test upstream changes before they are merged and published, but those are explicitly non-publishable test artifacts and cannot become a supported compatibility set.

Reusable runtime capabilities should be implemented in the owning upstream project and retain their existing package and container defaults.

independently deployable Foreman/Katello, Candlepin and Pulp components;

Why do I want this if I need them to work together?

I agree that the original wording was misleading.

The components are installed, upgraded and qualified as one compatible stack. “Separate” only means that each component retains the process and operational boundaries that already exist upstream: its own image, Service, configuration, scaling policy and failure boundary where applicable.

It does not mean that arbitrary versions can be combined, or that Katello becomes a standalone service outside Foreman. Compatibility is represented by a tested, digest-pinned release set covering the complete stack.

support for external PostgreSQL, Redis/Valkey, Artemis and object storage;

I agree this can be a nice to have, but is that strictly a goal of or necessary to run in Kubernetes? We can easily over scope this effort and muddy the initial intent. External support can also apply to our podman-based installation method. I would ask that we separate architectural features that benefit both from being tied directly into Kubernetes.

Agreed. I have separated generic runtime capabilities from Kubernetes-specific behavior.

External PostgreSQL, Valkey, object storage, TLS configuration and explicit migration commands are generic container capabilities that may benefit foremanctl and other deployment methods as well. This repository only owns their Kubernetes consumption: Secrets, Services, Jobs, NetworkPolicies, scheduling and release ordering.

The initial chart expects PostgreSQL and Valkey to be operated outside the application chart. That is an initial ownership boundary, not a claim that external services are unique to Kubernetes or must be implemented differently for Kubernetes.

S3-compatible storage remains an optional Pulp backend and a priority for removing the RWX storage requirement.

I also removed the proposed external Artemis deployment and the Candlepin HA experiment. The current chart enforces exactly one Candlepin replica and retains Candlepin’s current embedded messaging and non-clustered Quartz behavior. The repository no longer exposes an external-broker configuration or claims Candlepin horizontal availability without a qualified upstream contract.

separate and observable database migrations;

What does this mean?

It means one finite Kubernetes Job for each schema owner: Candlepin, Pulp and Foreman/Katello.

Each Job uses the corresponding application image and configuration, has a bounded deadline, and must finish before the new application workloads are submitted. Its result and logs can therefore be inspected independently of a long-running application Deployment.

The underlying migration commands are not intended to be Kubernetes-specific. Where possible, the same entry points and ordering should also be reusable by foremanctl and other supervisors.

High-level architecture

This is missing what we think of as foreman-proxy, and is missing Foreman talking to Pulp.

The architecture diagram and documentation now include both.

Foreman reaches Pulp through a private control endpoint, while public Pulp content is exposed separately. Smart Proxies are represented as independent execution or edge services communicating with Foreman through the normal registration and feature-discovery interfaces.

Helm charts and values schemas;

Why Helm? I’m not questioning it per say, but this is an example of a choice that would benefit from some discussion and vetting over other directions (e.g. operators).

Helm is currently used as the packaging and rendering layer. It provides inspectable manifests, a typed values schema, a conventional manual installation path and a reusable substrate for lifecycle tooling.

The optional ForemanRelease operator does not replace the Helm charts. It handles decisions that a normal Helm invocation cannot safely infer:

• dependency validation;
• migration-before-rollout ordering;
• fencing concurrent release operations;
• checkpointing progress across controller restarts;
• coordinating the application and execution-proxy releases;
• stopping after an irreversible migration failure instead of automatically rolling application manifests back across a changed schema.

I have documented this division explicitly. The Helm choice is not intended to prevent further discussion or evaluation of other approaches. Keeping rendering and lifecycle management separate should make that discussion easier.

A deeper integration could be discussed later if the Foreman team considers a common deployment frontend useful.

Agreed. foremanctl and the Kubernetes project should be considered different orchestrators for the same published application images and generic runtime contracts.

Image compatibility, health semantics, migration ordering, certificate handling and generic functional validation should converge where practical. Neither project should introduce a private application mode merely to satisfy its own supervisor.

A common deployment frontend may make sense later, once both implementations have produced enough practical experience to define the shared boundary.

The important distinction is that application compatibility would primarily depend on the published container images and Kubernetes interfaces rather than on the availability of Katello packages for the node operating system.

I can’t tell if you are saying this is novel to the Kubernetes approach or just re-stating the intent of foremanctl with container images. Can you clarify what this is trying to say?

This was not meant to claim something novel compared with foremanctl.

The intended point is that the Kubernetes nodes do not need Foreman or Katello packages installed on their host operating system. Application dependencies remain inside the published images. This is the same general container portability goal as foremanctl.

Kubernetes adds its own qualification dimensions—Kubernetes version, node architecture, container runtime, storage implementation, ingress controller, security policy and scheduling—but it does not create a new application packaging model. I have clarified this wording.

I would therefore also appreciate feedback on whether this separation—central application services in Kubernetes and network-local provisioning through existing Smart Proxies—matches how the Foreman community would expect larger installations to evolve.

Basically yes, I would expect that we will have on-cluster smart-proxies and off-cluster smart-proxies.

That is now the documented model.

The supplied on-cluster profile is deliberately narrow: a singleton Smart Proxy for Remote Execution SSH and Ansible.

DHCP, DNS, TFTP, BMC and other network-local services remain on separately operated Smart Proxies near the networks they manage. Additional on-cluster profiles can be added when their state, privileges, networking and lifecycle requirements are understood, but generic proxy functionality is not embedded in the Foreman web pods.

Create an issue in the owning upstream for every generic runtime or correctness change.

We do not require this today in our repositories, as a pull request can serve this purpose, and opening an issue followed immediately by a pull request can feel repetitive and overkill. Can we relax this a bit?

Relaxed.

A focused pull request is sufficient when it contains the motivation, behavior and test evidence. A separate issue is useful for larger design work, public API changes, compatibility policy or changes spanning several projects, but it is not required merely to duplicate a pull request.

Cross-project work can instead be linked through the shared GitHub Project you suggested.

Proposed contribution model

I think this generally reads as a good list, I’d incorporate what’s necessary in a contributing markdown file, with what is specific to the repository keeping the Foreman standard model as the baseline.

I added a CONTRIBUTING.md that uses the normal Foreman contribution model as its baseline and documents only repository-specific rules:

• application changes remain in their owning upstream projects;
• the Kubernetes repository owns orchestration and Kubernetes qualification;
• an issue is not mandatory before every pull request;
• implemented-but-unrun tests must not be described as passing;
• experimental compatibility sets must not be presented as supported.

Based on the wider discussion, I also removed the separate Katello event-daemon Deployment and its Kubernetes-specific runner patch. Katello event processing remains part of the upstream Foreman/Katello runtime unless Katello eventually publishes a supported independently supervised process contract.

The testing strategy now distinguishes three layers:

1. upstream component tests in the owning repositories;
2. deployment-independent functional validation through theforeman/smoker;
3. Kubernetes-specific lifecycle, migration, recovery, security and failure testing in the disposable Kind suite.

Smoker integration is documented as a future promotion gate, not as completed work.

The current static Helm rendering, schema, contract and ShellCheck suites pass. The full disposable Kind qualification and Smoker validation are still pending, so the repository continues to state clearly that no supported image set exists yet.

For the initial publication, I have prepared a clean single-commit snapshot rather than publishing the development history containing discarded experiments.

There is one GitHub bootstrap issue remaining: because the repository is completely empty, GitHub refuses to create a fork or pull request because there is no Git content or base branch. Could you initialize main with a small README commit? Once that exists, I can create a fork and submit the complete snapshot as the initial pull request.

[ekohl]

@jakduch

• Preserve an explicitly configured non-default Pulp API port when constructing Pulp clients.

I started Align on pulp_content_url by ekohl · Pull Request #9878 · Katello/katello · GitHub for this, but for a lack of time I never finished it. IMHO that should still be done.

Note that some content (notably containers) is actually served by the API. But it may actually be good for Foreman to always communicate via the public ingress rather than some private endpoint that is special.

[nikos_moum]

@ekohl

Biggest easy win: we got rid of the Katello event deamon and that also removes the need for Artemis in Candlepin.

Katello may no longer need Artemis, but Candlepin itself uses it for message-based async jobs. If you keep a single Candlepin node then Artemis can run embedded in the JVM as it does today, but if you ever need to run multiple Candlepin nodes, then Artemis will have to run as an external service (this is how the hosted/managed Candlepin works today)

[ehelms]

@jakduch

There is one GitHub bootstrap issue remaining: because the repository is completely empty, GitHub refuses to create a fork or pull request because there is no Git content or base branch. Could you initialize main with a small README commit? Once that exists, I can create a fork and submit the complete snapshot as the initial pull request.

Done – https://github.com/theforeman/foreman-kubernetes/blob/develop/README.md

Please note, or consider, the larger the initial PR diff, the longer it will take to review. The description of the work thus far sounds like you’ve done quite a lot already. You may want to consider whether there is a division of the current work that could result in multiple PRs to reduce the review burden.

[jakduch]

Thanks, that makes sense. I have been working on this for some time, and the implementation has grown beyond an initial Helm chart. I am therefore splitting it into focused, single-commit changes along functional boundaries.

The current implementation includes:

• typed Helm values with JSON Schema validation;
• separate Foreman web, Dynflow orchestrator and worker workloads;
• separate Candlepin and Pulp API, content and worker workloads;
• generated Foreman, Katello, Candlepin and Pulp configuration;
• external PostgreSQL and separate TLS-enabled Valkey endpoints;
• dependency checks and ordered database migrations before workload rollout;
• a private mTLS Pulp control endpoint with separate public content routes;
• filesystem and S3-compatible Pulp storage;
• ingress, default-deny network policies and restricted Pod security;
• horizontal scaling, disruption budgets, topology spreading and scheduling controls;
• a separate Remote Execution and Ansible Smart Proxy deployment;
• coordinated installation, upgrades and migration failure recovery;
• backup and restore of databases, Secrets, Foreman data, execution data and Pulp content;
• a namespaced release controller with leader election, resumable operations, drift detection, metrics and Kubernetes events;
• digest-pinned application and execution-proxy compatibility sets.

The containerization tests cover Helm rendering and schema validation, non-root runtime identities, readiness and health checks, migration failure and roll-forward, Pod replacement, Secret and certificate rotation, Remote Execution and Ansible jobs, Pulp file/Python/Debian/RPM/container content, Webhooks, virt-who, object storage, restricted network access, recovery into a clean namespace and controlled upgrades.

The complete end-to-end qualification is still in progress. The disposable Kind environment and its lifecycle scenarios are implemented, while several runtime capabilities first need to be merged and published by their owning upstream projects.

The upstream compatibility work currently waiting for merge is distributed as follows:

• Foreman — 4 PRs (1, 2, 3, 4)
• Katello — 3 PRs (1, 2, 3)
• Candlepin OCI Images — 1 PR (1)
• Pulpcore Packaging — 1 PR (1)
• Pulp OCI Images — 1 PR (1)
• Pulp Smart Proxy — 1 PR (1)
• Smart Proxy Remote Execution SSH — 1 PR (1)
• Foreman KubeVirt — 1 PR (1)
• Fog KubeVirt — 1 PR (1)

For anyone who would like to inspect the implementation as a whole rather than through the individual review slices, I am also publishing the complete current integration snapshot in my public fork:

GitHub - jakduch/foreman-kubernetes at full-prototype

The branch represents the current development state and shows how the charts, release controller, compatibility metadata, lifecycle tooling and integration tests fit together. Upstream review will remain divided into focused, single-commit pull requests.

I will continue presenting the Kubernetes work in smaller functional units so each part can be reviewed independently and the dependencies remain explicit.

[jakduch]

@ekohl Thanks again for pointing me in this direction. I picked up both parts and have now published the work as small, single-commit PRs.

For Pulp:

• pulpcore-packaging #3186 restores django-storages and its boto3 extra for EL9/EL10. Its CI is green.
• pulpcore-packaging #3189 adds a stable pulpcore-storage-s3 metapackage. The builds pass; repoclosure currently waits for #3186 to be merged and published.
• puppet-pulpcore #418 is the successor to #379. It adds optional S3-compatible storage configuration while keeping filesystem storage as the default and static files local. The complete test matrix is green.
• pulp-oci-images #117 installs and verifies the S3 runtime in the image; it currently waits for the new package to be published.

Together these replace the unfinished implementation from puppet-pulpcore #379 and pulpcore-packaging #1760, split so that the packaging dependency can land first.

For Foreman, foreman #11314 revives the framework part of #9339: it enables Active Storage, adds the official schema and keeps the generic blob routes disabled so consumers must retain authorization control. The required AWS SDK RPMs are split by dependency into #14203, #14204, #14205, #14206 and #14207.

Once the Active Storage foundation lands, I plan to follow it with the S3 service configuration and then migrate the actual consumers separately: core LDAP avatars, Katello upload staging and the relevant foreman_rh_cloud paths. That should let us remove shared filesystem requirements incrementally without weakening access control or making the initial PR too large.

[ehelms]

Thanks @jakduch for all of the details. I think taking a cursory glance of everything, where you are at in the journey with the code and across the code bases the changes you are trying to implement a few thoughts come to mind.

The biggest challenge will be organizing and presenting it in a cohesive way that allows folks, when they have time, to review the changes. As you can imagine, and likely tell, a lot of contributors and maintainers are working on a lot of things with decent backlogs across the ecosystem. With a heavy focus on getting foremanctl complete, the foreman-installer maintained until it can be retired through feature parity in foremanctl.

To that end, I think we need to think about how to present the set of changes in a meaningful, ordered way. You’v put a lot out there already, now we need to order it into a story that can be followed. What do you think about if we create a Github Project to track this similar to our Containerization one?

To begin with I would mark all the related issues across projects, put them in “To Do” and then have at least a “Review” column where the issues that need to be reviewed first are, and leave dependent issues in “To Do”.

[ehelms]

I hit send too soon. I also wanted to re-iterate balancing PR size, as even in the age of AI, smaller diffs are easier and quicker to review, and also easier to accept and merge versus a large PR might hold up small independent parts because of bigger, more complex parts that need to undergo more detailed review.

For example, Add project foundation and architecture by jakduch · Pull Request #2 · theforeman/foreman-kubernetes · GitHub is well themed, but still large. This could be split up into some much smaller, easier to review parts (almost every document could be it’s own PR). I realize as you’ve been working on this for a while, that you are eager to get it out there and strike a balance. For something this big, and new amongst the community, consider erring on the side of smaller is better.

[jakduch]

Thanks, I think a GitHub Project would be a very helpful way to organize this. The changes currently span several repositories, and reviewing them as isolated pull requests makes the intended dependency order difficult to see.

I also take your point about PR size. PR #2 was intended to establish a coherent foundation, but it combines several parts that can be reviewed independently. I will split it into a smaller ordered series covering the repository and tooling foundation, the architecture overview, individual design decisions and contracts, and the initial deployment structure. Where practical, individual documents will have their own PR.

I would suggest putting the general upstream prerequisites in “Review” first — changes such as migration entry points, readiness contracts, storage support, non-root execution and read-only filesystem compatibility. These are useful beyond Kubernetes and should continue to work automatically in existing standalone deployments. Dependent component integrations and Kubernetes changes can remain in “To Do” until their prerequisites are accepted.

I see foremanctl and foreman-kubernetes as two consumers of the same upstream images and runtime contracts. foremanctl remains focused on the supported single-host container deployment and eventual feature parity with foreman-installer, while the Kubernetes project provides cluster-specific orchestration. The upstream applications should not require users to select which environment they run in.

I can collect the existing issues and pull requests, map their dependencies, propose the initial review order, and help maintain the project board as the work progresses.

[ehelms]

Here’s the project: Foreman on Kubernetes · GitHub