RFC: Applying crypto policies to Foreman containers

RFC: Applying crypto policies to Foreman containers

Context and Problem Statement

Containerized Foreman needs a way to apply the host’s selected crypto policy to its services. Ideally, users would continue using update-crypto-policies on the host, then run foremanctl deploy to apply the change to their containers.

We cannot simply mount the host’s generated files into every container. Those files depend on the OS and crypto libraries that generated them, and we need to support configurations such as EL10 containers on EL9 hosts. Podman provides automatic handling for plain FIPS, but does not propagate other policy selections or FIPS subpolicies.

This proposal covers configuration managed by update-crypto-policies. Custom-written policies, application-specific TLS configuration, and automatic synchronization outside deployment are out of scope.

Proposal

I’m proposing that each container image generate its own crypto-policy configuration. This avoids requiring every image to use exactly the same base OS version.

During foremanctl deploy, we would:

  1. Read the host’s selected crypto-policy string, unless the user supplies a container-policy override.
  2. Start a temporary container from each distinct target image, using the exact image version that will run the service (or that is already in storage).
    • For existing deployments, this step uses the container image that exists in storage.
    • I originally wanted to use the running container itself to generate the crypto files, however, podman does not support adding bind-mounts to already running containers. The container thus needs to be recreated anyway, so it’s simpler to always use the image first to generate the crypto config files, and then create the permanent container after.
  3. Run update-crypto-policies as non-root via the ephemeral containers, using base_dir to write into a temporary directory (and --no-reload since it’s pointless in an ephemeral container).
  4. Export the generated crypto configuration to the main Foreman host machine.
  5. Run the containers with their respective crypto policy bind mounts.

Configuration would be regenerated when images change. Each relevant image must provide update-crypto-policies and its execution dependencies.

Policy compatibility

A host selection may not exist in the container’s OS. For example, an EL9 host could use LEGACY:NO-SHA1 while an EL10 image lacks NO-SHA1.

In that case, deployment would fail with an explanation and direct the user to a new container-policy override option. The override would be persisted, with some way to return to host inheritance.

Matching policy names means using each image’s native interpretation, not guaranteeing identical restrictions across OS versions. This is a compromise to deal with the complexities of trying to sync up crypto policies between different OS versions.

FIPS

We would use the same generation path for FIPS and supported FIPS subpolicies. Explicit mounts at /etc/crypto-policies/config and /etc/crypto-policies/back-ends would take precedence over Podman’s automatic plain-FIPS mounts without disabling the underlying FIPS runtime state.

This mount behavior needs testing on supported Podman versions. Overrides must remain compatible with the deployment’s FIPS requirements.

Take a look at container-libs/common/pkg/subscriptions/subscriptions.go at c231aab00595fbd79ae05d8f01028f6e49b2e999 · podman-container-tools/container-libs · GitHub for some comments on FIPS from podman.

Alternative Designs

  • One shared generator image: Reduces the number of ephemeral containers that must run update-crypto-policies, but requires compatible OS and crypto-package versions across the generator and all consuming images. A common base image is expected to be created regardless, however one shared generator image forces every container image to update together when there is an OS update. I also think it requires this common base image to be a prerequisite development item. I’d like to start development of this feature sooner rather than later to make the Foreman 5.2 deadline.
  • Build locally configured image layers: Instead of bind-mounting the generated crypto policy files, build them into a new layer in the container image. Provides a place to install missing tools and bake in configuration, but adds image-building and lifecycle management. We can revisit this later if the bind mounting introduces unexpected issues.
  • Generating crypto config from already-running containers: I wanted to avoid ephemeral containers altogether, but as discussed above, bind-mounts must be declared at container image running time. Locally rebuilding container images with crypto config is one way to avoid this bind-mounting complexity, but we must also consider the overhead of building container images. At the moment, I’m assuming that local container builds are slower, but I have no data to back that up.

Decision Outcome

TBD, following community feedback.

I want us as a project to begin implementing the solution for this overall crypto-policy-matching issue sooner rather than later to ensure that the Foreman 5.2 deadline is not a problem.

Impacts/Prerequisites

  • IoP images need the crypto-policies-scripts RPM.
  • FIPS testing is necessary to qualify the proposal up above.

Open questions

  • When do we reload the crypto config?
    • The crypto config needs to be regenerated when container images are updated, and when the requested crypto policy changes. Is it easy via foremanctl to detect those specific situations? Or, since updating those files may be cheap, is it better to simplify the workflow and always regenerate the crypto config at foremanctl deploy time?
  • What do we do if an image we do not control stops shipping the crypto-policies-scripts RPM in the container?
    • Perhaps we can either start maintaining that container ourselves since we need a joint base container anyway. Or, we can look into how local rebuilds can solve this.

Could we have the foremanctl pull-images command check this and do the needful? That wouldn’t solve the problem when you updated images manually, however.

pull-images to me is where new content comes in but does not necessarily introduce downtime. We could recreate the containers then, but then we’d have to take the Foreman offline. And, yeah, I agree too that we should catch manual updates to container images.

Let me kick off with organization question. Aren’t we solving something that the platform team should be doing? This is not an easy task and we might not be the only team that needs this. Now, I can think about many reasons why it’s us who will have to do this, so I can rephrase this to: is this implementation a temporary solution with the plan that the platform team will solve this for us, or are we seeking for a permanent solution?

This is directly against how podman operates, it is sound if we treat this as a short-term workaround tho. I prefer this over other options you considered. An unfortunate consequence is that this will inevitably slow down the deployments and upgrades performance. I can imagine the tooling is fast, it is all about dropping some configuration files after all, but we have multiple containers and Ansible tasks can easily add up.

Have you explored any ideas around doing this purely in the runtime? What I mean by that is making sure that all our containers do have an ENTRYPOINT that includes a starting script that does this before process startup:

  • Read environmental variables with intended settings
  • Execute update-crypto-policies tool
  • Possibly do other preparations in the future
  • Run the container process

See, one thing that comes to my mind is that FIPS might be not the only thing that will have to solve this way and while bind-mounting works for this use case, it might not for something else. A script that performs preparation work prior process startup is super easy to implement, we start containers in parallel (mostly) so it should not be an issue too.

Hi @lzap ,

I view this as a workaround for a broader issue, just like you mentioned. I’ve heard that podman has requests to make it easier to sync up the container host’s crypto policies with containers. However, there is nothing solid there yet, so this new feature would serve our immediate deadlines.

I totally agree with the downsides. There were some discussions outside of Discourse and the general history went:

  1. Create containers with specific crypto policies in them.
    a. This is not flexible enough, so it was strucken off the list.
  2. Use a single generator image to create crypto files for all containers
    a. This forces matching up all container images in a deployment, which I find to be inflexible from a deployment standpoint.
  3. My proposal here.

Using ENTRYPOINT is interesting, I haven’t explored that specifically. However, doesn’t it still hit the limitation where we do not want to be using root on container images? I would 100% prefer to perform this update at runtime if possible, but I haven’t found a way around that root problem.

I’ll say, at least, the scope of this work has been mostly limited to what update-crypto-policies manages. For the sake of Foreman 5.2. Golang might need some work too for IoP containers, but that is to be determined.

If, though, we can perform this runtime config without root worries… well I’d enjoy looking down that route too.

I was merely exploring other options, if you say this is a non-negotiable requirement there is not much we can do about it. IDK how exactly is the requirement, but we could run the crypto updates and then drop privileges. But I can imagine security tools flagging our containers because USER 0:0 statement and there could be problems with that (spawning sub-processes from where podman expect just a single process).

In this light, I see your proposal as a better one.

I like this approach since its done at build time when being root is acceptable. But that would mean more tooling changes and management as you have pointed out. Given the non-root + containers with diff base images constraints I see your proposal as most practical.

What is base_dir and where is it documented how to use it?
Is that an official API that’s supported? man update-crypto-policies doesn’t mention it, and it sounds like something that might silently break the next day.

@iballou I second @lzap’s concerns regarding creating a workaround. This seems to try to recreate the system level feature of crypto policies into containers. I do believe that this needs to be rather handled on the system level (i.e. podman) rather that within the workload (i.e. container). Do we know how K8s/OpenShift solves crypto policies?

@Partha_Aji and I were just discussing this yesterday. It isn’t something official as far as I can tell from the docs, but there is support for an environment variable in the code:

https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/blob/master/python/update-crypto-policies.py?ref_type=heads#L61

We can ask about having it documented, or introduced as a command-line argument.


Alternatively, @evgeni, I don’t know much about the origin of the “container may not run root commands” requirement. Do you know if there’s any chance we could run update-crypto-policies as root temporarily during the generation step, grab the files, and then start the container officially with the default nonroot access?

My immediate thought is that not all root activities within containers are blocked because the original design, with the single generator image, would have needed to run update-crypto-policies as root too. I didn’t see mention of base_dir in the original design notes.

Of course there is also the option to build intermediate container images as mentioned in my first post which may work around this, but I’m not sure that we need to go that far unless running update-crypto-policies ends up truly being an issue.

I’ve received word from the container engine team that there is no help right now with matching host and container crypto policies. We’ve received some suggestions from crypto-related engineers at Red Hat. Their primary recommendation was to build container images with specific policies built into them, which we found too inflexible. The generator idea was another from them.

From a quick Google, it looks like OpenShift generally relies on the container builder to set their crypto policies. FIPS seems to be handled automatically, but I think that’s thanks to the container engine. I happened to find this open issue Make it easier to change crypto policy · Issue #607 · coreos/fedora-coreos-tracker · GitHub, which tells a little bit of the story. I’m seeing some information about “init containers” too, which sounds quite similar to this design.

I agree that this seems like something that the general EL platform should take care of for us. However, we do have wishes in the future to run also on other OSes (like Debian?) which may hit this problem separately, so there is some precedence to having a solution there. At the end of the day, though, we want to get this solved for EL for Foreman 5.2, and this deadline is coming up quickly.

Instead of running this in every container, I’d propose to have this generated once per base. The mapping could be detected from containers metadata and/or mapped within the foremanctl’s deployment.

Having the requirement for each container to include update-crypto-policies increases its footprint. Note, that update-crypto-policies requires Python and shell. Also note that compiled languages run just on a very minimalist base system that includes just the binary and necessary libraries. That reduces its footprint and security surface.

Images that are distroless, like Hummingbird, might not work with crypto-policies at all. Some do not package shell.

This is a mixing of the original single generator container design proposal and my proposal up above. It would give us the flexibility to use containers that do not share a common base.

It does mean that we need to solve the “base container image” problem for images that we do not control, like the Postgres container. However, I think we’re already planning on doing so. It just means that this implementation in foremanctl has prerequisite build-team work. I posted an RFC link earlier about the intermediate container image idea.

Would we even have an option then to generate crypto policies from a base image for Hummingbird? It sounds like we wouldn’t have a base to begin with and we’d have to perform some building ourselves to generate the right crypto files.

That could bring us to my design variation mentioned above: building a layer on top of the Foreman containers with the crypto policies in them at deployment time. However, since we do not have images right now that fit this category, it strikes me as a future issue.

Maybe I have an answer to this: Using a Project Hummingbird container image | Project Hummingbird

The builder variant of Hummingbird seems to be a potential generator in case we use Hummingbird in the future.

Not Evgeni, but answering for him since I was involved. I had talks at least 1 Satellite customer that has internal company policies against this. Note that OpenShift (Kubernetes too?) also doesn’t support running as root. With both in mind, there is an automated test to verify containers don’t have root: foremanctl/tests/container_user_test.py at master · theforeman/foremanctl · GitHub

This also means there is no standardized way for admins to set security policies.

Something that hasn’t been mentioned here, but the use case of managing this. Some cases an admin might like:

  • PQC-only
  • PQC and classic
  • Allow older TLS for older RHEL versions

As the people building the images, we can’t determine those ahead of time for the user.

One addition: make the ephemeral generation cacheable, keyed on image digest plus the effective policy string (including any persisted container-policy override). That avoids rerunning update-crypto-policies on every deploy when neither has changed, while still regenerating when either the host policy or the image changes. Logging the effective policy per image would also make it obvious when an override is masking host inheritance.