RFC: Packit-based nightlies for everyone

evgeni · September 1, 2025, 1:54pm

RFC: Packit-based nightlies for everyone

Decision Due Date: 2025-09-16

Context and Problem Statement

Today most Foreman plugins and many other repositories in the Foreman project implement Packit-based builds for PRs, resulting in a very convinient way to test out a single PR in an RPM-based (production-like) deployment.

However, once a PR is merged, the changes become uninstallable^[1] until a release happens.

For a couple of selected packages, we have implemented a “nightly” workflow, where pseudo-releases are created on Jenkins and packaged into the “nightly” repository together with the normal releases of all the other projects.

As the “nightly” process requires quite some management overhead for setup (Jenkins jobs, triggers, adjustment to packaging specs, to name a few) and then later for doing stable releases correctly (ensuring all nightly packages get a “real” release), we have not added any new packages to the list of nightly built packages and asked the maintainers to “release often” instead.

Nightly packages are also incompatible with our “package bumping” workflow, as it requires the packages to be present on rubygems.org.

Proposal

Build commits to main branch into a central repository

Packit supports triggering builds on commit and building them into a user-defined repository.

We have implemented this for foreman_rh_cloud:

github.com/theforeman/foreman_rh_cloud

.packit.yaml

330ad54d7


      
          - <<: *copr
            trigger: commit
            branch: develop
            owner: '@theforeman'
            project: develop

The build result can be seen in the @theforeman/develop COPR.

We propose to implement this for all repositories that utilize Packit today.
The implementation requires two changes:

a trigger: commit block in .packit.yaml
the GitHub repository being added to the allowlist of sources for @theforeman/develop COPR

Packages from this COPR can then be used on-demand by everyone who wants to “grab the latest develoment snapshot”, by enabling the copr using dnf copr enable @theforeman/develop rhel-9-x86_64 and installing/upgrading packages using dnf install <pkg> and dnf upgrade <pkg>.

This requires no further changes to the packaging specs and does not affect the “released” package set.

Automated testing of development snapshots

To automatically test those development snapshots we can add a dedicated pipeline that runs in parallel to our normal nightly pipelines and that performs the whole “luna” set of tests, but utilizing the development builds instead of the ones from the normal packaging repository.

Keeping the development snapshots in a separate repository ensures they do not accidentally leak into a stable release because there was no bump to a proper plugin release.

Alternative Designs

Instead of consuming the packages from a dedicated copr repository and test them in a dedicated pipeline, we could have a process that injects the packages into the “regular” staging repository for nightly. This way they get tested in our regular testing pipelines and are automatically exposed to all users when the tests pass and the staging repository gets published as “nightly” for our users.

The downside is that this would require more work during branching, as those development snapshots should not leak into stable releases we publish.

Decision Outcome

TBD. What was the decision that was ultimately made and relevant outcomes with links.

Impacts

For “building” this will require changes to all repositories that have Packit enabled.
A GitHub project will be created for tracking the effort.
Given the rather small change to the Packit configuration, and the requirement to adjust COPR configuration, the work will be performed by the authors of the proposal.

For “testing”: Depends on the implementation/design picked above. When dedicated pipeline, that needs to be implemented and will not impact any existing workflows.

Unless you open a new PR to create a new build, or take the Packit result of an unrelated PR. ↩︎

ekohl · September 2, 2025, 12:32pm

Today we have nightly packages for Katello. If we implement this, do we want to keep those or move completely to a “nightly only contains released packages” model? This could make Katello more “just” any other plugin without its own special release process.

If so, then it may be good to introduce a pipeline that runs with nightly packages enabled.

evgeni · September 2, 2025, 1:40pm

I think migrating the existing nightly packages to this workflow would be a logical next step, but nothing that I planned to dictate just yet.

The current nightly packages (foreman, foreman-installer, foreman-proxy, foreman-selinux, katello, hammer_cli, hammer_cli_foreman, hammer_cli_katello) have several additional gatings and workflows attached to their nightly status, which we’d need to consider:

the nightly source generation (which then gets packaged) only happens when CI passes in Jenkins – Packit generates a source on every commit, even if CI would be red (I personally do not think it’s a major loss, but a change nevertheless)
the CI runs produce lock files (Gemfile.lock, package-lock.json) that we use to bump dependencies in foreman-packaging to versions that are “newer but have passed CI” – I think this is an artifact that we should keep in some way (but I am sure we can generate it somewhere else).

ehelms · September 3, 2025, 7:38pm

Generally I really like this idea as it feels simpler to get a hold of a nightly version of a package compared to our current model. What I worry about is investing into this at the cost of container investment which we have identified as the direction we would like to go.

evgeni · September 4, 2025, 7:28am

Given the current containers are still based on RPM packages, they too would benefit from better tested RPMs

I think the “build” part is cheap enough to not take away precious resources from the containerization effort, while still providing value.

The “additional pipeline” could also be implemented cheaply, but would require people to analyze failures if there are any. (I’d argue time spent here will be saved analyzing the same failures later on, so it’s a win)

Odilhao · September 8, 2025, 12:25pm

I was the one supposed to post this @evgeni . One of the goals to move from Jenkins is to also ensure that Devs can look more closely to nightly builds, right now anyone that needs to understand the shape of nightly need to look at 2 Jenkins jobs.

evgeni · September 8, 2025, 7:37pm

Well, yeah, but that RFC won’t change the Jenkins shenanigans. We still need to fix those.

ekohl · September 9, 2025, 8:45am

It took me a while to realize you were talking about stop using centos jenkins now that Duffy API is available on the Internet · Issue #1828 · theforeman/foreman-infra · GitHub