Nightly Packaging Unification Proposal

Development RFCs
1

Nightly Package+

Goal

My goal is to have a single workflow for building and releasing nightly built packages that is straightforward conceptually and allows for more nightly package additions without heavy effort.

Package Structure

If a package is a nightly package, then it is required to have the following criteria:

1. It will have no sources checked into foreman-packaging

The project repo for the nightly package will be git checked out (locally or on jenkins) and the sources will be generated from that.

2. It will be based upon the version given in the project’s Github repo

If the version listed in the git repo is 0.14.0, then the specfile for the nightly built package will be 0.14.0. Conceptually this can be thought of as the goal we’re aiming for rather than a representation of the project as it stands today.

3. Nightly package specs will have a specific release field

The prerelease macro will be used to signify a nightly package. Any package being built in a nightly fashion will have the following release field:

Release: %{?prerelease:0.}%{release}%{?prerelease}%{?dist}

The prerelease macro is populated during obal nightly, with .<timestamp>git<githhash>. The timestamp is in the format YYYYMMDDHHMMSS. The githash consists of the first seven characters in the commit hash.

an example:

rubygem-katello-3.9.0-0.6.20181201120000gitabcdef0.el7

Current Infrastructure

We have 9 nightly packages today.

$ koji list-targets \
  | grep -E 'nightly-(nonscl-)?rhel7' \
  | grep -v pulp \
  | awk '{ print $1 }' \
  | xargs -I '{}' koji list-tagged --latest --inherit {} \
  | grep -E '[0-9]git' \
  | awk '{print $1}'
foreman-installer-1.20.0-0.201808070324gitb4ef170.0.1.develop.el7
foreman-proxy-1.20.0-0.201808020010gita8f5b8e.0.1.develop.el7
foreman-selinux-1.20.0-0.201808060304gitfc6c210.0.1.develop.el7
foreman-1.20.0-0.201808061457git9f3477a.0.2.develop.el7
tfm-rubygem-hammer_cli-0.14.pre.develop-1.201807201904git0350ccb.el7
tfm-rubygem-hammer_cli_foreman-0.14.pre.develop-1.201808010919gitc3c8fa2.el7
katello-installer-base-3.9.0-0.201808062246gita2cd105.2.el7
tfm-rubygem-hammer_cli_katello-0.14.0-1.201808070939git54e9f0b.el7
tfm-rubygem-katello-3.9.0-0.201808061815git5820dc3.2.el7

Let’s take a closer look at each:

1. foreman-installer

packaging_trigger_installer_develop jenkins job: builds the source, archives it, then triggers RPM and deb build jobs.

release_nightly_build_deb jenkins job: exists to trigger another job

packaging_build_deb_coreproject jenkins job: does the building

release_nightly_build_rpm jenkins job: exists to trigger another job

packaging_build_rpm jenkins job: does the building

2. foreman-proxy

test_proxy_develop jenkins job: runs tests, builds the source, archives it, then triggers RPM and deb build jobs.

release_nightly_build_deb jenkins job: exists to trigger another job

packaging_build_deb_coreproject jenkins job: does the building

release_nightly_build_rpm jenkins job: exists to trigger another job

packaging_build_rpm jenkins job: does the building

3. foreman-selinux

packaging_trigger_selinux_develop: builds the source, archives it, then triggers RPM build job.

release_nightly_build_rpm jenkins job: exists to trigger another job

packaging_build_rpm jenkins job: does the building

4. foreman

test_develop jenkins job: runs tests, builds the source, archives it, then triggers RPM and deb build jobs.

release_nightly_build_deb jenkins job: exists to trigger another job

packaging_build_deb_coreproject jenkins job: does the building

release_nightly_build_rpm jenkins job: exists to trigger another job

packaging_build_rpm jenkins job: does the building

5. hammer-cli

test_hammer_cli jenkins jobs: runs tests, builds the source, archives it, then triggers RPM build job (no debian?).

release_nightly_build_rpm jenkins job: exists to trigger another job

packaging_build_rpm jenkins job: does the building

6. hammer-cli-foreman

test_hammer_cli_foreman jenkins job: runs tests, builds the source, archives it, then triggers RPM build job (no debian?).

release_nightly_build_rpm jenkins job: exists to trigger another job

packaging_build_rpm jenkins job: does the building

7. katello-installer

katello-installer-nightly-release jenkins workflow job: builds the source, archives it, triggers RPM build job.

packaging_build_rpm jenkins job: does the building

8. hammer-cli-katello

hammer-cli-katello-master-release jenkins workflow job: builds the source, archives it, then builds the RPM.

9.katello

katello-master-release jenkins workflow job: builds the source, archives it, then builds the RPM.

Bad Smells

  • The parent jobs (beginning job for each chain) don’t all confirm to foreman-infra jenkins job naming conventions.
  • Sometimes the source is generated and archived in a test job, other times in a release job.
  • Groups of jobs that are always seen together
    • release_nightly_build_rpm --> packaging_build_rpm
    • release_nightly_build_deb --> packaging_build_deb_coreproject
  • Multiple jobs chained together are harder to follow and debug since you have to chase down logs across jobs.

Proposal

Remedies

  1. Job Renaming
current name proposed name
packaging_trigger_installer_develop foreman-intaller-develop-release
test_proxy_develop foreman-proxy-develop-test --> foreman-proxy-develop-release
packaging_trigger_selinux_develop foreman-selinux-develop-release
test_develop foreman-develop-test --> foreman-develop-release
test_hammer_cli hammer-cli-master-test --> hammer-cli-master-release
test_hammer_cli_foreman hammer-cli-foreman-master-test --> hammer-cli-foreman-master-release
katello-installer-nightly-release katello-installer-master-release
hammer-cli-katello-master-release
katello-master-release

  1. Reuse code via jenkins-job-builder rather than having multiple generic build jobs with lots of parameters chained together. This will allow us to cut down on the number of jobs and have one jenkins job per nightly package release, dropping the chained jobs anti pattern.

  2. Use obal to build nightly packages based on a local checkout of the source repository rather than depending on archives stored on jenkins. We can still continue to archive the built source files. The source files will be archived in the -release job for a given package.

Limitations & Complications

I can imagine a scenario where a *-nightly-release job starts, then code is merged into the * repo while the job is running. Since the job runs git checkout in multiple places, is it possible that the code being tested could differ from the code that gets packaged into a source file?

The best solution I can come up with quickly is to disallow concurrent runs of the given jenkins job and git checkout based on a hash instead of a named pointer like master or develop. If we disable concurrent runs of jenkins jobs, do subsequent runs get queued or does jenkins drop them?

Completion Strategy

  1. Refactor obal and foreman-packaging to facility building of a nightly package both in jenkins or locally on a developer’s machine.
  1. Build a JJB groovy template and/or methods for releasing nightly deb packages to replace the release_nightly_build_deb --> packaging_build_deb_coreproject job chain.
  2. Build a JJB groovy template and/or methods for releasing nightly RPM packages to replace the release_nightly_build_rpm --> packaging_build_rpm job chain.
  3. Build a single JJB groovy template from both 2) and 3) for those nightly packages that build for both deb and RPM.
  4. Document how to become a nightly package and how does a nightly package do a standard release.

Questions

What will happen when I open a pull-request against foreman-packaging for a nightly package?

  1. Github hook will trigger packaging_test_pull_request and foreman-packaging-rpm-pr-test as it does today.
    1a. packaging_test_pull_request does its usual deb thing.
    1b. foreman-packaging-rpm-pr-test will need to be updated to know how to build sources. this shouldn’t be difficult given that will be groovy methods that can be called.
  2. Merge pull-request. This triggers foreman-packaging-release which builds the source (from the merged commit hash), then builds a new nightly RPM on Koji.

Please feel free to ask questions, comment, provide feedback.

If I don’t receive any strong disagreements regarding this proposal by 7 September, then I will assume people are okay with this plan and I will continue working on it. Thanks!

2 Likes
2

Is this change going to enable plugins to have nightlies as well? If so, where can I subscribe for those?

1 Like
3

Yes, plugins can opt-in to be built nightly provided they follow the Package Structure rules.

Just let us know, and we’ll add a nightly release job for the plugin.

4

Adding a section on how does a nightly package release a real release would help plugin maintainers to understand the implications to the workflow. The current workflow most are used to is releasing to nightly and then cherry picking it back to other releases. I think this new workflow would have plugin maintainers with nightly packages releasing to release branches directly and skipping rpm/develop.

5

This tight coupling currently exists for the core packages. This means that there’s the expectation from the project maintainers (hammer for example) to keep them synced up and a bump from .0.14-develop to 0.15-develop also means a packaging PR. Note that in practice this is already the reality because we no longer override the version in the spec file (breaking all hammer nightly builds a few weeks ago).

We do use %prerelease in some places. It also has a special meaning in gem building. However, I do prefer simplicity. Can we introduce one single macro so you can simplify --define nightly <version> and the specs would know how to generate the correct RPMs? It would mean we wouldn’t have to modify files in place anymore which is much cleaner.

Obal has no knowledge of Debian building whatsoever. That means we must still archive the source file. I don’t see a proposal about a standardized place to do this. Given some pipelines only have a -release job, would that be it?

Yay for this. The single jobs was the old pattern to reduce code duplication but now that we’re fully using JJB we can remove the code duplication at another layer giving a much clearer view to the user.

Release process

I don’t see anything about actual releases. Release Process - Foreman describes the process and we will need to add all nightly packages there. It means they can no longer be actually released into nightly repositories. This makes the whole nightly process unusable for plugins which have their own release cycle. Not a problem per se because we don’t have that now, but important to be aware of. I’d also argue we need to revisit the whole cherry picking process and document it. AFAIK it’s not really documented anywhere. Because of this I’d propose to consider plugins out of scope for this PR.

Another matter which I noticed while doing releases: we don’t always sync back releases into the main RPM/deb changelog. You don’t see 1.18.0 released in the 1.19 package changelog. It’s true that those changelogs are linear while our branching model isn’t. It also makes you wonder if changelogs have any value at all. If git log is the only way to really see what’s going on, should we stop pretending and get rid of them altogether? See 1.18 vs 1.19 vs develop.

6

That’s a good suggestion, thanks @ehelms!

I’ve added step 5 in the Completion Strategy section.

7

Thanks for the question, @iNecas. I’ve added a step 5 to the Completion Strategy section to inform developers on how to opt-in to nightly releases if they desire.

8

This tight coupling currently exists for the core packages. This means that there’s the expectation from the project maintainers (hammer for example) to keep them synced up and a bump from .0.14-develop to 0.15-develop also means a packaging PR. Note that in practice this is already the reality because we no longer override the version in the spec file (breaking all hammer nightly builds a few weeks ago).

Yeah, some nightly packages already follow this coupling, but I wanted to state it explicitly as a requirement for being a nightly package.

Can we introduce one single macro so you can simplify --define nightly and the specs would know how to generate the correct RPMs? It would mean we wouldn’t have to modify files in place anymore which is much cleaner.

That is a really good idea! I was able to make it work in this fashion easily. I’ve updated my two current PRs listed in the OP. Thanks!

Obal has no knowledge of Debian building whatsoever.

Part of the initiative for this proposal is teaching obal how to deal with debian packages (at least nightly, at a minimum). That can be seen in step 2 of the Completion Strategy section.

I don’t see a proposal about a standardized place to [archive sources]. Given some pipelines only have a -release job, would that be it?

Ah, you’re right. I didn’t specify. Sources will be generated and archived in the -release job for a given package. I’ll update the doc with that information.

I don’t see anything about actual releases.

I’ve added a step 5 to the Completion Strategy section about documenting this. Does that address your concern? If you’d prefer something more descriptive, please let me know what information you’re looking for and we can work together to come up with something.

Thanks for the feedback!

9

I’ve not heard any large opposition to the proposed plan. Thanks everyone who popped in for questions or comments.

My next step is the break the Completion Strategy section down into management chunks so that I can get started working.

Thanks again!

1 Like