Packaging Automation Proposal RFC

This is a proposal for taking our packaging repository to the next level of tooling and automation.


This proposal is inspired by the work that my team has been doing and refining across downstream packaging, the Rails 5.1 SCL packaging and within the Pulp community. We’ve had great success with managing many packages, automating the addition of new packages, updates, test automation and release automation. We believe this can greatly benefit foreman packaging and help to unify the various packaging efforts that are on going in related communities.

The proposal will in general reference the RPM side of things as that is the aspect the team has worked with the most and crafted these tooling and workflows with. However, that does not mean this is limited to RPMs. What lies below should be applicable to Debian packaging as well and more depends on the input from the Debian packagers.


The bulk of this proposal revolves around a move to using Ansible for tooling to provide scripted tasks and easy CI integration. A majority of this tooling is already written and used in other repositories that my team manages and we are working on putting them into a single shared repository to allow re-use in foreman-packaging as well. The following is a breakdown of the various parts of the proposal and changes therein.

Ansible Playbooks

Ansible provides an easy to understand and reference DSL for performing tasks that touch a variety of concerns. Further, the framework for orchestrating and organizing a set of tasks makes it a good use case for building out release functionality. As I mentioned, the team has built a set of tasks for scratch builds, releases, adding and updating packages that are already in use in multiple packaging repositories. The functionality exists and is ready to be used to make both maintainers and developers lives easier when performing rote packaging tasks.

Another goal of this proposal, mentioned below, is a heavier use of CI. Ansible playbooks provide a nice clean, easy to follow output along with additional behaviors like performance output. This allows us to write complex tasks in a single language and run it either locally or create lightweight CI jobs that have little to no logic other than calling the same playbooks a developer would. This makes debugging or performing actions manually much less burdensome.

Ansible Inventory

When first learning Ansible, the documentation talks about managing hosts and the state of them. The inventory is that of a set of hosts. However, the Ansible inventory is not restricted to hosts. The inventory is just collections of things grouped together in a user defined logical manner that can have tasks performed on them. This realization has led us to use an Ansible inventory to define the set of packages and attributes about them as variables. This makes adding new packages, updating existing and releasing packages easy as you can perform actions across the inventory or by limiting the set of packages. This has so produced great results so far.

The proposal here is then to add (what we call a package_manifest.yml) that serves as an Ansible inventory and the playbooks act upon. This inventory would contain packages organized by repository. This is similar to the per-tag configuration found in rel-eng/tito.props but gives the power to use Ansible directly as intended without having to write a similar system in ruby, bash or more Ansible.

Stop using Git Tags

The use of tito with git tagging has had some usefulness but places a burden on maintainers and CI systems. The workflow today requires that after merging a packaging update a maintainer must then use tito tag to create both a new commit and tag on the mainline branch. These both must then be pushed up to the remote branch before being released. This presents both overhead and makes creating automated release pipelines cumbersome as they would have to modify the codebase. Further, the tagging structure provides no real benefit in the context of the packaging repository. Originally the tito project created tagging with the idea of using a spec file inside of the project source code and thereby capturing the git history.


The previous work leads toward this final aspect of the proposal which is to add significant gains in continuous integration and delivery. The Foreman, plugins and Katello require maintaining a large number of packages. The actions to maintain those are fairly routine: add a new package, update a package, release a new version of a package. A human should not be required to complete these actions as they are definable. A human should interact where it’s important by reviewing pull request code, and approving. After that, automation should be able to handle the rest.

I propose to create (or update) the following CI jobs:

  • Update the pull request scratch build job
  • Create a package release job
  • Add a job that examines and reports on changes to Foreman’s Gemfile.lock
  • Add a job that examines and reports on changes to Foreman’s package.json

These jobs would be written:

  • Using Jenkins pipelines
  • “Code” written in Ansible playbooks and executed by Jenkins

Finally, the goal with this proposal is to reduce the packaging burden on maintainers by creating maintainable code for performing packaging release work and using that to automate the jobs that keep packagers busy but are routine. Further, the hope is this should help us also move faster and keep dependencies up to date more often. Please raise questions, comments and concerns you have so that we may begin putting this proposal into action.



Sounds good, evn though I didn’t go to much into details. One suggestion
while looking into this would be possibility of having nightlies for
plugins? Maybe phase 2 for this, but keeping that in mnd would help a lot
the plugin developers and users

– Ivan

st 10. 1. 2018 v 3:06 odesílatel Eric D Helms <> napsal:

1 Like

Sounds good.

+1 I like that idea

Anything that simplifies and reduces the time we spend handling packing is a big :+1: from me.

In that spirit I wrote a PR to update the foreman.spec dependencies based on package.json:

More automation is always :+1: from me too - I do have some structure questions though…

  • Will the Ansible code be part of foreman-infra or some other repo?
  • Do we still plan to separate out the JJB config to a new repo or top-level of foreman-infra? (seems like a good time to do so)
  • Some repo management is done by Puppet today, do we plan to port this over?
  • Will we be doing any of this from our Foreman instance via foreman_ansible or will it all be from Jenkins?

I have some answers…

  • The Ansible code will be part of a separate repository and it’s own tool that we are working on
  • Yes - but for now there is a simple symlink at the top of foreman-infra to make it easier for developers
  • Which repository management do you mean?
  • All from Jenkins

Note this is currently not planning to touch repository management, but rather just the packaging management of spec files, RPM builds, etc.

I was referring to @ekohl’s comment on this thread, I’m not actually sure where the code lives right now. But I believe that is management of the repos, so I guess that’s a separate/later thing anyway.

Is there any reason not to have this in foreman-infra, or foreman-packaging? It feels a bit like we’re spreading knowledge/code over more repos than are really needed, at least from an outside viewpoint (I appreciate this may be bikeshedding, and can always be consolidated later).

The problem with storing this in foreman-packaging is that we branch foreman-packaging which means that your scripts and tooling gets forked repeatedly and begins to change per branch. This should really live outside of and act upon the packaging repository (i.e. the packaging repository is “configuration” or “data”).

I consider foreman-infra to be the place where code that manages infrastructure lives and not the place where all things tooling should go. Packaging is not something I consider infrastructure but more so build and/or deployment related. I already am not a big fan of JJB being in the repository :slight_smile:

Lastly, and this point is easy to lose throughout my proposal, the team is currently managing multiple packaging repositories across our community and adjacent communities: Rails 5.1 packaging repository, Foreman packaging repository, Pulp packaging repository and downstream packaging. By creating similar structures and workflows across these, we are able to build up a set of tooling that works across them and thus can be centralized. I believe this leads to keeping this packaging tooling (e.g the Ansible playbooks and binary to call them per the proposal) within it’s own project repository to simplify and allow use across multiple concerns.

All fair points, especially about Foreman packaging. I guess the only place we diverge is that I see foreman-infra as the place we store all our automation, so perhaps the name is slightly off. Still, so long as we clearly document / signpost this new repo so contributors can find it and get involved, I’m good :slight_smile:

My PR to automate updating foreman.spec does indeed belong in another repo, but currently we don’t have that (yet).

Personally I’d really like if we could ship our tooling as a python package on pypi and even package it into RPMs but we’ll have to see how feasible that actually is.

Towards this RFC effort, we’ve done cursory work to build a prioritized list of items to tackle in three phases. For further review, here is that list. If you have other ideas, areas we missed please let us know.

Phase 1

  • Smarter automated scratch build testing on PRs using Github Pull Request Builder (P1)
    • move to Jenkins pipelines
    • Add repoclosure test to PRs (P1)
    • Quick access to Koji build links from PR and thus to scratch builds (P1)
  • Dropping the use of git tags in fpackaging (P1)
  • Auto-release job after PR merge (P1)
    • Handle changelog creation as a result of dropping tags (P1)
  • converting scripts that live in f-packaging into Obal (P1)
    • does this mean everything should be Ansible?
    • helper scripts inside the repository?
  • Obalize fpackaging – add package_manifest.yml (P1)

Phase 2

Phase 3

  • automated packaging PRs when new gems/npms/dependencies are released or updated (e.g. fog updates, passenger updates) (P3)
  • Move to Copr (P3)
  • Host Koji via community hosting service through OSAS (P3)
  • Move packages into sub-directories by concerns (P3)
  • Add client repositories (P3)
  • Add proxy repositories and support more OSes? (P3)
  • Hammer client repository (P3)

For those unfamiliar with Obal: it’s is an effort to make a wrapper around ansible scripts for common operations relating to packaging. It’s being integrated in the pulp packaging workflow as well.

The PR to restructure foreman-packaging rpm/develop has been merged:

This means all open PRs to that branch need a rebase. We’ve modified the scripts in the repo to use new locations but we might have missed something. You could also have your own scripts which might need modifications.

1 Like