This is a proposal for taking our packaging repository to the next level of tooling and automation.
Background
This proposal is inspired by the work that my team has been doing and refining across downstream packaging, the Rails 5.1 SCL packaging and within the Pulp community. We’ve had great success with managing many packages, automating the addition of new packages, updates, test automation and release automation. We believe this can greatly benefit foreman packaging and help to unify the various packaging efforts that are on going in related communities.
The proposal will in general reference the RPM side of things as that is the aspect the team has worked with the most and crafted these tooling and workflows with. However, that does not mean this is limited to RPMs. What lies below should be applicable to Debian packaging as well and more depends on the input from the Debian packagers.
Proposal
The bulk of this proposal revolves around a move to using Ansible for tooling to provide scripted tasks and easy CI integration. A majority of this tooling is already written and used in other repositories that my team manages and we are working on putting them into a single shared repository to allow re-use in foreman-packaging as well. The following is a breakdown of the various parts of the proposal and changes therein.
Ansible Playbooks
Ansible provides an easy to understand and reference DSL for performing tasks that touch a variety of concerns. Further, the framework for orchestrating and organizing a set of tasks makes it a good use case for building out release functionality. As I mentioned, the team has built a set of tasks for scratch builds, releases, adding and updating packages that are already in use in multiple packaging repositories. The functionality exists and is ready to be used to make both maintainers and developers lives easier when performing rote packaging tasks.
Another goal of this proposal, mentioned below, is a heavier use of CI. Ansible playbooks provide a nice clean, easy to follow output along with additional behaviors like performance output. This allows us to write complex tasks in a single language and run it either locally or create lightweight CI jobs that have little to no logic other than calling the same playbooks a developer would. This makes debugging or performing actions manually much less burdensome.
Ansible Inventory
When first learning Ansible, the documentation talks about managing hosts and the state of them. The inventory is that of a set of hosts. However, the Ansible inventory is not restricted to hosts. The inventory is just collections of things grouped together in a user defined logical manner that can have tasks performed on them. This realization has led us to use an Ansible inventory to define the set of packages and attributes about them as variables. This makes adding new packages, updating existing and releasing packages easy as you can perform actions across the inventory or by limiting the set of packages. This has so produced great results so far.
The proposal here is then to add (what we call a package_manifest.yml) that serves as an Ansible inventory and the playbooks act upon. This inventory would contain packages organized by repository. This is similar to the per-tag configuration found in rel-eng/tito.props but gives the power to use Ansible directly as intended without having to write a similar system in ruby, bash or more Ansible.
Stop using Git Tags
The use of tito with git tagging has had some usefulness but places a burden on maintainers and CI systems. The workflow today requires that after merging a packaging update a maintainer must then use tito tag to create both a new commit and tag on the mainline branch. These both must then be pushed up to the remote branch before being released. This presents both overhead and makes creating automated release pipelines cumbersome as they would have to modify the codebase. Further, the tagging structure provides no real benefit in the context of the packaging repository. Originally the tito project created tagging with the idea of using a spec file inside of the project source code and thereby capturing the git history.
CI/CD
The previous work leads toward this final aspect of the proposal which is to add significant gains in continuous integration and delivery. The Foreman, plugins and Katello require maintaining a large number of packages. The actions to maintain those are fairly routine: add a new package, update a package, release a new version of a package. A human should not be required to complete these actions as they are definable. A human should interact where it’s important by reviewing pull request code, and approving. After that, automation should be able to handle the rest.
I propose to create (or update) the following CI jobs:
- Update the pull request scratch build job
- Create a package release job
- Add a job that examines and reports on changes to Foreman’s Gemfile.lock
- Add a job that examines and reports on changes to Foreman’s package.json
These jobs would be written:
- Using Jenkins pipelines
- “Code” written in Ansible playbooks and executed by Jenkins
Finally, the goal with this proposal is to reduce the packaging burden on maintainers by creating maintainable code for performing packaging release work and using that to automate the jobs that keep packagers busy but are routine. Further, the hope is this should help us also move faster and keep dependencies up to date more often. Please raise questions, comments and concerns you have so that we may begin putting this proposal into action.
Thanks,
Eric
from me.