Future of debian packaging

This feature was released in Puppet 3.7.0:

That added support in the yum package provider. Then this was released in Puppet 6.11.0:

Now that Puppet 5 is EOL, I think this gives us a good reason to bump up the minimum version to 6.11.0 if we want to use this.

Sounds good!

Note that even today we don’t have the same provides on Hammer packages (not RPMs nor debs). That is something that needs to be done first. We do have foreman-proxy-plugin-%{plugin_name}. I do suspect that this is still broken for plugins with underscores since Debian doesn’t allow that.

Regardless of whether CentOS 8 Stream will have 2.7 soon or not, I don’t think this is a good process for us to update our dependencies. I think we should instead create a new process. After every branching, developers should allocate some time to update ruby on their dev setups and fix all issues we find. Let’s also upgrade to the newest Rails, let’s update all fog adapters etc. We should formalize how we delete our Gemfile.lock and update packaging repo accordingly for the next upcoming release.

Relying on the distro pushing us to upgrade should be the very, very last resort solution :slight_smile: I know we rely on system ruby on deb distros, but devs should be testing with newer version, before it hits stable distributions.

Previously it was usually Michael who added new Ruby versions to Jenkins because Ubuntu moved to a newer version. Much longer ago it was Dominic who added versions to support newer versions on Fedora. Traditionally distributions have driven updates. If we only support EL, I’m afraid we would stagnate.

There are two parts of this. One - developers should update their dev setups regularly, so we stabilize new versions as part of regular development. Two - we need to reflect that in our pipelines. I understand Ubuntu previously forced us to work on the second part. But the point is, we could do better and should not rely on any distro to force us. We should be a step ahead and know the Foreman works with the newer versions. Therefore updating Jenkins would not cause any changes in the codebase. So what I propose is formalize our update process instead. This definitely shouldn’t be the strongest argument for keeping debian builds.

I agree that it’s not the strongest argument, but I haven’t seen developers push for it. I wouldn’t mind seeing that happen though.

There is still a possibility, not advertised tho, to install from gems. I wonder if it makes sense to drop debian packaging in favor of gem installation so we would actually cover more non-RH OSes than before. This would open door to testing on rolling-based distros like Gentoo/Arch where the most fresh software resides if we want to really live on the edge.

Just a thought tho, it’s a lot of work I guess.

Note that Debian packaging for Foreman itself is already bundling all gems. Only foreman-proxy and foreman-installer use system gems. This is the reason the foreman.deb is so huge and builds so slow. Also why you need to do a lot less work to keep dependencies up to date and why Michael often sent in PRs to limit it to some newer versions: regular breakages in newer gem versions broke releases. Because RPM packages were pinned, they didn’t suffer from this.

On Monday, @Dirk, @Baboon and myself had a short chat how Netways can help out (it terms of human power) with Debian packaging.

The current plan is that @Baboon will join me in reviewing and improving our packaging, especially:

  • semi-regular review sessions to avoid situations where PRs linger around too long and contributors are annoyed (sorry lzap!)
  • reducing “time to package”: simplify adding new dependencies and plugins by providing templates and helpers (similar to what we do for RPM today)
  • increasing test reporting: today it’s rather hard to see why the CI failed, we need to make it better approachable by the users

We’ll start with the first two points ASAP, and hope to get to number three after that.

8 Likes

Great news! Once you all feel more comfortable, is there a chance you’d be able to take over the ansible-runner builds too? That’s something we currently struggle with in foreman_ansible.

This was on my agenda similar to having a client repo containing openscap, subscription-manager, …
But like Evgeni said let them start with the simpler things.

I hope to add more human power in the future because we hope to have new trainees every year and when they have been taught the basics of packaging they need some practice so it is a win-win situation.

2 Likes

This is good news. With CentOS switching everyone to streams, this may leave a lot of users switching to Ubuntu which may drive interest in deb packages for foreman.

Thanks guys, please keep en eye on improving documentation so it’s up to date.

Hello guys,

I’m also interested in getting Debian packaging improved, especially for AzureRM plugins.
I’m a Debian contributor so I’m quite experienced with Debian packaging but not at all with Ruby ecosystem.
Right now I’m struggling understanding how I can build a plugins package on my machine because folders being present in GIT repo does not have any sources so ofc package build is failling immediately because of missing files:

acecile@lattitude:~/dev/github/theforeman/foreman-packaging/plugins/ruby-foreman-puppet$ fakeroot debian/rules build
cp cache/* /usr/share/foreman/vendor/cache/
cp: cannot stat 'cache/*': No such file or directory
make: *** [debian/rules:15: build] Error 1
1 Like

I’ve seen quite an activity on the IRC channel, so I assume you are looking into this with @evgeni ?

Yes, people on IRC have been really helpful and I managed to open a PR adding azure plugins packaging.
The package a’has ben confirmed working on a my production 2.3.3 server

3 Likes

I wanted to draw some attention to some updates in this area over the past few months. Improvements we’ve made:

  • automatic Debian builds on PR merge – Debian CI Jobs
  • speedier Debian builds – no link, but @evgeni did various improvements within packaging and CI to reduce build times (in one case I believe from 2 hours to < 30 minutes)
3 Likes

Yes, 2h → 30 min (build code improvements) → 15 min (build infra improvements)

5 Likes

We’ll take it. :slight_smile: