Future of debian packaging

Hello,

Michael, who was doing most of the debian packaging work, unfortunately left the project and while there is some information about how to update an existing package there is no documentation on how to create new packages:

I am not familiar with debian packaging at all and I need to get two new dependencies into our repos to finish Redfish feature. While I was able to create the package on my debian system using gem2deb tool and build it, I am not able to get the CI passing using trial and error commits. I guess I am stuck:

Any ideas how to proceed? What is our plan in this regard? Can we reach out to the community to find volunteers with debian packaging expertise?

3 Likes

Iā€™m afraid weā€™re long term unable to keep the same quality for rpm and deb based distros. Iā€™d suggest the following:

  1. letā€™s see if thereā€™s someone in the community, who could help building deb packages and maintain our https://github.com/theforeman/foreman-packaging/tree/deb/develop

  2. if not, based on the last community survey (127 responses), 28.3% of our users have Foreman installed on Debian/Ubuntu. Thereā€™s 54.4% of users, who said ther main OS of hosts is Debian or Ubuntu. It seems even people who mainly use deb based distros can install Foreman on rpm based distro. Therefore Iā€™d propose in the next survey, asking a question whether it would be problem to migrate.

  3. if we donā€™t see blockers, we should think about how to make the transition simple, e.g. by providing some tooling. IMHO migrate Foreman web app is not that complex, given most of the state is stored in the PG database. The hardest part would probably be migrating smart proxies (all tftp content, dhcp/dns/puppet deployment). This of course requires a bit deeper analysis.

Please donā€™t take it as Iā€™m suggesting dropping deb packages right-away, perhaps weā€™ll find someone who will take care of it and we stop at step 1. However I think it would be better to admit, our deb installations suffer from several issues

  1. we no longer have enough experts in this area
  2. we package Foreman differently there, dependencies are in a big bundle so to update e.g. fog, we need to release a new minor version of Foreman
  3. smart_proxy_dynflow is deployed differently, leading to problems usually in REX/ansible that we donā€™t have capacity to fix
  4. plugins like katello, virt_who_configure, openscap are not available in deb repos at all

Also just to be clear, Iā€™m not saying we should limit deb based distros on client machines (hosts) in any way. Client repo should remain.

Ironically, today that doesnā€™t exist.

Ok, client repo should be created and maintained instead :slight_smile:

I think dropping Debian packaging should be a very, very last resort solution. And Iā€™d do everything to avoid that.

Why?

  • Debian/Ubuntu usually provides newer Ruby versions quicker than CentOS, which allows us to ship actual releases using these Ruby versions earlier and thus possibly finding and fixing bugs quicker.
  • Shipping on two completely different targets forces us to architect the app in a more modular and more standards-compliant way.
  • We actually have plugins today, that are only shipped on Debian (foreman_datacenter, I am planing to change that, but itā€™s not done yet).
  • Providing Debian packages eases the entry of primarily Debian-using users into the community.

At the same time, I completely agree with yā€™all, weā€™re today lacking people who would actively maintain the packaging and the packaging is more complicated than it should be.

I have an outstanding todo-item to talk to a few people, who offered help with the former, and also have a few ideas how to fix the later. So please, donā€™t ditch it just yet :wink:

1 Like

Thank you, @evgeni, for your extensive help with getting my Debian packages through the pipeline. I learned some things, Iā€™ve updated the README a bit in the process, hopefully our know-how is better now.

I think we can keep the status-quo for now and survive until we implement container-based deployment some day? :slight_smile:

Is this still true with CentOS 8 or CentOS Stream?

Do you have examples of this you could provide? Iā€™m curious what it is about Debian vs EL that requires a more modular deployment. Also curious, what about having two different OSes enforces the standards-compliant way mentioned? (That say a pure CentOS 7 and 8 install target would not)

This part is directed at the entire thread.

Playing devilā€™s advocate a moment to examine the trade-offs in the other direction as well. The argument one might make for focusing on a limited set of deployment methods is that it allows optimization and focus on those deployment methods, reduces testing matrices and time spent on variance. Also, reduces the number of developers require to maintain them thereby freeing them up to focus on user facing features and provide more value to the community. The current majority of developers are familiar with EL platforms, deploy their development environments on them thereby making the majority of our expertise centered around said platform.

Given the entire Katello user community deploys the server on EL, I am curious how often we get requests to provide it on Debian or if users accept that they must deploy on EL and do so.

Other questions that come to mind:

  • What if any benefit would Katello have for providing deployment on Debian support given the amount of work and required dependencies?
  • With container talk present again in the community, would we further increase our matrix by having EL and Debian containers alongside EL and Debian packages?
  • Is there a disservice to the community by providing a supported OS that is only a subset of the community features? Do users install on Debian and then realize they have to start over because they want features not available on Debian?

Even with the recent progress on containers, this wonā€™t be possible soon. Do you accept the status quo for another e.g. 2 years? Or would you help other contributors to package their stuff for deb now? Iā€™m happy the issue got resolved for you, however other devs will face the same issue in future again. I think we need to find someone who will own this part, otherwise we should proceed with step 2 - put the migration question to the community survey.

As we have a very diverse group of customers it comes up very often (also for SLES), but most are fine using CentOS at the end. I think I had only two project which were canceled or changed because of this limitation in all the years.

I donā€™t see Ruby 2.7 on CentOS 8, and have no idea how to access Stream. There is an 2.7 SCL for CentOS 7, tho.

Debian forces us to jump to the new Ruby (Bullseye will have 2.7), which I quite like :wink:

Also, Debian is usually quick with new PostgreSQL versions, so we have that tested there ā€œfor freeā€, before switching it on CentOS.

One example is the way we handle different package names for plugins. Having this for Debian already, I think it enabled us to implement it for CentOS 8 quicker, as we just had to look at all places that distinguished between CentOS and Debian, and add another elif. Compared to the current efforts to get Katello on EL8ā€¦ (Granted, once we had EL7 and EL8, the ā€œbenefitā€ of having Debian was reduced for that particular exercise).

While most developers deploy their setups on EL, I wouldnā€™t go as far as saying they are familiar with EL packaging (and this is what this thread is about, no offense dear devs!). What is true is that our tools and CI is just so much better for RPM, and this the developers donā€™t need to be familiar with packaging.

  • Growing user base ā€“ while I agree with Dirk, paying customers (who buy support anyways) wonā€™t care for the odd EL box in their Debian fleet, I think for a pure community user having it available on ā€œtheirā€ distribution is a win. Whether this justifies the huge invest weā€™d need to do is a different question (Iā€™m leaning to ā€œnoā€)
  • No, we should have only one container base (EL8, probably). Users donā€™t have to dig inside the container, contrary to a normal OS.
  • This sounds like our docs need a ā€œplanning a deploymentā€ section, which discusses why one would chose which base.

Ruby 2.7 was released in RHEL Appstream with 8.3 which has not landed in CentOS so there is only 2.6 at the moment.

# dnf module list ruby
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:10:26 ago on Tue Dec  1 14:05:52 2020.
CentOS-8 - AppStream
Name  Stream   Profiles    Summary                                              
ruby  2.5 [d]  common [d]  An interpreter of object-oriented scripting language 
ruby  2.6      common [d]  An interpreter of object-oriented scripting language 

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled
1 Like

On the other hand, wouldnā€™t it be nice if we could have a standard naming scheme for packages? (e.g. foreman-plugin-katello, foreman-plugin-remote-execution) This would allow less mental overhead when trying to think of what package you need to install based on what platform you are on.

Delightful idea.

We have these provides for RPMs (for ages) but we never did the same thing for debs.

I still would like to support an OS thatā€™s on a higher release cadence than RHEL. Another way to achieve that is to support Fedora again. RHEL is such a slow moving target that it encourages bad behaviors, like staying on old versions.

For example, if we could drop EL7 today, we would be able to rely count on HTTP2 being supported. We could experiment with mpm_event as an Apache worker. All those things should improve experience by being more responsive.

We could add that to debs, but does Puppet support installing by ā€œprovidesā€? Last time I looked the answer was: no.

This feature was released in Puppet 3.7.0:

That added support in the yum package provider. Then this was released in Puppet 6.11.0:

Now that Puppet 5 is EOL, I think this gives us a good reason to bump up the minimum version to 6.11.0 if we want to use this.

Sounds good!

Note that even today we donā€™t have the same provides on Hammer packages (not RPMs nor debs). That is something that needs to be done first. We do have foreman-proxy-plugin-%{plugin_name}. I do suspect that this is still broken for plugins with underscores since Debian doesnā€™t allow that.

Regardless of whether CentOS 8 Stream will have 2.7 soon or not, I donā€™t think this is a good process for us to update our dependencies. I think we should instead create a new process. After every branching, developers should allocate some time to update ruby on their dev setups and fix all issues we find. Letā€™s also upgrade to the newest Rails, letā€™s update all fog adapters etc. We should formalize how we delete our Gemfile.lock and update packaging repo accordingly for the next upcoming release.

Relying on the distro pushing us to upgrade should be the very, very last resort solution :slight_smile: I know we rely on system ruby on deb distros, but devs should be testing with newer version, before it hits stable distributions.

Previously it was usually Michael who added new Ruby versions to Jenkins because Ubuntu moved to a newer version. Much longer ago it was Dominic who added versions to support newer versions on Fedora. Traditionally distributions have driven updates. If we only support EL, Iā€™m afraid we would stagnate.

There are two parts of this. One - developers should update their dev setups regularly, so we stabilize new versions as part of regular development. Two - we need to reflect that in our pipelines. I understand Ubuntu previously forced us to work on the second part. But the point is, we could do better and should not rely on any distro to force us. We should be a step ahead and know the Foreman works with the newer versions. Therefore updating Jenkins would not cause any changes in the codebase. So what I propose is formalize our update process instead. This definitely shouldnā€™t be the strongest argument for keeping debian builds.