Michael, who was doing most of the debian packaging work, unfortunately left the project and while there is some information about how to update an existing package there is no documentation on how to create new packages:
I am not familiar with debian packaging at all and I need to get two new dependencies into our repos to finish Redfish feature. While I was able to create the package on my debian system using gem2deb tool and build it, I am not able to get the CI passing using trial and error commits. I guess I am stuck:
Any ideas how to proceed? What is our plan in this regard? Can we reach out to the community to find volunteers with debian packaging expertise?
if not, based on the last community survey (127 responses), 28.3% of our users have Foreman installed on Debian/Ubuntu. Thereās 54.4% of users, who said ther main OS of hosts is Debian or Ubuntu. It seems even people who mainly use deb based distros can install Foreman on rpm based distro. Therefore Iād propose in the next survey, asking a question whether it would be problem to migrate.
if we donāt see blockers, we should think about how to make the transition simple, e.g. by providing some tooling. IMHO migrate Foreman web app is not that complex, given most of the state is stored in the PG database. The hardest part would probably be migrating smart proxies (all tftp content, dhcp/dns/puppet deployment). This of course requires a bit deeper analysis.
Please donāt take it as Iām suggesting dropping deb packages right-away, perhaps weāll find someone who will take care of it and we stop at step 1. However I think it would be better to admit, our deb installations suffer from several issues
we no longer have enough experts in this area
we package Foreman differently there, dependencies are in a big bundle so to update e.g. fog, we need to release a new minor version of Foreman
smart_proxy_dynflow is deployed differently, leading to problems usually in REX/ansible that we donāt have capacity to fix
plugins like katello, virt_who_configure, openscap are not available in deb repos at all
Also just to be clear, Iām not saying we should limit deb based distros on client machines (hosts) in any way. Client repo should remain.
I think dropping Debian packaging should be a very, very last resort solution. And Iād do everything to avoid that.
Why?
Debian/Ubuntu usually provides newer Ruby versions quicker than CentOS, which allows us to ship actual releases using these Ruby versions earlier and thus possibly finding and fixing bugs quicker.
Shipping on two completely different targets forces us to architect the app in a more modular and more standards-compliant way.
We actually have plugins today, that are only shipped on Debian (foreman_datacenter, I am planing to change that, but itās not done yet).
Providing Debian packages eases the entry of primarily Debian-using users into the community.
At the same time, I completely agree with yāall, weāre today lacking people who would actively maintain the packaging and the packaging is more complicated than it should be.
I have an outstanding todo-item to talk to a few people, who offered help with the former, and also have a few ideas how to fix the later. So please, donāt ditch it just yet
Thank you, @evgeni, for your extensive help with getting my Debian packages through the pipeline. I learned some things, Iāve updated the README a bit in the process, hopefully our know-how is better now.
I think we can keep the status-quo for now and survive until we implement container-based deployment some day?
Is this still true with CentOS 8 or CentOS Stream?
Do you have examples of this you could provide? Iām curious what it is about Debian vs EL that requires a more modular deployment. Also curious, what about having two different OSes enforces the standards-compliant way mentioned? (That say a pure CentOS 7 and 8 install target would not)
This part is directed at the entire thread.
Playing devilās advocate a moment to examine the trade-offs in the other direction as well. The argument one might make for focusing on a limited set of deployment methods is that it allows optimization and focus on those deployment methods, reduces testing matrices and time spent on variance. Also, reduces the number of developers require to maintain them thereby freeing them up to focus on user facing features and provide more value to the community. The current majority of developers are familiar with EL platforms, deploy their development environments on them thereby making the majority of our expertise centered around said platform.
Given the entire Katello user community deploys the server on EL, I am curious how often we get requests to provide it on Debian or if users accept that they must deploy on EL and do so.
Other questions that come to mind:
What if any benefit would Katello have for providing deployment on Debian support given the amount of work and required dependencies?
With container talk present again in the community, would we further increase our matrix by having EL and Debian containers alongside EL and Debian packages?
Is there a disservice to the community by providing a supported OS that is only a subset of the community features? Do users install on Debian and then realize they have to start over because they want features not available on Debian?
Even with the recent progress on containers, this wonāt be possible soon. Do you accept the status quo for another e.g. 2 years? Or would you help other contributors to package their stuff for deb now? Iām happy the issue got resolved for you, however other devs will face the same issue in future again. I think we need to find someone who will own this part, otherwise we should proceed with step 2 - put the migration question to the community survey.
As we have a very diverse group of customers it comes up very often (also for SLES), but most are fine using CentOS at the end. I think I had only two project which were canceled or changed because of this limitation in all the years.
I donāt see Ruby 2.7 on CentOS 8, and have no idea how to access Stream. There is an 2.7 SCL for CentOS 7, tho.
Debian forces us to jump to the new Ruby (Bullseye will have 2.7), which I quite like
Also, Debian is usually quick with new PostgreSQL versions, so we have that tested there āfor freeā, before switching it on CentOS.
One example is the way we handle different package names for plugins. Having this for Debian already, I think it enabled us to implement it for CentOS 8 quicker, as we just had to look at all places that distinguished between CentOS and Debian, and add another elif. Compared to the current efforts to get Katello on EL8ā¦ (Granted, once we had EL7 and EL8, the ābenefitā of having Debian was reduced for that particular exercise).
While most developers deploy their setups on EL, I wouldnāt go as far as saying they are familiar with EL packaging (and this is what this thread is about, no offense dear devs!). What is true is that our tools and CI is just so much better for RPM, and this the developers donāt need to be familiar with packaging.
Growing user base ā while I agree with Dirk, paying customers (who buy support anyways) wonāt care for the odd EL box in their Debian fleet, I think for a pure community user having it available on ātheirā distribution is a win. Whether this justifies the huge invest weād need to do is a different question (Iām leaning to ānoā)
No, we should have only one container base (EL8, probably). Users donāt have to dig inside the container, contrary to a normal OS.
This sounds like our docs need a āplanning a deploymentā section, which discusses why one would chose which base.
Ruby 2.7 was released in RHEL Appstream with 8.3 which has not landed in CentOS so there is only 2.6 at the moment.
# dnf module list ruby
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:10:26 ago on Tue Dec 1 14:05:52 2020.
CentOS-8 - AppStream
Name Stream Profiles Summary
ruby 2.5 [d] common [d] An interpreter of object-oriented scripting language
ruby 2.6 common [d] An interpreter of object-oriented scripting language
Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled
On the other hand, wouldnāt it be nice if we could have a standard naming scheme for packages? (e.g. foreman-plugin-katello, foreman-plugin-remote-execution) This would allow less mental overhead when trying to think of what package you need to install based on what platform you are on.
We have these provides for RPMs (for ages) but we never did the same thing for debs.
I still would like to support an OS thatās on a higher release cadence than RHEL. Another way to achieve that is to support Fedora again. RHEL is such a slow moving target that it encourages bad behaviors, like staying on old versions.
For example, if we could drop EL7 today, we would be able to rely count on HTTP2 being supported. We could experiment with mpm_event as an Apache worker. All those things should improve experience by being more responsive.
Note that even today we donāt have the same provides on Hammer packages (not RPMs nor debs). That is something that needs to be done first. We do have foreman-proxy-plugin-%{plugin_name}. I do suspect that this is still broken for plugins with underscores since Debian doesnāt allow that.
Regardless of whether CentOS 8 Stream will have 2.7 soon or not, I donāt think this is a good process for us to update our dependencies. I think we should instead create a new process. After every branching, developers should allocate some time to update ruby on their dev setups and fix all issues we find. Letās also upgrade to the newest Rails, letās update all fog adapters etc. We should formalize how we delete our Gemfile.lock and update packaging repo accordingly for the next upcoming release.
Relying on the distro pushing us to upgrade should be the very, very last resort solution I know we rely on system ruby on deb distros, but devs should be testing with newer version, before it hits stable distributions.
Previously it was usually Michael who added new Ruby versions to Jenkins because Ubuntu moved to a newer version. Much longer ago it was Dominic who added versions to support newer versions on Fedora. Traditionally distributions have driven updates. If we only support EL, Iām afraid we would stagnate.
There are two parts of this. One - developers should update their dev setups regularly, so we stabilize new versions as part of regular development. Two - we need to reflect that in our pipelines. I understand Ubuntu previously forced us to work on the second part. But the point is, we could do better and should not rely on any distro to force us. We should be a step ahead and know the Foreman works with the newer versions. Therefore updating Jenkins would not cause any changes in the codebase. So what I propose is formalize our update process instead. This definitely shouldnāt be the strongest argument for keeping debian builds.
I know we rely on system ruby on deb distros, but devs should be testing with newer version, before it hits stable distributions.