With Rails 6 around the corner, it is time to start planning our upgrade to it. In the past major Rails version upgrades took a lot of effort, and there are other considerations as well that are needed to be taken into account. During the meeting in Brno, we had a small working group (@kgaikwad, @shiramax, @evgeni and myself) to try and map the steps needed to get to Rails 6. Below are the outcomes of our discussion, feel free to chime in with your opinions and suggestions.
Rails 6 has two requirements which also need to be taken into consideration:
The minimal supported Ruby version is 2.5. This means that when we move to Rails 6 we will need to drop support for Debian Stretch and Ubuntu Xenial which ship with Ruby 2.3.
The minimal supported PostgreSQL version is 9.3. CentOS 7 only ships with 9.2, which means we will need to either migrate to an SCL version of PostgreSQL for CentOS 7 or drop support for it and require CentOS 8 as a prerequisite (which ships version 10 by default). We will likely need to upgrade PostgreSQL version in any case during this timeframe to support Pulp 3 in Katello - as it requires PostgreSQL 9.6 at least.
We have outlined the following proposed stages to get migrated to Rails 6 in a gradual manner so we can avoid having a huge PR with all changes that will need to be constantly rebased:
We are currently running on Rails 5.2.1, while the latest version is 5.2.3. During the 1.23 time-frame we should be upgrading to the latest 5.2 version so we donāt need to fix changes needed for it while working on the 6 upgrade. The tasks needed to be done for this are tracked in https://projects.theforeman.org/issues/25601 (thanks @mmoll!).
In 1.24 we should get all of the backwards-compatible changes that are needed for Rails 6 compatibility, and ideally with as many backward-incompatible changes already included behind guard conditions. We will use a similar setting as we did for the rails 5.2 upgrade to allow for working with both versions. The tasks needed for Rails 6 compatibility are tracked at Tracker #24837: Rails 6.0 Tracker - Foreman (thanks again @mmoll!).
Once initial changes are ready, we will enable testing with Rails 6 on CI without making it fail tests, to ensure no new Rails 6 incompatible changes are added and indicate the progress on this effort.
As we identify the needed changes, we will create a guide for plugin authors indicating steps needed to ensure compatibility.
During this time frame we should also look into the steps needed for upgrading to PostgreSQL 10 using SCL on Centos 7. Some insights can be gained from the process done to migrate MongoDB from system default to SCL for katello. We thought if we are going to need scl anyways, it would be beneficial to standardize on the same version that will be in Centos 8 and is already in all Debian-based distros. PostgreSQL 10 also offers some significant improvements with regards to performance.
1.25 will ship with only Rails 6 enabled. This means that the supported distros will be Centos 7 and 8, Debian Buster (10), which should be released by then, and Ubuntu Bionic (1804). If all goes well, 1.25 should ship around February 2020.
Additional considerations:
Is it reasonable to expect debian stretch and ubuntu xenial users to be able to upgrade their base OS in this time frame (about 9 months)? If not, should we postpone the upgrade to allow a longer timeframe for upgrades?
What is the supported life of the PostgreSQL 10 SCL? Will it be long enough until we can completely drop CentOS 7 support or will we be stuck with an unsupported SCL or have to build our own?
Would it be better to not try supporting Rails 6 on CentOS 7? This would save the effort of having to handle upgrades from system packages to SCL during the upgrade. On the other hand, this may delay the Rails 6 upgrade, require dropping CentOS 7 support earlier, and/or delay Pulp 3 adoption in Katello.
We will need to make sure Candlepin can also work with PG 10.
Who is going to be involved in this effort? I agreed to coordinate it, but this will likely require help from more developers, including at least one from the @packaging team.
What did we miss? There are certainly other things we did not think of that may affect the proposed plan.
Ugly, but it may be an option to only support CentOS 7 with a remote DB that is new enough. That can be a CentOS 8 server giving you some time to migrate the application server. Not sure we want to go there though given itāll be very complicated for us to test.
Also good to note that at this point we donāt know when CentOS 8 will be released (like Debian 10.
I also would like to deprecate running on Debian 16.04 in the 1.23 release, even if we only drop it with 1.25. This should set the right expectations on users to upgrade to 18.04 which is already a year old at the time of writing so they can start planning We canāt deprecate Debian 9 if thereās no newer release yet though.
Makes sense to deprecate Xenial already, opened PR to the nightly manual. I hope that both Centos 8 and Debian 10 are released within the next month or two so we can agree on when is reasonable to drop Centos 7 and Debian 9.
Switching to CentOS8-only without supporting CentOS7 postgresql SCL is pretty hard as we still donāt know when CentOS8 will be released. Of course, we donāt know āhow stableā CentOS8 is, too.
I guess, there is more work to do to run foreman + katello on CentOS8. Therefore, I would recommend to let foreman + katello run on CentOS7 for at least this year. Are there any project guidelines / policies regarding this?
I think we learned from the ruby SCL, that theyāre horribly maintained. I would try to avoid more SCL software at any cost. Can we just ship a new database version in a container?
One aspect of this discussion I am missing is an outline of why we need or want to upgrade to Rails 6.0 other than itās ānewerā. There are a lot of considerations that have to be made to upgrade, and generally .0 releases are the least stable in a given release stream. What are the benefits to Foreman we forsee with Rails 6 that help to outweigh the outlined costs?
While I like your attitude and I would love to wait until .1 I am bit afraid that Rails is such a fast moving target that quickly after 6.1 there will be Rails 7.0. Skipping a Rails release might be problematic for the future - upstream seem to drop old releases pretty quickly.
However delaying it a bit might not hurt, as well as making necessary compatible Rails 6 changes in advance.
5.0 was released on June 30, 2016
5.1 was released on April 27, 2017
5.2 was released on April 9, 2018
This means itās typically a year between Y-releases and at the current rate 3 or so years between X.0 releases with Rails 6 still being in release candidate phase.
Unfortunately, thatās not true. The ruby SCL hasnāt been updated in quite a while even though there are known bugs. How hard can it be to bump the point release of the ruby version?
This is certainly possible, we donāt have to deprecate CentOS 7 right away once we support CentOS 8. This still comes with some cost as we would need to be building packages for both archs and handle the various differences between the two.
That said, supporting CentOS 7 for longer means we will need to figure out some solution for a newer Postgres version, either using SCL or external DB (or some other way), not only for supporting this effort but also for Pulp 3 support (which requires at least version 9.6).
The main goal of starting this discussion now is to prepare in advance for the upgrade, as well as lower the cost of jumping multiple versions as weāve done in the past when upgrading major rails version (e.g. 3.2->4.1, 4.2->5.1). It is possible to argue it is better to skip the .0 release and go straight to .1, but these upgrade have taken quite a significant effort in the past due to the amount of changes needed for both versions, so it does not come with no cost.
Once Rails 6.0 is out (which we expect to be very soon), 5.2 will no longer receive bug fixes, and once 6.1 is out (which looking at the history you gave would be in about a year), 5.2 will only receive significant security fixes, meaning by that time we should preferably migrate to a newer version. The timeline Iāve outlined hopefully gets us on 6.0 in about 9 months.
We can decide to stretch this effort for longer or even skip the 6.0 release completely, but if the reason to do so is avoiding the PG upgrade, this might also mean we need to postpone Pulp 3 support for longer.
But thatās not how SCLs work - they follow the same workflow as in RHEL. Packages are not rebased until there is a bug reported via Bugzilla or there is a security issue that gets fixed by security response team.
Letās drop discussing subjectively if this is good or bad, I would just like to know - are there any bugs that were relevant to Foreman? Why we did not report them to the SCL team?
I did. If this were an open source project, Iād consider it unmaintained.
The point Iām trying to make is: We need a solution where we can control the versions being used.
Let me correct: looks like SCL team is regularly rebasing Ruby in Red Hat SCLs (rh-ruby*) so this should be āautomaticā. But filing a BZ wonāt hurt, itās definitely not every single minor release.
With the concerns raised here and with 1.23 branching in two weeks, I think the most reasonable course of action going forward would be to postpone my proposed timeline by one release.
This will allow us a bit longer to adapt our processes for CentOS 8 and Debian 10 once theyāre released and give users a bit longer time to plan their underlying OS upgrades.
The remaining question is regarding supporting a newer PostgreSQL, which we may want to do sooner to allow using Pulp 3 in Katello. There are two options I can see in that regard, but there may be others:
Decide that pulp 3 will only be supported on CentOS 8. This will save the effort of having to handle migration from system to SCL version of Postgres in CentOS 7, with the downside that it may be longer before we have a Katello release that can use leverage Pulp 3.
Decide to migrate to the SCL version of Postgres for CentOS 7. This will require additional effort to make sure that we change usersā existing setups and changes needed in packaging and installer to use the SCL version, but will allow us to start working with Pulp 3 sooner rather than later.
@Justin_Sherrill and @Partha_Aji - since you have been working on pulp 3 support recently, would be good to get some feedback from you (or anyone else from @katello team) regarding what would be better.
Iād still like to highlight that all the tasks blocking #24837 and #25601 would be fine to be implemented with Rails 5.2 right now and would help with every Rails update (and possibly other dependencies) in the future.
Another important topic would be the removal of Turbolinks (#23449 and for background: #20961) or alternatively an update to Turbolinks 5, which hopefully can get circumvented.
With 1.23 already branched, it would be very good to work on getting 1.24 ready for upgrading to the latest rails 5.2 release, and start looking into merging backward-compatible changes required for rails 6.
Support for parallel testing seems like it could be a great benefit, considering how long our tests take to run. And naturally Iām interested in what Webpacker means for us, too (should/can we get rid of the Foreman gem?)
We canāt deprecate Debian 9 if thereās no newer release yet though.