Foreman 2.0 branching

According to Foreman <strike>1.25</strike> 2.0 schedule and planning, we should be branching for foreman 2.0 in two weeks from now, meaning next week we will be the stabilization week
There are multiple infrastructure changes we wanted to incorporate in 2.0, and I’m concerned some of them might not be ready in time and may risk our ability to branch on time.

Some highlights include (feel free to add more in a comment if I missed something significant):

  • Moving Proxy to SCL
  • Switching default Foreman web server to Puma
  • Changing Foreman to use PostgreSQL 10 from SCL
  • Rails 6 upgrade
  • Changing default Dynflow implementation to sidekiq+redis based
  • Pulp 3 support for File and Docker content in Katello

Given where we are, there are several options we can take:

  1. postpone branching by a couple of weeks in the hope that this will be enough to finish merging everything into nightlies
  2. branch on schedule but elongate the RC phase and allow cherry-picking into the stable branch of commits related to these changes
  3. decide to postpone some of these (which?) to 2.1 and focus on the others to get them in on time.

I would appreciate getting input from others (@core specifically but not just) before making a decision on one of these options.
Developers involved in these change, please also share the status of the change you are working on currently and if you feel it could be merged this week, if two more weeks will be enough to get it in, or if you prefer not to risk the 2.0 release and push it to the next release so it has more time to mature in nightlies.

To give some insight of where the mentioned highlights are at:

I found an issue where the assets are served via Puma instead of by Apache itself. This will mean a higher load on the application server.

Another issue is that external authentication via Apache mods (Kerberos, Keycloak) will probably break because they use environment variables which I don’t think you can use with a reverse proxy.

Since this hasn’t landed in nightly I’m hesitant to switching the default and I prefer to focus on other tasks. The support is still there and users can switch by using --foreman-passenger false as an installer option.

This will depend on the PostgreSQL 10 SCL change above. We also need to update PostgreSQL on our CI:

Add ruby-gitlab-sidekiq-fetcher by adamruzicka · Pull Request #4196 · theforeman/foreman-packaging · GitHub (deb)
https://github.com/theforeman/foreman-packaging/pull/4230 (deb)
Fixes #28421 - drop dynflowd service by ezr-ondrej · Pull Request #4442 · theforeman/foreman-packaging · GitHub (rpm)
https://github.com/theforeman/puppet-foreman/pull/761
Fixes #28525: Install redis from SCL to prevent selinux issue by ehelms · Pull Request #434 · theforeman/foreman-installer · GitHub

Experimental support is present in the installer, but it’s missing various things. I’ve created Tracker #28736: Use Pulp 3 for File and Container content in Katello - Installer - Foreman which is blocked by:

  • Bug #28711: /pulp/content is not being served over http with pulp3
  • Bug #28696: with pulp3 installed on a main katello server, apache should be configured to help serve docker content
  • Bug #28655: support fetching files via /pulp/isos with pulp3
  • Feature #28654: support client cert auth with pulp3

There’s also Feature #28695: Support pulp3 upgrade/migration - Installer - Foreman.

1 Like

Not on the list, but work by @ehelms is the seed refactor.

(missing is the db:seed removal from foreman-installer in the katello hooks)

I think we tried this approach in the past and we kept postponing the branching, since it was never good enough, meanwhile more things got in and needed stabilization

I’d prefer this option. But if we take it, we need to have “second deadline” for these features and be prepared to stop accepting changes unless they are absolutely critical. Even if it means the feature is not perfect.

That’s the easiest and the cleanest solution, but if we don’t do these critical changes now, we may end up in a very similar situation in 3 months. Like with dynflow switch to sidekiq+redis.

1 Like

Given the many changes we still have around packaging and the installer, cherry picking may be very painful. We also don’t have end to end CI pipelines in stable branches, other than the releases. That can make cherry picking packaging and installer high risk. Because of that I’m open to postponing.

One potential risk of postponing now is that the conferences can easily add a lot more delay.

I know @Justin_Sherrill is also working hard on making sure we can get Pulp 3 in which requires installer changes. It’s not mentioned here either, but good to be aware of.

Right now I’m prioritizing merging the big changes so we can branch in time and then on branching day itself make the call.

We went over the status of all of these items during today’s release meeting and decided that we are not comfortable that we can branch in a week and half, due to these and other issues. see Release meeting 2020-01-15 for the details.