Currently we build and ship ARM packages for debian-based distros. Some preliminary results from the community survey seem to indicate they are hardly used - just one user (other than our own @Gwmngilfen) has reported so far that they use Foreman on ARM.
The build process for ARM is much slower than for x86, making the release process currently take longer since we don’t push the packages until all builds are done. This also makes the build pipeline for debian more complex, since it needs to handle building on both arm and x86.
Given this situation, and the fact the foreman should still be installable on ARM from source, I suggest that we stop building and shipping arm packages. Please vote below on your preferred course of action (or comment if I missed some point):
Drop ARM packaging work completely
Continue to build ARM as best-effort, separately from the main build pipelines (i.e. not blocking)
Continue building and shipping ARM packages only for released version and not nightlies
Continue building and shipping ARM packages for all releases including nightlies
First of all, I really like having the option to support multiple architectures since it forces you not to code to one arch.
That said, it is much slower to build. Currently one issue is the number of build slots. To solve this, there’s also another option: add more build slaves.
Changing the build process means someone needs to step up to do the work.
Making it non-blocking does sound nice, but I worry that nobody will look at the failures and it’ll bit rot. Not building nightlies but only releases has a similar risk. We’ve already seen that when nightlies were broken it adds a huge extra load on the release process. When you add ARM specific failures I wouldn’t know how to fix them.
Overall I’m leaning to dropping ARM packaging work.
If you and Tomer, who work on Foreman releases the most, say it’s too much work and would prefer dropping, I’m with you (can’t change my vote). It seems there’s no strong push back so far. I think if someone steps up, volunteer to do the work and helps with releasing, that would change my mind again
Given that Foreman itself is a Rails app, I don’t expect us to handle many arm-specific problems in our codebase. It might be different in packaging itself where we bundle gems, though.
May I propose keeping full support for arm64 (aarch64) and dropping all older arm(el/hf/whatever abomination)?
And then upgrading our ARM-hw to faster builds as soon as we don’t need to run different arm-subarches?
The first only makes arm non-blocking for nightlies, while the second completely removed them from nightly builds.
There was at least 60% agreement for removing them from nightly in the poll, so i tend to think the second one should be merged, but since there was no clear agreement I opened PRs for both options.
The other reason that increases the load on builders are the broken -release jobs. The have triggers on foreman-packaging merges. This means we build a new installer nightly whenever we merge something to packaging. It’s a bug in the Jenkins plugin, but cause a lot of additional builds.
This can be seen in foreman-installer | Package Info | koji for example. Note we often build from the same git hash on the same day. If we fix these issues we get a cleaner history and reduce the load on our infra. I think we can also re-introduce ARM builds after that.
