Dropping ARM packages

#1

Hello,

Currently we build and ship ARM packages for debian-based distros. Some preliminary results from the community survey seem to indicate they are hardly used - just one user (other than our own @Gwmngilfen) has reported so far that they use Foreman on ARM.
The build process for ARM is much slower than for x86, making the release process currently take longer since we don’t push the packages until all builds are done. This also makes the build pipeline for debian more complex, since it needs to handle building on both arm and x86.

Given this situation, and the fact the foreman should still be installable on ARM from source, I suggest that we stop building and shipping arm packages. Please vote below on your preferred course of action (or comment if I missed some point):

  • Drop ARM packaging work completely
  • Continue to build ARM as best-effort, separately from the main build pipelines (i.e. not blocking)
  • Continue building and shipping ARM packages only for released version and not nightlies
  • Continue building and shipping ARM packages for all releases including nightlies

0 voters

Do you still use rsync to mirror our repositories?
#2

I had a big internal debate about this.

First of all, I really like having the option to support multiple architectures since it forces you not to code to one arch.

That said, it is much slower to build. Currently one issue is the number of build slots. To solve this, there’s also another option: add more build slaves.

Changing the build process means someone needs to step up to do the work.

Making it non-blocking does sound nice, but I worry that nobody will look at the failures and it’ll bit rot. Not building nightlies but only releases has a similar risk. We’ve already seen that when nightlies were broken it adds a huge extra load on the release process. When you add ARM specific failures I wouldn’t know how to fix them.

Overall I’m leaning to dropping ARM packaging work.

#3

If you and Tomer, who work on Foreman releases the most, say it’s too much work and would prefer dropping, I’m with you (can’t change my vote). It seems there’s no strong push back so far. I think if someone steps up, volunteer to do the work and helps with releasing, that would change my mind again :slight_smile:

#4

Given that Foreman itself is a Rails app, I don’t expect us to handle many arm-specific problems in our codebase. It might be different in packaging itself where we bundle gems, though.

May I propose keeping full support for arm64 (aarch64) and dropping all older arm(el/hf/whatever abomination)?

And then upgrading our ARM-hw to faster builds as soon as we don’t need to run different arm-subarches?

#5

We already did that with 1.19. We do have an arm32 builder in Jenkins but that hasn’t built a packaging job in a month. Not since:

#6

Oh cool, this means we can actually drop that 32 bit node now?

Would increasing the arm64 nodes from 6core/4G to 8core/8G help us? Maybe @mmoll knows :slight_smile:

#7

We could do that. I kept it for possible re-builds of the last package for proxy-support which is not arch neutral (ruby-rkereros).

No.

#8

:crying_cat_face:
That would’ve been too easy

#9

I opened two PR (hopefully correctly…) following this discussion:


The first only makes arm non-blocking for nightlies, while the second completely removed them from nightly builds.
There was at least 60% agreement for removing them from nightly in the poll, so i tend to think the second one should be merged, but since there was no clear agreement I opened PRs for both options.

#10

I just discovered why the load increased so much: we were triggering builds twice.

We already merged:

The other reason that increases the load on builders are the broken -release jobs. The have triggers on foreman-packaging merges. This means we build a new installer nightly whenever we merge something to packaging. It’s a bug in the Jenkins plugin, but cause a lot of additional builds.

This can be seen in
http://koji.katello.org/koji/packageinfo?packageID=695 for example. Note we often build from the same git hash on the same day. If we fix these issues we get a cleaner history and reduce the load on our infra. I think we can also re-introduce ARM builds after that.