Bootstrapping Foreman RPM packaging

Development
1

This is the journey I had recently. I had intended to write a blog, but I rarely finish those so I decided to write it as a Discourse post instead. You may recognize the writing style reflects that.

My problem is that I have an old Puppet 7 server running on CentOS Stream 8. Both are end of life, but it was OK to leave it for a bit because it’s firewalled. Now that the CA certificate also expired I really need to take action.

My plan is to build a new OpenVox 8 server running on CentOS Stream 10 so I need Foreman Proxy there. Today it’s not built on EL 10 so we need to bootstrap the build environment.

Spoiler: today I can’t fully recreate the buildroot. At the end there are various issues I found.

Allowing Foreman to bootstrap itself

I’m assuming you’re on a supported Fedora version, have a Fedora Account and used fkinit to log into Kerberos. With all of that in place we can move to the Foreman specific parts.

Clone the packaging repository if you haven’t already:

$ git clone https://github.com/theforeman/foreman-packaging -b rpm/develop

Note: at the time of writing this relies on the bootstrap conditional in foreman and related support in obal.

Foreman packaging is largely automated using obal. There are multiple ways of installing it, but I maintain a copr repository which I think is the easiest way. First install obal:

# dnf copr enable ekohl/obal
# dnf install obal

That was all the important set up, so let’s get to the real work.

To build in your own namespace you can use either always pass -e copr_project_user=MyUser every command or locally modify package_manifest.yaml. In the commands below I’ve assumed a local modification so you can copy-paste commands.

The first important part is that we are bootstrapping, which is always awkward because you need a build artifact to build itself.
The Foreman buildroot needs foreman-build to set the right version macros, but that is a subpackage of the foreman package.

With that out of the way we can create the COPR project in bootstrap state:

$ obal copr-project foreman-copr -e bootstrap=true

Now build foreman:

$ obal release foreman

Once that built successfully you reconfigure the project for the final configuration.

$ obal copr-project foreman-copr

Now we can start building all packages. There are 2 ways to go about this:

  • Run obal release --skip-failed-build packages until it all passes
  • Call various release groups

I’m opting for the latter since it’s faster in runtime. To understand what’s going on it’s important to understand that the package manifest is actually an Ansible inventory file. Each package is a “host” and those “hosts” are in various groups.

For example, packages contains foreman_packages, plugin_packages, foreman_client_packages, katello_packages and foreman_release_packages as children. Those all correlate to the various repositories we have. In turn, foreman_packages has foreman_core_packages, foreman_proxy_packages, foreman_nodejs_packages and foreman_installer_packages as children. And so forth.

We’ll also use --nowait to build all packages within a group in parallel.

$ obal release --nowait ruby_core_packages_tier1

Now it takes quite a while to even submit all of these, let alone build. Once submitted, start watching all builds concurrently:

$ copr list-builds --output-format text-row $USER/foreman-nightly-staging  | awk '!/succeeded/ { print $1 }' | xargs copr watch-build

Once it’s complete and it does come up with failures, you should analyze them. The ones I found are described at the end. Otherwise, proceed with building tier 2 is similar to tier 1:

$ obal release --nowait ruby_core_packages_tier2

Similar to the above, you should monitor the builds and once completed, you can kick off another set of builds that can run in parallel:

$ obal release --nowait rails_core_packages hammer_core_packages other_core_packages foreman_nodejs_packages

Once everything is built, you can release the top level to build the final packages:

$ obal release --nowait foreman foreman_installer_packages foreman_proxy_packages

Now you should have built the core repository from scratch. Building the plugins and later Katello repositories are left as an exercise for the reader.

Issues found

Building EL 10

In my initial attempt I started brave and wanted to build on EL 10 so I used RHEL 10 buildroots. I haven’t submitted a PR for this and it’s for another post, but I’ll list some of the issues I found below.

A buildroot needs a foreman-build package, but that’s built from foreman. I’ve modified foreman.spec to introduce an RPM conditional called bootstrap.
This allows calling rpmbuild --with bootstrap to only build the foreman-build and foreman-plugin packages. The packaging PR is still open. It also really benefits from an obal PR that’s also still open.

Once that was in place, I shifted to building dependencies and found issues:

Building on EL 9 ARM

At this point I shifted my whole effort to just being able to bootstrap a Foreman environment. I still wanted some challenge so I decided to look at building on ARM instead. We used to have Debian builds on ARM, but never had RPMs because we lacked the builders. These days we build on copr which has them. So why not.

I found that several nodejs packages failed to build due to NPM 6+ and the cache was still using the old layout:

  • nodejs-babel-plugin-transform-class-properties
  • nodejs-babel-preset-env
  • nodejs-babel-preset-react
  • nodejs-buffer
  • nodejs-compression-webpack-plugin
  • nodejs-react-bootstrap
  • nodejs-react-intl
  • nodejs-sass
  • nodejs-sass-loader
  • nodejs-style-loader
  • nodejs-use-deep-compare-effect
  • nodejs-webpack-cli

I then had the thought of removing unused nodejs packages because maybe that resolved some of the issues. A first round found a few unused packages but none that failed to build previously. While doing so I also found that the remove script needed a patch for the line width.

Fixing them is relatively easy: regenerate them with npm2rpm in the right environment, but I haven’t done that.

Then I also found that various go packages were skipped because it doesn’t use the OS provided go architectures (PR is open for dynflow-utils and pcp-mmvstatsd. The latter doesn’t build on EL9 x86_64 either and never has. There is also similar work in the client tools repo, but I didn’t touch that because it also builds on SLES which is another can of worms.

Chronological list

The above is a nice write up, but in reality my process was:

4 Likes
2

So now I know what the higher goal of all those PRs was. :grin: Great effort and great post on this. Thanks, Ewoud!

1 Like
3

I’ve taken another stab at building EL10 and submitted the branch I was working on as a draft PR:

For now I don’t intend to actively work on this, but I’d welcome anyone else to work on this. I’ve tried to leave notes on where I was.

1 Like
4

I had a brief look at the NPM situation.

  • Both nodejs-babel-plugin-transform-class-properties and nodejs-babel-preset-react are unused after we dropped foreman_acd and I’ve added their removal to Remove unused nodejs packages by ekohl · Pull Request #12820 · theforeman/foreman-packaging · GitHub.
  • nodejs-babel-preset-env is updated in Bump nodejs-babel-preset-env to 7.9.5 by evgeni · Pull Request #12421 · theforeman/foreman-packaging · GitHub.
  • nodejs-buffer opened a PR to unbundle: Convert nodejs-buffer to single style package by ekohl · Pull Request #12875 · theforeman/foreman-packaging · GitHub
  • nodejs-compression-webpack-plugin is still used by Foreman and has a non-trivial dependency tree. Probably needs to be rebuilt.
  • nodejs-react-bootstrap is only used by Katello, so technically should probably live in packages/katello instead of packages/foreman. Foreman itself uses it indirectly via patternfly-react which vendors it. Ideally speaking it’s all migrated to PF5.
  • nodejs-react-intl used by Foreman (and plugins) with a non-trivial dependency tree. Probably needs to be rebuilt
  • nodejs-sass is a direct dependency of Foreman and needs to be rebuilt
  • nodejs-sass-loader only has a single dependency (neo-async) that has no dependencies. Probably easier to unbundle this and avoid the nasty parts
  • nodejs-style-loader we’re using version 1.3 which has dependencies, but version 3 and 4 don’t have any so if we can update then that’s even easier.
  • nodejs-use-deep-compare-effect is only used by katello so should probably live in packages/katello. It only has 2 dependencies, but 1 is @babel/runtime. We’re blocked by https://issues.redhat.com/browse/RHEL-120511 which is why we need to vendor it now. Interesting that it mentions you should really use React.useEffect if you can. I wonder why Katello needs this.
  • nodejs-webpack-cli is used by Foreman and has a non-trivial dependency tree. Probably needs to be rebuilt.
5

I couldn’t help myself and I’ve now spent some time on figuring out the NodeJS situation. It turns out that building the RPM on EL10 doesn’t deal well with an absolute NPM cache path. Drop NPM < 6 support, SCL support and use a relative cache directory by ekohl · Pull Request #87 · theforeman/npm2rpm · GitHub includes that a fix to use a relative directory.

Another issue I found is that when you start to really build the foreman package then obal thinks there is already a build (because we had a bootstrap build). You can force a rebuild with:

obal release --copr-rebuild foreman

With that the EL10 build currently fails with:

+ /usr/bin/rake webpack:compile DATABASE_URL=nulldb://nohost
2025-12-24T13:33:58 [I|app|] Rails cache backend: File
2025-12-24T13:33:58 [W|app|] You are trying to replace import_subnets from . Adding allowed actions from plugin permissions to the existing one.
npx --max_old_space_size=2048 webpack --config /builddir/build/BUILD/foreman-3.18.0-develop/config/webpack.config.js --bail
[webpack-cli] Failed to load '/builddir/build/BUILD/foreman-3.18.0-develop/config/webpack.config.js' config
[webpack-cli] Error: Cannot find module 'webpack-stats-plugin'
Require stack:
- /builddir/build/BUILD/foreman-3.18.0-develop/config/webpack.config.js
- /usr/lib/node_modules_22/webpack-cli/lib/webpack-cli.js
- /usr/lib/node_modules_22/webpack-cli/lib/bootstrap.js
- /usr/lib/node_modules_22/webpack-cli/bin/cli.js
- /usr/lib/node_modules_22/webpack/bin/webpack.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1383:15)
    at defaultResolveImpl (node:internal/modules/cjs/loader:1025:19)
    at resolveForCJSWithHooks (node:internal/modules/cjs/loader:1030:22)
    at Module._load (node:internal/modules/cjs/loader:1192:37)
    at TracingChannel.traceSync (node:diagnostics_channel:322:14)
    at wrapModuleLoad (node:internal/modules/cjs/loader:237:24)
    at Module.require (node:internal/modules/cjs/loader:1463:12)
    at require (node:internal/modules/helpers:147:16)
    at Object.<anonymous> (/builddir/build/BUILD/foreman-3.18.0-develop/config/webpack.config.js:9:25)
    at Module._compile (node:internal/modules/cjs/loader:1706:14) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/builddir/build/BUILD/foreman-3.18.0-develop/config/webpack.config.js',
    '/usr/lib/node_modules_22/webpack-cli/lib/webpack-cli.js',
    '/usr/lib/node_modules_22/webpack-cli/lib/bootstrap.js',
    '/usr/lib/node_modules_22/webpack-cli/bin/cli.js',
    '/usr/lib/node_modules_22/webpack/bin/webpack.js'
  ]
}
rake aborted!
Command failed with status (2): [npx --max_old_space_size=2048 webpack --config /builddir/build/BUILD/foreman-3.18.0-develop/config/webpack.config.js --bail]
/builddir/build/BUILD/foreman-3.18.0-develop/lib/tasks/webpack_compile.rake:18:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.1.0/exe/rake:27:in `<top (required)>'

I wonder if this has something to do with the new layout where it places files in /usr/lib/node_modules_22 instead of /usr/lib/node_modules.

Build at least the EL 9 build itself now passes, which proves you can bootstrap the whole chroot this way.