Resurrection of the client-side infrastructure upgrade effort

A general question:
What would be the acceptance criteria for this PR?
Obviously we should be able to spin up a dev machine and compile the JS for packaging.
Any constraints for size or timing?

I’d say “don’t get any worse than we already are” is a reasonable starting point.
also, make sure it doesn’t break Katello and other important plugins

General update:
It looks like the best way to move forward would be to start WP5 configuration from scratch and see what we are missing there, instead of trying to adapt the current configuration and custom plugins. The reasoning is that the purpose of the plugins was to create module federation with older WP versions. Now that we can use WP5’s module federation, the need for the complex configuration is not there anymore.

foreman-js on WP5 dilemma:
Theoretically we can expose all the libraries under foreman-js as shared modules, and consume the same copy of the module in all foreman plugins. The downside of this approach is that we can’t tree-shake the library, effectively loading everything it brings into browser’s memory.
Alternative approach would be creating a framework that will reuse the same library version in all plugins during the compilation phase, but each plugin would “bake in” all the parts of the library it uses, and ship as part of its own module.

Additional reading for the dilemma: [Question] What's the best practices of shared dependencies? · Issue #2219 · module-federation/module-federation-examples · GitHub

This was the whole design of the foreman-js library. From the start we knew we couldn’t tree shake it, since you don’t know what plugins are going to use.

I suppose this is also the reason that importmaps expose all module separate, then it’s up to the client to retrieve what it needs instead of some huge amount that you may or may not need.

I think this is going to duplicate more than having a common library. Every plugin that uses webpack will pull in react and patternfly.

Overall I’d say start with foreman-js as a shared library (accepting we can’t tree shake) and then figure out a (long term) path to import maps to avoid webpack as much as possible. Relying on native browser functionality is IMHO still a better path forward, even if it’ll take a long time to get there.

In the medium term we should critically look at what foreman-js pulls in. If it pulls in multiple versions of the same library, then figure out if we can align those on a single (latest) version for example.

Agreed. That’s the path I am taking right now.

The main obstacle I want to overcome right now is the ability to load the modules list dynamically.
By default the host application (the one that consumes other modules) should define in its configuration all the modules that it intends to consume. Since we don’t have the full list during the webpack build time, I want to understand how this list can be populated in runtime.

Update: there is an example of using dynamic loading of components here
Now I need to wrap this code into rails helpers and load the JS module from the plugin using the dynamic loaders.

Another update:
I am in the middle of adapting Foreman’s helpers to support the new set of webpack’s output.
Currently there is jsbundling-rails way, where first all the webpack assets are compiled and then they are served as static JS files from the rails’ assets pipeline. This puts some restrictions though, since dynamic links inside webpack generated JS will not work.
As a first step I will try to disable the fingerprinting, to see that everything actually loads correctly. Once I have a running Foreman instance, I will start adapting the solution to production-ready environment.

1 Like

Hi everyone, Shim and I made a working version of Foreman with webpack 5! Please help us test it and make any improvement suggestions to the new configuration.

We have successfully achieved a stable configuration of Foreman and its plugins using Webpack 5. In this revamped setup, we’ve embraced module federation to streamline the loading process. Because we are using module federation, each plugin will be built separately and will be dynamically loaded. The configuration for each plugin will be the same, and is located in foremans webpack.config.

One of the major changes in this configuration is the shift towards utilizing files from the public folder instead of relying on the webpack dev server to serve them. This change serves two primary purposes: first, it eliminates the need to run a webpack dev server for each plugin, and second, it simplifies the loading process within Ruby. After this change, every plugin will have a folder in public/webpack with its webpack assets (@packaging). This change also removes the webpack-rails gem.

This transition has necessitated adjustments to the loading order and the implementation of a few workarounds:

  1. Since the plugins are loaded asynchronously, we have introduced the loadPlugin global event that will be fired once all the plugins are loaded. Which means, JavaScript scripts/files loaded with content_for(:javascripts) will now load only after plugins are loaded.
  2. The react_component will search for its component only after all plugins and JavaScript scripts have loaded, otherwise it might not find the component.
  3. In Katello: We’ve made crucial changes to the bastion code to ensure it loads in the correct order. Without this adjustment, Angular pages may not function as expected. Specifically, this resolves these errors:
  • BASTION_MODULES not being defined and issues with angular.element(document).ready, which is no longer effective due to dynamic script loading.
  • angular.bootstrap can only run after all angular.module(... were run.
  • Unknown provider: $rootScopeProvider ← $rootScope making angular pages not load
  • webpack_asset_paths(plugin_name, extension: 'js') for Javascript files should not be used anymore, and plugins that use it should replace it with content_for(:javascripts) { webpacked_plugins_js_for(plugin_name) }. For the next Foreman version we will override the javascript_include_tag function to make sure plugins are loaded correctly.
  • webpack_asset_paths for CSS files should not be used as the css will already be loaded.
  1. Since we no longer run webpack as a server, we will remove webpack_dev_server and webpack_dev_server_https options.

We are sharing the react_app/components/HostDetails/Templates folder for a specific reason. Without this sharing, each plugin creates its own file for webpack/assets/javascripts/react_app/components/HostDetails/Templates/CardItem/CardTemplate/index.js. This isolated approach prevents the React Context from being properly passed to the plugins, resulting in crashes on the host details page’s details tab.

Other changes:

  1. SimpleNamedModulesPlugin was a simple version of webpack 3 SimpleNamedModulesPlugin, we removed it and we are using optimization:{ moduleIds: 'named',} as recommended by webpack 5 instead, @packaging will that work for you?
  2. Webpack 5 no longer automatically includes the buffer package, so we need to import it manually in webpack/assets/javascripts/react_app/common/globalIdHelpers.js.

While we’ve achieved significant improvements with this configuration, we’re aware of a few challenges that still need attention, now or in the future:

  1. Lengthy webpack compile times (sometimes up to 5 minutes).

  2. Increased JavaScript file sizes.

  3. Longer page loading times for certain pages (for example: /foreman_tasks/tasks, /products).

Relevant PRS to use the new configuration:
Foreman: webpack 5 by MariaAga · Pull Request #9834 · theforeman/foreman · GitHub
Katello: webpack5 by MariaAga · Pull Request #10735 · Katello/katello · GitHub

To ensure a clean run with Webpack 5, please follow these steps in the Foreman folder:

rm -rf node_modules public/webpack/ public/assets/
npm install

And then:

Run webpack and rails separately:

npx webpack --config config/webpack.config.js --watch
bundle exec foreman start rails

Or together:

bundle exec foreman start

As we move forward, our goal is to enable plugins to consume Foreman as a module via package.json while leveraging the expose and shared features in the webpack’s module federation configuration. We believe this approach will help reduce JavaScript file sizes and prevent Foreman CSS from being duplicated in every plugin, mitigating issues related to CSS file size and compatibility with older plugins.


Can you expand on this a bit? Is this only meant to indicate changes in the development environment set up? What will be in this folder? If yes, it would bring some continuity but I am curious then what happens in production where we already use this folder structure to deliver compressed and minimized assets. For example, every plugin in production has a public/webpack/<plugin name>/ folder with these assets.

I don’t know how this affects packaging and production set up to be honest. Can you help explain what you think it would affect?

Based on this and the above comments about public/webpack, do I have it correct that this means:

  • Rails serving up the assets now in development
  • npx webpack is just a file watcher that initiates a compile when things change
  • Using Apache in front of a development setup will now correctly work and mimic production better

Is this only on first generation or this means in development set up a change to a file might trigger a 5 minute wait while it compiles?

In production:

In development:

We couldnt see why it was excatly needed which is why I tagged packaging, as it looks like a solution made for packaging.


If a file changes it will only trigger 1 webpack recompile, so for example if I change a file in Katello, only Katello will recompile and not the other plugins so it will take less time. Currently we don’t have a solution for this, we think that exporting Foreman to a module will help with that but did not do any testing with that.

If I recall correctly, it’s sort of related to packaging. Without the plugin the files would be stored by their absolute path and when you add plugins to that it would all break. You would see /buildroot/BUILD/... as paths, because that’s what Koji uses. So rather than that, it would be stored by their module name. It’s very likely that by now webpack has defaulted to a better solution just because they now properly support modules, but I’m just going by memory and not looking anything up.

I am trying to think through the best way to roll this out and I have a few questions to help.

  1. Does this change require all plugins to also change simultaneously?
  2. Are there any parts of this change that can be made independently ahead of time?
  3. Is there any way by which we can gate the implementation on a feature flag? Such that we can merge the changes and once ready across the ecosystem flip a switch and rebuild?