Resurrection of the client-side infrastructure upgrade effort

[Shimon_Shtein]

A general question:
What would be the acceptance criteria for this PR?
Obviously we should be able to spin up a dev machine and compile the JS for packaging.
Any constraints for size or timing?

[jeremylenz]

@Shimon_Shtein

Any constraints for size or timing?

I’d say “don’t get any worse than we already are” is a reasonable starting point.
also, make sure it doesn’t break Katello and other important plugins

[Shimon_Shtein]

General update:
It looks like the best way to move forward would be to start WP5 configuration from scratch and see what we are missing there, instead of trying to adapt the current configuration and custom plugins. The reasoning is that the purpose of the plugins was to create module federation with older WP versions. Now that we can use WP5’s module federation, the need for the complex configuration is not there anymore.

foreman-js on WP5 dilemma:
Theoretically we can expose all the libraries under foreman-js as shared modules, and consume the same copy of the module in all foreman plugins. The downside of this approach is that we can’t tree-shake the library, effectively loading everything it brings into browser’s memory.
Alternative approach would be creating a framework that will reuse the same library version in all plugins during the compilation phase, but each plugin would “bake in” all the parts of the library it uses, and ship as part of its own module.

Additional reading for the dilemma: [Question] What's the best practices of shared dependencies? · Issue #2219 · module-federation/module-federation-examples · GitHub

[ekohl]

@Shimon_Shtein

Theoretically we can expose all the libraries under foreman-js as shared modules, and consume the same copy of the module in all foreman plugins. The downside of this approach is that we can’t tree-shake the library, effectively loading everything it brings into browser’s memory.

This was the whole design of the foreman-js library. From the start we knew we couldn’t tree shake it, since you don’t know what plugins are going to use.

I suppose this is also the reason that importmaps expose all module separate, then it’s up to the client to retrieve what it needs instead of some huge amount that you may or may not need.

@Shimon_Shtein

Alternative approach would be creating a framework that will reuse the same library version in all plugins during the compilation phase, but each plugin would “bake in” all the parts of the library it uses, and ship as part of its own module.

I think this is going to duplicate more than having a common library. Every plugin that uses webpack will pull in react and patternfly.

Overall I’d say start with foreman-js as a shared library (accepting we can’t tree shake) and then figure out a (long term) path to import maps to avoid webpack as much as possible. Relying on native browser functionality is IMHO still a better path forward, even if it’ll take a long time to get there.

In the medium term we should critically look at what foreman-js pulls in. If it pulls in multiple versions of the same library, then figure out if we can align those on a single (latest) version for example.

[Shimon_Shtein]

@ekohl

Overall I’d say start with foreman-js as a shared library (accepting we can’t tree shake)

Agreed. That’s the path I am taking right now.

The main obstacle I want to overcome right now is the ability to load the modules list dynamically.
By default the host application (the one that consumes other modules) should define in its configuration all the modules that it intends to consume. Since we don’t have the full list during the webpack build time, I want to understand how this list can be populated in runtime.

[Shimon_Shtein]

Update: there is an example of using dynamic loading of components here
Now I need to wrap this code into rails helpers and load the JS module from the plugin using the dynamic loaders.

[Shimon_Shtein]

Another update:
I am in the middle of adapting Foreman’s helpers to support the new set of webpack’s output.
Currently there is jsbundling-rails way, where first all the webpack assets are compiled and then they are served as static JS files from the rails’ assets pipeline. This puts some restrictions though, since dynamic links inside webpack generated JS will not work.
As a first step I will try to disable the fingerprinting, to see that everything actually loads correctly. Once I have a running Foreman instance, I will start adapting the solution to production-ready environment.

[MariaAga]

Hi everyone, Shim and I made a working version of Foreman with webpack 5! Please help us test it and make any improvement suggestions to the new configuration.

We have successfully achieved a stable configuration of Foreman and its plugins using Webpack 5. In this revamped setup, we’ve embraced module federation to streamline the loading process. Because we are using module federation, each plugin will be built separately and will be dynamically loaded. The configuration for each plugin will be the same, and is located in foremans webpack.config.

One of the major changes in this configuration is the shift towards utilizing files from the public folder instead of relying on the webpack dev server to serve them. This change serves two primary purposes: first, it eliminates the need to run a webpack dev server for each plugin, and second, it simplifies the loading process within Ruby. After this change, every plugin will have a folder in public/webpack with its webpack assets (@packaging). This change also removes the webpack-rails gem.

This transition has necessitated adjustments to the loading order and the implementation of a few workarounds:

1. Since the plugins are loaded asynchronously, we have introduced the loadPlugin global event that will be fired once all the plugins are loaded. Which means, JavaScript scripts/files loaded with content_for(:javascripts) will now load only after plugins are loaded.
2. The react_component will search for its component only after all plugins and JavaScript scripts have loaded, otherwise it might not find the component.
3. In Katello: We’ve made crucial changes to the bastion code to ensure it loads in the correct order. Without this adjustment, Angular pages may not function as expected. Specifically, this resolves these errors:

• BASTION_MODULES not being defined and issues with angular.element(document).ready, which is no longer effective due to dynamic script loading.
• angular.bootstrap can only run after all angular.module(... were run.
• Unknown provider: $rootScopeProvider ← $rootScope making angular pages not load

• webpack_asset_paths(plugin_name, extension: 'js') for Javascript files should not be used anymore, and plugins that use it should replace it with content_for(:javascripts) { webpacked_plugins_js_for(plugin_name) }. For the next Foreman version we will override the javascript_include_tag function to make sure plugins are loaded correctly.
• webpack_asset_paths for CSS files should not be used as the css will already be loaded.

5. Since we no longer run webpack as a server, we will remove webpack_dev_server and webpack_dev_server_https options.

We are sharing the react_app/components/HostDetails/Templates folder for a specific reason. Without this sharing, each plugin creates its own file for webpack/assets/javascripts/react_app/components/HostDetails/Templates/CardItem/CardTemplate/index.js. This isolated approach prevents the React Context from being properly passed to the plugins, resulting in crashes on the host details page’s details tab.

Other changes:

1. SimpleNamedModulesPlugin was a simple version of webpack 3 SimpleNamedModulesPlugin, we removed it and we are using optimization:{ moduleIds: 'named',} as recommended by webpack 5 instead, @packaging will that work for you?
2. Webpack 5 no longer automatically includes the buffer package, so we need to import it manually in webpack/assets/javascripts/react_app/common/globalIdHelpers.js.

While we’ve achieved significant improvements with this configuration, we’re aware of a few challenges that still need attention, now or in the future:

1. Lengthy webpack compile times (sometimes up to 5 minutes).

2. Increased JavaScript file sizes.

3. Longer page loading times for certain pages (for example: /foreman_tasks/tasks, /products).

Relevant PRS to use the new configuration:
Foreman: webpack 5 by MariaAga · Pull Request #9834 · theforeman/foreman · GitHub
Katello: webpack5 by MariaAga · Pull Request #10735 · Katello/katello · GitHub

To ensure a clean run with Webpack 5, please follow these steps in the Foreman folder:

rm -rf node_modules public/webpack/ public/assets/
npm install

And then:

Run webpack and rails separately:

npx webpack --config config/webpack.config.js --watch
bundle exec foreman start rails

Or together:

bundle exec foreman start

As we move forward, our goal is to enable plugins to consume Foreman as a module via package.json while leveraging the expose and shared features in the webpack’s module federation configuration. We believe this approach will help reduce JavaScript file sizes and prevent Foreman CSS from being duplicated in every plugin, mitigating issues related to CSS file size and compatibility with older plugins.

[ehelms]

@MariaAga

One of the major changes in this configuration is the shift towards utilizing files from the public folder instead of relying on the webpack dev server to serve them. This change serves two primary purposes: first, it eliminates the need to run a webpack dev server for each plugin, and second, it simplifies the loading process within Ruby. After this change, every plugin will have a folder in public/webpack with its webpack assets (@packaging).

Can you expand on this a bit? Is this only meant to indicate changes in the development environment set up? What will be in this folder? If yes, it would bring some continuity but I am curious then what happens in production where we already use this folder structure to deliver compressed and minimized assets. For example, every plugin in production has a public/webpack/<plugin name>/ folder with these assets.

@MariaAga

SimpleNamedModulesPlugin was a simple version of webpack 3 SimpleNamedModulesPlugin, we removed it and we are using optimization:{ moduleIds: 'named',} as recommended by webpack 5 instead, @packaging will that work for you?

I don’t know how this affects packaging and production set up to be honest. Can you help explain what you think it would affect?

@MariaAga

Run webpack and rails separately:

npx webpack --config config/webpack.config.js --watch
bundle exec foreman start rails

Based on this and the above comments about public/webpack, do I have it correct that this means:

• Rails serving up the assets now in development
• npx webpack is just a file watcher that initiates a compile when things change
• Using Apache in front of a development setup will now correctly work and mimic production better

@MariaAga

• Lengthy webpack compile times (sometimes up to 5 minutes).

Is this only on first generation or this means in development set up a change to a file might trigger a 5 minute wait while it compiles?

[MariaAga]

@ehelms

What will be in this folder?

In production:

In development:

@ehelms

I don’t know how this affects packaging and production set up to be honest. Can you help explain what you think it would affect?

We couldnt see why it was excatly needed which is why I tagged packaging, as it looks like a solution made for packaging.

@ehelms

Based on this and the above comments about public/webpack, do I have it correct that this means:

• Rails serving up the assets now in development
• npx webpack is just a file watcher that initiates a compile when things change
• Using Apache in front of a development setup will now correctly work and mimic production better

Yes

@ehelms

Is this only on first generation or this means in development set up a change to a file might trigger a 5 minute wait while it compiles?

If a file changes it will only trigger 1 webpack recompile, so for example if I change a file in Katello, only Katello will recompile and not the other plugins so it will take less time. Currently we don’t have a solution for this, we think that exporting Foreman to a module will help with that but did not do any testing with that.

[ekohl]

@MariaAga

@ehelms

I don’t know how this affects packaging and production set up to be honest. Can you help explain what you think it would affect?

We couldnt see why it was excatly needed which is why I tagged packaging, as it looks like a solution made for packaging.

If I recall correctly, it’s sort of related to packaging. Without the plugin the files would be stored by their absolute path and when you add plugins to that it would all break. You would see /buildroot/BUILD/... as paths, because that’s what Koji uses. So rather than that, it would be stored by their module name. It’s very likely that by now webpack has defaulted to a better solution just because they now properly support modules, but I’m just going by memory and not looking anything up.

[ehelms]

I am trying to think through the best way to roll this out and I have a few questions to help.

1. Does this change require all plugins to also change simultaneously?
2. Are there any parts of this change that can be made independently ahead of time?
3. Is there any way by which we can gate the implementation on a feature flag? Such that we can merge the changes and once ready across the ecosystem flip a switch and rebuild?

[MariaAga]

@ehelms

• Does this change require all plugins to also change simultaneously?

Probably not, most plugins should work without any change (for now Katello is the exception)

@ehelms

• Are there any parts of this change that can be made independently ahead of time?

Most changes are necessary to make Foreman work with webpack 5.
I will try to separate all non necessary changes out of the PR, and continue discussion on this topic there.

@ehelms

• Is there any way by which we can gate the implementation on a feature flag? Such that we can merge the changes and once ready across the ecosystem flip a switch and rebuild?

I think it will be too complicated to build and would rather focus that time on optimizing Foreman with webpack 5

[Shimon_Shtein]

Status update:
We have identified a dependency on node version >= 14.
So to move forward, the plan is:

1. Switch to node 14 in our current setup (should be possible without extensive changes to the code)
2. Switch to webpack 5 (by merging the PR)
   Once these two steps are done, we will have a lot of additional upgrade options unlocked, such as:

• Switching to newer node versions (node 14 is EOL already) >= 16 recommended
• Switching to patternfly 5 (requires node 16)
• Switching to newer version of npm / or deciding on another package manager (rails 7 defaults to yarn for Webpack based JS stack)

There are more code improvements that would be unlocked once webpack 5 PR is merged, but I didn’t mention them here, since this post is dealing with infrastructure upgrade path.

This approach should keep the changes incremental, and not tie them all into a single mess.

Currently @ehelms is investigating the possibility to switch to node 14.

[ekohl]

@Shimon_Shtein

Switching to newer node versions (node 14 is EOL already) >= 16 recommended

EL9 ships with NodeJS 16 (and 18 & 20 in modules) so once we’re at this stage one blocker to upgrading has been removed.

@Shimon_Shtein

Currently @ehelms is investigating the possibility to switch to node 14.

Add NodeJS 14 flattened external repo

Use NodeJS 14 flat repos

[MariaAga]

Update - all tests are passing in the Webpack 5 pull request!
Reviews are needed as it’s a big change to the application loading process.
After webpack 5 is in, the next steps for the UI infrastructure will be converting foremanReact into a module to be consumed by plugins to improve efficiency, and to upgrade PF4 to PF5 (which shouldn’t take too long thanks to automation).

Your feedback and collaboration during this phase are greatly appreciated.

To test, you will need these 2 PRS:

webpack 5

webpack5

running rm -rf node_modules package-lock.json public/webpack (for a clean start)

npm install

and then foreman start
or rake webpack:compile and foreman start rails.
Webpack no longer runs as a server and will create static files.