Webpack 5 merge this Friday (2024-01-26)

ekohl · January 22, 2024, 4:08pm

tl;dr: https://github.com/theforeman/foreman/pull/9834 & friends will be merged this Friday.

A while back an effort was started to update our frontend: Resurrection of the client-side infrastructure upgrade effort. A big milestone is the update Webpack from version 3 to version 5. Care has been taken to test for regressions, but it still requires merging various PRs at the same time. That also means you, as a developer, need to pull in all those changes at the same time. It’s also recommended to run these steps (from Resurrection of the client-side infrastructure upgrade effort - #36 by MariaAga):

# Clean up old stuff
rm -rf node_modules package-lock.json public/webpack
# Fresh install
npm install
# Start development again
foreman start

The list of PRs that I know of:

@MariaAga and @evgeni correct me if I missed anytthing.

MariaAga · January 22, 2024, 4:36pm

Thanks Ewoud for the post!
Adding that if someone has a machine with not a lot of RAM I found it useful setting NODE_OPTIONS='--max-old-space-size=2024' (or 4048) if foreman start crashes. (its not something webpack specific, I also use this when running js tests).

Also once this is merged and working we should remove the deprecated use of webpack_asset_paths, webpacked_plugins_css_for, css_tags_for in plugins.

ekohl · January 26, 2024, 10:44am

This is happening now. I’ve just merged the Foreman core PR and others will follow soon.

Partha_Aji · January 26, 2024, 10:09pm

If you are hitting any webpack 5 related errors like below while this is getting ironed out among other plugins

No such file or directory @ rb_sysopen - /home/vagrant/foreman/public/webpack/manifest.json

I’d recommend basing your develop or master branches off the following commit hashes for foreman and katello

cd foreman; git checkout 066dd49a9ec121038bd93ac7d03f28ea0fc4b69c; git checkout -b webpack3
cd ../katello; git checkout cc1b7cc0a624e00a6b59b5bae2be298a8ba6331e; git checkout -b webpack3

ekohl · January 27, 2024, 1:44pm

If you are hitting any errors, please share them here so we’re aware of it. I’d like to avoid a situation where everyone involved thinks it works but all other developers have implemented workarounds.

evgeni · January 29, 2024, 9:48am

So the merge happened, the packages were built, and the pipelines ran.
Things mostly work, so let’s see what does not:

`@import '~@theforeman/vendor/…` (FIXED)

In https://github.com/theforeman/foreman/pull/9834 we changed @import '~@theforeman/vendor/… in our SCSS files to @import 'node_modules/@theforeman/vendor/…, which I didn’t truly understand, but RPM builds were broken without it, so we did it. Now the same thing started creeping up in plugins, and I went poking a bit. It seems to be dependent on the path the various node_modules have on disk, as Debian builds (which bundle everything) work fine with either notation, while RPM wanted node_modules. CI for Core accepted both, while CI for Plugins wanted node_modules.
I came up with Refs #37102 - correct imports from `@theforeman/vendor` by evgeni · Pull Request #10018 · theforeman/foreman · GitHub which seems to fix RPM builds and Plugins CI, without breaking Core CI and Core RPM.
I didn’t test Debian (or devel setups), but I hope it’s fine?

`Can't resolve 'buffer'` when building REX RPM (FIXED)

When building REX RPM it fails with:

[webpack-cli] ModuleNotFoundError: Module not found: Error: Can't resolve 'buffer' in '/builddir/build/BUILD/foreman_remote_execution-12.0.5/usr/share/foreman/webpack/assets/javascripts/react_app/common'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
	- add a fallback 'resolve.fallback: { "buffer": require.resolve("buffer/") }'
	- install 'buffer'

This seems due to the fact that Webpack < 5 had buffer in the fallbacks, and now does not anymore.
It does not happen in CI (or Debian) as an npm i does pull in buffer, but we do not ship that as an RPM.
We use Buffer in globalIdHelpers:

github.com

theforeman/foreman/blob/89c610451c9ea22eb426826e405518e04d68972f/webpack/assets/javascripts/react_app/common/globalIdHelpers.js#L9


      
          /*
           * Helpers for dealing with GlobalIDs as returned by GraphQL. When `id` property is
           * retrieved through GraphQL, the value is not a "native" ID under which the
           * backend knows the record in question.
           *
           * The GlobalID contains a version number, model class and the id itself, put
           * together into a single string and encoded as base64.
           */
          import { Buffer } from 'buffer';
          
          const idSeparator = '-';
          const versionSeparator = ':';
          const defaultVersion = '01';
          
          // Decodes the provided GlobalID, returns a three element array consisting of
          // version, type name and id as strings
          export const decode = globalId => {
            const s = Buffer.from(globalId, 'base64').toString();
            let i = s.indexOf(versionSeparator);

I think this can be fixed by explicitly adding buffer to Foreman’s package.json (as that’s where we use it).

`Field 'browser' doesn't contain a valid alias configuration` (FIXED)

When building katello or ansible, builds fail with errors like:

[webpack-cli] ModuleNotFoundError: Module not found: Error: Can't resolve 'dequal' in '/usr/lib/node_modules/use-deep-compare-effect/dist'
    at /usr/lib/node_modules/webpack/lib/Compilation.js:2016:28
    at /usr/lib/node_modules/webpack/lib/NormalModuleFactory.js:798:13
    at eval (eval at create (/usr/lib/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:10:1)
    at /usr/lib/node_modules/webpack/lib/NormalModuleFactory.js:270:22
    at eval (eval at create (/usr/lib/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:9:1)
    at /usr/lib/node_modules/webpack/lib/NormalModuleFactory.js:434:22
    at /usr/lib/node_modules/webpack/lib/NormalModuleFactory.js:116:11
    at /usr/lib/node_modules/webpack/lib/NormalModuleFactory.js:670:25
    at /usr/lib/node_modules/webpack/lib/NormalModuleFactory.js:855:8
    at /usr/lib/node_modules/webpack/lib/NormalModuleFactory.js:975:5
    at /usr/lib/node_modules/webpack/node_modules/neo-async/async.js:6883:13
    at /usr/lib/node_modules/webpack/lib/NormalModuleFactory.js:958:45
    at finishWithoutResolve (/usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:372:11)
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:461:15
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:519:5
    at eval (eval at create (/usr/lib/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:15:1)
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:519:5
    at eval (eval at create (/usr/lib/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:27:1)
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/DescriptionFilePlugin.js:89:43
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:519:5
    at eval (eval at create (/usr/lib/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/forEachBail.js:39:13
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/AliasPlugin.js:148:14
    at next (/usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/forEachBail.js:35:3)
    at forEachBail (/usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/forEachBail.js:49:9)
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/AliasPlugin.js:61:5
    at _next0 (eval at create (/usr/lib/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:8:1)
    at eval (eval at create (/usr/lib/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:30:1)
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:519:5
    at eval (eval at create (/usr/lib/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:519:5
    at eval (eval at create (/usr/lib/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:519:5
    at eval (eval at create (/usr/lib/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:27:1)
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/DescriptionFilePlugin.js:89:43
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:519:5
    at eval (eval at create (/usr/lib/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:519:5
    at eval (eval at create (/usr/lib/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:15:1)
    at /usr/lib/node_modules/webpack/node_modules/enhanced-resolve/lib/DirectoryExistsPlugin.js:41:15
    at processTicksAndRejections (internal/process/task_queues.js:81:21)
resolve 'dequal' in '/usr/lib/node_modules/use-deep-compare-effect/dist'
  Parsed request is a module
  using description file: /usr/lib/node_modules/use-deep-compare-effect/package.json (relative path: ./dist)
    Field 'browser' doesn't contain a valid alias configuration
    resolve as module
      looking for modules in /builddir/build/BUILD/katello-4.12.0.pre.master/usr/share/foreman/node_modules
        single file module
          using description file: /builddir/build/BUILD/katello-4.12.0.pre.master/package.json (relative path: ./usr/share/foreman/node_modules/dequal)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
              /builddir/build/BUILD/katello-4.12.0.pre.master/usr/share/foreman/node_modules/dequal doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              /builddir/build/BUILD/katello-4.12.0.pre.master/usr/share/foreman/node_modules/dequal.js doesn't exist
            .json
              Field 'browser' doesn't contain a valid alias configuration
              /builddir/build/BUILD/katello-4.12.0.pre.master/usr/share/foreman/node_modules/dequal.json doesn't exist
            .wasm
              Field 'browser' doesn't contain a valid alias configuration
              /builddir/build/BUILD/katello-4.12.0.pre.master/usr/share/foreman/node_modules/dequal.wasm doesn't exist
        /builddir/build/BUILD/katello-4.12.0.pre.master/usr/share/foreman/node_modules/dequal doesn't exist
      looking for modules in /builddir/build/BUILDROOT/rubygem-katello-4.12.0-0.1.pre.master.20240126171217git7091b11.el8.x86_64/usr/share/gems/gems/katello-4.12.0.pre.master/node_modules
        single file module
          using description file: /builddir/build/BUILDROOT/rubygem-katello-4.12.0-0.1.pre.master.20240126171217git7091b11.el8.x86_64/usr/share/gems/gems/katello-4.12.0.pre.master/package.json (relative path: ./node_modules/dequal)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
              /builddir/build/BUILDROOT/rubygem-katello-4.12.0-0.1.pre.master.20240126171217git7091b11.el8.x86_64/usr/share/gems/gems/katello-4.12.0.pre.master/node_modules/dequal doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              /builddir/build/BUILDROOT/rubygem-katello-4.12.0-0.1.pre.master.20240126171217git7091b11.el8.x86_64/usr/share/gems/gems/katello-4.12.0.pre.master/node_modules/dequal.js doesn't exist
            .json
              Field 'browser' doesn't contain a valid alias configuration
              /builddir/build/BUILDROOT/rubygem-katello-4.12.0-0.1.pre.master.20240126171217git7091b11.el8.x86_64/usr/share/gems/gems/katello-4.12.0.pre.master/node_modules/dequal.json doesn't exist
            .wasm
              Field 'browser' doesn't contain a valid alias configuration
              /builddir/build/BUILDROOT/rubygem-katello-4.12.0-0.1.pre.master.20240126171217git7091b11.el8.x86_64/usr/share/gems/gems/katello-4.12.0.pre.master/node_modules/dequal.wasm doesn't exist
        /builddir/build/BUILDROOT/rubygem-katello-4.12.0-0.1.pre.master.20240126171217git7091b11.el8.x86_64/usr/share/gems/gems/katello-4.12.0.pre.master/node_modules/dequal doesn't exist
rake aborted!

Even tho our RPM for use-deep-compare-effect does bundle dequal (ansible fails with a different Node module, but the same error). It seems /usr/lib/node_modules/use-deep-compare-effect/node_modules is not part of the list of node_modules Webpack uses for resolving this.
I think this can be fixed by adding a bare node_modules at the end of resolve.modules in

github.com

theforeman/foreman/blob/89c610451c9ea22eb426826e405518e04d68972f/config/webpack.config.js#L235-L240


      
          var configModules = config.resolve.modules || [];
          // make webpack to resolve modules from core first
          configModules.unshift(path.resolve(__dirname, '..', 'node_modules'));
          // add plugin's node_modules to the reslver list
          configModules.push(path.resolve(pluginRoot, 'node_modules'));
          config.resolve.modules = configModules;

But I didn’t have time to verify this.

Plugins CI can’t load `/webpack/foreman-vendor.(js|css)`

No idea why the tests try to load those URLs or why they do not resolve

ekohl · January 29, 2024, 11:16am

I didn’t check other plugins yet, but I can at least see foreman_discovery fail on the deprecation of webpacked_plugins_css_for.

evgeni · January 29, 2024, 1:07pm

github.com/theforeman/foreman

Refs #37102 - add back generic node_modules resolve path

theforeman:develop ← theforeman:include-node-modules

opened 12:31PM - 29 Jan 24 UTC

evgeni

+1 -0

evgeni · January 29, 2024, 1:58pm

github.com/theforeman/foreman

Refs #37102 - explicitly depend on buffer

theforeman:develop ← evgeni:buffer

opened 01:57PM - 29 Jan 24 UTC

evgeni

+1 -0

we use it in react_app/common/globalIdHelpers.js so we should explicitly depend… on it

jeremylenz · January 29, 2024, 3:10pm

If you are getting Error: Unknown option '--host' when trying to start the server with bundle exec foreman start, remove the --host option from your .env file. Refs #37102 - filter out webpack host argument by MariaAga · Pull Request #10024 · theforeman/foreman · GitHub

evgeni · January 30, 2024, 8:16am

Welp, turned out all RPM builds didn’t load Webpack JS assets for plugins (even if they were compiled correctly).

github.com/theforeman/foreman

Refs #37102 - detect plugins webpack by looking for remoteEntry.js

theforeman:develop ← evgeni:detect-webpack

opened 08:14AM - 30 Jan 24 UTC

evgeni

+6 -1

Since we switched to Webpack 5, there are no more manifest.json in the plugins w…ebpack folder, but we need some token to detect that we gotta load webpack stuff.

evgeni · January 30, 2024, 8:27am

I think it’s because of

github.com

theforeman/foreman/blob/b5e5dfb4108d36210a1691af45d6d8b980f4f6d1/test/test_helper.rb#L65-L69


      
          module ReactjsHelper
            def read_webpack_manifest
              {"assetsByChunkName" => {"foreman-vendor" => ["foreman-vendor.js", "foreman-vendor.css"]}}
            end
          end

MariaAga · January 30, 2024, 1:26pm

What kind of failure? I can see the pages from foreman_discovery with a deprecation warning in the logs

ekohl · January 30, 2024, 5:58pm

Deprecations are fatal in tests, so it fails the test suite. I also see errors on

Error:
DiscoveredHostsTest::using Create Host link::with a Organization selected#test_0001_it passes it on:
ActionController::RoutingError: No route matches [GET] "/webpack/foreman-vendor.css"
    lib/foreman/middleware/logging_context_request.rb:11:in `call'

Perhaps these are related.

evgeni · January 30, 2024, 6:31pm

Oh, and one thing (that is not blocking, but will be confusing to users if they ever look), the console is full with Component name already taken warnings, as reported by Ron in

evgeni · January 31, 2024, 7:15am

Great, with all those things in, we’re down to one error in Katello.

The “Sync Status” (/katello/sync_management) page is broken, as the console now contains the following error:

63:2 Uncaught ReferenceError: localize is not defined

This comes from this change here:

MariaAga · February 5, 2024, 1:33pm

fixed and merged this Refs #37104 - fix sync_management by MariaAga · Pull Request #10873 · Katello/katello · GitHub

evgeni · February 5, 2024, 1:57pm

Yeah, kinda forgot to mention that, but since last Friday all plugins are rebuilt with Webpack 5 (on Debian and RPM) and seem to be working fine, at least as much our CI is concerned.

MariaAga · February 9, 2024, 3:53pm

Opened Fixes #37154 - run componentRegistry.registerMultiple once by MariaAga · Pull Request #10042 · theforeman/foreman · GitHub

@import '~@theforeman/vendor/… (FIXED)

Can't resolve 'buffer' when building REX RPM (FIXED)

Field 'browser' doesn't contain a valid alias configuration (FIXED)

Plugins CI can’t load /webpack/foreman-vendor.(js|css)

`@import '~@theforeman/vendor/…` (FIXED)

`Can't resolve 'buffer'` when building REX RPM (FIXED)

`Field 'browser' doesn't contain a valid alias configuration` (FIXED)

Plugins CI can’t load `/webpack/foreman-vendor.(js|css)`