Redesigning the javascript stack

rfc

#1

Problems the redesign should solve

  1. It is difficult to understand the build process.
  2. We need to do npm2rpm for each npm dependency we use in core.
  3. Plugins can’t build themselves, they need Foreman to get build.
  4. We need to rebuild plugins after each core build.
  5. The build process copies the react_app folder from core to each plugin bundle.
  6. The build process creates an importing alias for plugins from foremanReact to the core react_app folder. While it works when running foreman itself, it doesn’t work in the plugins dev environment (test, lint, storybook).
    Plugins find themselvs mocking each piece of code they are using from core, see: katello mocks
  7. To set up a plugin environment there are some deps and configurations need to be set up (jest, babel, eslint, storybook). Those configs get copied from plugin to plugin with small changes so we end up with different env config for each plugin so it cannot be maintained.

Packages Architecture

The foreman has multiple packages, each package can build individually.

  1. @theforeman/env
    Provides the basic environment core and plugins needs.

    1. Dependencies

      1. babel
      2. eslint
      3. Jest
      4. Storybook
    2. Source Exports

      1. babel configuration to extend
      2. eslint configuration to extend
      3. Jest configuration to extend
    3. Bin Exports

      1. tfm-lint
      2. tfm-test
      3. tfm-storybook
    4. Publish to

      1. npm
  2. @theforeman/vendor
    Foreman 3rd party supported libraries collection to use in core and plugins.
    Already have a working solution: https://github.com/sharvit/foreman-js
    In the end, it got split into 3 packages vendor-core, vendor-dev and vendor.
    See the vendor discution here: RFC: Changing how we handle Webpack Building

    1. dependencies

      1. […node_modules]
    2. devDependencies:

      1. webpack
      2. @theforeman/env
    3. Scripts

      1. npm run build
    4. Build

      1. Use webpack to build dist files. (manifest.json, bundle.jd, bundle.css)
    5. Exports

      1. dist/vendor.bundle.js
      2. dist/vendor.bundle.css
      3. scss/mixins.scss
      4. scss/variables.scss
      5. scss/mixins.scss
      6. scss/vendor.scss
      7. webpack-plugin.js
    6. Deploy

      1. npm with ./dist (and source)
      2. rpm with ./dist (and source)
  3. @theforeman/builder
    Build production and dev dist files for core and plugins.
    Each plugin can use it to create it’s own build.

    1. dependencies

      1. Webpack
    2. devDependencies:

      1. @theforeman/env
    3. Configuration:

      1. Has webpack configurations and webpack-plugins inside
      2. Read the minimal configuration file: tfm-builder.config.js from consumer root.
        This file will export a json with the following fields
        1. entry (extctly like webpack.entry)
        2. output (extctly like webpack.output)
    4. bin

      1. tfm-build
      2. tfm-dev-server
      3. tfm-builder-analyze
    5. Publish

      1. npm
      2. rpm?
  4. @theforeman/react-application
    Will contain the core react_app folder so it can be shared with core and plugins.
    Can live in the core repo.

    1. dependencies

      1. @theforeman/vendor
    2. devDependencies:

      1. @theforeman/builder
      2. @theforeman/env
    3. Scripts

      1. npm run build
      2. npm run storybook
      3. npm test
      4. npm run lint
    4. Publish

      1. Deploy to npm-registry with ./dist
      2. Deploy to rpm with ./dist
    5. Build

      1. Use @theforeman/builder to produce react-application.bundle.js
        1. tfm-builder.config.js
          1. entry: react-application.js
          2. output: ./dist/react-application.bundle.js
  5. Plugins

    1. dependencies

      1. @theforeman/vendor
      2. @theforeman/react-application
    2. devDependencies:

      1. @theforeman/builder
      2. @theforeman/env
    3. Scripts

      1. npm run build
      2. npm run storybook
      3. npm test
      4. npm run lint
    4. Build

      1. Use @theforeman/builder to produce [plugin].bundle.js
      2. tfm-builder.config.js
        1. entry: webpack/index.js
        2. output: ./public/webpack/[plugin].bundle.js
    5. Publish

      1. Publish the gem together with public/webpack/
  6. Foreman

    1. dependencies

      1. @theforeman/vendor
      2. @theforeman/react-application
    2. devDependencies:

      1. @theforeman/builder
      2. @theforeman/env
    3. Scripts

      1. npm start
      2. npm run build
      3. npm run storybook
      4. npm test
      5. npm run lint
    4. Build

      1. Use @theforeman/builder to produce application.bundle.js
      2. tfm-builder.config.js
        1. entry: webpack/assets/application.js
        2. output: ./public/webpack/application.bundle.js
        3. devServer: { port, bind }
    5. Publish

      1. Publish the gem together with public/webpack/
    6. Run dev

      1. Use @theforeman/builder to run tfm-dev-server

Managin the sub-packages

I belive we should use lerna to create a monorepo with all the sub-packages.
I already created such a repo under my githhub account: https://github.com/sharvit/foreman-js
This repo contains some sub-packages (vendor, vendor-core, vendor-dev).

Installing plugins in development

Current behavior:

  • For-each plugin
    • cd <plugin-path> && npm install
  • Webpack configuration that can resolve plugins

Wanted behavior:

  • For-each plugin
    • cd <plugin-path> && npm link
    • cd <root> && npm link <plugin-name>
      The result will be a symlink in node_modules to the actual plugin path
      ./node_modules/katello -> ../katello

Assets to serve

  1. public/webpack/manifest.json
    1. vendor.bundle.(js,css)
    2. react-application.bundle.(js,css)
    3. application.bundle.(js,css)
  2. For each-plugin
    1. <plugin.path>/public/webpack/manifest.json
      <plugin.name>.(js,css)

Serving assets in development

@theforeman/builder supply a script tfm-dev-server that can run webpack-dev-server.
The tfm-dev-server will know how to build and serve the core with plugins based on their configurations.

Other packages we can/want to put in the monorepo

  1. react-redux-test-utils
  2. react-ellipsis-with-tooltip
  3. slot-and-fill

Cheers!


State of React for Plugins
State of React for Plugins
Deep Dive - RFC: Redesigning the javascript stack
#2

@sharvit thanks for looking into this, I know this is quite a lot of work…

I think its hard for the average developer to understand all of this, and I wonder if it makes sense to have some code examples (e.g. add support in the https://github.com/theforeman/foreman_plugin_template) so its obvious as a developer what it means.

Overall, the direction make sense to me, biggest benefits are that we are not “abusing” webpack and using a “normal” npm package build and dependency workflows. many bonus points for also eliminating the need for per npm package vs just one package for each type (vendor, core, plugin, dev env etc ).

Looking forward to see this in action, perhaps a deep dive when you are ready?

thanks!
Ohad


#3

@sharvit Thanks for the effort!, looks great
It’s very informative and in some parts a bit complicated, so scheduling a deep dive would be helpful to make it clear.


#4

Join me tomorrow for a deep-dive about this topic, it’s a great opportunity to share your concerns and ask questions

It will be recorded as well.


#5