Redesigning the javascript stack

sharvit · April 1, 2019, 2:46pm

Problems the redesign should solve

It is difficult to understand the build process.
We need to do npm2rpm for each npm dependency we use in core.
Plugins can’t build themselves, they need Foreman to get build.
We need to rebuild plugins after each core build.
The build process copies the react_app folder from core to each plugin bundle.
The build process creates an importing alias for plugins from foremanReact to the core react_app folder. While it works when running foreman itself, it doesn’t work in the plugins dev environment (test, lint, storybook).
Plugins find themselvs mocking each piece of code they are using from core, see: katello mocks
To set up a plugin environment there are some deps and configurations need to be set up (jest, babel, eslint, storybook). Those configs get copied from plugin to plugin with small changes so we end up with different env config for each plugin so it cannot be maintained.

Packages Architecture

The foreman has multiple packages, each package can build individually.

@theforeman/env
Provides the basic environment core and plugins needs.
1. Dependencies
  1. babel
  2. eslint
  3. Jest
  4. Storybook
2. Source Exports
  1. babel configuration to extend
  2. eslint configuration to extend
  3. Jest configuration to extend
3. Bin Exports
  1. tfm-lint
  2. tfm-test
  3. tfm-storybook
4. Publish to
  1. npm
@theforeman/vendor
Foreman 3rd party supported libraries collection to use in core and plugins.
Already have a working solution: https://github.com/sharvit/foreman-js
In the end, it got split into 3 packages vendor-core, vendor-dev and vendor.
See the vendor discution here: RFC: Changing how we handle Webpack Building
1. dependencies
  1. […node_modules]
2. devDependencies:
  1. webpack
  2. @theforeman/env
3. Scripts
  1. npm run build
4. Build
  1. Use webpack to build dist files. (manifest.json, bundle.jd, bundle.css)
5. Exports
  1. dist/vendor.bundle.js
  2. dist/vendor.bundle.css
  3. scss/mixins.scss
  4. scss/variables.scss
  5. scss/mixins.scss
  6. scss/vendor.scss
  7. webpack-plugin.js
6. Deploy
  1. npm with ./dist (and source)
  2. rpm with ./dist (and source)
@theforeman/builder
Build production and dev dist files for core and plugins.
Each plugin can use it to create it’s own build.
1. dependencies
  1. Webpack
2. devDependencies:
  1. @theforeman/env
3. Configuration:
  1. Has webpack configurations and webpack-plugins inside
  2. Read the minimal configuration file: tfm-builder.config.js from consumer root.
    This file will export a json with the following fields
    1. entry (extctly like webpack.entry)
    2. output (extctly like webpack.output)
4. bin
  1. tfm-build
  2. tfm-dev-server
  3. tfm-builder-analyze
5. Publish
  1. npm
  2. rpm?
@theforeman/react-application
Will contain the core react_app folder so it can be shared with core and plugins.
Can live in the core repo.
1. dependencies
  1. @theforeman/vendor
2. devDependencies:
  1. @theforeman/builder
  2. @theforeman/env
3. Scripts
  1. npm run build
  2. npm run storybook
  3. npm test
  4. npm run lint
4. Publish
  1. Deploy to npm-registry with ./dist
  2. Deploy to rpm with ./dist
5. Build
  1. Use @theforeman/builder to produce react-application.bundle.js
    1. tfm-builder.config.js
      1. entry: react-application.js
      2. output: ./dist/react-application.bundle.js
Plugins
1. dependencies
  1. @theforeman/vendor
  2. @theforeman/react-application
2. devDependencies:
  1. @theforeman/builder
  2. @theforeman/env
3. Scripts
  1. npm run build
  2. npm run storybook
  3. npm test
  4. npm run lint
4. Build
  1. Use @theforeman/builder to produce [plugin].bundle.js
  2. tfm-builder.config.js
    1. entry: webpack/index.js
    2. output: ./public/webpack/[plugin].bundle.js
5. Publish
  1. Publish the gem together with public/webpack/
Foreman
1. dependencies
  1. @theforeman/vendor
  2. @theforeman/react-application
2. devDependencies:
  1. @theforeman/builder
  2. @theforeman/env
3. Scripts
  1. npm start
  2. npm run build
  3. npm run storybook
  4. npm test
  5. npm run lint
4. Build
  1. Use @theforeman/builder to produce application.bundle.js
  2. tfm-builder.config.js
    1. entry: webpack/assets/application.js
    2. output: ./public/webpack/application.bundle.js
    3. devServer: { port, bind }
5. Publish
  1. Publish the gem together with public/webpack/
6. Run dev
  1. Use @theforeman/builder to run tfm-dev-server

Managin the sub-packages

I belive we should use lerna to create a monorepo with all the sub-packages.
I already created such a repo under my githhub account: https://github.com/sharvit/foreman-js
This repo contains some sub-packages (vendor, vendor-core, vendor-dev).

Installing plugins in development

Current behavior:

For-each plugin
- cd <plugin-path> && npm install
Webpack configuration that can resolve plugins

Wanted behavior:

For-each plugin
- cd <plugin-path> && npm link
- cd <root> && npm link <plugin-name>
  The result will be a symlink in node_modules to the actual plugin path
  ./node_modules/katello -> ../katello

Assets to serve

public/webpack/manifest.json
1. vendor.bundle.(js,css)
2. react-application.bundle.(js,css)
3. application.bundle.(js,css)
For each-plugin
1. <plugin.path>/public/webpack/manifest.json
  <plugin.name>.(js,css)

Serving assets in development

@theforeman/builder supply a script tfm-dev-server that can run webpack-dev-server.
The tfm-dev-server will know how to build and serve the core with plugins based on their configurations.

Other packages we can/want to put in the monorepo

Cheers!

ohadlevy · April 4, 2019, 6:38am

@sharvit thanks for looking into this, I know this is quite a lot of work…

I think its hard for the average developer to understand all of this, and I wonder if it makes sense to have some code examples (e.g. add support in the https://github.com/theforeman/foreman_plugin_template) so its obvious as a developer what it means.

Overall, the direction make sense to me, biggest benefits are that we are not “abusing” webpack and using a “normal” npm package build and dependency workflows. many bonus points for also eliminating the need for per npm package vs just one package for each type (vendor, core, plugin, dev env etc ).

Looking forward to see this in action, perhaps a deep dive when you are ready?

thanks!
Ohad

amirfefer · April 8, 2019, 9:00am

@sharvit Thanks for the effort!, looks great
It’s very informative and in some parts a bit complicated, so scheduling a deep dive would be helpful to make it clear.

sharvit · April 17, 2019, 10:08am

Join me tomorrow for a deep-dive about this topic, it’s a great opportunity to share your concerns and ask questions

It will be recorded as well.

sharvit · April 21, 2019, 6:06am

sharvit · January 23, 2020, 3:27pm

I think it’s a good time for a status update about the redesign.

@theforeman/env
Deprecated and it’s content split into multiple packages:
- @theforeman/builder - the babel configuration, planned to live in the env but moved the the builder.
- @theforeman/eslint-plugin-foreman - eslint moved to this package.
- @theforeman/test - jest and the testing common logic moved to this package.
- @theforeman/stories - storybook and the storybook common logic moved to this package.
@theforeman/vendor
- We already released version 4 of the vendor, it contains almost all the node_modules core uses.
- The vendor already in used in all the plugins (that uses npm)
- feat(vendor-core): adds jed, intl and react-intl and adds support for async imports - This PR adds the remaining modules from foreman to vendor (while also bring support for async loadable modules).
@theforeman/builder
- Under development, see related PRs:
@theforeman/react-application
- The plan is adding logic into @theforeman/builder so it would be able to build the react_app/foremanReact int a standalone library that can be consumed by the main build process. I will update with a final design soon.

Cheers!