What's up with package.json in plugins?

[lzap]

Hey,

this is bugging me quite often now and I want to know more. There are several plugins I currently maintain that have package.json now since some JS components were introduced. When dependencies are bumped in Foreman core, the ones mentioned in plugins do not match, but dev setup still do work fine somehow.

But then I hit problems when I try to perform package update (RPM), we have more strict rules and the packaging script carries versions over from package.json to SPEC. I don’t particularly like catching up with dependencies manually, that is a time wasted. There should be some tool that could maintain this, isn’t package.json similar to bundler that does the same job? Do we want to write some script or instructions for plugin authors to keep them clean?

It’s a mess for me, granted, I do not fully understand how this is supposed to work. I have a feeling that core already moves on when I perform some change to correct it: Clean up package.json by lzap · Pull Request #560 · theforeman/foreman_discovery · GitHub

Can you recommend a good solution to this?

[ekohl]

A previous related discussion on this topic:

Foreman-js - why keep all meta-packages on the same version?

For our foreman-js meta-packages (like @theforeman/vendor), I was told to update them all together when updating one, so in this PR, I updated @theforeman/test to 6.0.0, but needed to update the other @theforeman/* packages to 6.0.0 as well. Seeing something like katello-nightly-rpm-pipeline 756 failed makes me wonder if there is a way these can be updated independently. @theforeman/builder updated to 6.0.0, so Katello’s version was no longer available in packaging. But there were no breaking c…

The question there was never answered.

[lzap]

What is package.json for?

Do we need it for development?

For production?

I have a node_modules in each of my plugins, that is 1.3 GB per plugin is this correct? Is it possible to reuse this content?

Can core provide some kind of “standard” package.json for all plugins? Or at least some base dependency that can be imported by all?

If we really need a separate package.json and node_modules per plugin, can we write a tool that would bump these automatically for plugin authors?

If I compare to Ruby, it’s much better experience:

• We have a Gemfile
• We pin most (all) gems in the Gemfile
• We do not commit Gemfile.lock into the git because we already did the job in the Gemfile
• Plugins simply require foreman gem dependency to get all the dependencies
• Plugins add unique dependencies on their own
• As long as Gemfile and gemspec is okay, packaging is relatively easy

[ezr-ondrej]

@lzap

What is package.json for?

Mostly development, but also used partially at build time.

@lzap

I have a node_modules in each of my plugins, that is 1.3 GB per plugin is this correct? Is it possible to reuse this content?

Somewhere. It is reused AFAIK in the build process and in the browser so it is not reused only for development and tests.

@lzap

If we really need a separate package.json and node_modules per plugin, can we write a tool that would bump these automatically for plugin authors?

I think we could have dynamic dependencies there, that would inherit it from foreman, but that is quite complex.

This is very complicated issue and ppl are trying to figure out a way out: Rails 7 and new frontend approaches POC

But it won’t be fixed in a day given we’ve driven ourselves into that mess three years

[Ron_Lavi]

Thanks @lzap for the post, there is a lot of things to do better,
it seems that our frontend build tools and configuration with Rails is made patch by patch to fix things that didn’t work, and to deal with it we need help from people who understand rails and the assets-pipeline, packaging, and frontend development, Rails is bringing updated approaches to frontend in Rails7 as detailed in Rails 7 and new frontend approaches POC and we must update our deprecated stack but as I said it’s not one person or one group to handle it.

What is package.json for?

Do we need it for development?

For production?

Basically, it’s the same as the Gemfile for Rails,
in development you need it for testing, linter, building the frontend code on changes,
in production, you need to build your plugin code and iiuc, when someone wants to install a plugin, e.g: yum install foreman-tasks all of the frontend assets should be already compiled.

I have a node_modules in each of my plugins, that is 1.3 GB per plugin is this correct? Is it possible to reuse this content?

That’s something I really dislike, and hopefully, we could overcome it via using properly the assets-pipeline / module-federation - discussing it in Rails 7 and new frontend approaches POC

Can core provide some kind of “standard” package.json for all plugins? Or at least some base dependency that can be imported by all?

That’s the goal of foreman-js I believe, and it made the work much better as you don’t need to package each and every module your plugin uses, and you don’t need to care if Foreman core is using another version of your module, because on runtime it will use the core’s module and your plugin could break - all of that is already fixed.

I also went over many plugins and reduced the dependencies they have in package.json so now there are plugins that don’t even have dependencies, just devDependencies.

If we really need a separate package.json and node_modules per plugin, can we write a tool that would bump these automatically for plugin authors?

foreman-js works with semver and treats each version like other npm packages, which might not suite really for Foreman and plugins that don’t want to bump it always, and just accept the newer version of it. one solution could be to use @theforeman/vendor >= version so it will accept all versions above, or @theforeman/vendor: '*' to accept all.

If I compare to Ruby, it’s much better experience:

• We have a Gemfile

There’s package.json

• We pin most (all) gems in the Gemfile

Foreman-js does it for us

• We do not commit Gemfile.lock into the git because we already did the job in the Gemfile

package-lock.json is also ignored

• Plugins simply require foreman gem dependency to get all the dependencies

that’s something I would like to see in plugins as well, but basically we also use foreman-js for it,
but I would like to avoid the 1.5 GB in each plugin.

• Plugins add unique dependencies on their own

that’s also achievable with pagack.json of plugins today, not that I would encourage it too much.
there is a post-install script in foreman that looks for all of the plugins and install their node-modules.

• As long as Gemfile and gemspec is okay, packaging is relatively easy

I believe that as long as the plugin just use all the foreman-js supported modules, and accepts any version of foreman-js, then packaging is also very easy.

[lzap]

Thanks for info, so my takeaway is all of this is really needed, we would like to improve in the future.

Can we do something now to make life of a plugin maintainer a little bit easier?

We literally have one sentence in the How to write a plugin docs:

theforeman/foreman/blob/develop/developer_docs/how_to_create_a_plugin.asciidoc#adding-3rd-party-js-libraries

Then our plugin template github repo contains this which is I assume not valid (2 yo):

theforeman/foreman_templates/blob/master/package.json

And then doing a followup information what to do when preparing RPM/DEB packaging?

For me this is always trial and error and poking other people. Until we improve the tech, I think we can at least streamline the process. I am fine following a guideline if there is anything that is kept up to date.

[Ron_Lavi]

Yeah, I totally agree with you,
We need better documentation on processes so we won’t rely on poking people around