Webpack plugin assets in .deb

tbrisker · June 28, 2018, 1:52pm

Hello,

It would appear that .deb packages for plugins have not been compiling and/or including webpack assets.
Yesterday, the following commit was merged to attempt to fix the issue:

however there are still two things we need to solve for it to work properly:

link from $plugin_path/public/webpack to /var/lib/foreman/public/webpack/$plugin_name, or package them differently so they are present at the correct location (in which case we will likely need a link the other way around)
not crucial, but right now the plugin also contains the vendor and bundle files from foreman core which aren’t needed.

My knowledge of debian packing is even smaller then my knowledge of rpms, so it would be good if someone more knowledgeable then me takes a look at this. Thanks!

mmoll · June 28, 2018, 8:33pm

Why is this needed? Can this also be solved with some different copying?

tbrisker · July 1, 2018, 7:53am

Right now, to combine all the plugins’ manifests into one, Foreman looks inside each plugin’s folder for the manifest and assets:

github.com

theforeman/foreman/blob/develop/config/initializers/assets.rb#L114


#
# Otherwise we need to combine all the chunks from the various webpack
# manifests. This is the main foreman manifest and all plugins that may
# have one. We then store this in the webpack-rails manifest using our
# monkey patched function.
unless config.webpack.dev_server.enabled
  if (webpack_manifest_file = Dir.glob("#{Rails.root}/public/webpack/manifest.json").first)
    webpack_manifest = JSON.parse(File.read(webpack_manifest_file))


    Foreman::Plugin.all.each do |plugin|
      manifest_path = File.join(plugin.path, 'public', 'webpack', plugin.id.to_s, 'manifest.json')


      if File.file?(manifest_path)
        Rails.logger.debug { "Loading #{plugin.id} webpack asset manifest from #{manifest_path}" }
        assets = JSON.parse(File.read(manifest_path))


        webpack_manifest['assetsByChunkName'][plugin.id.to_s] = assets['assetsByChunkName'][plugin.id.to_s] if assets['assetsByChunkName'].key?(plugin.id.to_s)
      end
    end


    Webpack::Rails::Manifest.manifest = webpack_manifest

The files also have to be available in /var/lib/foreman/public to be downloadable by the browser iiuc.
In rpms, if i’m not mistaken (@dLobatog correct if i am), this is solved by symlinking the plugin’s dir into the foreman dir:

github.com

theforeman/foreman-packaging/blob/rpm/develop/packages/foreman/foreman/foreman.spec#L1151


%%{?-s:/usr/bin/%%{?scl:%%{scl}-}rake %%{-r*}%%{!?-r:plugin:assets:precompile[%%{-n*}%%{!?-n:%%{gem_name}}]} RAILS_ENV=production --trace} \\
%%{?-a:/usr/bin/%%{?scl:%%{scl}-}rake db:create db:schema:load SCHEMA=%{_datadir}/%{name}/schema_plugin.rb RAILS_ENV=development --trace} \\
%%{?-a:/usr/bin/%%{?scl:%%{scl}-}rake db:migrate RAILS_ENV=development --trace} \\
%%{?-a:/usr/bin/%%{?scl:%%{scl}-}rake plugin:apipie:cache[%%{gem_name}] RAILS_ENV=development cache_part=resources OUT=%%{buildroot}%%{%{name}_apipie_cache_plugin} --trace} \\
\\
popd \\
rm -rf ./usr \\
%%{?-a:mkdir -p %%{buildroot}%%{foreman_dir}/public/apipie-cache/plugin} \\
%%{?-a:ln -s %%{%{name}_apipie_cache_plugin} %%{buildroot}%%{%{name}_apipie_cache_foreman}} \\
%%{?-s:[ -e %%{buildroot}%%{%{name}_webpack_plugin} ] && mkdir -p %%{buildroot}%%{foreman_dir}/public/webpack} \\
%%{?-s:[ -e %%{buildroot}%%{%{name}_webpack_plugin} ] && ln -s %%{%{name}_webpack_plugin} %%{buildroot}%%{%{name}_webpack_foreman}}
EOF


%clean
rm -rf %{buildroot}


%files
%defattr(-,root,root,0755)
%doc CHANGELOG Contributors README.md VERSION
%license LICENSE
%dir %{_datadir}/%{name}

The other option would be to change the way foreman core looks for the plugins, which may work but I haven’t tested (and would require cherry-picking to 1.18).

mmoll · July 1, 2018, 11:39am

In the current .deb, /var/lib/foreman/public/webpack/foreman_remote_execution/manifest.json is included, as the whole webpack folder gets copied after building. Is this sufficient or are there more files directly from the plugin required?

tbrisker · July 1, 2018, 11:49am

The only files that actually need to be copied from that folder (or packaged for that matter) are manifest.json and foreman_remote_execution*.
However we also need a symlink from inside the plugin’s root directory (/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_remote_execution-1.5.3/public/webpack for the current plugin and ruby versions) to the same folder:

ln -s /var/lib/foreman/public/webpack/ /usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_remote_execution-1.5.3/public/webpack

Where I got lost in the debian packaging is how to make this link work without having to change the versions in it every time, as well as how to only copy some of the files to the public folder (the asset compilation also generates vendor.js and bundle.js that aren’t needed as they are shipped in foreman.core).

mmoll · July 1, 2018, 12:36pm

I’m giving up on this.

I and others invested days or even weeks on make this whole webpack and plugins stuff somehow work, but every step is resulting in more and more “requirements”.

Specifically about that problem: The symlink in question is also dependent on the Ruby (ABI) version which requires this symlink to be added and removed in the plugin DEB’s postinst, as Ruby versions are potentially different per distro. I guess this can be made work somehow by using bundler to query parts of the needed paths.

tbrisker · July 1, 2018, 12:46pm

bundle show foreman_remote_execution gives the path to the gem. I’m not familiar enough with the way deb packages work to fix this, which is why I asked for your assistance. I have verified that once the symlink is in place the webpack code works on nightlies.

The other option is to drop webpack support in debians, meaning the ui will not work for many plugins.

Believe me, I understand your frustration, I’ve also had my share of fighting with npm and webpack, but we’re finally to a point where plugins are able to leverage webpack to create better UI for our users, and I would hate to see that break because of a missing symlink. If you do not want to deal with it, can you please point me in the direction of what needs to be done for this to work?

mmoll · July 1, 2018, 12:51pm

it’s in plugins/ruby-foreman-remote-execution/debian/postinst. The removal of the symlink needs a new plugins/ruby-foreman-remote-execution/debian/postrm, see debian/stretch/foreman/foreman.postrm for an example of the parameters, although I think it just can remove the symlink with each of the params.