Attempt to fix the broken katello nightly

sharvit · January 21, 2020, 10:32am

I could verify the centos7-katello-nightly box is broken, once you open it you will see a javascript error in the login page while the page is still functioning:

katello:fills-ccdbc5a5bf8461855f61.js:1 Uncaught DOMException: Failed to execute 'define' on 'CustomElementRegistry': the name "react-component" has already been used with this registry
    at Object../webpack/assets/javascripts/react_app/common/MountingService.js (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/katello/katello:fills-ccdbc5a5bf8461855f61.js:1:27287)
    at o (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/vendor-4f7f4f3563d146a3e24d.js:1:411)
    at Object.../../../../../BUILDROOT/tfm-rubygem-katello-3.15.0-0.4.pre.master.20200119210854git796b07d.el7.noarch/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.0.pre.master/webpack/fills_index.js (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/katello/katello:fills-ccdbc5a5bf8461855f61.js:1:6343)
    at o (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/vendor-4f7f4f3563d146a3e24d.js:1:411)
    at window.webpackJsonp (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/vendor-4f7f4f3563d146a3e24d.js:1:284)
    at https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/katello/katello:fills-ccdbc5a5bf8461855f61.js:1:1

When I navigate to /foreman_tasks/tasks I receive a blank page with more JS errors:

foreman-tasks-8f031ef96597f7cc6551.js:1 Uncaught TypeError: (0 , i.translate) is not a function
    at Object../webpack/assets/javascripts/react_app/components/common/charts/LineChart/index.js (foreman-tasks-8f031ef96597f7cc6551.js:1)
    at o (bootstrap 860ad639982f0cf45c34:54)
    at Object../webpack/assets/javascripts/react_app/components/componentRegistry.js (foreman-tasks-8f031ef96597f7cc6551.js:1)
    at o (bootstrap 860ad639982f0cf45c34:54)
    at Object.../../../../../BUILDROOT/tfm-rubygem-foreman-tasks-0.17.3-1.fm1_25.el7.noarch/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.3/webpack/index.js (foreman-tasks-8f031ef96597f7cc6551.js:1)
    at o (bootstrap 860ad639982f0cf45c34:54)
    at window.webpackJsonp (bootstrap 860ad639982f0cf45c34:25)
    at foreman-tasks-8f031ef96597f7cc6551.js:1
./webpack/assets/javascripts/react_app/components/common/charts/LineChart/index.js @ foreman-tasks-8f031ef96597f7cc6551.js:1
o @ bootstrap 860ad639982f0cf45c34:54
./webpack/assets/javascripts/react_app/components/componentRegistry.js @ foreman-tasks-8f031ef96597f7cc6551.js:1
o @ bootstrap 860ad639982f0cf45c34:54
../../../../../BUILDROOT/tfm-rubygem-foreman-tasks-0.17.3-1.fm1_25.el7.noarch/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.3/webpack/index.js @ foreman-tasks-8f031ef96597f7cc6551.js:1
o @ bootstrap 860ad639982f0cf45c34:54
window.webpackJsonp @ bootstrap 860ad639982f0cf45c34:25
(anonymous) @ foreman-tasks-8f031ef96597f7cc6551.js:1
componentRegistry.js:60 Uncaught Error: Component not found: ForemanTasks among ReactApp, SearchBar, AutoComplete, DonutChart, LineChart, StatisticsChartsList, PowerStatus, NotificationContainer, ToastNotifications, StorageContainer, PasswordStrength, BreadcrumbBar, FactChart, Pagination, Layout, EmptyState, BarChart, ChartBox, ComponentWrapper, ConfigReports, DiffModal, TemplateInput, ExternalLogout, Slot, RelativeDateTime, LongDateTime, ShortDateTime, IsoDate, FormField, InputFactory, ModelsTable, Editor, TemplateGenerator, LoginPage
    at Object.getComponent (componentRegistry.js:60)
    at Object.markup (componentRegistry.js:94)
    at Pi (MountingService.js:22)
    at Object.Si (MountingService.js:11)
    at tasks:75
getComponent @ componentRegistry.js:60
markup @ componentRegistry.js:94
Pi @ MountingService.js:22
Si @ MountingService.js:11
(anonymous) @ tasks:75
katello:fills-ccdbc5a5bf8461855f61.js:1 Uncaught DOMException: Failed to execute 'define' on 'CustomElementRegistry': the name "react-component" has already been used with this registry
    at Object../webpack/assets/javascripts/react_app/common/MountingService.js (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/katello/katello:fills-ccdbc5a5bf8461855f61.js:1:27287)
    at o (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/vendor-4f7f4f3563d146a3e24d.js:1:411)
    at Object.../../../../../BUILDROOT/tfm-rubygem-katello-3.15.0-0.4.pre.master.20200119210854git796b07d.el7.noarch/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.0.pre.master/webpack/fills_index.js (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/katello/katello:fills-ccdbc5a5bf8461855f61.js:1:6343)
    at o (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/vendor-4f7f4f3563d146a3e24d.js:1:411)
    at window.webpackJsonp (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/vendor-4f7f4f3563d146a3e24d.js:1:284)
    at https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/katello/katello:fills-ccdbc5a5bf8461855f61.js:1:1

Navigating to /subscriptions giving a similar result with a blank page and the JS errors:

katello-717267ff57e4a17eb947.js:1 Uncaught DOMException: Failed to execute 'define' on 'CustomElementRegistry': the name "react-component" has already been used with this registry
    at Object../webpack/assets/javascripts/react_app/common/MountingService.js (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/katello/katello-717267ff57e4a17eb947.js:1:391569)
    at o (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/vendor-4f7f4f3563d146a3e24d.js:1:411)
    at Object.../../../../../BUILDROOT/tfm-rubygem-katello-3.15.0-0.4.pre.master.20200119210854git796b07d.el7.noarch/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.0.pre.master/webpack/index.js (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/katello/katello-717267ff57e4a17eb947.js:1:354686)
    at o (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/vendor-4f7f4f3563d146a3e24d.js:1:411)
    at window.webpackJsonp (https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/vendor-4f7f4f3563d146a3e24d.js:1:284)
    at https://centos7-katello-nightly.sharvit-fedorabook-t480s.example.com/webpack/katello/katello-717267ff57e4a17eb947.js:1:1
./webpack/assets/javascripts/react_app/common/MountingService.js @ katello-717267ff57e4a17eb947.js:1
o @ bootstrap 860ad639982f0cf45c34:54
../../../../../BUILDROOT/tfm-rubygem-katello-3.15.0-0.4.pre.master.20200119210854git796b07d.el7.noarch/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.0.pre.master/webpack/index.js @ katello-717267ff57e4a17eb947.js:1
o @ bootstrap 860ad639982f0cf45c34:54
window.webpackJsonp @ bootstrap 860ad639982f0cf45c34:25
(anonymous) @ katello-717267ff57e4a17eb947.js:1
componentRegistry.js:60 Uncaught Error: Component not found: katello among ReactApp, SearchBar, AutoComplete, DonutChart, LineChart, StatisticsChartsList, PowerStatus, NotificationContainer, ToastNotifications, StorageContainer, PasswordStrength, BreadcrumbBar, FactChart, Pagination, Layout, EmptyState, BarChart, ChartBox, ComponentWrapper, ConfigReports, DiffModal, TemplateInput, ExternalLogout, Slot, RelativeDateTime, LongDateTime, ShortDateTime, IsoDate, FormField, InputFactory, ModelsTable, Editor, TemplateGenerator, LoginPage
    at Object.getComponent (componentRegistry.js:60)
    at Object.markup (componentRegistry.js:94)
    at Pi (MountingService.js:22)
    at Object.Si (MountingService.js:11)
    at subscriptions:71

What does it mean?

Looking closer at the error: the name "react-component" has already been used with this registry.
It happens because we are trying to define a new web-component called “react-component” but this web-component already defined before.

There is an issue with our build where the source code of the foreman/webpack/assets/javascripts/react_app/* gets copied to each plugin bundle.
See: Bug #27195: Plugin stylesheet bundles contain styles from core - Foreman

We know about this issue for a long while but no one actually tried to fix it because:

Somehow everything used to work well
Nobody had the capacity or the willingness to fix that issue
I suggested a long-term solution which includes redesigning the js-build
It is in a development process, see the discussion we had about that:
RFC: Changing how we handle Webpack Building
Webpacking plugins, let's do it right
Redesigning the javascript stack

I try to fix that issue in this PR but it wasn’t stable enough because of the usage of the webpack-common-chunk-plugin, see: Fixes #27195 - react_app folder as a common chunk by sharvit · Pull Request #7320 · theforeman/foreman · GitHub

Trying to reproduce on a `centos7-luna-devel`

Unfortunately, I cannot reproduce this error in a devel box centos7-luna-devel.

I tried running a development server

bundle exec foreman start

No Errors

I tried building production and running a production server:

RAILS_ENV=production bundle exec rake assets:precompile
RAILS_ENV=production bundle exec rake webpack:compile
RAILS_ENV=production bundle exec foreman start rails

No Errors

I tried to remove foreman/public/webpack/katello* and build katello/public/webpack:

rm -rf public/webpack/katello*
RAILS_ENV=production bundle exec rake plugin:assets:precompile[katello]
RAILS_ENV=production bundle exec foreman start rails

No Errors but when I navigate to /subscriptions I get a blank page with a JS error:

react-dom.production.min.js:187 TypeError: Cannot read property 'canDeleteManifest' of undefined
    at t.value (SubscriptionsPage.js:176)
    at gi (react-dom.production.min.js:173)
    at mi (react-dom.production.min.js:172)
    at Gs (react-dom.production.min.js:247)
    at Bs (react-dom.production.min.js:216)
    at Ds (react-dom.production.min.js:210)
    at Ns (react-dom.production.min.js:206)
    at react-dom.production.min.js:114
    at t.unstable_runWithPriority (scheduler.production.min.js:20)
    at co (react-dom.production.min.js:113)
Ni @ react-dom.production.min.js:187
Ui.n.callback @ react-dom.production.min.js:200
Do @ react-dom.production.min.js:125
zo @ react-dom.production.min.js:125
(anonymous) @ react-dom.production.min.js:239
t.unstable_runWithPriority @ scheduler.production.min.js:20
co @ react-dom.production.min.js:113
$s @ react-dom.production.min.js:230
Ns @ react-dom.production.min.js:206
(anonymous) @ react-dom.production.min.js:114
t.unstable_runWithPriority @ scheduler.production.min.js:20
co @ react-dom.production.min.js:113
fo @ react-dom.production.min.js:114
po @ react-dom.production.min.js:113
As @ react-dom.production.min.js:204
enqueueSetState @ react-dom.production.min.js:127
w.setState @ react.production.min.js:13
(anonymous) @ i18nProviderWrapperFactory.js:19
Promise.then (async)
I18nProviderWrapper(Connect(Component)) @ i18nProviderWrapperFactory.js:18
Ho @ react-dom.production.min.js:129
mi @ react-dom.production.min.js:167
Gs @ react-dom.production.min.js:247
Bs @ react-dom.production.min.js:216
Ds @ react-dom.production.min.js:210
As @ react-dom.production.min.js:204
il @ react-dom.production.min.js:263
sl @ react-dom.production.min.js:263
(anonymous) @ react-dom.production.min.js:272
Is @ react-dom.production.min.js:208
fl @ react-dom.production.min.js:272
render @ react-dom.production.min.js:273
b @ MountingService.js:21
t.mount @ MountingService.js:11
(anonymous) @ subscriptions:71
SubscriptionsPage.js:176 Uncaught (in promise) TypeError: Cannot read property 'canDeleteManifest' of undefined
    at t.value (SubscriptionsPage.js:176)
    at gi (react-dom.production.min.js:173)
    at mi (react-dom.production.min.js:172)
    at Gs (react-dom.production.min.js:247)
    at Bs (react-dom.production.min.js:216)
    at Ds (react-dom.production.min.js:210)
    at Ns (react-dom.production.min.js:206)
    at react-dom.production.min.js:114
    at t.unstable_runWithPriority (scheduler.production.min.js:20)
    at co (react-dom.production.min.js:113)

At this point, I am very confused

I tried to remove foreman/public/webpack/foreman-tasks* and build foreman-tasks/public/webpack:

rm -rf public/webpack/foreman-tasks*
RAILS_ENV=production bundle exec rake plugin:assets:precompile[foreman-tasks]
RAILS_ENV=production bundle exec foreman start rails

Same like the previous, error when navigation to /subscriptions

Questions I still can’t answer

Why now? It used to work before. AFAIK it used to work after we introduced the web-component.
How to reproduce it in a devel machine?

I will keep updating here about the effort to fix it.

Marek_Hulan · January 21, 2020, 12:47pm

Thanks Avi for the write up and update. If that helps I can give you a production machine, that’s broken, for experimenting. There are two more broken pages - editing a hostgroup reported back in November, @amirfefer any progress with the fix? The second (sub)page is the ansible roles tab in host form, no content is rendered there.

sharvit · January 21, 2020, 1:15pm

Thanks, I can spin up a production machine and reproduce the error but I can’t do that with a devel machine so I can’t update the code, rebuild and see if it fixes the issue.

Just created this PR so it won’t try to register the component if it’s already registered, but as I said, I can’t be sure it is going to fix the problem:
https://github.com/theforeman/foreman/pull/7374

sharvit · January 21, 2020, 2:07pm

Another PR created by @tbrisker to protect the component-registry as well:
https://github.com/theforeman/foreman/pull/7376

sharvit · January 23, 2020, 10:05am

I can reproduce the issue in development now

Seems that another PR I created a while ago reproduce the same issue with the code duplication.
It’s a PR that moves the remaining npm modules from foreman to the vendor.
https://github.com/theforeman/foreman/pull/7233

It uses a different vendor version which isn’t merged yet:
https://github.com/theforeman/foreman-js/pull/81

Using this environment I could find more issues related to code duplication and opened this PR to fix them:
https://github.com/theforeman/foreman/pull/7379

When using this PR I can verify that plugin works again.

Now, once I have an environment where I can reproduce the code duplication issue, I can try to fix the real issue wich is the duplication itself.

ezr-ondrej · January 24, 2020, 12:35pm

I don’t like the window variables. I believe it should work without them, but good job fixing it!

tbrisker · January 26, 2020, 9:23am

Looks like we are now failing on I18N issue. On katello react pages i’m seeing:

Error: "No translation key found."

dcnpgettext jed.js:260

gettext jed.js:172

w I18n.js:62

h helpers.js:87

h helpers.js:87

hp katello-2979b6eb77d74a1a9a8d.js:1

value katello-2979b6eb77d74a1a9a8d.js:1

React 7

unstable_runWithPriority scheduler.production.min.js:20

React 6

a i18nProviderWrapperFactory.js:19

[react-dom.production.min.js:187:194](webpack:///.-core/node_modules/react-dom/cjs/react-dom.production.min.js)

Error: No translation key found.

on tasks pages i’m getting:

TypeError: "e is undefined"
p urlHelpers.js:55
$r foreman-tasks-4687f4329e4a96cab1b7.js:1
React 6
unstable_runWithPriority scheduler.production.min.js:20
React 6
a i18nProviderWrapperFactory.js:19

sharvit · January 26, 2020, 11:26am

Thanks @tbrisker,

Created this PR to fix the issue:
https://github.com/theforeman/foreman/pull/7389

tbrisker · January 26, 2020, 7:56pm

New nightlies have been released but looks like the issue continues.
The strange thing is that for some reason when katello calls propsToSnakeCase function from foreman helpers, the call stack actually goes to the translateObject function instead.
Looking at the source, is it possible that the propsToSnakeCase function gets tree shaken out since nothing in foreman bundle uses it, and when katello bundle is built it remains, causing a mismatch between the object mappings?

tbrisker · January 26, 2020, 7:59pm

Looking at the compiled bundle.js, propsToSnakeCase is indeed not there.

sharvit · January 27, 2020, 7:27am

Nice! It can bypassed by forcing to use it somewhere

sharvit · January 27, 2020, 7:51am

Please work!!
https://github.com/theforeman/foreman/pull/7390

tbrisker · January 27, 2020, 12:52pm

Updating here as well for those not following the PRs:
Looks like the issue we’re now seeing is an outcome of the babel 7 upgrade which was merged in
https://github.com/theforeman/foreman/pull/7178
The default handling of js modules changed from transpiling them to commonjs in babel 6 to keeping them as es6 modules in babel 7.
Webpack 3 knows how to tree-shake (i.e. remove unused code) es6 modules, but not commonjs modules.
The unfortunate outcome for us here is that all functions that weren’t used by core got removed from the core bundle when foreman gets built. This includes, in this case, propsToSnakeCase which was only used by katello and not in core. However, when katello gets built, these functions don’t get tree-shaken out, resulting in katello js code expecting the function to be available in the core bundle (using a very descriptive name such as W.e). However, since the missing function was removed, the uglification resulted in a different function being assigned to W.e, meaning that when katello tries to call it, it actually triggers a different function, in this case translateObject, which obviously fails as it is used for something completely different.

sharvit · January 27, 2020, 1:15pm

Thanks @tbrisker, I could just verify that updating to commonjs fixes the issue

PRs soon

sharvit · January 27, 2020, 1:18pm

https://github.com/theforeman/foreman-js/pull/121

sharvit · January 27, 2020, 1:42pm

https://github.com/theforeman/foreman/pull/7393

tbrisker · January 27, 2020, 2:42pm

https://github.com/theforeman/foreman-packaging/pull/4653

tbrisker · January 27, 2020, 7:14pm

I have just tested with the newly updated packages on koji, and looks like katello pages are working properly again!
Tomorrow we’ll rebuild all webpack-using plugins (unless someone beats me to it earlier - by all means please do ) so they get updated to the latest builder, as there are still issues with them.

tbrisker · January 28, 2020, 8:59am

Opened pr to rebuild all other plugins:
https://github.com/theforeman/foreman-packaging/pull/4657

Attempt to fix the broken katello nightly

What does it mean?

Trying to reproduce on a centos7-luna-devel

Questions I still can’t answer

Trying to reproduce on a `centos7-luna-devel`