I’ve opened Bug #23721: Nightlies javascript does not work - Foreman to keep track of this. At this moment, nightlies assets do not work. I have not investigated the cause yet.
Any help is appreciated!
On a related note the katello pipeline is also failing (https://ci.theforeman.org/blue/rest/organizations/jenkins/pipelines/katello-nightly-release/runs/254/nodes/137/steps/144/log/?start=0)
not ok 30 publish content view
# (in test file fb-content-katello.bats, line 215)
# `hammer content-view publish --organization="${ORGANIZATION}" \' failed with status 70
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb planned: 0.0/1, 0%, elapsed: 00:00:00
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.12903225806451613/1, 12%, 0.1/s, elapsed: 00:00:02, ETA: 00:00:14
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.22580645161290322/1, 22%, 0.1/s, elapsed: 00:00:04, ETA: 00:00:14
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.3064516129032258/1, 30%, 0.0/s, elapsed: 00:00:06, ETA: 00:00:14
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.3225806451612903/1, 32%, 0.0/s, elapsed: 00:00:08, ETA: 00:00:18
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.3709677419354839/1, 37%, 0.0/s, elapsed: 00:00:10, ETA: 00:00:18
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.46774193548387094/1, 46%, 0.0/s, elapsed: 00:00:12, ETA: 00:00:14
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.5161290322580645/1, 51%, 0.0/s, elapsed: 00:00:14, ETA: 00:00:16
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.5967741935483871/1, 59%, 0.0/s, elapsed: 00:00:17, ETA: 00:00:13
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.6935483870967742/1, 69%, 0.0/s, elapsed: 00:00:19, ETA: 00:00:10
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:21, ETA: 00:00:02
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:23, ETA: 00:00:02
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:25, ETA: 00:00:02
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:28, ETA: 00:00:03
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:30, ETA: 00:00:04
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:32, ETA: 00:00:06
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:34
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:36
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:38
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:40
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:43
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:45
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:47
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:49
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb running: 0.9032258064516129/1, 90%, 0.0/s, elapsed: 00:00:51
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb error: 0.9193548387096774/1, 91%, 0.0/s, elapsed: 00:00:53, ETA: 00:02:09
# Task 59c85dba-f7bf-4cfe-8b68-470d0742b8cb error: 0.9193548387096774/1, 91%, 0.0/s, elapsed: 00:00:53, ETA: 00:02:09
# Error: ERF12-4115 [ProxyAPI::ProxyException]: Unable to get classes from Puppet for KT_Test_Organization_Library_Test_CV_4 ([RestClient::ServiceUnavailable]: 503 Service Unavailable) for proxy https://centos7-katello-bats-ci.localdomain:9090/puppet
# {"content_view_id"=>4,
# "content_view_version_id"=>5,
# "composite_version_auto_published"=>[],
# "composite_view_publish_failed"=>[],
# "composite_auto_publish_task_id"=>[]}
I am pretty sure it’s not a problem with the compilation itself.
It looks like sudo foreman-rake webpack:compile shouldn’t actually work, it ran it after the assets already compiled and the /webpack source code already removed. And it don’y actually have /node_modules since they compiled into the vendor.js and then removed?
What I do see is this error:
vendor-082a89412fcf1fa4db57.js:formatted:41890 Uncaught TypeError: Cannot set property 'actionHeaderCellFormatter' of undefined
From the compiled version of the vendor,js:
Yw.propTypes = {
children: Ww.a.node,
className: Ww.a.string
},
Yw.defaultProps = {
children: null,
className: ""
};
var Xw = Yw;
Qv[!0].actionHeaderCellFormatter = rv,
Qv[!0].customHeaderFormattersDefinition = sv,
Qv[!0].defaultSortingOrder = nk,
Qv[!0].selectionCellFormatter = pv,
Qv[!0].selectionHeaderCellFormatter = kv,
Qv[!0].sortableHeaderCellFormatter = Tv,
Qv[!0].tableCellFormatter = Nv,
Qv[!0].inlineEditFormatterFactory = Xv,
Qv[!0].Actions = dk,
Qv[!0].Button = wk,
Qv[!0].Cell = Ok,
Qv[!0].Checkbox = Mk,
Qv[!0].DropdownKebab = Vk,
Qv[!0].Heading = Zk,
Qv[!0].PfProvider = Pw,
Qv[!0].InlineEditRow = gw,
Qv[!0].TableInlineEditHeaderRow = _w,
Qv[!0].SelectionCell = qw,
Qv[!0].SelectionHeading = Xw,
Qv[!0].TABLE_ALIGN = Jv,
Qv[!0].TABLE_ALIGNMENT_TYPES = Zv,
Qv[!0].TABLE_SORT_DIRECTION = ek,
Qv[!0].TABLE_SORT_DIRECTIONS = tk;
It’s basically the compiled version of the Table/index.js file from patternfly-react.
The file look like:
import actionHeaderCellFormatter from './Formatters/actionHeaderCellFormatter';
...
import { Table } from './Table';
...
Table.actionHeaderCellFormatter = actionHeaderCellFormatter;
export { Table, actionHeaderCellFormatter };
Should be fine, but inside the ./Formatters/actionHeaderCellFormatter:
Maybe this is the problem? The fact they are importing each-other!
import { Table } from '../index';
I’m keep playing with it, and trying to achieve the same error in a dev environment with no success.
1 Like
New version of patternfly-react was released 2 days ago:
https://github.com/patternfly/patternfly-react/releases/tag/patternfly-react%402.3.5
That could be related. I’m digging into what has changed there.
I thought we are pinning pf-react? or atleast limited by the packaging
effort for it, so what would be really helpful is to either get:
a. log of when nightly is built (hopefully with an error in there)
b. have a way to reproduce the build environment to try it out?
I’m not sure when, but i guess some pr change it (pinning pf) and we didn’t noticed.
a. I don’t think it was a build error
b. +1
There’s a log in http://koji.katello.org/koji/taskinfo?taskID=100943 for the absolute latest nightly, or simply visit http://ci.theforeman.org/job/release_nightly_build_rpm/lastBuild/
To reproduce the build environment you may use the instructions in https://github.com/theforeman/foreman-packaging/tree/rpm/develop#howto-build-multiple-packages
@Tomas_Strachota could you reproduce it?
I’m not sure why this set of commands do not reproduce it for me:
vagrant up centos7-devel
vagrant ssh centos7-devel
cd foreman
RAILS_ENV=production bundle exec rails assets:precompile
RAILS_ENV=production bundle exec rails webpack:compile
RAILS_ENV=production bundle exec rails plugin:assets:precompile[katello]
RAILS_ENV=production bundle exec rails server
Because that uses npm install rather than our RPM NPM packages. You’re probably pulling in different versions.
My suspicion is that we’re using a feature from a library that’s only in a recent version. Our packages match package.json but if that’s not accurate then that’s likely the source of the problem. I’ve seen before that webpack happily compiles a pack that’s then invalid.
I am spending to much time trying to reproduce it with no success.
I can’t point at the problem with confidence but my instincts tell me it’s about the pattrnfly-react dual-importing issue.
I created this PR to patternfly-react and I hope it will fix the issue 
https://github.com/patternfly/patternfly-react/pull/378
@dLobatog there is a way I can copy the node_modules folder from the build to the foreman folder?
If you want to inspect the output bundle, you may just create a new VM with forklift with Foreman nightly and look in /usr/share/foreman.
If you are convinced the dual-import is the issue, you could go to foreman-packaging, downgrade patternfly-react to a version that doesn’t have this bug, and build Foreman RPMs, install them and test them. I don’t think it’d be easy to reproduce this in development without knowing the root cause in production.
The node_modules in the build is just a temporary link from /usr/lib/node_modules/ to the Foreman directory, during the build stage. (https://github.com/theforeman/foreman-packaging/blob/rpm/develop/packages/foreman/foreman/foreman.spec#L973 contains the exact commands that do this.
After that happens and webpack --bail --config config/webpack.config.js is called, the bundle generated is what will be used.
If you want to get to that stage, the way to reproduce this exactly is to simply go ahead and build the packages. Follow the instructions in https://github.com/theforeman/foreman-packaging/tree/rpm/develop#howto-build-multiple-packages , a script like this will do it: https://gist.github.com/dLobatog/de3530b865f61e294f7bec0eb892a178 .
Once you’re able to build the packages, you can try to inspect the bundle, etc by putting some “debug” commands in the foreman.spec - by that I mean ls, cat and friends.
Of course this would be complicated in a system that’s not Fedora/Centos/RHEL - you can probably use a Forklift vm for this, configuring mock and tito as the README says.
It was as easy as installing Foreman from nightlies. Would it help if I
gave you access to machine with such setup? Or do you need semi-dev setup
so you can debuh further?
Where would you find node_modules in a nightly installation?
usr/share/foreman just contains the webpack:compile bundle in public/webpack. /usr/lib/node_modules is clean, we don’t require you to install any nodejs dep as a runtime dependency.
I think a hacky alternative to building the packages is to create a nightly VM with forklift, disable all repos but Foreman, then 'yum install nodejs*, then ln -s /usr/lib/node_modules /usr/share/foreman/node_modules, but they wouldn’t be used by Foreman in production anyway.
Installing foreman-assets is probably the easiest way. It has the same Requires as the main package has as BuildRequires. There can be a slight difference if a newer package is pulled into the repo, but we don’t do it that often. If that would fix it, the next nightlies would include that fix anyway.
Also note that you can use mock shell to enter the chroot you build packages in.
TBH: I find this process fairly complicated, even after 2(?) years of working with it, I don’t believe the average developer feels “in control” of the end result, and it is still a fairly large pain point.
I think its time to abandon the current way of packaging npm assets for ui proposes, and simply keep the node_module (for foreman and each plugin) as part (or separate) release tar.
Honestly, I don’t understand why we are willing to spend so many cycles to make this process work.
+1 for separate release tar.
having a tar with node_modules would really help in this case.
I created those PRs:
https://github.com/theforeman/foreman/pull/5641
https://github.com/Katello/katello/pull/7416
I hope it will fix the issue once it will get merged.
@dLobatog thanks for the information, I will play with it in order to know the process better and expand my toolbox.
Right now I just can’t spend more time on it (unless my solution won’t work and I will have no choice 
)
Can we merge the PR and trigger the nightly build manually?
I agree that the current process is not great, but I’m also not convinced shipping a tarbal is the one we want. How do you deal with plugins that ship webpack assets. Will each have their own tarball or are they not allowed to have any of their own dependencies?
If the problem is that you want to know what is used to build the assets run on any centos 7 machine:
cat > /etc/yum.repos.d/foreman-staging.repo <<EOF
[foreman-staging]
name=Foreman Staging
baseurl=http://koji.katello.org/releases/yum/foreman-nightly/RHEL/7/x86_64/
gpgcheck=0
EOF
yum -y install epel-release foreman-release-scl
yum -y install foreman-assets
tar -caf /tmp/node_modules.tar.gz /usr/lib/node_modules
It looks like we broke the mashing process when branching so our builds aren’t propagating to the above yum repo. That with only including rails 5.1.6 (while foreman depends on 5.1.4) means you need a temporary workaround to enable a repo with 5.1.4.
wget https://copr.fedorainfracloud.org/coprs/g/theforeman/tfm-ror51/repo/epel-7/group_theforeman-tfm-ror51-epel-7.repo -O /etc/yum.repos.d/tfm-ror51.repo
Note this overwrites the normal repo we install.
@ekohl I believe plugins should come with their own node-modules,
our build process should (and it does afaik) create a vendor.js with list of shared modules, plugins who use the same modules (and same version) can use them from the vendor.js instead of build it into their own plugin-bundle.js.
After the build, production env don’t need the node-modules anymore because everything got compiled into vendor and bundles.
Just saving snapshot (for core and each plugin) a moment before the build starts can give tons of benefits.