Status of RPM packaging for npm modules in Foreman

[dLobatog]

Hi devs,

Just wanted to give you an update on packaging for npm & webpack
incidentally. We have two PRs in core that are dabbling in a more modern
way of doing frontend development in Foreman.

Fixes #12364 - Use NPM+Webpack to handle external assets

Fixes #13424 - c3 patternfly react implementation

At the moment, most frontend modules are no longer served as gemified
assets but rather as npm modules. It's difficult for us to develop on a
modern stack of tools & get a nice UX experience so the idea is to move
towards a model where npm will provide all the frontend stuff you need.
Read more on: https://github.com/theforeman/rfcs/pull/3

···

--------------------------

Now, regarding packaging (which really has been blocking the two
aforementioned PRs and potential frontend development):

The issue we face is that we need to serve the node_modules (folder
generated after running npm install) dependencies with Foreman somehow.
I’ve been trying several things, with different degrees of success.

1. Use npm2rpm [1] to create a package for every npm module. Only
   ‘direct’ dependencies are specified in these packages, and they store
   merely the dependency itself.

• Upsides:

  • ‘the right way’ according to rpm guidelines
  • no confusion about dependencies and simpler RPM specs

• Downsides:

  • this creates close to 1000 new packages
  • we’d have to serve several different versions of the same package,
    which the OS would refuse to install (e.g if we have uglify-js 0.0.1, and
    1.0.4 in our repos, the former would be superseded by the latter)
  • impossible to review this amount of packages, it must be automated

• Success with POC? No

2. Create ‘nodejs-bundle-xxxxx’ packages which contain all of the
   dependencies for a certain package. e.g webpack Provides
   bundle-webpack-core, etc…

• Upsides:
  • similar to how foreman node_modules is structured in development
• Downsides:

  • conflicting versions for subdependencies force us to keep track
    of the ‘dependency tree’ in the package. For instance in this spec
    https://github.com/dLobatog/foreman-packaging/blob/5fc7bb8e831fa74f579a41e9bf9f367f6a9527ff/nodejs-webpack/nodejs-webpack.spec
    we can’t just unpack all ‘async’ to
    %{buildroot}/%{nodejs_sitelib} as there are several ‘async’
    packages, and the last one to be unpacked will prevail.

  • these packages become huge and unwieldy to review

  • we have to jump hoops to make it work. for instance we have to
    call ‘%nodejs_fixdep -r dependency’ to allow dependencies to not
    conflict (this doesn’t always work)

  • lots of ‘cruft’ is packed, like examples directories, etc…

  • Success with POC? No. I stopped when I figured I had to keep track
    of the dependency tree before unpackaging.

3. Put node_modules in a new package (foreman-npm-assets?) that
   foreman-assets %install section puts in the right place?

• Upsides:

  • relatively easy to do
  • jenkins can automate it

• Downsides:

  • less control or information for users about what’s installed with
    foreman, although we could specify the dependencies under
    “Provides: bundled-npm(xxxx)”

• Success with POC: haven’t tried

4. Make Foreman tar file include node_modules so it’s part of the
   foreman package itself

• Upsides:

  • easiest to do
  • zero work required by devs
  • closest to development

• Downsides:

  • lack of knowledge/control about deps for users, like 3.
  • no possibility of depending on system nodejs-xxx packages, but I
    don’t think we want that.

• Success with POC: Yes!
  https://github.com/dlobatog/foreman-packaging/tree/chainbuild-webpack
  is able to run webpack compile just fine after that. We’d need even
  less than what’s on that branch to do it.

---

I got kind of stuck with 2. because of the tree dependencies thing, and
honestly I don’t see much of a benefit to it now, especially considering
the amount of work there’s involved.

If we can redo this somehow I’d definitely go with 4 which is really
easy and I can’t see how that’s a problem for us. It’d be a problem if
we were to put Foreman in a Linux distribution but we’re not aiming for
that. It is as well more manageable than any other model which requires
hundreds of specs + sources in our foreman-packaging repo.

Aside from that…

Why don’t we go with 4 at least until we find a better/feasible model? It
would allow development to continue (the two PRs I mentioned + more).
There’s plenty of time until 1.13 so I don’t see why we can’t move
forward, merge these PRs and figure out other packaging strategies if we
need to.

---

If you got this far, you surely know if I’m missing something or you can
make some suggestions about which would be the best process, so please
reply.

New ideas, comments, critiques, welcome

[1] - https://github.com/dlobatog/npm2rpm

Best,

–
Daniel Lobato Garcia

@dLobatog

Daniel Lobato – Black box of a budding computer scientist

bio – Daniel Lobato

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

[ohadlevy]

> Hi devs,
>
> Just wanted to give you an update on packaging for npm & webpack
> incidentally. We have two PRs in core that are dabbling in a more modern
> way of doing frontend development in Foreman.
> https://github.com/theforeman/foreman/pull/3433
> https://github.com/theforeman/foreman/pull/3603
>
> At the moment, most frontend modules are no longer served as gemified
> assets but rather as npm modules. It's difficult for us to develop on a
> modern stack of tools & get a nice UX experience so the idea is to move
> towards a model where npm will provide all the frontend stuff you need.
> Read more on: https://github.com/theforeman/rfcs/pull/3
>
> --------------------------
>
> Now, regarding packaging (which really has been blocking the two
> aforementioned PRs and potential frontend development):
>
> The issue we face is that we need to serve the node_modules (folder
> generated after running npm install) dependencies with Foreman somehow.
> I've been trying several things, with different degrees of success.
>
> 1. Use npm2rpm [1] to create a package for every npm module. Only
> 'direct' dependencies are specified in these packages, and they store
> merely the dependency itself.
>
> - Upsides:
> - 'the right way' according to rpm guidelines
> - no confusion about dependencies and simpler RPM specs
> - Downsides:
> - this creates close to 1000 new packages
> - we'd have to serve several different versions of the same package,
> which the OS would refuse to install (e.g if we have uglify-js
> 0.0.1, and
> 1.0.4 in our repos, the former would be superseded by the latter)
> - impossible to review this amount of packages, it must be automated
>
> - Success with POC? No
>
> 2. Create 'nodejs-bundle-xxxxx' packages which contain all of the
> dependencies for a certain package. e.g webpack Provides
> bundle-webpack-core, etc…
>
> - Upsides:
> - similar to how foreman node_modules is structured in development
> - Downsides:
> - conflicting versions for subdependencies force us to keep track
> of the 'dependency tree' in the package. For instance in this spec
>
> https://github.com/dLobatog/foreman-packaging/blob/5fc7bb8e831fa74f579a41e9bf9f367f6a9527ff/nodejs-webpack/nodejs-webpack.spec
> we can't just unpack all 'async' to
> %{buildroot}/%{nodejs_sitelib} as there are several 'async'
> packages, and the last one to be unpacked will prevail.
> - these packages become huge and unwieldy to review
> - we have to jump hoops to make it work. for instance we have to
> call '%nodejs_fixdep -r dependency' to allow dependencies to not
> conflict (this doesn't always work)
> - lots of 'cruft' is packed, like examples directories, etc…
>
> - Success with POC? No. I stopped when I figured I had to keep track
> of the dependency tree before unpackaging.
>
> 3. Put node_modules in a new package (foreman-npm-assets?) that
> foreman-assets %install section puts in the right place?
>
> - Upsides:
> - relatively easy to do
> - jenkins can automate it
>
> - Downsides:
> - less control or information for users about what's installed with
> foreman, although we could specify the dependencies under
> "Provides: bundled-npm(xxxx)"
>
> - Success with POC: haven't tried
>
> 4. Make Foreman tar file include node_modules so it's part of the
> foreman package itself
> - Upsides:
> - easiest to do
> - zero work required by devs
> - closest to development
>
> - Downsides:
> - lack of knowledge/control about deps for users, like 3.
> - no possibility of depending on system nodejs-xxx packages, but I
> don't think we want that.
>
> - Success with POC: Yes!
> https://github.com/dlobatog/foreman-packaging/tree/chainbuild-webpack
> is able to run webpack compile just fine after that. We'd need even
> less than what's on that branch to do it.
>
> ----------------------------------------
>
> I got kind of stuck with 2. because of the tree dependencies thing, and
> honestly I don't see much of a benefit to it now, especially considering
> the amount of work there's involved.
>
> If we can redo this somehow I'd definitely go with 4 which is really
> easy and I can't see how that's a problem for us. It'd be a problem if
> we were to put Foreman in a Linux distribution but we're not aiming for
> that. It is as well more manageable than any other model which requires
> hundreds of specs + sources in our foreman-packaging repo.

> Aside from that…
>
> Why don't we go with 4 at least until we find a better/feasible model? It
> would allow development to continue (the two PRs I mentioned + more).
> There's plenty of time until 1.13 so I don't see why we can't move
> forward, merge these PRs and figure out other packaging strategies if we
> need to.
>

I would be extremely happy to move forward one way or another. I honestly
want our developers to get unblocked asap.

I would ask that we define npm packaging as a blocker for 1.13 release, and
merge the https://github.com/theforeman/foreman/pull/3433 (or a variant of
it) and would prefer option 3 than option 4 if its easily doable, as i do
think foreman tar should not include 3rd party code in general.

> ---------------------------------------
>
> If you got this far, you surely know if I'm missing something or you can
> make some suggestions about which would be the best process, so please
> reply.
>

thanks Daniel, its obvious you put a lot of effort into this, I tend to
agree with your logic above, mostly because assets (javascript, css etc)
gets minified/compress meaning all of your packages are build time only
(they never get to users), further, since option 2 (the one we initially
agreed upon in the pr discussion) already bundle some of the dependencies,
I would prefer option 3 (bundle it all in one package) as does it really
matters if break in into one bundle vs multiple bundles?

thanks,
Ohad

···

On Thu, Jul 21, 2016 at 12:46 PM, Daniel Lobato Garcia wrote:

New ideas, comments, critiques, welcome

[1] - https://github.com/dlobatog/npm2rpm

Best,

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[dLobatog]

Update - I tried to make a POC for this quickly

foreman-node_modules - proof of concept [wip]

I think it could work if there's a way of telling RPM to not care about
the requirements -i.e: it notices there are multiple npm(tap-stream)
versions and complains about that…

To be fair I didn't expect RPM to read the contents of
%{nodejs_sitelib} and figure out all the dependencies!

···

On 07/21, Daniel Lobato Garcia wrote:

3. Put node_modules in a new package (foreman-npm-assets?) that
   foreman-assets %install section puts in the right place?

• Upsides:

  • relatively easy to do
  • jenkins can automate it

• Downsides:

  • less control or information for users about what’s installed with
    foreman, although we could specify the dependencies under
    “Provides: bundled-npm(xxxx)”

• Success with POC: haven’t tried

–
Daniel Lobato Garcia

@dLobatog

Daniel Lobato – Black box of a budding computer scientist

bio – Daniel Lobato

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

[dLobatog]

Update - we can avoid keeping track of the dependency tree by storing it
in the npm cache, then npm is able to install the module offline.
This also removes the 'jump hoops' & 'packages are unwieldy to review'
disadvantages. Props to Dominic Cleal on this as he came up with this idea.

Check how that's done here:

Webpack dependency

npm2rpm nightly is able to produce specs that do this (& download
sources automatically) - https://github.com/dlobatog/npm2rpm

I have tried to build Foreman itself using these packages, but
I also have to create packages for the devDependencies that are
webpack plugins before it can run 'webpack:compile'. It looks to me like
it'll build just fine after that. Notice we don't require any changes in
the pkg:generate_source task that generates the .tar file

···

On 07/21, Daniel Lobato Garcia wrote: > Hi devs, > > Just wanted to give you an update on packaging for npm & webpack > incidentally. We have two PRs in core that are dabbling in a more modern > way of doing frontend development in Foreman. > https://github.com/theforeman/foreman/pull/3433 > https://github.com/theforeman/foreman/pull/3603 > > At the moment, most frontend modules are no longer served as gemified > assets but rather as npm modules. It's difficult for us to develop on a > modern stack of tools & get a nice UX experience so the idea is to move > towards a model where npm will provide all the frontend stuff you need. > Read more on: https://github.com/theforeman/rfcs/pull/3 > > -------------------------- > > Now, regarding packaging (which really has been blocking the two > aforementioned PRs and potential frontend development): > > The issue we face is that we need to serve the node_modules (folder > generated after running npm install) dependencies with Foreman somehow. > I've been trying several things, with different degrees of success. > > 1. Use npm2rpm [1] to create a package for every npm module. Only > 'direct' dependencies are specified in these packages, and they store > merely the dependency itself. > > - Upsides: > - 'the right way' according to rpm guidelines > - no confusion about dependencies and simpler RPM specs > - Downsides: > - this creates close to 1000 new packages > - we'd have to serve several different versions of the same package, > which the OS would refuse to install (e.g if we have uglify-js 0.0.1, and > 1.0.4 in our repos, the former would be superseded by the latter) > - impossible to review this amount of packages, it must be automated > > - Success with POC? No > > 2. Create 'nodejs-bundle-xxxxx' packages which contain all of the > dependencies for a certain package. e.g webpack Provides > bundle-webpack-core, etc... > > - Upsides: > - similar to how foreman node_modules is structured in development > - Downsides: > - conflicting versions for subdependencies force us to keep track > of the 'dependency tree' in the package. For instance in this spec > https://github.com/dLobatog/foreman-packaging/blob/5fc7bb8e831fa74f579a41e9bf9f367f6a9527ff/nodejs-webpack/nodejs-webpack.spec > we can't just unpack all 'async' to > %{buildroot}/%{nodejs_sitelib} as there are several 'async' > packages, and the last one to be unpacked will prevail. > - these packages become huge and unwieldy to review > - we have to jump hoops to make it work. for instance we have to > call '%nodejs_fixdep -r dependency' to allow dependencies to not > conflict (this doesn't always work) > - lots of 'cruft' is packed, like examples directories, etc.. > > - Success with POC? No. I stopped when I figured I had to keep track > of the dependency tree before unpackaging.

---

At this point I have working POCs for 3 and 4, however since we’re so close
to being done with 2, and the person with the most packaging experience
here backs it up, I think we can go with this (and it won’t delay us).
Right now all that’s left is:

• Rebasing the current nodejs-xxx PRs
• Submitting new nodejs-xxx PRs for the webpack plugins (they’re
  automatically created using npm2rpm, nbd)
• Merging the Webpack PR & a PR to the foreman.spec that sets the right
  BuildRequires and runs webpack:compile.
• Last but not least: Celebrate!

That all could be done in a matter of a few hours - if I can catch some
time I’ll do all of it (except actually merging the PRs) this afternoon.

---

Thanks to all that have commented here for your help & suggestions!

3. Put node_modules in a new package (foreman-npm-assets?) that
   foreman-assets %install section puts in the right place?

• Upsides:

  • relatively easy to do
  • jenkins can automate it

• Downsides:

  • less control or information for users about what’s installed with
    foreman, although we could specify the dependencies under
    “Provides: bundled-npm(xxxx)”

• Success with POC: haven’t tried

4. Make Foreman tar file include node_modules so it’s part of the
   foreman package itself

• Upsides:

  • easiest to do
  • zero work required by devs
  • closest to development

• Downsides:

  • lack of knowledge/control about deps for users, like 3.
  • no possibility of depending on system nodejs-xxx packages, but I
    don’t think we want that.

• Success with POC: Yes!
  https://github.com/dlobatog/foreman-packaging/tree/chainbuild-webpack
  is able to run webpack compile just fine after that. We’d need even
  less than what’s on that branch to do it.

---

I got kind of stuck with 2. because of the tree dependencies thing, and
honestly I don’t see much of a benefit to it now, especially considering
the amount of work there’s involved.

If we can redo this somehow I’d definitely go with 4 which is really
easy and I can’t see how that’s a problem for us. It’d be a problem if
we were to put Foreman in a Linux distribution but we’re not aiming for
that. It is as well more manageable than any other model which requires
hundreds of specs + sources in our foreman-packaging repo.

Aside from that…

Why don’t we go with 4 at least until we find a better/feasible model? It
would allow development to continue (the two PRs I mentioned + more).
There’s plenty of time until 1.13 so I don’t see why we can’t move
forward, merge these PRs and figure out other packaging strategies if we
need to.

---

If you got this far, you surely know if I’m missing something or you can
make some suggestions about which would be the best process, so please
reply.

New ideas, comments, critiques, welcome

[1] - https://github.com/dlobatog/npm2rpm

Best,

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

–
Daniel Lobato Garcia

@dLobatog

Daniel Lobato – Black box of a budding computer scientist

bio – Daniel Lobato

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

[Dominic_Cleal2]

> Why don't we go with 4 at least until we find a better/feasible
> model? It
> would allow development to continue (the two PRs I mentioned + more).
> There's plenty of time until 1.13 so I don't see why we can't move
> forward, merge these PRs and figure out other packaging strategies if we
> need to.
>
> I would ask that we define npm packaging as a blocker for 1.13 release,
> and merge the https://github.com/theforeman/foreman/pull/3433 (or a
> variant of it)

Please don't merge patches to develop that will knowingly break the
packaging of Foreman for a long period. It will leave the project with
reduced testing for all releases, and it becomes increasingly difficult
to fix. Use another branch or keep the pull request open.

> If you got this far, you surely know if I'm missing something or you can
> make some suggestions about which would be the best process, so please
> reply.
>
>
> thanks Daniel, its obvious you put a lot of effort into this, I tend to
> agree with your logic above, mostly because assets (javascript, css etc)
> gets minified/compress meaning all of your packages are build time only
> (they never get to users), further, since option 2 (the one we initially
> agreed upon in the pr discussion) already bundle some of the
> dependencies, I would prefer option 3 (bundle it all in one package) as
> does it really matters if break in into one bundle vs multiple bundles?

Yes, it's easier to partially update or add packages if it's split up,
particularly in stable releases.

···

On 21/07/16 13:01, Ohad Levy wrote: > On Thu, Jul 21, 2016 at 12:46 PM, Daniel Lobato Garcia > <elobatocs@gmail.com > wrote:

–
Dominic Cleal
dominic@cleal.org

[dLobatog]

>
> > 3. Put node_modules in a new package (foreman-npm-assets?) that
> > foreman-assets %install section puts in the right place?
> >
> > - Upsides:
> > - relatively easy to do
> > - jenkins can automate it
> >
> > - Downsides:
> > - less control or information for users about what's installed with
> > foreman, although we could specify the dependencies under
> > "Provides: bundled-npm(xxxx)"
> >
> > - Success with POC: haven't tried
>
> Update - I tried to make a POC for this quickly
> https://github.com/theforeman/foreman-packaging/pull/1240
>
> I think it could work if there's a way of telling RPM to not care about
> the requirements -i.e: it notices there are multiple npm(tap-stream)
> versions and complains about that…
>
> To be fair I didn't expect RPM to read the contents of
> %{nodejs_sitelib} and figure out all the dependencies!

Update 2:

It's working! - http://i.imgur.com/c8CPRqq.gif

Brief summary:

The way foreman-node_modules works is by dumping node_modules in
/opt/foreman/node_modules, then copying that sources to the Foreman root
to ensure webpack:compile can run. Aside from that the Foreman package
itself now has a BuildDependency on node itself (& libuv) to run
webpack.

That way we ensure no other program finds stuff it shouldn't on %{nodejs_sitelib}
and we don't conflict with the nodejs-XXX packages provided by the
linux distribution.

It's not building in Jenkins, but it's just because I have not put the
sources for foreman-node_modules anywhere yet. You should be able to
build it with mockchain locally.

I think this solution would be a good compromise considering the
difficulties 2. (bundled-npms) had and also it's much less intrusive -
other packages need not to care about our npm deps at all.

···

On 07/22, Daniel Lobato Garcia wrote: > On 07/21, Daniel Lobato Garcia wrote:

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

[ohadlevy]

>
> >
> > > 3. Put node_modules in a new package (foreman-npm-assets?) that
> > > foreman-assets %install section puts in the right place?
> > >
> > > - Upsides:
> > > - relatively easy to do
> > > - jenkins can automate it
> > >
> > > - Downsides:
> > > - less control or information for users about what's installed
with
> > > foreman, although we could specify the dependencies under
> > > "Provides: bundled-npm(xxxx)"
> > >
> > > - Success with POC: haven't tried
> >
> > Update - I tried to make a POC for this quickly
> > https://github.com/theforeman/foreman-packaging/pull/1240
> >
> > I think it could work if there's a way of telling RPM to not care about
> > the requirements -i.e: it notices there are multiple npm(tap-stream)
> > versions and complains about that…
> >
> > To be fair I didn't expect RPM to read the contents of
> > %{nodejs_sitelib} and figure out all the dependencies!
>
> Update 2:
>
> It's working! - http://i.imgur.com/c8CPRqq.gif
>

> Brief summary:
>
> The way foreman-node_modules works is by dumping node_modules in
> /opt/foreman/node_modules, then copying that sources to the Foreman root
> to ensure webpack:compile can run. Aside from that the Foreman package
> itself now has a BuildDependency on node itself (& libuv) to run
> webpack.
>
> That way we ensure no other program finds stuff it shouldn't on
%{nodejs_sitelib}
> and we don't conflict with the nodejs-XXX packages provided by the
> linux distribution.
>
> It's not building in Jenkins, but it's just because I have not put the
> sources for foreman-node_modules anywhere yet. You should be able to
> build it with mockchain locally.
>
> I think this solution would be a good compromise considering the
> difficulties 2. (bundled-npms) had and also it's much less intrusive -
> other packages need not to care about our npm deps at all.
>

It sounds great but I'm really looking for Dominic feedback here. I hope
it's enough to unblock merging webpack and can be improved if needed.

Thanks for your efforts!
Ohad
> >
> > –
> > Daniel Lobato Garcia
> >
> > @dLobatog
> > blog.daniellobato.me
> > daniellobato.me
> >
> > GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
> > Keybase: https://keybase.io/elobato
>
>
>
> –
> Daniel Lobato Garcia
>
> @dLobatog
> blog.daniellobato.me
> daniellobato.me
>
> GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
> Keybase: https://keybase.io/elobato
>
> –
> You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.

···

On Jul 22, 2016 9:01 PM, "Daniel Lobato Garcia" wrote: > On 07/22, Daniel Lobato Garcia wrote: > > On 07/21, Daniel Lobato Garcia wrote: > For more options, visit https://groups.google.com/d/optout.

[dLobatog]

> >
> > > 3. Put node_modules in a new package (foreman-npm-assets?) that
> > > foreman-assets %install section puts in the right place?
> > >
> > > - Upsides:
> > > - relatively easy to do
> > > - jenkins can automate it
> > >
> > > - Downsides:
> > > - less control or information for users about what's installed with
> > > foreman, although we could specify the dependencies under
> > > "Provides: bundled-npm(xxxx)"
> > >
> > > - Success with POC: haven't tried
> >
> > Update - I tried to make a POC for this quickly
> > https://github.com/theforeman/foreman-packaging/pull/1240
> >
> > I think it could work if there's a way of telling RPM to not care about
> > the requirements -i.e: it notices there are multiple npm(tap-stream)
> > versions and complains about that…
> >
> > To be fair I didn't expect RPM to read the contents of
> > %{nodejs_sitelib} and figure out all the dependencies!
>
> Update 2:
>
> It's working! - http://i.imgur.com/c8CPRqq.gif
>
> Brief summary:
>
> The way foreman-node_modules works is by dumping node_modules in
> /opt/foreman/node_modules, then copying that sources to the Foreman root
> to ensure webpack:compile can run. Aside from that the Foreman package
> itself now has a BuildDependency on node itself (& libuv) to run
> webpack.
>
> That way we ensure no other program finds stuff it shouldn't on %{nodejs_sitelib}
> and we don't conflict with the nodejs-XXX packages provided by the
> linux distribution.
>
> It's not building in Jenkins, but it's just because I have not put the
> sources for foreman-node_modules anywhere yet. You should be able to
> build it with mockchain locally.
>
> I think this solution would be a good compromise considering the
> difficulties 2. (bundled-npms) had and also it's much less intrusive -
> other packages need not to care about our npm deps at all.

More updates -

bundled npms are building fine in f24, however on el7 they're not
building yet. Part of the reason is that I have not been able to build
an npm registry cache that satisfies both.

A bit worrisome - the version of npm used in el7 isn't able to build the
cache for certain versions of some packages. For example:

nodejs-bundle-babel-preset-es2015

It might mean we have to use older versions so that older node/npm can
deal with it of some libraries.

BTW we would've missed these errors if we had gone with
packaging options 3 or 4!

Please follow
https://github.com/theforeman/foreman-packaging/milestone/1 if you want
to be on top of things or want to help

···

On 07/22, Daniel Lobato Garcia wrote: > On 07/22, Daniel Lobato Garcia wrote: > > On 07/21, Daniel Lobato Garcia wrote:

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

–
Daniel Lobato Garcia

@dLobatog

Daniel Lobato – Black box of a budding computer scientist

bio – Daniel Lobato

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

[tbrisker]

Hi,

I meant to reply to this much sooner but it somehow slipped between the
cracks.
First, a big thank-you to Daniel for all the effort put into getting this
done.
I think either 3 or 4 make the most sense, but at this point anything that
will let us move forward is fine with me.
I still haven't managed to understand why we insist on building a ton of
RPMs just so we can provide them to Koji.
All the node modules are only needed for one step in building the rpm and
are never used for anything else or provided to users.
I would understand the need to separately package them if they were needed
for running Foreman, but this is not the case.
We are literally wasting weeks of time and I have no idea how many hours of
engineering work for something that can be achieved in a simple tar ball or
a single package containing everything.
Lets finish this, please, one way or another, so we can concentrate on
building a better Foreman for our community.

Tomer

···

On Fri, Jul 22, 2016 at 10:26 PM, Ohad Levy wrote:

On Jul 22, 2016 9:01 PM, “Daniel Lobato Garcia” elobatocs@gmail.com > wrote:

On 07/22, Daniel Lobato Garcia wrote:

On 07/21, Daniel Lobato Garcia wrote:

3. Put node_modules in a new package (foreman-npm-assets?) that
   foreman-assets %install section puts in the right place?

• Upsides:

  • relatively easy to do
  • jenkins can automate it

• Downsides:

  • less control or information for users about what’s installed
    with

   foreman, although we could specify the dependencies under
   "Provides: bundled-npm(xxxx)"

• Success with POC: haven’t tried

Update - I tried to make a POC for this quickly
https://github.com/theforeman/foreman-packaging/pull/1240

I think it could work if there’s a way of telling RPM to not care about
the requirements -i.e: it notices there are multiple npm(tap-stream)
versions and complains about that…

To be fair I didn’t expect RPM to read the contents of
%{nodejs_sitelib} and figure out all the dependencies!

Update 2:

It’s working! - http://i.imgur.com/c8CPRqq.gif

Brief summary:

The way foreman-node_modules works is by dumping node_modules in
/opt/foreman/node_modules, then copying that sources to the Foreman root
to ensure webpack:compile can run. Aside from that the Foreman package
itself now has a BuildDependency on node itself (& libuv) to run
webpack.

That way we ensure no other program finds stuff it shouldn’t on
%{nodejs_sitelib}
and we don’t conflict with the nodejs-XXX packages provided by the
linux distribution.

It’s not building in Jenkins, but it’s just because I have not put the
sources for foreman-node_modules anywhere yet. You should be able to
build it with mockchain locally.

I think this solution would be a good compromise considering the
difficulties 2. (bundled-npms) had and also it’s much less intrusive -
other packages need not to care about our npm deps at all.

It sounds great but I’m really looking for Dominic feedback here. I hope
it’s enough to unblock merging webpack and can be improved if needed.

Thanks for your efforts!
Ohad

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

–
You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Have a nice day,
Tomer Brisker
Red Hat Engineering

[Tomas_Strachota]

> Hi,
>
> I meant to reply to this much sooner but it somehow slipped between the
> cracks.
> First, a big thank-you to Daniel for all the effort put into getting
> this done.
> I think either 3 or 4 make the most sense, but at this point anything
> that will let us move forward is fine with me.
> I still haven't managed to understand why we insist on building a ton of
> RPMs just so we can provide them to Koji.
> All the node modules are only needed for one step in building the rpm
> and are never used for anything else or provided to users.
> I would understand the need to separately package them if they were
> needed for running Foreman, but this is not the case.
> We are literally wasting weeks of time and I have no idea how many hours
> of engineering work for something that can be achieved in a simple tar
> ball or a single package containing everything.
> Lets finish this, please, one way or another, so we can concentrate on
> building a better Foreman for our community.
>
> Tomer
>
>
>
> >
> > >
> > > > 3. Put node_modules in a new package (foreman-npm-assets?) that
> > > > foreman-assets %install section puts in the right place?
> > > >
> > > > - Upsides:
> > > > - relatively easy to do
> > > > - jenkins can automate it
> > > >
> > > > - Downsides:
> > > > - less control or information for users about what's installed with
> > > > foreman, although we could specify the dependencies under
> > > > "Provides: bundled-npm(xxxx)"
> > > >
> > > > - Success with POC: haven't tried
> > >
> > > Update - I tried to make a POC for this quickly
> > >https://github.com/theforeman/foreman-packaging/pull/1240
> > >
> > > I think it could work if there's a way of telling RPM to not care about
> > > the requirements -i.e: it notices there are multiple npm(tap-stream)
> > > versions and complains about that…
> > >
> > > To be fair I didn't expect RPM to read the contents of
> > > %{nodejs_sitelib} and figure out all the dependencies!
> >
> > Update 2:
> >
> > It's working! -http://i.imgur.com/c8CPRqq.gif
> >
>
> > Brief summary:
> >
> > The way foreman-node_modules works is by dumping node_modules in
> > /opt/foreman/node_modules, then copying that sources to the Foreman root
> > to ensure webpack:compile can run. Aside from that the Foreman package
> > itself now has a BuildDependency on node itself (& libuv) to run
> > webpack.
> >
> > That way we ensure no other program finds stuff it shouldn't on %{nodejs_sitelib}
> > and we don't conflict with the nodejs-XXX packages provided by the
> > linux distribution.
> >
> > It's not building in Jenkins, but it's just because I have not put the
> > sources for foreman-node_modules anywhere yet. You should be able to
> > build it with mockchain locally.
> >
> > I think this solution would be a good compromise considering the
> > difficulties 2. (bundled-npms) had and also it's much less intrusive -
> > other packages need not to care about our npm deps at all.
> >
>
> It sounds great but I'm really looking for Dominic feedback here. I
> hope it's enough to unblock merging webpack and can be improved if
> needed.
>
> Thanks for your efforts!
> Ohad
> > >
> > > –
> > > Daniel Lobato Garcia
> > >
> > > @dLobatog
> > >blog.daniellobato.me <http://blog.daniellobato.me>
> > >daniellobato.me <http://daniellobato.me>
> > >
> > > GPG:http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
> > > Keybase:https://keybase.io/elobato
> >
> >
> >
> > –
> > Daniel Lobato Garcia
> >
> > @dLobatog
> >blog.daniellobato.me <http://blog.daniellobato.me>
> >daniellobato.me <http://daniellobato.me>
> >
> > GPG:http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
> > Keybase:https://keybase.io/elobato
> >
> > –
> > You received this message because you are subscribed to the Google Groups "foreman-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email toforeman-dev+unsubscribe@googlegroups.com
> <mailto:foreman-dev%2Bunsubscribe@googlegroups.com>.
> > For more options, visithttps://groups.google.com/d/optout.
>
> –
> You received this message because you are subscribed to the Google
> Groups "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to foreman-dev+unsubscribe@googlegroups.com
> <mailto:foreman-dev+unsubscribe@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> –
> Have a nice day,
> Tomer Brisker
> Red Hat Engineering
>
> –
> You received this message because you are subscribed to the Google
> Groups "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to foreman-dev+unsubscribe@googlegroups.com
> <mailto:foreman-dev+unsubscribe@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

···

On 07/26/2016 05:25 PM, Tomer Brisker wrote: > On Fri, Jul 22, 2016 at 10:26 PM, Ohad Levy > wrote: > On Jul 22, 2016 9:01 PM, "Daniel Lobato Garcia" > wrote: > > On 07/22, Daniel Lobato Garcia wrote: > > > On 07/21, Daniel Lobato Garcia wrote:

[Tomas_Strachota]

+1 for option 4. From what I saw it's common approach in other projects.
It's simple enough for start and we can re-evaluate later if we find it
problematic.

T.

···

On 07/26/2016 05:25 PM, Tomer Brisker wrote: > Hi, > > I meant to reply to this much sooner but it somehow slipped between the > cracks. > First, a big thank-you to Daniel for all the effort put into getting > this done. > I think either 3 or 4 make the most sense, but at this point anything > that will let us move forward is fine with me. > I still haven't managed to understand why we insist on building a ton of > RPMs just so we can provide them to Koji. > All the node modules are only needed for one step in building the rpm > and are never used for anything else or provided to users. > I would understand the need to separately package them if they were > needed for running Foreman, but this is not the case. > We are literally wasting weeks of time and I have no idea how many hours > of engineering work for something that can be achieved in a simple tar > ball or a single package containing everything. > Lets finish this, please, one way or another, so we can concentrate on > building a better Foreman for our community. > > Tomer > > > On Fri, Jul 22, 2016 at 10:26 PM, Ohad Levy > wrote: > > On Jul 22, 2016 9:01 PM, "Daniel Lobato Garcia" > wrote: > > > > On 07/22, Daniel Lobato Garcia wrote: > > > On 07/21, Daniel Lobato Garcia wrote: > > > > > > > 3. Put node_modules in a new package (foreman-npm-assets?) that > > > > foreman-assets %install section puts in the right place? > > > > > > > > - Upsides: > > > > - relatively easy to do > > > > - jenkins can automate it > > > > > > > > - Downsides: > > > > - less control or information for users about what's installed with > > > > foreman, although we could specify the dependencies under > > > > "Provides: bundled-npm(xxxx)" > > > > > > > > - Success with POC: haven't tried > > > > > > Update - I tried to make a POC for this quickly > > >https://github.com/theforeman/foreman-packaging/pull/1240 > > > > > > I think it could work if there's a way of telling RPM to not care about > > > the requirements -i.e: it notices there are multiple npm(tap-stream) > > > versions and complains about that... > > > > > > To be fair I didn't expect RPM to read the contents of > > > %{nodejs_sitelib} and figure out all the dependencies! > > > > Update 2: > > > > It's working! -http://i.imgur.com/c8CPRqq.gif > > > :) > > Brief summary: > > > > The way foreman-node_modules works is by dumping node_modules in > > /opt/foreman/node_modules, then copying that sources to the Foreman root > > to ensure webpack:compile can run. Aside from that the Foreman package > > itself now has a BuildDependency on node itself (& libuv) to run > > webpack. > > > > That way we ensure no other program finds stuff it shouldn't on %{nodejs_sitelib} > > and we don't conflict with the nodejs-XXX packages provided by the > > linux distribution. > > > > It's not building in Jenkins, but it's just because I have not put the > > sources for foreman-node_modules anywhere yet. You should be able to > > build it with mockchain locally. > > > > I think this solution would be a good compromise considering the > > difficulties 2. (bundled-npms) had and also it's much less intrusive - > > other packages need not to care about our npm deps at all. > > > > It sounds great but I'm really looking for Dominic feedback here. I > hope it's enough to unblock merging webpack and can be improved if > needed. > > Thanks for your efforts! > Ohad > > > > > > -- > > > Daniel Lobato Garcia > > > > > > @dLobatog > > >blog.daniellobato.me > > >daniellobato.me > > > > > > GPG:http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 > > > Keybase:https://keybase.io/elobato > > > > > > > > -- > > Daniel Lobato Garcia > > > > @dLobatog > >blog.daniellobato.me > >daniellobato.me > > > > GPG:http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 > > Keybase:https://keybase.io/elobato > > > > -- > > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > > To unsubscribe from this group and stop receiving emails from it, send an email toforeman-dev+unsubscribe@googlegroups.com > . > > For more options, visithttps://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google > Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to foreman-dev+unsubscribe@googlegroups.com > . > For more options, visit https://groups.google.com/d/optout. > > > > > -- > Have a nice day, > Tomer Brisker > Red Hat Engineering > > -- > You received this message because you are subscribed to the Google > Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to foreman-dev+unsubscribe@googlegroups.com > . > For more options, visit https://groups.google.com/d/optout.

[ohadlevy]

> +1 for option 4. From what I saw it's common approach in other projects.
> It's simple enough for start and we can re-evaluate later if we find it
> problematic.
>

due to the fact that number 3 is not that simple either, I'm starting to
lean towards 4 as well, as it will allow us to unblock yet will not break
any packaging / testing etc, we could still decide its a blocker for 1.13
release regardless.

Ohad

···

On Wed, Jul 27, 2016 at 11:27 AM, Tomas Strachota wrote:

T.

On 07/26/2016 05:25 PM, Tomer Brisker wrote:

Hi,

I meant to reply to this much sooner but it somehow slipped between the
cracks.
First, a big thank-you to Daniel for all the effort put into getting
this done.
I think either 3 or 4 make the most sense, but at this point anything
that will let us move forward is fine with me.
I still haven’t managed to understand why we insist on building a ton of
RPMs just so we can provide them to Koji.
All the node modules are only needed for one step in building the rpm
and are never used for anything else or provided to users.
I would understand the need to separately package them if they were
needed for running Foreman, but this is not the case.
We are literally wasting weeks of time and I have no idea how many hours
of engineering work for something that can be achieved in a simple tar
ball or a single package containing everything.
Lets finish this, please, one way or another, so we can concentrate on
building a better Foreman for our community.

Tomer

On Fri, Jul 22, 2016 at 10:26 PM, Ohad Levy <ohadlevy@gmail.com >> mailto:ohadlevy@gmail.com> wrote:

On Jul 22, 2016 9:01 PM, "Daniel Lobato Garcia" <elobatocs@gmail.com >>     <mailto:elobatocs@gmail.com>> wrote:
>
> On 07/22, Daniel Lobato Garcia wrote:
> > On 07/21, Daniel Lobato Garcia wrote:
> >
> > > 3. Put node_modules in a new package (foreman-npm-assets?) that
> > > foreman-assets %install section puts in the right place?
> > >
> > >   - Upsides:
> > >      - relatively easy to do
> > >      - jenkins can automate it
> > >
> > >   - Downsides:
> > >      - less control or information for users about what's

installed with
> > > foreman, although we could specify the dependencies under
> > > “Provides: bundled-npm(xxxx)”
> > >
> > > - Success with POC: haven’t tried
> >
> > Update - I tried to make a POC for this quickly
> >https://github.com/theforeman/foreman-packaging/pull/1240
> >
> > I think it could work if there’s a way of telling RPM to not care
about
> > the requirements -i.e: it notices there are multiple
npm(tap-stream)
> > versions and complains about that…
> >
> > To be fair I didn’t expect RPM to read the contents of
> > %{nodejs_sitelib} and figure out all the dependencies!
>
> Update 2:
>
> It’s working! -http://i.imgur.com/c8CPRqq.gif
>

> Brief summary:
>
> The way foreman-node_modules works is by dumping node_modules in
> /opt/foreman/node_modules, then copying that sources to the Foreman
root
> to ensure webpack:compile can run. Aside from that the Foreman
package
> itself now has a BuildDependency on node itself (& libuv) to run
> webpack.
>
> That way we ensure no other program finds stuff it shouldn’t on
%{nodejs_sitelib}
> and we don’t conflict with the nodejs-XXX packages provided by the
> linux distribution.
>
> It’s not building in Jenkins, but it’s just because I have not put
the
> sources for foreman-node_modules anywhere yet. You should be able to
> build it with mockchain locally.
>
> I think this solution would be a good compromise considering the
> difficulties 2. (bundled-npms) had and also it’s much less
intrusive -
> other packages need not to care about our npm deps at all.
>

It sounds great but I'm really looking for Dominic feedback here. I
hope it's enough to unblock merging webpack and can be improved if
needed.

Thanks for your efforts!
Ohad
> >
> > --
> > Daniel Lobato Garcia
> >
> > @dLobatog
> >blog.daniellobato.me <http://blog.daniellobato.me>
> >daniellobato.me <http://daniellobato.me>
> >
> > GPG:

http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
> > Keybase:https://keybase.io/elobato
>
>
>
> –
> Daniel Lobato Garcia
>
> @dLobatog
>blog.daniellobato.me http://blog.daniellobato.me
>daniellobato.me http://daniellobato.me
>
> GPG:
http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
> Keybase:https://keybase.io/elobato
>
> –
> You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
> To unsubscribe from this group and stop receiving emails from it,
send an email toforeman-dev+unsubscribe@googlegroups.com
mailto:foreman-dev%2Bunsubscribe@googlegroups.com.
> For more options, visithttps://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to foreman-dev+unsubscribe@googlegroups.com
<mailto:foreman-dev+unsubscribe@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout.

–
Have a nice day,
Tomer Brisker
Red Hat Engineering

–
You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev+unsubscribe@googlegroups.com
mailto:foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.