Be sure to import your product content data for Katello 3.6

Development
1
Hi all,

A change
<https://github.com/Katello/katello/commit/946b2990c9a054babc9ad563e1b14c4161207b3e> landed in Katello yesterday which introduced a new pattern of importing product content information from Candlepin into the Katello DB. This is in support of some new feature work and means that Candlepin will no longer be consulted for that information until it's time to be refreshed. All areas of Katello which use product content data are affected such as the Red Hat Repositories page and several APIs.

In order to populate your local cache, please run `rake
katello:upgrades:3.6:import_product_content`. From that point on the cache should be updated through ordinary use of Katello - importing manifests, etc. This step is not necessary unless you have a manifest imported and/or custom products created.

I'm working on a change for the installer to do the same import and I'll update this thread once it's ready for review & test.

Let me know if you have problems, questions, or concerns.

Thanks,

Jonathon
2
This sounds great! How long on average are you seeing the import take?

··· On Dec 6, 2017 10:07 AM, "Jonathon Turel" <jturel@gmail.com> wrote:

Hi all,

A change
<https://github.com/Katello/katello/commit/946b2990c9a054babc9ad563e1b14c4161207b3e>
landed in Katello yesterday which introduced a new pattern of importing
product content information from Candlepin into the Katello DB. This is in
support of some new feature work and means that Candlepin will no longer be
consulted for that information until it's time to be refreshed. All areas
of Katello which use product content data are affected such as the Red Hat
Repositories page and several APIs.

In order to populate your local cache, please run `rake
katello:upgrades:3.6:import_product_content`. From that point on the
cache should be updated through ordinary use of Katello - importing
manifests, etc. This step is not necessary unless you have a manifest
imported and/or custom products created.

I'm working on a change for the installer to do the same import and I'll
update this thread once it's ready for review & test.

Let me know if you have problems, questions, or concerns.

Thanks,

Jonathon

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3
Perhaps this is unrelated and not even possible, but I'd like to know if we could change database provisioning/seeding to not require candlepin. Currently for the organization a matching org in candlepin is created. That's a dependency that ideally shouldn't be there. Do you see this as an option?


··· On Wed, Dec 06, 2017 at 07:07:05AM -0800, Jonathon Turel wrote:
A change
<https://github.com/Katello/katello/commit/946b2990c9a054babc9ad563e1b14c4161207b3e>
landed in Katello yesterday which introduced a new pattern of importing
product content information from Candlepin into the Katello DB. This is in
support of some new feature work and means that Candlepin will no longer be
consulted for that information until it's time to be refreshed. All areas
of Katello which use product content data are affected such as the Red Hat
Repositories page and several APIs.

In order to populate your local cache, please run `rake
katello:upgrades:3.6:import_product_content`. From that point on the cache
should be updated through ordinary use of Katello - importing manifests,
etc. This step is not necessary unless you have a manifest imported and/or
custom products created.

I'm working on a change for the installer to do the same import and I'll
update this thread once it's ready for review & test.

Let me know if you have problems, questions, or concerns.
4
Perhaps this is unrelated and not even possible, but I'd like to know if we could change database provisioning/seeding to not require candlepin. Currently for the organization a matching org in candlepin is created. That's a dependency that ideally shouldn't be there. Do you see this as an option?

I think this is a separate discussion that should be split off.

··· On Dec 7, 2017 6:04 AM, "Ewoud Kohl van Wijngaarden" < ewoud@kohlvanwijngaarden.nl> wrote:

On Wed, Dec 06, 2017 at 07:07:05AM -0800, Jonathon Turel wrote:

A change
<https://github.com/Katello/katello/commit/946b2990c9a054bab
c9ad563e1b14c4161207b3e>

landed in Katello yesterday which introduced a new pattern of importing
product content information from Candlepin into the Katello DB. This is in
support of some new feature work and means that Candlepin will no longer be
consulted for that information until it's time to be refreshed. All areas
of Katello which use product content data are affected such as the Red Hat
Repositories page and several APIs.

In order to populate your local cache, please run `rake
katello:upgrades:3.6:import_product_content`. From that point on the cache
should be updated through ordinary use of Katello - importing manifests,
etc. This step is not necessary unless you have a manifest imported and/or
custom products created.

I'm working on a change for the installer to do the same import and I'll
update this thread once it's ready for review & test.

Let me know if you have problems, questions, or concerns.

--
You received this message because you are subscribed to the Google Groups "foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
5

This sounds great! How long on average are you seeing the import take?


In my environment I have two orgs each with a manifest, totaling 270 products. It's been taking about a minute to run the rake task (including the startup time) which results in the creation of 4027 Content records, and 4934 ProductContent records (which maps a Product to a Content as you might guess). The numbers are different because a single Content can be shared among products. I think the performance is pretty good <https://media.giphy.com/media/8kqrtQiz9YqnS/giphy.gif>.


··· On Thursday, December 7, 2017 at 5:31:44 AM UTC-5, Eric Helms wrote:

  


On Dec 6, 2017 10:07 AM, "Jonathon Turel" <jtu...@gmail.com <javascript:>> > wrote:

Hi all,

A change
<https://github.com/Katello/katello/commit/946b2990c9a054babc9ad563e1b14c4161207b3e>
landed in Katello yesterday which introduced a new pattern of importing
product content information from Candlepin into the Katello DB. This is in
support of some new feature work and means that Candlepin will no longer be
consulted for that information until it's time to be refreshed. All areas
of Katello which use product content data are affected such as the Red Hat
Repositories page and several APIs.

In order to populate your local cache, please run `rake
katello:upgrades:3.6:import_product_content`. From that point on the cache
should be updated through ordinary use of Katello - importing manifests,
etc. This step is not necessary unless you have a manifest imported and/or
custom products created.

I'm working on a change for the installer to do the same import and I'll
update this thread once it's ready for review & test.

Let me know if you have problems, questions, or concerns.

Thanks,

Jonathon

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/d/optout.

6
I'd have to look into that, but I think I see the merit. Maybe opening an issue is appropriate here?


··· On Thursday, December 7, 2017 at 6:04:45 AM UTC-5, Ewoud Kohl van Wijngaarden wrote:

On Wed, Dec 06, 2017 at 07:07:05AM -0800, Jonathon Turel wrote:
A change
<
https://github.com/Katello/katello/commit/946b2990c9a054babc9ad563e1b14c4161207b3e>

landed in Katello yesterday which introduced a new pattern of importing
product content information from Candlepin into the Katello DB. This is
in
support of some new feature work and means that Candlepin will no longer
be
consulted for that information until it's time to be refreshed. All areas
of Katello which use product content data are affected such as the Red
Hat
Repositories page and several APIs.

In order to populate your local cache, please run `rake
katello:upgrades:3.6:import_product_content`. From that point on the
cache
should be updated through ordinary use of Katello - importing manifests,
etc. This step is not necessary unless you have a manifest imported
and/or
custom products created.

I'm working on a change for the installer to do the same import and I'll
update this thread once it's ready for review & test.

Let me know if you have problems, questions, or concerns.

Perhaps this is unrelated and not even possible, but I'd like to know if
we could change database provisioning/seeding to not require candlepin.
Currently for the organization a matching org in candlepin is created.
That's a dependency that ideally shouldn't be there. Do you see this as
an option?

7

Is this also meant to be ran on dev boxes? I don't have foreman-tasks
running so I get the following:

2017-12-20T02:58:34 [app] [W] foreman-tasks service not running or is not
ready yet

Thanks,
Walden

···

On Thu, Dec 7, 2017 at 1:51 PM, Jonathon Turel <jturel@gmail.com> wrote:

On Thursday, December 7, 2017 at 6:04:45 AM UTC-5, Ewoud Kohl van > Wijngaarden wrote:

On Wed, Dec 06, 2017 at 07:07:05AM -0800, Jonathon Turel wrote:
>A change
><https://github.com/Katello/katello/commit/946b2990c9a054ba
bc9ad563e1b14c4161207b3e>
>landed in Katello yesterday which introduced a new pattern of importing
>product content information from Candlepin into the Katello DB. This is
in
>support of some new feature work and means that Candlepin will no longer
be
>consulted for that information until it's time to be refreshed. All
areas
>of Katello which use product content data are affected such as the Red
Hat
>Repositories page and several APIs.
>
>In order to populate your local cache, please run `rake
>katello:upgrades:3.6:import_product_content`. From that point on the
cache
>should be updated through ordinary use of Katello - importing manifests,
>etc. This step is not necessary unless you have a manifest imported
and/or
>custom products created.
>
>I'm working on a change for the installer to do the same import and I'll
>update this thread once it's ready for review & test.
>
>Let me know if you have problems, questions, or concerns.

Perhaps this is unrelated and not even possible, but I'd like to know if
we could change database provisioning/seeding to not require candlepin.
Currently for the organization a matching org in candlepin is created.
That's a dependency that ideally shouldn't be there. Do you see this as
an option?

I'd have to look into that, but I think I see the merit. Maybe opening an
issue is appropriate here?

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

It should be run on dev boxes that were built before the installer was
updated [1]. Regarding the foreman-tasks: I suppose that we actually do not
need to require foreman-tasks to be running in order to run that rake task.
If I recall correctly it did work fine for me so long as I had the rails
server running in another process. Let me know if that works and if not I
can open a PR with the fix.

Jonathon

[1]

···

On Tue, Dec 19, 2017 at 10:02 PM, Walden Raines <wraines@redhat.com> wrote:

Is this also meant to be ran on dev boxes? I don't have foreman-tasks
running so I get the following:

2017-12-20T02:58:34 [app] [W] foreman-tasks service not running or is not
ready yet

Thanks,
Walden

On Thu, Dec 7, 2017 at 1:51 PM, Jonathon Turel <jturel@gmail.com> wrote:

On Thursday, December 7, 2017 at 6:04:45 AM UTC-5, Ewoud Kohl van >> Wijngaarden wrote:

On Wed, Dec 06, 2017 at 07:07:05AM -0800, Jonathon Turel wrote:
>A change
><Fixes #21680 - Cache product content in the Katello DB · Katello/katello@946b299 · GitHub
bc9ad563e1b14c4161207b3e>
>landed in Katello yesterday which introduced a new pattern of importing
>product content information from Candlepin into the Katello DB. This is
in
>support of some new feature work and means that Candlepin will no
longer be
>consulted for that information until it's time to be refreshed. All
areas
>of Katello which use product content data are affected such as the Red
Hat
>Repositories page and several APIs.
>
>In order to populate your local cache, please run `rake
>katello:upgrades:3.6:import_product_content`. From that point on the
cache
>should be updated through ordinary use of Katello - importing
manifests,
>etc. This step is not necessary unless you have a manifest imported
and/or
>custom products created.
>
>I'm working on a change for the installer to do the same import and
I'll
>update this thread once it's ready for review & test.
>
>Let me know if you have problems, questions, or concerns.

Perhaps this is unrelated and not even possible, but I'd like to know if
we could change database provisioning/seeding to not require candlepin.
Currently for the organization a matching org in candlepin is created.
That's a dependency that ideally shouldn't be there. Do you see this as
an option?

I'd have to look into that, but I think I see the merit. Maybe opening an
issue is appropriate here?

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout\.

--
You received this message because you are subscribed to a topic in the
Google Groups "foreman-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/foreman-dev/mNAikpyNDFs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout\.

9

I was running foreman via foreman start in another process when I got the
above. Anything else I am missing?

Thanks,
Walden

···

On Tue, Dec 19, 2017 at 10:18 PM, JT <jturel@gmail.com> wrote:

It should be run on dev boxes that were built before the installer was
updated [1]. Regarding the foreman-tasks: I suppose that we actually do not
need to require foreman-tasks to be running in order to run that rake task.
If I recall correctly it did work fine for me so long as I had the rails
server running in another process. Let me know if that works and if not I
can open a PR with the fix.

Jonathon

[1] https://github.com/Katello/katello-installer/commit/
a03749491f344458a5b52acadcf21526c66b7c65

On Tue, Dec 19, 2017 at 10:02 PM, Walden Raines <wraines@redhat.com> > wrote:

Is this also meant to be ran on dev boxes? I don't have foreman-tasks
running so I get the following:

2017-12-20T02:58:34 [app] [W] foreman-tasks service not running or is not
ready yet

Thanks,
Walden

On Thu, Dec 7, 2017 at 1:51 PM, Jonathon Turel <jturel@gmail.com> wrote:

On Thursday, December 7, 2017 at 6:04:45 AM UTC-5, Ewoud Kohl van >>> Wijngaarden wrote:

On Wed, Dec 06, 2017 at 07:07:05AM -0800, Jonathon Turel wrote:
>A change
><https://github.com/Katello/katello/commit/946b2990c9a054ba
bc9ad563e1b14c4161207b3e>
>landed in Katello yesterday which introduced a new pattern of
importing
>product content information from Candlepin into the Katello DB. This
is in
>support of some new feature work and means that Candlepin will no
longer be
>consulted for that information until it's time to be refreshed. All
areas
>of Katello which use product content data are affected such as the Red
Hat
>Repositories page and several APIs.
>
>In order to populate your local cache, please run `rake
>katello:upgrades:3.6:import_product_content`. From that point on the
cache
>should be updated through ordinary use of Katello - importing
manifests,
>etc. This step is not necessary unless you have a manifest imported
and/or
>custom products created.
>
>I'm working on a change for the installer to do the same import and
I'll
>update this thread once it's ready for review & test.
>
>Let me know if you have problems, questions, or concerns.

Perhaps this is unrelated and not even possible, but I'd like to know
if
we could change database provisioning/seeding to not require candlepin.
Currently for the organization a matching org in candlepin is created.
That's a dependency that ideally shouldn't be there. Do you see this as
an option?

I'd have to look into that, but I think I see the merit. Maybe opening
an issue is appropriate here?

--
You received this message because you are subscribed to the Google
Groups "foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "foreman-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/to
pic/foreman-dev/mNAikpyNDFs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

10

Sorry for the delay, Walden. PR with the correction is here: https://github.com/Katello/katello/pull/7131