Katello 3.15 doesn't show available package updates anymore

iballou · June 4, 2020, 4:47pm

The Pulp folks have brought a recent Pulp 2 issue + PR to my attention: https://github.com/pulp/pulp/pull/3990

It looks like there might be a regression relating to the latest version of Pulp 2 that you are both on. I haven’t seen anyone confirm it besides the PR owner, but I think the PR could help since the regression it targets relates to recent Katello changes in 3.15.1.1.

If testing the Pulp PR above seems too risky for your Katello server, then I’ll watch the issue for more confirmation from other Pulp devs and let you know. If you do want to test the Pulp PR, just make sure you restart your Pulp 2 services, re-upload your package profile, and regenerate applicability after applying the patch.

gvde · June 4, 2020, 5:02pm

With the upload I guess you mean ‘katello-package-upload -f’.

How do I “regenerate applicability”? I thought that is an automated process…

gvde · June 4, 2020, 5:19pm

O.K. I found this solution at RedHat. Applied the patch, restarted all katello services and run the GenerateApplicability task. Now it shows the correct counts for my CentOS 7 counts. I didn’t even have to do the package upload.

So far no bad side effects from the patch.

iballou · June 4, 2020, 6:00pm

@gvde glad the applicability is working again.

For @Laszlo (or anyone else following along), after applying the Pulp PR I mentioned above and restarting all of your Pulp services, you can do the following to regenerate applicability for all hosts:

First: foreman-rake console
Then:

ForemanTasks.sync_task(::Actions::Katello::Host::GenerateApplicability, Host.all)

poulterer · June 4, 2020, 6:07pm

I tried this with an unpleasant result. Any insight?

irb(main):004:0> ForemanTasks.sync_task(::Actions::Katello::Host::GenerateApplicability, Host.all)
Traceback (most recent call last):
        8: from lib/tasks/console.rake:5:in `block in <top (required)>'
        7: from (irb):4
        6: from foreman-tasks (0.17.5) lib/foreman_tasks.rb:58:in `sync_task'
        5: from foreman-tasks (0.17.5) lib/foreman_tasks.rb:27:in `trigger_task'
        4: from foreman-tasks (0.17.5) lib/foreman_tasks.rb:48:in `rails_safe_trigger_task'
        3: from foreman-tasks (0.17.5) lib/foreman_tasks.rb:49:in `block in rails_safe_trigger_task'
        2: from foreman-tasks (0.17.5) lib/foreman_tasks.rb:29:in `block in trigger_task'
        1: from foreman-tasks (0.17.5) lib/foreman_tasks.rb:23:in `trigger'
RuntimeError (The Dynflow world was not initialized yet. If your plugin uses it, make sure to call Rails.application.dynflow.require! in some initializer)

iballou · June 4, 2020, 6:12pm

What version of Katello are you on?

I have a quick workaround that should work:

Open up /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/rails/configuration.rb
-> You might need to update the dynflow version
Search for self.rake_tasks_with_executor =
See the line self.rake_tasks_with_executor = %w(db:migrate db:seed)
Add console to the line like so:
self.rake_tasks_with_executor = %w(db:migrate db:seed console)
Open up the console again and see if it works now.

poulterer · June 4, 2020, 6:17pm

No, I am on v. 3.14.1 and Foreman 1.24.3. I seem to be having a similar problem and was following this thread, too.

poulterer · June 4, 2020, 6:22pm

@iballou — your fix seemed to have worked in 3.14.1, too. Thanks much for the tip.

iballou · June 4, 2020, 6:25pm

Good to hear, applicability is working for your CentOS 7.8 client again? I’m curious which of the patches you applied since I was just about to ask you about updating to 3.15.1.1

poulterer · June 4, 2020, 6:34pm

I only applied the “console” addition to configuration.rb.
Since we are still running 1.24.x, I did not think that 3.15.x was an option.

Refreshing applicability has now made the errata visible.

iballou · June 4, 2020, 6:36pm

You’re right, you’d have to upgrade your Foreman too it seems. Anyway, glad that worked out without an upgrade.

Laszlo · June 5, 2020, 11:17am

After applying the patch from pulp the applicability is working again.

Thank you very much @iballou for your help!

sjansen · June 5, 2020, 5:36pm

i can confirm that the pulp patches work. I stumbeld over this Post after seeing all my EL7 based Systems fails to show applicable updates and wrong “Needs Reboot” status.

caseybea · June 17, 2020, 10:19pm

If I upgrade to 3.16RC3 will these issues be addressed? (sounds like it?, indicated above 3.15.1.1 has the fixes)?

Just upgraded to 3.15 from 3.14 and having this exact same behavior.

iballou · June 18, 2020, 2:08pm

The issue is in Pulp and it looks like that Pulp PR hasn’t been merged in yet. I would try applying the Pulp PR to fix the issue as long as you’re already on 3.15.1.1.

iballou · June 18, 2020, 2:25pm

And afterwards, run this command in the foreman console from above:

ForemanTasks.sync_task(::Actions::Katello::Host::GenerateApplicability, Host.all)

John_Beranek · June 18, 2020, 2:36pm

Not sure how worthwhile a “Me too” is at this point, but me too. Getting “Combined Profile Update” tasks ending with a warning for just some content hosts, and some delayed/incorrect applicability data.

Not sure whether it’s related, but also getting “Publish content view” tasks ending up in a paused status during our weekly automated run since upgrade to 3.15.

caseybea · June 18, 2020, 2:43pm

How do I apply the pulp PR? I am at a loss looking at the pulp PR site mentioned here.

John_Beranek · June 18, 2020, 2:46pm

For me:

cd /usr/lib/python2.7/site-packages/
wget https://patch-diff.githubusercontent.com/raw/pulp/pulp/pull/3990.patch
patch -p3 < 3990.patch
foreman-maintain service restart

caseybea · June 18, 2020, 3:10pm

Success!!! Thank you.

Some notes for my situation:

(A) Small tweak to your patch block (note: the directory: need: …/pulp)

cd /usr/lib/python2.7/site-packages/pulp
wget https://patch-diff.githubusercontent.com/raw/pulp/pulp/pull/3990.patch
patch -p3 < 3990.patch
foreman-maintain service restart

(B)
I still had to fix the configuration.rb file mentioned a few blocks above and add the “console” text, as when I tried the foreman-rake regeneration bit, it blew up on me too.

©
It still errored out, but this time with a timeout of sorts. Regardless, after executing I do see the missing data now.

[root@katello ~]# foreman-rake console
Loading production environment (Rails 5.2.1)
irb(main):001:0> ForemanTasks.sync_task(::Actions::Katello::Host::GenerateApplicability, Host.all)
Traceback (most recent call last):
        8: from lib/tasks/console.rake:5:in `block in <top (required)>'
        7: from (irb):1
        6: from foreman-tasks (1.1.1) lib/foreman_tasks.rb:58:in `sync_task'
        5: from foreman-tasks (1.1.1) lib/foreman_tasks.rb:27:in `trigger_task'
        4: from foreman-tasks (1.1.1) lib/foreman_tasks.rb:48:in `rails_safe_trigger_task'
        3: from foreman-tasks (1.1.1) lib/foreman_tasks.rb:49:in `block in rails_safe_trigger_task'
        2: from foreman-tasks (1.1.1) lib/foreman_tasks.rb:29:in `block in trigger_task'
        1: from foreman-tasks (1.1.1) lib/foreman_tasks.rb:39:in `block (2 levels) in trigger_task'
Timeout::Error (The time waiting for task 1847f3cf-ce1f-4375-9d31-b633d1768d01 to finish exceeded the 'foreman_tasks_sync_task_timeout' (120s))
irb(main):002:0>