It seems this patch took care of the one part of the problem, other part still remains.
This morning bulk generate applicability tasks again began lasting 0-1 seconds and 90% of Content hosts/Installable updates column shows all 0’s again.
And no errors were logged. I think this is when this starts happening.
And this corresponds with time when repos are synced and new packages and errata are introduced.
If I don’t manually force recalculation via console, all tasks from now on will run with success but 0-1 seconds. Looking into tasks it seems it stops running for groups of 50 hosts, but starts doing it one by one host.
The last task that lasted 7 minutes for many hosts:
Id:abd18ea8-a639-47e9-872b-a3d9701fee51
Label:Actions::Katello::Applicability::Hosts::BulkGenerate
Duration:6 minutes
Raw input:
{
"host_ids": [
9563,
6740,
1450,
5637,
9521,
7311,
2637,
2997,
8422,
11316,
9812,
5264,
5000,
8934,
8355,
5944,
7261,
3565,
9397,
3510,
3022,
9137,
1014,
11494,
11366,
6982,
6571,
8524,
7004,
9098,
10282,
6107,
4035,
9180,
3858,
7227,
10896,
5642,
2850
],
"current_request_id": null,
"current_timezone": "UTC",
"current_user_id": 1,
"current_organization_id": null,
"current_location_id": null
}
Raw output:
{}
External Id:885f1514-b4de-4aea-bf53-377790368267
After sync plan ran, all bulk generate applicability tasks run per one host:
Id:2471d2e8-7d15-4fdd-95bf-549acb8fee71
Label:Actions::Katello::Applicability::Hosts::BulkGenerate
Duration:0 seconds
Raw input:
{
"host_ids": [
8297
],
"current_request_id": null,
"current_timezone": "UTC",
"current_user_id": 1,
"current_organization_id": null,
"current_location_id": null
}
Raw output:
{}
External Id:8d48d340-5b95-4abb-8e04-95b5595261a7
And when these bulk generate tasks run for one host, that host has data on Content hosts/Installable updates. That task is truly successful.
Looking at all of this the problem seems to be that “Bulk Generate Applicability” task is not doing the “bulk” part.
As sync plan syncs new packages and errata, the old one expires and is no longer relevant, so it resets to all 0’s, and “Bulk generate applicability” doesn’t get around 10 000+ hosts in 24 hours doing it one by one(instead of bulk), before new sync is triggered and previous calculations expire.
To confirm this theory, I’ve switched sync plan to weekly. Next sync will be Monday morning. I’ll check over the weekend if Content hosts/Installable updates is populated. If they are over the weekend, and on Monday after sync they aren’t, theory is confirmed.