Already asked this about a week back on Matrix, asking here again for better visibility in case someone has additional insights.
Problem:
When we promote new CV versions, all hosts that consume that CV in the LCE triggers a Katello::Host::ContentFacet update event, which in turn causes foreman-webhooks to trigger a host_updated event for every host. (Somewhere in this process, a “Generate Applicability” task gets created for every host, though that is somewhat expected)
Since we utilize webhooks to trigger the smart-proxy-shellhooks plugin on the integrated smart-proxy, this leads to a lot of tasks being created at once on the Foreman host, potentially causing a self-dos through OOM-killer (happened already to us). Additionally, we have observed that this also causes a DOS on some external services that we need to talk to through shellhooks.
Expected outcome:
No self-DOS on the Foreman server and no DOS of external services.
Foreman and Proxy versions:
3.9.3
Foreman and Proxy plugin versions:
foreman-tasks 9.0.4
foreman_expire_hosts 8.2.0
foreman_hooks 0.3.17
foreman_puppet 6.2.0
foreman_remote_execution 12.0.5
foreman_scc_manager 3.0.0
foreman_snapshot_management 3.0.0
foreman_templates 9.4.0
foreman_webhooks 3.2.2
katello 4.11.1
puppetdb_foreman 6.0.2
shellhooks 0.9.2-3
Distribution and version:
RHEL8.10
Other relevant data:
I primarily have two questions:
Is it intended behavior that errata applicability generation triggers host_update webhook events? To me this sounds rather unintuitive and definetly was unexpected.
We have the “Generate errata status from directly-installable content” setting enabled (true/yes), is this what is causing all the hosts to trigger the update at the same time on CV publish? If turning this setting off would result in the update events being distributed over a longer time-period, that would at least be a workaround could live with.
I’m assuming an update to a facet also counts as an update to a host. If so, this would explain why you get those events triggered by applicability generation.
So, if I understand this correctly, the behavior will be the same no matter what the generate errata status setting is set to?
Yes, this is what I am also assuming currently. I would initially have expected this to be handled by this PR, because I simply assumed that automatic actions like generate applicability are triggered via anonymus admin, but from what I am seeing, it looks like those generate applicability are all created by the user that initially triggered the CV publish.
Yes, this seems to be the case. I don’t see any User.as_anonymous_admin blocks in the relevant areas in Katello. Maybe if we added that it would solve your issue?
I would assume this would solve my issue, yes.
Probably would be a good idea to add a seperate hook for this type of event like with the host_facts_updated event?
I also tried to do a fix for it by simply wrapping everything in the update_applicability_counts function you linked above inside a User.as_anonymous_admin do block, but either I messed up somehow or that is not the correct place for this to happen since it still triggers all the host_updated events and webhooks.
