RFC: Disabling dependency solving by default for incremental update
Decision Due Date: August 12th
Written in collaboration with @dralley from Pulp.
Context and Problem Statement
When using Katello’s “incremental update” feature to apply security patches, users often experience significant performance issues. An operation that should be fast can take minutes or even hours to complete. In some cases, the process can lead to catastrophic failures, resulting in unkillable tasks that block all other content operations.
These issues are frequently caused by a default-enabled feature: server-side dependency solving. While this function was historically necessary, it has become a primary source of poor performance and instability. This post proposes that we change this default to “off” as an immediate measure, and perhaps phase it out entirely over the longer term.
The Origin of Default Dependency Solving
Katello’s incremental update feature allows administrators to apply specific patches to a stable Content View without incorporating all new changes from a repository. This is essential for production environments where stability is critical, but security patches must be applied.
Historically, vendor-supplied errata were often incomplete, lacking all necessary dependency packages. Applying such an erratum would result in a repository with broken dependencies, causing updates on client systems to fail.
To prevent this, Katello introduced server-side dependency solving. When an erratum was added via an incremental update, Katello / Pulp would analyze its packages, find any missing dependencies from the source repositories, and automatically include them in the new Content View version. This was a server-side solution for an upstream data quality issue.
Unfortunately, while this was occasionally useful historically, having this setting enabled by default causes significant problems:
-
Severe Performance Degradation: Dependency solving is resource-intensive. Instead of performing a simple metadata copy, the server must build a dependency graph from scratch for every operation by querying its database for all package metadata. This process can take from 20 minutes to several hours, severely slowing down patching workflows.
-
Functional Instability and Unpredictable Results: The feature is also a bit unstable and can also ignore user-defined Content View filters. For example, a filter designed to exclude all packages from RHEL 8.7 will be overridden if a security patch requires an 8.7 dependency, leading to untested packages being introduced into an environment. Plus, even with dependency solving, there are many circumstances where it simply cannot help, and you end up with broken dependencies regardless.
-
Architectural Mismatch: The dependency solver library, libsolv, was designed for client-side package management, where it calculates dependencies for a single host based on its currently installed packages. Katello uses it on the server to pre-solve dependencies for an entire repository without any knowledge of a client’s state. This mismatch between intended use case and how the library is actually being used leads to complex edge cases and creates a significant maintenance burden for developers.
But moreover the data quality issues have been slowly resolved over time as distros have improved their processes around the handling of errata. Thus, the original problems that motivated this feature have diminished significantly while the downsides of the workaround have remained or become more acute over time.
Furthermore, even in cases where dependency solving remains helpful, it is rarely the best solution. We have found that structuring filters differently, such as “excluding” patches newer than some date except for the desired ones (rather than “including” specific ones) tends to avoid the need for depsolving entirely, avoiding hiccups caused by data quality without taking on the substantial additional overhead depsolving incurs.
Proposal
Disabling dependency solving by default for incremental update.
The feature should remain as an explicit, advanced opt-in for specific edge cases. However, for the majority of users, changing this default will make Katello faster, more reliable, and more predictable.
Decision Outcome
TBD
Impacts
We would like to know which users are still using dependency solving today and how. We may be able to suggest ways to avoid dependency solving for a nice speed increase. Or, we may learn reasons that make dependency solving in Pulp still useful, which would help determine the general future of dependency solving in Pulp and Katello.
For the more distant future, we should determine if dependency solving is needed at all in Pulp & Katello. Considering the complications and inability to “fix” most complaints with the feature, it might be beneficial to remove it entirely if users don’t need to rely on it.
