As the title suggests, we’ve finally hit an issue with the generated Pulp bindings. For those new to this issue - the Pulp plugin client bindings on rubygems and pypi are generated against the newest version of Pulpcore. So far, we’ve never hit an issue with this, until now: [PULP-925] Ruby bindings: remote update returns empty RpmRpmRemoteResponse object · Issue #4178 · pulp/pulp_rpm · GitHub . The RPM plugin in Katello presented a critical error because we’re using pulp-rpm-client 3.32 with Pulpcore 3.85 instead of Pulpcore 3.91.
What are our options then?
Firstly, we’ve pinned the pulp-rpm-client bindings in Katello 4.19 to the 3.32.2 version that we know works. But, we cannot stay on this pinned version forever, especially if there are bugs.
We can patch the bindings to allow ourselves to upgrade. However, this is hacky and difficult to maintain.
Or - we can start generating new bindings. I’ve been talking with @pedro-psb about an idea where the Katello and smart_proxy_pulp (?) repositories could carry a copy of the Pulp bindings that are updated by some CI pipeline. That way, each version of Katello could be shipped with the proper clients, and we wouldn’t need to worry about packaging a huge RPM matrix of varying bindings. We just need to keep in mind that, to generate bindings, Pulp needs to be installed (and maybe running? See GitHub - pulp/pulp-openapi-generator).
However we do it, generating the Pulp client bindings ourselves is the safest way forward (maybe Pulp Glue is too, but I don’t want to get into that here). Pulp would like to stop shipping the bindings since they are broken when combined in certain ways, and Katello is one of the last stakeholders that they are waiting on.
I’d like to hear feedback on the basic idea above from packaging maintainers. There could be some pitfalls that we’re not seeing.
For the short term, I suggest that Katello 4.19.0 continues to pin pulp-rpm-client to 3.32.2. If the bindings packaging isn’t realistic to start in the short term, I recommend that we patch our bindings to unpin pulp-rpm-client for Katello 4.20 and backport the patch to Katello 4.19.0.
In our containerization strategy we are looking at going the other direction: pulp_smart_proxy. A Pulp plugin to present itself as a Smart Proxy. That allows admins to deploy a Pulp somewhere to provide the content bits and scale it independently.
It kind of breaks the whole “abstract differences away”. That intermediate layer should now be in the client gem.
Taking a step back: why do we need generated client bindings? If you download the OpenAPI spec it’s possible to generate the methods on the fly. Katello will need some way to retrieve that JSON. This is similar to the approach that apipie-bindings takes. You can miss some introspection options so I don’t know if that would be a problem.
Has this been an approach that’s been investigated? When I quickly look I don’t find any prior art.
And as Ewoud mentioned, smart_proxy_pulp is not present in the containerized deployment anymore (but it also never was a “real” proxy abstraction, it just told Katello where Pulp is).
Can the bindings (in generated form) live inside the Katello gem (and thus git repo)? Then we just drop the requirement on the “external” pulp_*_client gems and would be done, right?
For our downstream packaging, we have a pipeline that re-generates and re-packages the accompanying client, whenever we backport anything to a plugin component. We have had this pretty much since Pulp 3 was a thing. We have this because we had to:
When we add a new feature to pulp_deb, the time taken for that latest pulp_deb release to make it into a new Katello version, and then for that Katello version to make it into the downstream product can easily be months or even half a year. Understandably, product management does not accept waiting half a year to deliver new features to customers. And once you backport new features (with for example new API parameters) across many pulpcore versions (including plugin API breaking change releases), you really don’t have any choice but to generate new clients for the Pulp plugin-core combination you will actually deliver.
Our experience has been, that once the initial investment to set this up is done, we have been able to keep this going with a very moderate maintenance effort. Once per pulpcore Y-version upgrade, you want to check if the reference to the pulp-openapi-generator project that you are using is still up to date. In all of the Pulp 3 timeline there has been one major innovation regarding client generation, namely that there is now a pulpcore-manager command to generate the API schema of the installed instance without actually requiring that instance to run. (So you no longer need a running Pulp instance, a installed instance plus running pulpcore-manager is sufficient). We have not yet integrated that new feature into our Pipeline, because the old way still works for us, but I expect that is something we may want to change within a couple more pulpcore Y-version upgrades. Long story short: Every once in a blue moon one should expect some maintenance.
In case there is interest, I can offer a video session to show our client packaging implementation. Perhaps we could make it a session at the upcoming Virtual PulpCon 2025! - Community - Pulp Community ? If waiting a Month is undesirable because this is a hot topic right now, I am fine with scheduling an ad-hoc session any time past the 9th of November.
I think your generation of the bindings would make for a great Pulpcon presentation. I’m not sure how quickly we’ll end up actually reacting to this bindings issue anyway.
This is what @pedro-psb and I were thinking, I don’t see why they couldn’t live in the katello repository.
To Ewoud’s point though, it would be interesting if we could just use the OpenAPI spec directly. The only downside there would be that we’d need to make actual code changes. Carrying the bindings in the repo could be quicker, and this latest issue bumps the time criticality a bit to me. At that point too, I’d wonder if it would be better to figure out how we could use Pulp Glue, which would make our Pulp API compatibility even stronger.
This is great to hear. With all the changes going on, would it be better to wait to use generated bindings with the containerization effort? We could keep going for a little longer working around the bindings issues (I think). Without containerization, the smart proxy would need to get these bindings as well somehow, perhaps via the Katello repo even (sounds a bit strange, but it could work).
It would be really nice to stop using the generated bindings for Katello 4.21 at least, if not 4.20.
Actually - for smart proxies, in the short term would could simply continue using the same bindings that Katello uses like we do today. We don’t need to ship any bindings with the smart proxy. Sure we’d keep having the n-1 smart proxy issues, but that is easier to deal with than misgenerated bindings.
Kinda a side-question: how/when does the issue with the “bad” bindings arise?
Or maybe rather: does CI catch it so that it never lands in a “this was successfully tested” set of gems, that we use for our packaging bumps?
The best chance we have to catch any issues before building RPMs is with the Pulp VCR tests that run with unit tests on PRs. After that, it’s up to developers seeing issues in their development environments.
I don’t think there’s anything stopping new bindings gems from being packaged even if they are problematic. We would see the nightly failures after the fact, assuming it is caught be unit tests or BATS.
The VCR Pulp unit tests in Katello are probably the best way to tell if there is a bindings issue before making packaging updates. If the bindings have a new issue, the data coming from the recorded API responses could become garbage like they did in the issue that I just caught.
We had some chats today about the strategy for solving this issue. For the long term, we would benefit extra from having a smart client library like Pulp Glue that uses an OpenAPI spec to generate commands on the fly (essentially what Ewoud mentioned above).
Besides solving the pre-generated bindings issue here, we would also solve the issue of how to talk to older Pulps on capsules using the same client library. Pulp Glue, for example, carries “quirks” for talking with older Pulps that behave differently. With this, we could remove the monkey-patches that we carry in Katello to work with older Pulp smart proxies.
The amount of work that this would be needs to be explored. Perhaps folks from Pulp could give us some tips since they wrote Pulp Glue.
It may be worth exploring in parallel how much work it would take to ship generated bindings with Katello as a comparison.
