[RFC] Making things easier when working with custom products & Simple Content Access (SCA)

jeremylenz · November 21, 2022, 10:59pm

The problem

As a Katello user with custom (non-Red Hat) products, I need a way to restrict my hosts’ access to those products when I want to.

Background

Historically, the tools available to accomplish this restriction have been the following:

Don’t attach a subscription to the host or activation key for any products for which you want to restrict access.
Use content views and lifecycle environments to filter which products are available to hosts.
Use composite content views to combine content from different CVs
Use repository set overrides (“Override to disabled” / “Override to enabled”) for any repositories you wish to restrict further. You can do this for both activation keys and individual hosts.
Use the repository-level options Restrict to Architecture and Restrict to OS Version to further narrow down which repositories are available. (note: Restrict to Architecture works with CentOS and other non-RHEL systems, but Restrict to OS Version only works with RHEL.)

With Katello 4.6, Simple Content Access is now the default for new organizations.
In the future, entitlement-based subscription management (in other words, not using SCA) will go away entirely.
For organizations using SCA, restricting content through subscription attachments or lack thereof is not available because subscription attachment is not a thing anymore and custom products are always enabled by default.

You may at this point be thinking, “Why not just give us access to the Candlepin product content parameter that allows you to set a repository’s default enablement on an individual repo basis?” Two reasons:

When changing this for a given repository, it would affect all hosts which have access to that repository, immediately. There’s no way to make it affect “just new products” or “just hosts registered after this,” etc. This is why we had to revert that change originally in 2020.
It would be giving users yet another configuration option. We’ve been given the directive “don’t add any more of those.”

So we got together and brainstormed some other ways we could make SCA easier with custom products:

Proposed improvements

Automatically add disabled overrides for custom products to any newly-created activation key.

So, after creating a new AK, instead of this

it would look more like this:

And of course, you would then be able to remove any of the overrides that were automatically added, if desired.

Add a host bulk action to ‘Override all custom repositories to disabled.’ This would be easy to add to the individual host page as well, but would probably be more useful when performing on multiple hosts at a time.

Comments on the proposed improvements above are welcome, as well as lively discussions

rgp · November 22, 2022, 9:08am

I could be misunderstanding how this works, but doesn’t this essentially pollute hosts and activations keys with disabled custom repositories? What happens when you have 100s of custom repos? The repos also still end up in redhat.repo file on the host just disabled, right?

I can’t help but think there needs to be a way to assign products or repos as well as the ability to assign multiple content views to a host.

jeremylenz · November 22, 2022, 2:04pm

We’ve started building this. You’ll be able to register a host to multiple content views ~~within the next 1-2 releases~~ in Katello 4.8, if all goes according to plan.

(also, don’t forget about composite content views )

Bernhard_Suttner · November 22, 2022, 4:24pm

Another ideas how to solve this

Currently, “use sca” is a flag on the organization page. Probably it make sense in a pure RHEL environment to enable SCA and be happy with the new and easier way to manage repos.
As you already noted, with custom repos its different - and if you want to manage other OS like SUSE or even Debian / Ubuntu you always have custom repos.

First of all: I don’t think its easy for users to assign multiple content views. It was always pretty fine and easy for users to have exactly one content view assigned to one host and if you need more you can use composite content views. Additionally, host groups would need to be able to use multpile content views, too. It would be necessary to resolve pkg dependencies between these X assigned content views; publish new content view versions for more than one content view; remove duplicate pkgs. I would not change the current mode.

Idea 1: allow to set “use sca” on a per activation key base? Then you are free to select the mode you prefer.
Idea 2: as you have described in your first proposal, repositories should be disabled by default. One more thing which would be easier for a user would be, that only repostories are shown, which are part of the content view. Additionally, maybe some “quick actions” would improve the life of an admin, like: “enable the repository which delives package XYZ”, “Enable the repos which delivers package XYZ + all dependencies”

Bernhard_Suttner · November 22, 2022, 4:35pm

Furthermore regarding the “restrict to OS and restrict to architecture”. “restrict to OS” does currently only allow RHEL versions. Don’t know why this is the case right now. Don’t like this limitation. Additonally, I think they are on the wrong level. To “filter repos” you have activation keys and content views. This feature throws existing concepts overboard and make it pretty hard for admins to recognize why a repo is enabled or disabled. Maybe we should think about if these two settings should be where they are currently.

mhjacks · November 22, 2022, 4:48pm

I believe we should have the ability to create custom repos that are enabled by default, because the current proposal creates a non-trivial burden on either content views or activation keys. I understand that this may be a bit controversial because of the candlepin constraints Jeremy pointed out in his post, and that enabling a repo enables it whereever it is “visible.”

Let me spell out my reasoning a bit…

I use my Foreman/Katello installation to manage multiple versions of CentOS Stream (currently 8 and 9) and at least one version of Fedora. I have also used my Foreman/Katello to host RHEL content.

As I understand it, this means that I have to use Content Views to keep my content separate. So I have a Fedora (37) content view, two content views for CentOS 8 stream (I have some content filters to filter out the packages in EPEL that are problematic for coexistence with Foreman/Katello).

I use activation keys to register new hosts according to OS and usage (fedora 37 only, f37 with RPMfusion, C8s without Foreman, c8s with Foreman, c9s). My choices, then are:

Create a content view for each of these possible scenarios. (But content views don’t enable repo visibility in the new proposal.)
Each AK in the new proposal must effectively list each repo in the CV it references (since default visibility is changing). (The proposal, for this case, doesn’t help much since most of the repos I’m using I will have to explicitly enable anyway.)

In cases where RHEL content is mixed with non-RHEL content - which I suspect are the majority of Satellite uses - since EPEL is custom content - the same considerations would apply, I think.

I’m using Foreman Ansible Modules to maintain (most of) my configuration. This means I need to repeat a lot of repo names in different places (in the CVs and in the AKs), and making changes to add a new repo or remove one need to be propogated in at least two places; more for subvariants.

Now consider a world where we can set the repository’s default enabled status. I see the following benefits:

It’s straightforward to maintain the current “status quo” on custom repos - by setting the enabled flag on repos that you want enabled by default. (For custom reposets like Fedora and CentOS, many repos would be like this - fedora and updates, maybe EPEL for CentOS, etc.)
AK’s can vary repo status by exception, minimizing the state they would need to track (enable the repos disabled by default, disable any that were enabled by default).

Finally, it feels a little wrong to me to have an attribute that is obviously a characteristic of the repository but not to allow it to be set on the repository itself, but only by proxy through adding/removing the repo from content views, or specifying its state in an AK.

I recognize the risk in adding a repo with state enabled and having it “surprisingly” appear in every content view that mentions it. However, I believe this risk is largely mitigated by making the default repo state disabled.

Interested to hear other perspectives on this, and thanks for the time and attention.

mhjacks · November 22, 2022, 7:04pm

Another thought that occurred to me was to explore what “copied” activation keys do. I tried using my foreman-ansible-modules config mechanism to specify content_overrides on copied activation key creation but that didn’t work. Neither does it work to copy the key and then specify the repo “exceptions” (i.e. disable a few, enable a few).

Would it be outrageous to suggest that the activation key “copy” mechanism be extended by also allowing content overrides to be specified at copy time, such that the overrides would be “merged” with the overrides from the key being copied? This would allow for more “exception-based” management of repos at the AK level. I also recognize this might cause problems with idempotence, as it might not be clear what should happen if the original key changes state and the same params are passed to it again.

I’m not sure this solves more problems than it might create, just brainstorming about ways to honor the “no new options” directive.

gvde · November 22, 2022, 9:53pm

Well, this is continuing from Deprecation of legacy subscription management:

I still think, giving access to the enable setting for each repository would be useful. For the potential “side effects” of the current setting of a repository, I would see two simple solutions:

show a warning explaining why this is “dangerous” and the side effects. The user could then simply solve this by setting up the necessary overrides on all hosts (very easy via bulk operation) and activation keys (easy, but requires changes of all/many activation keys because there is no bulk operation for keys).
You could silently set the necessary overrides to keep the the current state, i.e. override the repo status on all hosts and keys, i.e. after the change of the default the repository has an override on all hosts and keys.

1 + 2 could be combined, i.e. warning the user and given the option to set the overrides. As additional option, the user could choose the remove the overrides where it’s already set to the new default, which would clean up a lot for me if you are using SCA and thus have already tons of disabled overrides for all the custom product repos we have.

As a practical example: let’s say I change the default for my postgresql14-el8 repository from the current (only possible) enabled to disabled.

The operation shows a warning: “if you change this it will disable the repository for all hosts and activation keys which have access to this repository and don’t have an enabled override”

Offer the option to override any host and any activation key, which does not have an override for this repository set, yet, to override enabled, thus maintaining enabled postgresql14-el8 repos at those places which don’t have it enabled by manual override already.

Also offer the option to remove any disabled override from any host and any activation key, which currently have an explicit disabled override for this repository, removing the manual override and thus reverting to the new default disable. (This would remove all those overrides I had to set after I have switched to SCA).

With the first option or both hosts and keys still have the same enabled repository set.

I would find that rather frustrating that only RHEL products should be able enable/disable a reasonable set by default. If I use any operating system, it’s reasonable to have some repositories like BaseOS, AppStream enabled and others like extras, powertools, disabled just like it’s for RHEL.

And for some products, I occasionally have to make testing/debug/development repos available for select hosts in addition to the production repos. Again, it would be extremely useful, if the production repo would be enabled by default and the others are disabled by default.

I mean, the option is there in the database for a reason. Because it’s necessary. Because some people at redhat put some thought into to what should be enabled by default and what not. I want to do the same for the products and repositories I use.

Because in the end, you have to do this anyway: all existing installations without SCA need a working migration to the new model, otherwise they’ll end up with all there hosts having access to lots of repositories which they didn’t have before.

And it can’t really be the idea of deprecating subscriptions management only for people requiring to rebuilt subscriptions using content views for all combinations of products used.

jtruestedt · November 23, 2022, 1:36pm

So with the current solution without SCA and a setup, where multiple OS and OS-Versions need to be maintained, often Composite Content Views are used to have different sets of custom repos in separate Content Views to update them without having to update all repositories.
In this case I often use Content Views to combine multiple products offering the same repo for different operating systems. With the SCA approach that would not work any more, meaning to have the same functionality I would end up with around 10 times more content views like currently.

To have an example:
Maintaining different OS:

RedHat 7
RedHat 8
RedHat 9
Ubuntu 18
Ubuntu 20
Ubuntu 22
Rockylinux 8
Rockylinux 9

And to simplify the amount of Custom Repositories, I have separate Content Views for:

Base-Repositories (OS, subscription-manager) - one for each OS major-release
EPEL - one CV combining EPEL for 7 and 8 (which are separate products)
Puppet 7 - one CV combining all Puppet 7 - repos (which are separate products)
Puppet 4 (because there is an legacy environment, where the modules are not yet updated to work with Puppet 7) - one CV combining all Puppet 7 - repos (which are separate products)
Docker-Repositories - one CV combining all Puppet 7 - repos (which are separate products)

Now I combine one composite Content View for each operating system with each CV → 8 composite Content Views and 12 “normal” content views.
Since I do not add the subscription for all the unwanted products in my activation keys, this is easy to maintain.
If SCA now means I have to separate those Content Views, I would still have 8 composite content views but to achieve the same functionality I end up with around 40 normal content views - and in this example there are not very many custom repos included.
So in the end it will be difficult to maintain all content views to have the correct patching states - which will only work, when automating it with Ansible or something.

So I can image that the overhead is somehow “acceptable” if I only have to use RHEL in 2-3 major releases but in the end I have the impression that SCA will make everything much more complex for every repository not provided by RedHat.

jtruestedt · November 23, 2022, 1:44pm

To comment also the suggestions:
I could disable/enable all unwanted repositories in the activationkeys but since products consist of multiple repositories, this will make it complex to keep activation keys up to date. If everything is enabled as default I need to adjust all activationkeys when a new OS joins my CVs.
If everything is disabled, this is probably better. But I will end up with repo-files containing very many repositories, which are disabled. Since then I only need to activate repos in new activationkeys, which will be very “crowded” if I do not have the filter mechanism to show only repos included by subscriptions. So this would be similar to the current workflow but still less user-friendly due to having many repositories to go through.

gvde · November 23, 2022, 3:05pm

Remember, that activation keys only affect the repository during registration of a new host. It does not change the repository set of existing clients. Thus, if you add a new repository to a product (e.g. you add postgresql 15 to your postgresql product), this will be enabled by default on all your existing clients if they have access through the content view.

The redhat.repo always contains all repositories which are part of the content view. Thus with SCA you will always have a lot of disabled repositories in your redhat.repo. The only way to reduce the number of repositories would be to remove them from the content view…

jtruestedt · November 23, 2022, 3:24pm

Ok so that would mean, that I have to split everything up into much more Content Views - which is not nice.

So maybe this would be an idea, where I could restrict the repositories to work only on certain OS. That would add the connection of a certain repository to a given operating system. Then there might be custom repositories, which are used in more than one OS but in this case I might need adjust Content Views.

jtruestedt · November 23, 2022, 3:39pm

Another special case, which just came to my mind:

What will be the future of the possibility to connect to the latest synced repositories?
So an activation key pointing to Library & Default Organization View → this is often used for development machines.
But this would mean, that these hosts would get ALL repositories set up in this organization (which could be a mix of RPM-repos of redhat based-distros and suse-repos). I think a mix of rpm and deb will not work anyway but that could mean that this kind of use case needs something like a content-view which is published daily or similar.

jeremylenz · December 2, 2022, 8:46pm

It seems like the two proposals in the original post may be useful for some people, but probably not enough. So I also want to add a third. (This third proposal was actually discussed previously internally, but it was decided that the first two were less complicated and more elegant…)

During a future Katello upgrade, Katello flips custom products from “enabled by default” to “disabled by default.” As part of this process, there is a migration:

Before: Custom repositories are enabled by default. Let’s say a host has 6 custom repositories. 4 are overridden to disabled, leaving 2 repositories enabled.

Migration: The 2 repositories (that were previously enabled by default) have content overrides automatically added (“override to enabled”). The 4 repositories that were previously disabled via content overrides are not touched.

After: Of the 6 custom repositories, 2 are overridden to enabled, and 4 are overridden to disabled. The same 2 repositories are enabled as before, while the same 4 repositories as before are disabled. Going forward, new custom repositories will be disabled by default.

This proposal meets the criteria of

No new configuration options for users; and
No change for a given host’s enabled repositories - no surprises.

Also, it’s not mutually exclusive with the other two proposals above. We could do all three, or just this one.

Would this be useful? More useful than the status quo?

gvde · December 2, 2022, 9:56pm

I have mentioned this „migration“ above in here also pointing out some additional options if you wanted to give users some more control.

rgp · December 3, 2022, 11:25am

Yes, that would be reasonable place to be.

However, I still think that there needs to be a way to filter what a repositories a hosts can see/use. Yes, it can be done by content views, but then you are forced to managed content views. If we have multiple OS like Fedora, CentOS Stream, AlmaLinux with many rpm repositories you will just get everything at once without a content views.

mhjacks · December 4, 2022, 1:40pm

To be clear, the status quo (which controls repo visibility with products) works decently well for the degenerate case of community distro content.

It’s not at all clear how users are benefitting from this change - a default is changing, and tools (subscriptions) we’ve had to make content determinations are now being “nerfed”. So for the case of managing multiple community distros, content views will now become mandatory, and the complexity of defining activation keys to use them also increases (since you have to explicitly enable every repo you want to use).

It can be done, of course, I just don’t see the advantage. Meanwhile, we’re being told that a setting which is clearly an attribute of the repository (it’s “enabled” status) can’t be set on the object itself, but has to be manipulated through other means (actions on host collections, AKs etc). I recognize the limitation of candlepin in setting something to “enabled”, but as a user I think that’s something I can handle.

jeremylenz · December 5, 2022, 2:11pm

This is because of the way the ‘Restrict to OS’ feature works. Candlepin compares the “required tags” of the entitlement certificate in /etc/pki/entitlement

rct cat-cert 1795965736968636904.pem

...
Content:
	Type: yum
	Name: Red Hat Satellite Client 6 for RHEL 8 x86_64 (RPMs)
	Label: satellite-client-6-for-rhel-8-x86_64-rpms
	Vendor: Red Hat
	URL: /Default_Organization/Library/content/dist/layered/rhel8/x86_64/sat-client/6/os
	GPG: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
	Enabled: False
	Expires: 1
	Required Tags: rhel-8-x86_64
	Arches: x86_64

with the “provided tags” of the product certificate in /etc/pki/product or /etc/pki/product-default:

rct cat-cert 479.pem

...
Product:
	ID: 479
	Name: Red Hat Enterprise Linux for x86_64
	Version: 8.7
	Arch: x86_64
	Tags: rhel-8,rhel-8-x86_64
	Brand Type: 
	Brand Name:

In this case, the entitlement certificate “requires” the tag rhel-8-x86_64. Since the product certificate “provides” all of the “required tags” (in this case, just the one), the Red Hat Satellite Client 6 for RHEL 8 x86_64 (RPMs) repository is available.

Note that I say “available” here as opposed to “enabled.” If the required tags were not provided, the repo would simply not be listed at all under subscription-manager repos.

Since CentOS and other non-RHEL Enterprise Linux distros do not include a product certificate, none of the required tags will ever be provided. This is why the feature only works with RHEL currently.

I’m not sure exactly what it would take to make this feature work with other OSes. I plan to ask the Candlepin team about it.

lumarel · December 15, 2022, 11:05pm

I don’t really have to add much to this as most things I’m also very much scared of are already mentioned.

One little addition, which is most likely a extreme edge case of my own is:
I’m using Katello in the release process of any new Rocky Linux release, which means I have added all and every repo in one of my Katello instances.
This means prod and staging for 2 major releases and all in all I end up there already with 137 repos.
Plus application repos from many different distributors and for testing 2 Ubuntu releases.
All in all I end up with 195 repos.
Currently managing everything with 2 lifecycle environments over 2 content views.

Beside that I have some machines assigned to no CV because they are the only machine leveraging the repo (or are unable to switch the CV because they are a Kubernetes node, and the networks there make Katello bug out on switching).
Also heavily using adding subscriptions for systems which need repos that most other machines don’t need (like Remi’s RPM repos for PHP)

Moving to SCA would mean to rethink everything, creating like at least 50 CVs with several CCVs on top to make it more manageable (or I will end up with a insane list of repos on every system repository sets view and having 200 repos in every systems redhat.repo file), touching literally every system because of the all repos being disabled by default (with having to generate a report, what is previously enabled where, and has to be enabled afterwards)

Tbh right now as the subscription-manager for Ubuntu is not ready for this, it’s not possible to move, but as soon as it is, it would be doable, will just take me days to reconfigure everything.

And everything because SCA is the new great thing… for RHEL
I really understand the move, but tbh there aren’t really many benefits for distros beside RHEL.

jeremylenz · December 16, 2022, 6:15pm

Can you elaborate why it’s a concern to have lots of repos in redhat.repo?

Can you elaborate why it’s a concern to have lots of items listed in Repository sets?

You either start with all custom products enabled, and must disable thru overrides (the status quo), or you start with custom products disabled, and must enable thru overrides (proposal #3 above.) I’m not sure I understand what concern you outlined here? Are you in favor of one or the other?