We’ve started building this. You’ll be able to register a host to multiple content views within the next 1-2 releases in Katello 4.8, if all goes according to plan.
(also, don’t forget about composite content views 
)
Another ideas how to solve this
Currently, “use sca” is a flag on the organization page. Probably it make sense in a pure RHEL environment to enable SCA and be happy with the new and easier way to manage repos.
As you already noted, with custom repos its different - and if you want to manage other OS like SUSE or even Debian / Ubuntu you always have custom repos.
First of all: I don’t think its easy for users to assign multiple content views. It was always pretty fine and easy for users to have exactly one content view assigned to one host and if you need more you can use composite content views. Additionally, host groups would need to be able to use multpile content views, too. It would be necessary to resolve pkg dependencies between these X assigned content views; publish new content view versions for more than one content view; remove duplicate pkgs. I would not change the current mode.
Idea 1: allow to set “use sca” on a per activation key base? Then you are free to select the mode you prefer.
Idea 2: as you have described in your first proposal, repositories should be disabled by default. One more thing which would be easier for a user would be, that only repostories are shown, which are part of the content view. Additionally, maybe some “quick actions” would improve the life of an admin, like: “enable the repository which delives package XYZ”, “Enable the repos which delivers package XYZ + all dependencies”
2 Likes
Furthermore regarding the “restrict to OS and restrict to architecture”. “restrict to OS” does currently only allow RHEL versions. Don’t know why this is the case right now. Don’t like this limitation. Additonally, I think they are on the wrong level. To “filter repos” you have activation keys and content views. This feature throws existing concepts overboard and make it pretty hard for admins to recognize why a repo is enabled or disabled. Maybe we should think about if these two settings should be where they are currently.
1 Like
I believe we should have the ability to create custom repos that are enabled by default, because the current proposal creates a non-trivial burden on either content views or activation keys. I understand that this may be a bit controversial because of the candlepin constraints Jeremy pointed out in his post, and that enabling a repo enables it whereever it is “visible.”
Let me spell out my reasoning a bit…
I use my Foreman/Katello installation to manage multiple versions of CentOS Stream (currently 8 and 9) and at least one version of Fedora. I have also used my Foreman/Katello to host RHEL content.
As I understand it, this means that I have to use Content Views to keep my content separate. So I have a Fedora (37) content view, two content views for CentOS 8 stream (I have some content filters to filter out the packages in EPEL that are problematic for coexistence with Foreman/Katello).
I use activation keys to register new hosts according to OS and usage (fedora 37 only, f37 with RPMfusion, C8s without Foreman, c8s with Foreman, c9s). My choices, then are:
- Create a content view for each of these possible scenarios. (But content views don’t enable repo visibility in the new proposal.)
- Each AK in the new proposal must effectively list each repo in the CV it references (since default visibility is changing). (The proposal, for this case, doesn’t help much since most of the repos I’m using I will have to explicitly enable anyway.)
In cases where RHEL content is mixed with non-RHEL content - which I suspect are the majority of Satellite uses - since EPEL is custom content - the same considerations would apply, I think.
I’m using Foreman Ansible Modules to maintain (most of) my configuration. This means I need to repeat a lot of repo names in different places (in the CVs and in the AKs), and making changes to add a new repo or remove one need to be propogated in at least two places; more for subvariants.
Now consider a world where we can set the repository’s default enabled status. I see the following benefits:
- It’s straightforward to maintain the current “status quo” on custom repos - by setting the enabled flag on repos that you want enabled by default. (For custom reposets like Fedora and CentOS, many repos would be like this - fedora and updates, maybe EPEL for CentOS, etc.)
- AK’s can vary repo status by exception, minimizing the state they would need to track (enable the repos disabled by default, disable any that were enabled by default).
Finally, it feels a little wrong to me to have an attribute that is obviously a characteristic of the repository but not to allow it to be set on the repository itself, but only by proxy through adding/removing the repo from content views, or specifying its state in an AK.
I recognize the risk in adding a repo with state enabled and having it “surprisingly” appear in every content view that mentions it. However, I believe this risk is largely mitigated by making the default repo state disabled.
Interested to hear other perspectives on this, and thanks for the time and attention.
1 Like
Another thought that occurred to me was to explore what “copied” activation keys do. I tried using my foreman-ansible-modules config mechanism to specify content_overrides on copied activation key creation but that didn’t work. Neither does it work to copy the key and then specify the repo “exceptions” (i.e. disable a few, enable a few).
Would it be outrageous to suggest that the activation key “copy” mechanism be extended by also allowing content overrides to be specified at copy time, such that the overrides would be “merged” with the overrides from the key being copied? This would allow for more “exception-based” management of repos at the AK level. I also recognize this might cause problems with idempotence, as it might not be clear what should happen if the original key changes state and the same params are passed to it again.
I’m not sure this solves more problems than it might create, just brainstorming about ways to honor the “no new options” directive.
Well, this is continuing from Deprecation of legacy subscription management:
I still think, giving access to the enable setting for each repository would be useful. For the potential “side effects” of the current setting of a repository, I would see two simple solutions:
-
show a warning explaining why this is “dangerous” and the side effects. The user could then simply solve this by setting up the necessary overrides on all hosts (very easy via bulk operation) and activation keys (easy, but requires changes of all/many activation keys because there is no bulk operation for keys).
-
You could silently set the necessary overrides to keep the the current state, i.e. override the repo status on all hosts and keys, i.e. after the change of the default the repository has an override on all hosts and keys.
1 + 2 could be combined, i.e. warning the user and given the option to set the overrides. As additional option, the user could choose the remove the overrides where it’s already set to the new default, which would clean up a lot for me if you are using SCA and thus have already tons of disabled overrides for all the custom product repos we have.
As a practical example: let’s say I change the default for my postgresql14-el8 repository from the current (only possible) enabled to disabled.
The operation shows a warning: “if you change this it will disable the repository for all hosts and activation keys which have access to this repository and don’t have an enabled override”
Offer the option to override any host and any activation key, which does not have an override for this repository set, yet, to override enabled, thus maintaining enabled postgresql14-el8 repos at those places which don’t have it enabled by manual override already.
Also offer the option to remove any disabled override from any host and any activation key, which currently have an explicit disabled override for this repository, removing the manual override and thus reverting to the new default disable. (This would remove all those overrides I had to set after I have switched to SCA).
With the first option or both hosts and keys still have the same enabled repository set.
I would find that rather frustrating that only RHEL products should be able enable/disable a reasonable set by default. If I use any operating system, it’s reasonable to have some repositories like BaseOS, AppStream enabled and others like extras, powertools, disabled just like it’s for RHEL.
And for some products, I occasionally have to make testing/debug/development repos available for select hosts in addition to the production repos. Again, it would be extremely useful, if the production repo would be enabled by default and the others are disabled by default.
I mean, the option is there in the database for a reason. Because it’s necessary. Because some people at redhat put some thought into to what should be enabled by default and what not. I want to do the same for the products and repositories I use.
Because in the end, you have to do this anyway: all existing installations without SCA need a working migration to the new model, otherwise they’ll end up with all there hosts having access to lots of repositories which they didn’t have before.
And it can’t really be the idea of deprecating subscriptions management only for people requiring to rebuilt subscriptions using content views for all combinations of products used.
2 Likes
So with the current solution without SCA and a setup, where multiple OS and OS-Versions need to be maintained, often Composite Content Views are used to have different sets of custom repos in separate Content Views to update them without having to update all repositories.
In this case I often use Content Views to combine multiple products offering the same repo for different operating systems. With the SCA approach that would not work any more, meaning to have the same functionality I would end up with around 10 times more content views like currently.
To have an example:
Maintaining different OS:
- RedHat 7
- RedHat 8
- RedHat 9
- Ubuntu 18
- Ubuntu 20
- Ubuntu 22
- Rockylinux 8
- Rockylinux 9
And to simplify the amount of Custom Repositories, I have separate Content Views for:
- Base-Repositories (OS, subscription-manager) - one for each OS major-release
- EPEL - one CV combining EPEL for 7 and 8 (which are separate products)
- Puppet 7 - one CV combining all Puppet 7 - repos (which are separate products)
- Puppet 4 (because there is an legacy environment, where the modules are not yet updated to work with Puppet 7) - one CV combining all Puppet 7 - repos (which are separate products)
- Docker-Repositories - one CV combining all Puppet 7 - repos (which are separate products)
Now I combine one composite Content View for each operating system with each CV → 8 composite Content Views and 12 “normal” content views.
Since I do not add the subscription for all the unwanted products in my activation keys, this is easy to maintain.
If SCA now means I have to separate those Content Views, I would still have 8 composite content views but to achieve the same functionality I end up with around 40 normal content views - and in this example there are not very many custom repos included.
So in the end it will be difficult to maintain all content views to have the correct patching states - which will only work, when automating it with Ansible or something.
So I can image that the overhead is somehow “acceptable” if I only have to use RHEL in 2-3 major releases but in the end I have the impression that SCA will make everything much more complex for every repository not provided by RedHat.
3 Likes
To comment also the suggestions:
I could disable/enable all unwanted repositories in the activationkeys but since products consist of multiple repositories, this will make it complex to keep activation keys up to date. If everything is enabled as default I need to adjust all activationkeys when a new OS joins my CVs.
If everything is disabled, this is probably better. But I will end up with repo-files containing very many repositories, which are disabled. Since then I only need to activate repos in new activationkeys, which will be very “crowded” if I do not have the filter mechanism to show only repos included by subscriptions. So this would be similar to the current workflow but still less user-friendly due to having many repositories to go through.
2 Likes
Remember, that activation keys only affect the repository during registration of a new host. It does not change the repository set of existing clients. Thus, if you add a new repository to a product (e.g. you add postgresql 15 to your postgresql product), this will be enabled by default on all your existing clients if they have access through the content view.
The redhat.repo always contains all repositories which are part of the content view. Thus with SCA you will always have a lot of disabled repositories in your redhat.repo. The only way to reduce the number of repositories would be to remove them from the content view…
Ok so that would mean, that I have to split everything up into much more Content Views - which is not nice.
So maybe this would be an idea, where I could restrict the repositories to work only on certain OS. That would add the connection of a certain repository to a given operating system. Then there might be custom repositories, which are used in more than one OS but in this case I might need adjust Content Views.
Another special case, which just came to my mind:
What will be the future of the possibility to connect to the latest synced repositories?
So an activation key pointing to Library & Default Organization View → this is often used for development machines.
But this would mean, that these hosts would get ALL repositories set up in this organization (which could be a mix of RPM-repos of redhat based-distros and suse-repos). I think a mix of rpm and deb will not work anyway but that could mean that this kind of use case needs something like a content-view which is published daily or similar.
It seems like the two proposals in the original post may be useful for some people, but probably not enough. So I also want to add a third. (This third proposal was actually discussed previously internally, but it was decided that the first two were less complicated and more elegant…)
- During a future Katello upgrade, Katello flips custom products from “enabled by default” to “disabled by default.” As part of this process, there is a migration:
Before: Custom repositories are enabled by default. Let’s say a host has 6 custom repositories. 4 are overridden to disabled, leaving 2 repositories enabled.
Migration: The 2 repositories (that were previously enabled by default) have content overrides automatically added (“override to enabled”). The 4 repositories that were previously disabled via content overrides are not touched.
After: Of the 6 custom repositories, 2 are overridden to enabled, and 4 are overridden to disabled. The same 2 repositories are enabled as before, while the same 4 repositories as before are disabled. Going forward, new custom repositories will be disabled by default.
This proposal meets the criteria of
- No new configuration options for users; and
- No change for a given host’s enabled repositories - no surprises.
Also, it’s not mutually exclusive with the other two proposals above. We could do all three, or just this one.
Would this be useful? More useful than the status quo?
I have mentioned this „migration“ above in here also pointing out some additional options if you wanted to give users some more control.
1 Like
Yes, that would be reasonable place to be.
However, I still think that there needs to be a way to filter what a repositories a hosts can see/use. Yes, it can be done by content views, but then you are forced to managed content views. If we have multiple OS like Fedora, CentOS Stream, AlmaLinux with many rpm repositories you will just get everything at once without a content views.
2 Likes
To be clear, the status quo (which controls repo visibility with products) works decently well for the degenerate case of community distro content.
It’s not at all clear how users are benefitting from this change - a default is changing, and tools (subscriptions) we’ve had to make content determinations are now being “nerfed”. So for the case of managing multiple community distros, content views will now become mandatory, and the complexity of defining activation keys to use them also increases (since you have to explicitly enable every repo you want to use).
It can be done, of course, I just don’t see the advantage. Meanwhile, we’re being told that a setting which is clearly an attribute of the repository (it’s “enabled” status) can’t be set on the object itself, but has to be manipulated through other means (actions on host collections, AKs etc). I recognize the limitation of candlepin in setting something to “enabled”, but as a user I think that’s something I can handle.
1 Like
This is because of the way the ‘Restrict to OS’ feature works. Candlepin compares the “required tags” of the entitlement certificate in /etc/pki/entitlement
rct cat-cert 1795965736968636904.pem
...
Content:
Type: yum
Name: Red Hat Satellite Client 6 for RHEL 8 x86_64 (RPMs)
Label: satellite-client-6-for-rhel-8-x86_64-rpms
Vendor: Red Hat
URL: /Default_Organization/Library/content/dist/layered/rhel8/x86_64/sat-client/6/os
GPG: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Enabled: False
Expires: 1
Required Tags: rhel-8-x86_64
Arches: x86_64
with the “provided tags” of the product certificate in /etc/pki/product or /etc/pki/product-default:
rct cat-cert 479.pem
...
Product:
ID: 479
Name: Red Hat Enterprise Linux for x86_64
Version: 8.7
Arch: x86_64
Tags: rhel-8,rhel-8-x86_64
Brand Type:
Brand Name:
In this case, the entitlement certificate “requires” the tag rhel-8-x86_64. Since the product certificate “provides” all of the “required tags” (in this case, just the one), the Red Hat Satellite Client 6 for RHEL 8 x86_64 (RPMs) repository is available.
Note that I say “available” here as opposed to “enabled.” If the required tags were not provided, the repo would simply not be listed at all under subscription-manager repos.
Since CentOS and other non-RHEL Enterprise Linux distros do not include a product certificate, none of the required tags will ever be provided. This is why the feature only works with RHEL currently.
I’m not sure exactly what it would take to make this feature work with other OSes. I plan to ask the Candlepin team about it.
3 Likes
I don’t really have to add much to this as most things I’m also very much scared of are already mentioned.
One little addition, which is most likely a extreme edge case of my own is:
I’m using Katello in the release process of any new Rocky Linux release, which means I have added all and every repo in one of my Katello instances.
This means prod and staging for 2 major releases and all in all I end up there already with 137 repos.
Plus application repos from many different distributors and for testing 2 Ubuntu releases.
All in all I end up with 195 repos.
Currently managing everything with 2 lifecycle environments over 2 content views.
Beside that I have some machines assigned to no CV because they are the only machine leveraging the repo (or are unable to switch the CV because they are a Kubernetes node, and the networks there make Katello bug out on switching).
Also heavily using adding subscriptions for systems which need repos that most other machines don’t need (like Remi’s RPM repos for PHP)
Moving to SCA would mean to rethink everything, creating like at least 50 CVs with several CCVs on top to make it more manageable (or I will end up with a insane list of repos on every system repository sets view and having 200 repos in every systems redhat.repo file), touching literally every system because of the all repos being disabled by default (with having to generate a report, what is previously enabled where, and has to be enabled afterwards)
Tbh right now as the subscription-manager for Ubuntu is not ready for this, it’s not possible to move, but as soon as it is, it would be doable, will just take me days to reconfigure everything.
And everything because SCA is the new great thing… for RHEL
I really understand the move, but tbh there aren’t really many benefits for distros beside RHEL.
Can you elaborate why it’s a concern to have lots of repos in redhat.repo?
Can you elaborate why it’s a concern to have lots of items listed in Repository sets?
You either start with all custom products enabled, and must disable thru overrides (the status quo), or you start with custom products disabled, and must enable thru overrides (proposal #3 above.) I’m not sure I understand what concern you outlined here? Are you in favor of one or the other?
1 Like
This is not really a technical problem, just a look and feel thing, even a very minor one. (In case you start looking for repos on a client machine via either dnf or subscription-manager, which nearly never happens)
As well very much a keeping the overview thing, a search and a filter for currently enabled repos would improve that of course, but still that is slower than just having the repos there, you enabled the subscriptions of and being able to see what is enabled or what needs to be enabled to get to the wanted result.
Correct, currently the systems get some repos disabled on the bootstrap via the Activation Key, after changing all repos to being visible to a machine all will be either enabled or disabled by default, the currently already overridden repos will stay the way they are, but the rest… Lots of manual work involved.
Or am I reading this wrong in your #3 proposal? Does it mean that all repos that previously were available but not changed to overridden disabled (so either default or enabled overridden), will be enabled overridden and everything else disabled? (looks like I didn’t think of a automatic migration then 
)
It wouldn’t remove the need of creating lots of CVs to keep the overview though, but at least it doesn’t have to be done while migrating to SCA, which sounds a lot better than I previously thought!
1 Like
Yes. So on a given host you’d still have the same repos available before and after the Katello upgrade / migration.
1 Like