Katello 4.9 Custom products changes with SCA

Now that Katello 4.9RC1 is out, I’m interested to see how upgrading goes from Katello 4.8.

On a Katello 4.9, custom products should be disabled by default, and you’ll have “enabled” overrides for all existing hosts and activation keys.

For an overview of the feature, see https://github.com/Katello/katello/pull/10502.

To read the looooong discussion with context, see [RFC] Making things easier when working with custom products & Simple Content Access (SCA)

The task that creates these runs during upgrade and calls the Candlepin API once for each host and activation key.

We haven’t really tested this yet on deployments with large numbers of hosts. Please open a Redmine issue or post here if you have any issues with the upgrade. Thanks!

1 Like


This is pretty much the TL;DR; of the whole switching to default disabled and switching to SCA thing.

Just wanted to report my status from the 4.9 RC1 upgrade. (1 org, no RH repos, non-SCA)
Upgrade completed without any errors, reran the rake task afterwards again to see if there are any further errors (none), checked repos, activation keys, CVs, hosts… basically everywhere I expected to see the enabled status, repos are now disabled default, and every repo on activation keys and hosts are overridden with the state before the upgrade, resetting sets it to disabled, both rpm based (Rocky) and deb based (Ubuntu) systems still show the correct enabled repos after a refresh.
Next I switched the org to SCA, as expected more repos show up on the systems, but all disabled, as expected, even Ubuntu systems work without any issues!
(checked the log the whole time, and I need to take some time in the future about export import stuff, just for learning purposes)

So thank you for you great work on this rather heated task! :beers:

One question: Is there anything or anything planned to reset all activation keys and hosts, so that the overridden disabled repos are just in the state default instead?

(and reverted to before the upgrade, to have the fun a second time with RC2, to be continued)


Great to hear the upgrade is working well! By the way, how many hosts and AKs do you have?

There’s a bulk action available from the old Content Hosts list page. Select Action > Manage Repository Sets. Then select everything and Select Action > Reset to Default.

For activation keys there isn’t any bulk action available.

1 Like

This is just my homelab prod system, 50 hosts (17 really active) and 4 activation keys (4 OSes).

There’s a bulk action available from the old Content Hosts list page. Select Action > Manage Repository Sets. Then select everything and Select Action > Reset to Default.

Yeah but that’s just for setting all selected repos for the selected hosts to the same state, right? What would be needed is a “if the repo is in state Disabled (overridden) reset to Default”, it does not value Enabled (overridden). I might be able to solve this with some bash hammer magic, or a more sophisticated Ansible playbook, but I think this would be needed by everyone afterwards. (or better said the handful of people who try to clean up afterwards)

1 Like

@jeremylenz As the Katello 4.9 release gets close I spent time again to think about the automatic switching:
So what happens if you upgrade to 4.9, all the host/activation key overrides get set, you think, great everything is done, doing manual cleanup of all unnecessary overrides (because yeah… sounds like a logical thing to do), but then 4.10 or 4.9.1 comes around and you have to rerun foreman-installer again.
Wouldn’t that mess up all the overrides? Because the disabled overrides got reset to being default, and the automatic (or manual trigger) helper on the end of foreman-installer will get detected and overridden to enabled.

Or am I misunderstanding something here?

And also what happens to new hosts/activation keys where you didn’t bother to put everything in a overridden state?

The overrides are created during an upgrade rake task, so I don’t think it would re-run on subsequent installer runs. Upgrade rake tasks are also namespaced to the version (4.9), so it also shouldn’t run again when you upgrade to 4.10.

New hosts and AKs will have custom repos disabled by default, unless you add overrides :slight_smile:

However, if you register a new host with an existing AK that has overrides, that host will of course get those overrides.

Okay I will test that when 4.9.1 is available :sweat_smile: I just suspect that it will convert the defaults to override enabled then, but not sure ^^

Yeah but if you rerun the conversion then manually (or it somehow gets triggered a 2nd time during the foreman-installer run) that will override the defaults to override enabled, while testing with 4.9 RC1 I had such a stopping brain moment, where I set everything up, removed some overrides as they were override disabled, and then reran the conversion manually, it basically messed up the whole machines repos, as the defaults were overridden to enabled ^^

I will try soon again, when the release anouncement was made, not sure if it is safe to install 4.9 right now :slight_smile: (the checklist would suggest there isn’t everything there up to now)

Yeah, the migration to add the overrides is supposed to run only once, and you’re not really supposed to run it yourself. If it does re-run on upgrade to 4.9.1, that will be a bug. Thanks for bringing that up; I’ll remind myself to check on that :slight_smile:

1 Like

Okay thanks got it, hopefully it will just run smoothly :slightly_smiling_face:

After double-checking in Foreman’s code, I think this should be fine.

Upgrade tasks are executed during rake upgrade:run: foreman/lib/tasks/upgrade.rake at ba1abfc447fd22447c79d7351d39507cba4b8b83 · theforeman/foreman · GitHub

The only upgrade tasks that execute are those “needing run”:


and then when the task completes successfully, it marks them as ran:

task.mark_as_ran! if run_task(task)

What this means is that a given upgrade rake task should only run once, regardless of version namespacing-- unless you clear out your database or something, or the task is marked as :always_run (which this one isn’t.) So, for upgrades, we should be all good.

However, I tested in the console on a new install:

irb(main):003:0> UpgradeTask.find_by(name: "katello:upgrades:4.9:update_custom_products_enablement").last_run_time
=> nil

This may mean there’s an issue in the following scenario:

  1. I perform a new install of Katello 4.9.0. The upgrade tasks never run, since it’s a new install. Default product enablement is disabled.
  2. I then upgrade to 4.9.1 or 4.10. The upgrade task’s last_run_time is nil, so it will be included in UpgradeTask.needing_run and create enabled overrides for all my custom repositories.

I’ve created Bug #36540: Upgrade rake task will create bad content overrides on post-4.9 Katello upgrades - Katello - Foreman

thanks for helping me think this out!


Am I going to cause myself trouble if I go into Candlepin Swagger and POST ?enabled=true to my /owners/{owner_key}/products/{product_id}/content/{content_id}s?

Probably. This would cause Katello’s ProductContent to be out of sync with Candlepin and show you confusing things on the web UI. It’s not supported to change a custom product’s default enablement as a user.

1 Like

Hi there!
I’m a bit late to the party and I was (kinda unpleasantly, but it’s OK :wink: ) surprised by the Release notes of Satellite 6.14 that Custom products are now disabled by Default.

I tried looking for a rationale behind it, but I have some issues finding it. And I find it interesting that a function that’s been in the product for many years (I only started using Foreman/Satellite in 2019) is changed so drastically without the explanation why

Don’t get me wrong, I love the work you all do on this and I’m a really happy user overall, but I don’t understand why this needed to be changed in the first place, let alone, why it wasn’t turned into a toggle that can be configured by the administrator.

The reason for me to want it on is that I extensively use CCVs/Activation Keys to decide which products I present to ‘endusers’ and I have all this documented in Ansible. So composing new (C)CVs wasn’t a lot of work for me, up until now. As the theforeman.foreman.activation_key module requires the label of each and every repo I need to add to a activation key. And those names can differ in each Satellite installation (as an MSP we manage multiple), it adds up. In my Lab environment alone I have 50 activation keys (generated with Ansible, but still, updating my code/these keys manually is not fun)

BTW, props for not retroactively disabling all existing, enabled, repos in activation keys :wink:

1 Like

Hey @Thulium-Drake

The rationale for changing the default enablement was basically that Red Hat is moving away from entitlement management in the tooling, in favor of SCA.

When adding a new custom product to the system, you have to take certain actions:

With (non-SCA) entitlement mode, people would attach a subscription, and then expect to have access to that repo immediately without having to take additional steps. Hence custom products being enabled by default.

Now that Simple Content Access is the new norm, enabled becomes a bad default – when adding a new custom product, all hosts will immediately have access to it, and you must then go disable it for any existing hosts and activation keys, except those which you want the new product to have access to.

The recurring theme is that when adding a new custom product, we want you to have to take action in order to give hosts access to it – not the other way around. So that was the main reason to flip the default enablement to disabled.

As for adding a user-definable toggle, that was explicitly discouraged by RH product folks, since it would have added “yet another configuration option” to Katello, and we already have plenty of those.

Regarding existing content overrides on your activation keys-- We did add one thing which may make it slightly easier, if you’re open to using the web UI. On the activation key detail Repository Sets tab, there is a new filter dropdown so that you can now select just “Custom” repos, and then select all items and override to Enabled.

1 Like

Tbh how it works now with SCA is pretty sleek, also for distros other than RHEL.

The only missing feature compared to RHEL would be Restrict to architecture / Restrict to OS version for custom repositories now, which would be very nice to have, but yeah I know that this is currently bound to the entitlements (or was it something else) and the priority/motivation to change/extend that is not that high.
I think to remember a message from Bernhard some time ago, so maybe they are already working on that.

1 Like

Thanks for the reply, I appreciate it! :slight_smile:

I haven’t installed a fresh Katello system in a while, but I can’t imagine this is actually maintainable, because it adds AKs * Custom Repos (and for non-RHEL products that means all of them) new buttons that can be misaligned between LCEs, different people misreading a instruction/mistyping a think in Ansible (and this has already bitten me in addressing this for a fresh Satellite installation).

I don’t agree with that ‘disabled by default’ is a bad practice, for me and my colleagues that’s the whole point of making CCVs, we only allow access to products that a enduser (system or group) is actually allowed to use.

Maybe I’m seeing things wrong, but for me SCA means something different, it meant taking away all of the complex subscription stuff and making RH sub compliancy a administrative requirement. Which, and I fully agree with that, significantly lessens the “technical administrative” burden and thus also calls to support. And requiring that we add repos to CCVs (no objection here) as well as AKs is basically bringing back all that nonsense again.

Also, for me, adding ‘yet another button’ to a versatile product is not an objection for me, but I understand the need from the product people to keep things managable.

Hi There,

New to this forum yet I’ve red tons of articles here already :sweat_smile:

I must admin that Thulium has a point there. The whole idea of using Ansible to automate such thing
is to… well… automate… make things easier, more secure yada yada yada…

Having a switch on the UI seems like a ‘compensation’, but for those having the entire infra as code

The idea of ‘disabled by default unless stated otherwise’ is a common and best practice, but imho… it would be best to find the right balance between ‘to secure’ and ‘usability’ (and i’m sure this has been thoughtfully considered), but it seems to make things more complicated at certain use-cases instead.

imo it would be best to make sure the admin/engineer using satellite has the option to manage this
to his/her/them will, both via UI and Cli/Ansible.

Is there anything specific that would help those who run things via Ansible only?

I know we do have an API for adding content overrides to an existing host. For the endpoint /api/v2/hosts/:id/subscriptions/content_override, if you add the limit_to_env parameter and a blank search, like this:


you can enable all repositories for that host without having to specify the content labels.

I checked the corresponding activation key endpoint, and it doesn’t yet have the limit_to_env parameter. update: I don’t think we’d need that for activation keys

If we added an Ansible module to Foreman Ansible modules for those two endpoints, that would perhaps help a bit. Thoughts?

1 Like

First, if you want extra clarification, let me know, I’m about to dump some code :wink:

You can find all of the code in GitHub - Thulium-Drake/ansible-role-foreman: Ansible toolkit for Foreman/Satellite

The task I use to make the AKs with the content overrides currently require the full label of the repository, like so:

- name: "Create Activation Keys for {{ key_lifecycle['os'] }} - {{ key_lifecycle['lifecycle'] }}"  # noqa name[template]
    username: "{{ foreman_admin_user }}"
    password: "{{ foreman_admin_password }}"
    server_url: "{{ foreman_url }}"
    validate_certs: "{{ foreman_validate_certs }}"
    organization: "{{ foreman_organization }}"
    name: "{{ key['name'] }}-{{ key_lifecycle['lifecycle'] }}"
    description: 'Managed by Ansible, your changes will be lost'
    lifecycle_environment: "{{ key_lifecycle['lifecycle'] }}"
    content_view: "{{ key['content_view'] }}"
    subscriptions: "{{ (foreman_simple_content_mode) | ternary(omit, key['subscriptions']) | default(omit) }}"
    content_overrides: "{{ key['content_overrides'] | default(omit) }}"
    host_collections: >-
      {{ key_lifecycle['host_collection'] +
      [key['name'] | lower] +
      [key['os'] | lower] }}
    release_version: "{{ key['release_version'] | default(omit) }}"
  loop: "{{ foreman_activation_keys }}"
    - key['os'] == key_lifecycle['os']
    - key['host_collections'] is not defined
    loop_var: 'key'
    label: "{{ key['name'] }}"

The relevant part of the input variables currently looks like this:

  - name: 'RedHat8-Base'
    os: 'redhat8'
    content_view: 'COV RedHat8-Base'
    release_version: '8'
    auto_attach: true
      - label: 'satellite-client-6-for-rhel-8-x86_64-rpms'
        override: 'enabled'
      - label: "{{ foreman_organization}}_Zabbix-EL8_Zabbix-EL8-Zabbix-{{ zabbix_version | replace('.', '_') }}"
        override: 'enabled'
      - label: "{{ foreman_organization}}_Zabbix-EL8_Zabbix-EL8-Zabbix-Non-supported-packages"
        override: 'enabled'

The main issue I see is that the content overrides require labels, which are different for each organization. product etc. However, it’s not possible to retrieve them from the WebUI (I must admin, I didn’t try with Hammer yet…) without inspecting the requests going out to the server.

It would already be a massive help if the content override input could either be the Repository name, or use the same ‘fuzzy’ matching that subscription-manager uses, e.g. the Zabbix repo would be *-Zabbix-6_4

The solution you mention also would be a very interesting addition, that’d make the picture complete. The fuzzy matching makes adding overrides easier, and having the ability to search and change all related repos for an AK eliminates the need for it in the usecase we have.

Thanks for sharing your thoughts! :slight_smile:



I’m doing pretty same thing as you:

  - name: "Create AK-PRODUCTION-ALMALINUX9 Activation Key"
      username: "{{ login }}"
      password: "{{ password }}"
      server_url: "{{ host_proxy }}"
      validate_certs: "{{ certs }}"
      organization: "{{ organization }}"
      lifecycle_environment: "{{ lifecycle_env }}"
      content_view: "CV-ALMALINUX9"
        - name: "AlmaLinux"
        - name: "Elrepo"
        - name: "RPMFusion"
        - name: "EPEL"
          - label: "Elrepo_elrepo-7-x64"
            override: disabled
          - label: "RPMFusion_rpmfusion-el7-x64"
            override: disabled
          - label: "EPEL_epel-7-x64"
            override: disabled  
      state: present

I’m on the latest 3.10 version with SCA enable by default and go this error:

FAILED! => {“changed”: false, “error”: {“displayMessage”: “The specified organization is in Simple Content Access mode. Attaching subscriptions is disabled”, “errors”: [“The specified organization is in Simple Content Access mode. Attaching subscriptions is disabled”]}, “msg”: “Error while performing add_subscriptions on activation_keys: 400 Client Error: Bad Request for url: https://foreman/katello/api/activation_keys/6/add_subscriptions”}

If i comment the subscriptions part it works, but every custom repo are disabled on my host.

Do you deal with this ?