Katello 4.9 Custom products changes with SCA

Now that Katello 4.9RC1 is out, I’m interested to see how upgrading goes from Katello 4.8.

On a Katello 4.9, custom products should be disabled by default, and you’ll have “enabled” overrides for all existing hosts and activation keys.

For an overview of the feature, see https://github.com/Katello/katello/pull/10502.

To read the looooong discussion with context, see [RFC] Making things easier when working with custom products & Simple Content Access (SCA)

The task that creates these runs during upgrade and calls the Candlepin API once for each host and activation key.

We haven’t really tested this yet on deployments with large numbers of hosts. Please open a Redmine issue or post here if you have any issues with the upgrade. Thanks!

1 Like


This is pretty much the TL;DR; of the whole switching to default disabled and switching to SCA thing.

Just wanted to report my status from the 4.9 RC1 upgrade. (1 org, no RH repos, non-SCA)
Upgrade completed without any errors, reran the rake task afterwards again to see if there are any further errors (none), checked repos, activation keys, CVs, hosts… basically everywhere I expected to see the enabled status, repos are now disabled default, and every repo on activation keys and hosts are overridden with the state before the upgrade, resetting sets it to disabled, both rpm based (Rocky) and deb based (Ubuntu) systems still show the correct enabled repos after a refresh.
Next I switched the org to SCA, as expected more repos show up on the systems, but all disabled, as expected, even Ubuntu systems work without any issues!
(checked the log the whole time, and I need to take some time in the future about export import stuff, just for learning purposes)

So thank you for you great work on this rather heated task! :beers:

One question: Is there anything or anything planned to reset all activation keys and hosts, so that the overridden disabled repos are just in the state default instead?

(and reverted to before the upgrade, to have the fun a second time with RC2, to be continued)


Great to hear the upgrade is working well! By the way, how many hosts and AKs do you have?

There’s a bulk action available from the old Content Hosts list page. Select Action > Manage Repository Sets. Then select everything and Select Action > Reset to Default.

For activation keys there isn’t any bulk action available.

1 Like

This is just my homelab prod system, 50 hosts (17 really active) and 4 activation keys (4 OSes).

There’s a bulk action available from the old Content Hosts list page. Select Action > Manage Repository Sets. Then select everything and Select Action > Reset to Default.

Yeah but that’s just for setting all selected repos for the selected hosts to the same state, right? What would be needed is a “if the repo is in state Disabled (overridden) reset to Default”, it does not value Enabled (overridden). I might be able to solve this with some bash hammer magic, or a more sophisticated Ansible playbook, but I think this would be needed by everyone afterwards. (or better said the handful of people who try to clean up afterwards)

1 Like

@jeremylenz As the Katello 4.9 release gets close I spent time again to think about the automatic switching:
So what happens if you upgrade to 4.9, all the host/activation key overrides get set, you think, great everything is done, doing manual cleanup of all unnecessary overrides (because yeah… sounds like a logical thing to do), but then 4.10 or 4.9.1 comes around and you have to rerun foreman-installer again.
Wouldn’t that mess up all the overrides? Because the disabled overrides got reset to being default, and the automatic (or manual trigger) helper on the end of foreman-installer will get detected and overridden to enabled.

Or am I misunderstanding something here?

And also what happens to new hosts/activation keys where you didn’t bother to put everything in a overridden state?

The overrides are created during an upgrade rake task, so I don’t think it would re-run on subsequent installer runs. Upgrade rake tasks are also namespaced to the version (4.9), so it also shouldn’t run again when you upgrade to 4.10.

New hosts and AKs will have custom repos disabled by default, unless you add overrides :slight_smile:

However, if you register a new host with an existing AK that has overrides, that host will of course get those overrides.

Okay I will test that when 4.9.1 is available :sweat_smile: I just suspect that it will convert the defaults to override enabled then, but not sure ^^

Yeah but if you rerun the conversion then manually (or it somehow gets triggered a 2nd time during the foreman-installer run) that will override the defaults to override enabled, while testing with 4.9 RC1 I had such a stopping brain moment, where I set everything up, removed some overrides as they were override disabled, and then reran the conversion manually, it basically messed up the whole machines repos, as the defaults were overridden to enabled ^^

I will try soon again, when the release anouncement was made, not sure if it is safe to install 4.9 right now :slight_smile: (the checklist would suggest there isn’t everything there up to now)

Yeah, the migration to add the overrides is supposed to run only once, and you’re not really supposed to run it yourself. If it does re-run on upgrade to 4.9.1, that will be a bug. Thanks for bringing that up; I’ll remind myself to check on that :slight_smile:

1 Like

Okay thanks got it, hopefully it will just run smoothly :slightly_smiling_face:

After double-checking in Foreman’s code, I think this should be fine.

Upgrade tasks are executed during rake upgrade:run: foreman/lib/tasks/upgrade.rake at ba1abfc447fd22447c79d7351d39507cba4b8b83 · theforeman/foreman · GitHub

The only upgrade tasks that execute are those “needing run”:


and then when the task completes successfully, it marks them as ran:

task.mark_as_ran! if run_task(task)

What this means is that a given upgrade rake task should only run once, regardless of version namespacing-- unless you clear out your database or something, or the task is marked as :always_run (which this one isn’t.) So, for upgrades, we should be all good.

However, I tested in the console on a new install:

irb(main):003:0> UpgradeTask.find_by(name: "katello:upgrades:4.9:update_custom_products_enablement").last_run_time
=> nil

This may mean there’s an issue in the following scenario:

  1. I perform a new install of Katello 4.9.0. The upgrade tasks never run, since it’s a new install. Default product enablement is disabled.
  2. I then upgrade to 4.9.1 or 4.10. The upgrade task’s last_run_time is nil, so it will be included in UpgradeTask.needing_run and create enabled overrides for all my custom repositories.

I’ve created Bug #36540: Upgrade rake task will create bad content overrides on post-4.9 Katello upgrades - Katello - Foreman

thanks for helping me think this out!


Am I going to cause myself trouble if I go into Candlepin Swagger and POST ?enabled=true to my /owners/{owner_key}/products/{product_id}/content/{content_id}s?

Probably. This would cause Katello’s ProductContent to be out of sync with Candlepin and show you confusing things on the web UI. It’s not supported to change a custom product’s default enablement as a user.

1 Like