Lookback to the FOSDEM and cfgmgmt camp 2023

I’m on my way back to home from Brussels and I thought I’d share some of my notes and observations from the conversations I had with other visitors. I hope my colleagues from Foreman team (@nofaralfasi @ekohl @evgeni @MariaAga @MariSvirik) and our friends from @atix will also add their perspectives.

This may be long posting, brace yourself.

So we started early in the morning at FOSDEM in the K building. This year we had probably the best position, the line of tables people saw at first after entering the ground floor. We were in between Postgresql and Bareos (one of their dev was ex-user of Foreman), in the same line with Ansible. We got our first visitors even before the schedule officially started.

We met existing users, ex-users, but also people who haven’t heard about Foreman before. @MariSvirik will share the statistics in her post, I got the feeling there were a lot of people who didn’t know the project, but using it would make their life much easier. I hope to see them joining our community soon, if you see newcomers on the forum asking questions regarding installation, please help :slightly_smiling_face:

The ex-user I spoke with used Foreman years ago (Foreman 1.14) and they loved it for automating their developers setups. They automated web servers installation through Puppet, shortening the deployment from almost a day to 30 minutes. When developers needed to quickly get a new setup for “beta” testing a new app and getting the field response, they didn’t have to wait a month for sysadmin to hand them over to a properly configured and secured environment. Another ex-user (5 years ago, used it mainly for baremetal through discovery) is searching for the engine for the new service they build, that would provision to OpenStack and would be able to trigger Ansible runs. He liked our configuration management capabilities we have around Ansible. Also the discovery rules were received very positively.

When I spoke with existing users, I tried to get feedback on the new UI. Everyone I spoke with liked the new host detail page (5 users at FOSDEM). Users of Katello for debian content complained about the missing Content tab on this page, we hope @atix could look into this soon. Everyone also agreed we can drop the old page in the near future, they didn’t find anything missing on the new page. List of puppet classes and compliance information was the most frequently asked additions, but they were not present on the old page either, so that does not block us from the removal.

Regarding the selectable columns, again, people generally liked it. They are looking forward to the moment we’ll completely merge All hosts and Content hosts page, wink wink @jeremylenz. The last bit that’s missing at this point, is merging and redesign of the bulk actions, again, something that @MariSvirik works on right now and she will comment on in her post. One suggestion was to add the possibility for the admin, to define the default set of columns for all users, who didn’t set their preference. RFE opened. Second commonly asked enhancement was to add all host sub-statuses as separate columns. I’m not sure about the feasibility, perhaps @ofedoren would know. Anyway, I’ve created RFE to track this. One user also asked for a customizable order of the columns, but it seemed only nice to have. Another comment was regarding the installable updates column, it does not have the fixed witdh, which does not look nice in case some host has installable packages and some does not - see this, the Content Hosts page displays that correctly.

I asked people who use Puppet about their experience. Most people very rarely look at facts, they are mainly interested in reports. When I asked about the possibility to drop View Chart button due to various problems with it, no one was concerned since they don’t really see a good use for it. Tracking here.

The team from the Swedish Linköping University pointed out few UX improvements. One was that our avatar support is nice, but we only display the pictures in the users list. It would be great to display this in the top right menu and for the host owner. Avatars for user groups would also be great, however harder to achieve. Similarly it would be nice to display them on the audits page. Note that our avatar support is currently limited just to users, who use LDAP authentication. If the LDAP provides images, Foreman stores them on the filesystem and can serve them. Well… not anymore, this is currently broken, Apache config does not ProxyPass the /images directory. Given how limited the current support is, I’ll open a separate poll on keeping or dropping this, based on that I’d open redmine issues to track addition to more places or the removal. Another topic was the missing DB constraint on host FQDN. I believe that’s due to our taxonomy mode, while it’s perfectly fine to have two hosts with FQDN set to machine.example.com in two different orgs, they end up in the same table and therefore we can’t have the constraint. As it turned out, our validation however does not reflect that and we don’t support this full multitenancy, therefore I opened the redmine issue for this easy fix.

There was a user still running some old 2.x version of Foreman that was concerned with upgrading since we dropped Smart Variables. We explained they can easily migrate to Parameters. The only missing functionality is validations, but they didn’t use that at all. This is rather a feedback for engineers, we need to better communicate our changes and document alternative solutions.

I also met with someone from Luxembourg University running a ~6 months old version with roughly 500 hosts. They experienced some performance issues, some processes consumed RAM a lot. I pointed them to the logrotate issue we had at some point, but it may have also been related to the Puppet’s JVM, for which had added some tuning recently. I asked for a process snapshot from when the problem occurs, so if they report it here, I’d like to ask all Puppet and performance gurus to help.

An interesting user from the credit card issuing company mentioned, they have a setup of 15-20k hosts (mostly VMware) and they use Puppet with Foreman to manage this infrastructure. They have built terraform modules for configuring Foreman (I think they are open source, but sadly don’t have a link, please share if you know where it lives). The main complaint was about the performance, our scoped search is great, but autocompletion takes a very long time. Namely, searching by facts is painful, since they have a lot of facts. As many others, they liked the new UI, but they don’t use it much (terraform)

I performed roughly 5 demos to people who didn’t know Foreman at all. They liked mostly the discovery workflow, our compliance management capabilities (foreman_openscap) and the use of Ansible for configuration management.

Very common questions from both existing users and newcomers were around the existence of a docker image, deployability to the kubernetes and similar. People who’d like to experiment with it would appreciate this greatly.

More people who deal with the content liked the Alternative Content Source functionality. Also the Content View diff was received very well. They didn’t have a hard time to quickly find out how to find only the changed packages in the diff view.

I had rather a longer discussion with the user known as @lumarel. He has been using the project for a year and a half now. He said it’s used as a manual step in the Rocky building pipeline, to verify the repo xml files are generated correctly. That means Katello/Pulp does a great job in validating the repositories it syncs. Reactions to the new host detail page, selectable columns on host index page and the new REX wizard were all positive. As we talked, we came across the possibility of a self-subscribed Foreman. We stopped recommending doing so quite some time ago, but apparently we’re not clear about that in the docs. We should explicitly mention this, probably in the installation guide (@docs team). According to this user, upgrades are also quite smooth. @lumarel is even testing our RC builds and reporting issues he finds, perhaps our release process could get such contributors more involved and make the user testing a formal step. Other UI feedback - the menu is hard to navigate and sadly the menu searching wouldn’t solve this, since he usually doesn’t know the label right away. Also he disliked the extra padding we have in new tables, the more compact the tables are, the better. The example he demonstrated this at was, the packages list on the old content host page and on the new one. Since this is always subjective, it may be interesting to have a settings “compact tables” with yes/no options. That would have to be respected by all tables (… if only we had one way to generate the tables). The last UI feedback I got from him is something I also find quite bad user experience, which is the fact that it takes 3 seconds for the menu to disappear after hovering the cursor off. I will open a separate poll for this with a hope, this could be changed if preferred by the majority. There was one more suggestion, adding one new notification type. Today, the user can subscribe to get notifications about successful repo syncing, but there’s no way to get a notification in case of a failure. Which I agree is even more important. I opened RFE for Katello. @lumarel regularly watches our demos from recordings and would even be willing to demo his use case.

A huge surprise for me was the user telling me about using Foreman for LXD containers deployment. He created a daemon that mimics VMware API, so he can create a VMware compute resource in Foreman. That daemon then performs all actions necessary to create the LXD. It is open source, sadly I don’t have the link nor I have the contact information. Dear user, if you’re reading this, please link your repo so others can take a look.

Overall I met many users who are working in our domain but didn’t know Foreman until now, despite we’re around for ~14 years. That means we need to work more on spreading the word.

Configuration management camp was, as always, very different from FOSDEM. However we still encountered some users not knowing the Foreman there. We had the Foreman room the first day after lunch. The audience was roughly around 20 people the whole day. Ewoud’s community state talk became an interactive session, where he surveyed our users. We realized we don’t really well document our support policy, as some user was still running with 2.x because he thought 3.x would be some major upgrade. At the same time we should be clearer about our “last two releases’’ support policy. @ekohl could you please make sure it’s somewhere in the new docs? The new docs are apparently already used by a lot of users. Similarly, we found out we need instructions for creating the plugin. While we have some developer docs for this, we don’t provide detailed instruction on how they should handle the packaging or how they should add the installer support. Ewoud showed his work in progress on the list of the plugins, which may help in this area. @ekohl can you please link this work if it’s already available somewhere?

We also discussed the project definition or mission statement if you will. The group has generally identified with “Customizable infrastructure automation console”. It was suggested that infrastructure may be a bit limiting term, but we didn’t come up with anything better. Not everyone agreed with saying the Foreman is typically the Source of truth as sometimes it defines the world, sometimes it just reflects it.

In Jan Bundesmann’s talk about deploying Openshift using Foreman, we briefly discussed the existing proposal of performing some steps like calling home after the actual reboot, which seemed would be a welcomed addition by many. Any news regarding this @lstejskal?

At the stand I met with @langesmalle who helped with the support for provisioning Ubuntu 22.04.3+. He showed us how the community can be nice, when he brought a bottle of special Belgian beer to @iballou (who was sadly not at the conference), because Ian helped him with some issues during the year. I think the bottle ended up with @ekohl, because he also helped the user with some issues at the conference.

I also spoke with Maxmilian from @atix about the new documentation. He made a good point that we should have all engineers listed in the foreman github org, so they can be pinged easily. I’ll work on this in the following days.

Someone also pointed out, we don’t have a host status widget on the dashboard. We only have that for configuration status. I’d say that would probably be a trivial addition yet probably the most important feature of the dashboard. I opened the RM issue, I hope someone will pick up this extremely easy fix.

Another suggestion was for foreman_webhooks and/or foreman_remote_execution. Users would love to have a webhook event for when the job starts executing. What is meant by that is not the actual creation of the job invocation object, but rather the moment the jobs starts to be executed (I guess it switched to running state?). This would be very helpful when jobs are scheduled to future or repeat regularly. Users could then call the notification system that would inform them via email that the patching has started. Redmine opened at here, @aruzicka or @ofedoren could you please look into this? Feedback from people who migrated or are in the process of migrating from foreman_hooks to foreman_webhooks was only positive. We also need to officially deprecate foreman_hooks. @ekohl, what would be the best progress here?

I met a lot of users interested or using our compliance management functionality. We should add at least a widget to the new host detail page representing the compliance status for SCAP policies. The plugin is currently not maintained, if you have the capacity or will to learn, please let me know, I’d help you get started. @MariSvirik works on the design already.

Also few discovery users mentioned that the customization flow is quite broken. They prefer the short form, where they only specify the host group. If we added the possibility to specify the name and potentially the comment, we may not need that option at all as the rest can be set from the host group. Looking at @lstejskal since he was playing with this recently.

During preparing and performing the demo (on nightly) I found the following bugs:

  • Hosts cloning broken (likely with katello only)
  • The new rhsm command we used in anaconda does not work (at least for centos stream 9) and the machine (without any error) ends up consuming content from upstream mirror. I had to force the registration snippet in the %post even for el9 to make it use the synced content.
  • Settings autoceompletion (search) fails with 500
  • Some pages have breadcrumbs that don’t have the host name clickable. That makes it hard to navigate back to the previous page (e.g. host detail). More details in the redmine issue
  • We no longer need the link to reports from the host detail page and it should be removed

That’s all I have for now as my train slowly arrives to Brno.

Thanks all who are working on this great project and contributed to our great community one way or the other! See you next year!


Do you know why they stopped using Foreman or did they simply move to a different job?

The one thing currently missing is template preview, but I know this is in the making (or did it already land in nightly). I also found one small thing missing last week and created an issue for it: Bug #36022: Reboot for build option in new host view missing - Foreman

I can offer some help here if needed, as I did this several times, but would need someone for the Debian part.

Just jumping in. Do you mean the overview? Foreman landscape

From some plugins we did this in the past:

1 Like

Everyone who no longer use Foreman simply changed job or project they work on. Those who migrated from Foreman to something else were people moving between Foreman, Satellite and Orcharhino.

Correct. It is also in 3.5 but you need to enable experimental featues in settings to see the Details tab. If you convince @MariaAga we may even cherry pick that as non-experimental for next 3.5.z release, the patch was tiny.

Good catch on the reboot missing.

If you open the PR against foreman/how_to_create_a_plugin.asciidoc at develop · theforeman/foreman · GitHub even with just rpm or installer steps, that would be a great start. Someone else could then fill in the deb steps based on the existing structure. Thanks!

Yes, that’s it, thanks!

Good list, thanks. @ofedoren, when you will have some time, could you please look at this so we finish the webhooks migration? It’s not urgent I guess, but probably the last missing step.

1 Like

thanks for the write-up, in made me remember how much fun those events can be… hopefully again in the near future :slight_smile:


“self-subscribed” as in “Foreman Server is registered as a managed host to Foreman”? If so, then I can create a PR to add a small recommendation/warning not to do this.


Yes, using the Github groups (please document how to add/remove people etc.) is a great idea because we hope it makes things easier to ping “technical people” during docs review to get a quick “ACK” and general feedback, for example on cherry-picking to which versions.

@docs Is this a private team? If it does not exist, can I/someone please create it and add me?

Overall, as commented on the thread in the events category, I had a blast at Ghent & truely enjoyed meeting everyone from the Foreman community/ecosystem.

I’ve just created Documentation team - TheForeman and made you owner of it, please add everyone :wink:

1 Like

Just as a draft, so you can let me know if this is what we want and to improve this together. I adjusted the existing part about packaging and added puppet and the installer. I included also examples for every step.

1 Like


some people point their repositories to “localhost”, to use Katello repo management capabilities for this same Katello instance. That way they would get a better control of the Katello rpms. However it comes with some surprises and gotchas (can’t list them now from top of my head). Typically this was achieved by configuring rhsm.conf to point to localhost (using FQDN due to x509 authn) and running subscription-manager register. We should be explicit about the fact, this is not recommended.

If people need this, they should use external Katello for that. Typically people having multiple instances have one they only use as single source of truth, other instances syncing from this main instance. That’s where content export/import functionality comes handy.

Thank you Marek for this post!
I would also like to say that it was an amazing experience for me. I am so grateful I had the opportunity to be there.
My favorite parts include talking to users and hearing about their use of foreman. One of the best conversations we had was during Ewoud’s community state talk in CfgMgmtCamp, where we had an open discussion about what foreman is for our users.

I agree with Marek about the new host details page, everyone I talked with agreed it was a great improvement from the old UI.
From what I’ve heard, each user has their own use cases for foreman, and some properties are more important than others to different users. I think the new manage columns feature is a great solution for that.

As the community demos maintainer, I also thought about how we can improve them, and make them more interactive. I will start to add a short update at the beginning of each demo about our latest news, like version releases.

Regarding the documentation for plugins, I would also be happy if we add more information about plugin maintenance, for example about packaging, releases, translations, etc.


That was me, I just do not know where in the code this needs to be done. So @aruzicka and @ofedoren : if you know where this has to be done and just do not have the time to implement it, I am willing to give it a try.

1 Like

Would something like Fixes #36063 - Emit event when job starts by adamruzicka · Pull Request #783 · theforeman/foreman_remote_execution · GitHub do the trick? It can be made more elaborate, but as I baseline it could do


Raised Fixes #36064 - Don't try to assign empty CV/LCE to a cloned host by jeremylenz · Pull Request #10448 · Katello/katello · GitHub, let me know if that fixes it @Marek_Hulan :slight_smile:

1 Like

Adding a few thoughts from cfgmgmtcamp :slight_smile:

I was there much shorter than y’all (only Tue/Wed really), so it’s a lot shorter too.

evaluating dependencies

On Tuesday Ben Ford from Puppet gave an interesting talk about vetting dependencies, which was focused on Puppet but the ideas really apply to any “fetch a defined set of code from a repo” like RubyGems, PyPI etc. The idea is to take a project (like a Puppet module) and look how well maintained it seems by performing various checks (like are the tags in the repo signed, do the releases on Puppet Forge match git tags, are there many open issues for a long time, etc).
I’ve then spent part of my Wednesday hacking on that tool and see how it can be used for other projects – it certainly can :slight_smile:
I would think this is something that we should try to adopt (whether this explicit tool, or something else) to make sure we’re not pulling in dependencies that are going to rot too quickly.

keeping ansible module parameters uptodate

Had a discussion with a user of our Ansible modules, and one of the frustrations they shared was that we’re often “behind” on possible parameters (or their values) to our modules, like when a new download policy was added or a new parameter was introduced to extend a resource model.

So far we had a very reactive approach on that, and I’d be very happy if we could “fix” it by getting the right input the moment a PR that changes the API is opened/merged. Maybe we could teach CI to use GitHub - Apipie/apipie-diff and inform interested parties?

keeping ansible roles parameters uptodate

Very related to the above, we have roles that effectively are wrappers around a single module to ease creation of multiple resources. Those roles need to know about all the parameters that a module accepts, and we often forget to keep them uptodate once we update the module.

I think we could create some automation that ensures they are in sync.


opened at User Avatar support in the Foreman, keep it or drop it?

opened at Menu hiding delay poll

1 Like

I’ll also convert my notes into a somewhat coherent post.

First of all, it was great seeing so many familiar faces again. I missed it and I’d say it mostly felt like it was no different than the last edition. And I say that as a plus. For me it started with catching up with people, and that was a huge theme.

I spoke to SimonPe (Simon Peeters) · GitHub for a while and discussed their setup. He now works at the University of Ghent and was happy to hear they use LISA, which I must admit I almost forgot about. They don’t use some functionality, like IP management, for political rather than technical reasons. One thing he has mentioned in presentations in prior years, but could be interesting to mention again is that they bake in the role a server has in the Puppet certificate during provisioning (by specifying it in the CSR attributes).

Then I spoke to a user who was overall happy, but wanted Katello to function like a real container registry where you can push to directly. Pulp can do this but Katello can’t do it (Feature #33901: docker push functionality in katello - Katello - Foreman is open). For now his workaround was to run some local registry and then sync from that into Katello.

With another user I had a discussion about a self-registered Katello instance, which is when Katello is registered to itself. That means pulling RPM content from its own published repositories. This is generally not recommended because during upgrades the repositories won’t be available. Technically it would probably work to download all packages dnf upgrade --downloadonly to retrieve a cache, but in practice it doesn’t work. When a Puppet package statement ensures it to latest, it trashes the yum/dnf cache and retrieves it again from the server. This can actually create a large load on your server, so it’s recommended against. I recalled I noticed one exception where it is the default. I have just opened a PR to change that: Default to using the installed version by ekohl · Pull Request #48 · voxpupuli/puppet-trusted_ca · GitHub. Another thing is that we should clearly document that this is a setup that’s known to break and should be considered unsupported. I’m not sure we do today.

Another user I spoke to wanted more documentation and more concrete examples. I do think the new documentation effort (GitHub - theforeman/foreman-documentation: Documentation for the Foreman Project and its ecosystem) should eventually improve that, but right now it’s a bit of a mess because there’s a lot duplication. For example, most content at Foreman :: Plugin Manuals is awfully outdated.

Then I think at cfgmgmtcamp I had a lot more in depth conversations with people.

With @Dirk I spoke about Ubuntu auto-install provisioning. He opened Feature #36027: Provide also /pub with Foreman only for Ubuntu Autoinstall - Foreman but there is a set of PRs opened by @bastian-src from ATIX already (Fixes #35270 - Enable boot image download by bastian-src · Pull Request #837 · theforeman/smart-proxy · GitHub / Fixes #35269 - Enable boot image download of installation media by bastian-src · Pull Request #9318 · theforeman/foreman · GitHub) which should properly solve it. It’s totally on me that I haven’t finished the review, mostly due to a lack of time. I would encourage other developers to review it, but otherwise I’ll try to wrap it up soon.

Then we also spoke about how you would replace Foreman’s SSL certificate and which installer options to use. Back in November 2019 I started a blog to explain it, but I never got around to finishing it. Something I’ll also put back on my action items.

On a personal note, I made a note to check out pulp/oci_env because I’m interested to see what their containerized development setup looks like and if we can use some of it.

Then in another Pulp talk @quba42 was interested in how Foreman’s installer sets up Pulp, which is done using puppet-pulpcore which means we’re unaffected when Pulp’s Ansible-based installer pulp-installer is discontinued after Pulp 3.22.

On the last day I spent my time at the Puppet contributor summit, where I spoke to Martin Alfke about Hiera Data Manager and how it could be integrated in Foreman. I suggested we could perhaps rewrite the API from a Ruby on Rails application to a Sinatra application so it could be reused inside the Smart Proxy. It may also be great to extend the Puppet tab on the new host detail page with that information. Future versions could then add HDM’s global search and even editing (which HDM does support). We’ve scheduled a chat online.

In addition to that I gave two talks.

In The Foreman Community update :: Config Management Camp 2023 Ghent :: pretalx I spoke about the Foreman community update. While I’m not a community manager, I felt it was good to at least do this. My slides were mostly irrelevant (I can still attach them later), but I will note that while preparing the list of releases made since last time I realized we didn’t even release Foreman 2.0 back then while now we’re about to branch Foreman 3.6. On the Katello side there was a similarly large list. Because I selected a full 50 minute slot, I couldn’t fill all of that by just giving updates. Instead, I decided to make it interactive. We had a great discussion where the community could update us developers and each other with a bit of Q&A. @Marek_Hulan already summarized this.

And in How Vox Pupuli built their Continuous Integration :: Config Management Camp 2023 Ghent :: pretalx I spoke about Vox Pupuli’s CI stack and how it’s built. It had a lot of information, so I’ll probably need to write it down in a blog series at some point. For now I linked to the slides on my own server, since they’ 28 MB (I guess somewhere in the HTML to PDF conversion there’s some inefficiency) and I couldn’t upload it.

And I’m sure I forgot some things. I certainly spoke to many people, quite a few over beers where I didn’t make any notes. Speaking of beer:

I did indeed happily accept the beer after debugging some virtual console issues with websockify in Foreman. This time it was a DNS problem. He suggested it should rest for a year or 2 before it’s optimal, so we have about that time to plan something where @iballou and I can meet in person to share it.


Thank you Marek for your thorough post and here are my two cents.

Small statistic*
Just during Saturday at Fosdem we had approximately 66 conversations:

  • 38 conversations with people that never heard about Foreman
  • 4 with people that heard or knew something about the project
  • 3 with people that used to work with Foreman
  • 21 with people that still use Foreman to some extent (either for all their flows, just some of them, via CLI, GUI)

*These were conversations happening with 1-3 people listening and were of different lengths and depths. Designed and demos we shared displayed also some plugins.

To get to the UX feedback, unfortunately (to my UX research needs) not all people I talked to used our interface for all or at least some of their flows which is understandable but it made feedback hunting more difficult :stuck_out_tongue: .

New host detail page
Overall feedback from the users that I talked to was positive. We showed them the new host detail page based on their interest and needs. The most popular part (for users and not users) was a tab that integrates Foreman with Ansible and lets users assign roles, and schedule jobs to run them.

Index page columns and bulk actions
People were very excited about our new functionality - column management on the host index page. We asked about the most important columns. Besides name and OS, people mentioned also host groups, environments, last reports, and last seen columns. That’s important to know for a creating a default view.

With the bulk actions it was a bit more difficult as not many people I spoke with use them. (It was due to a lack of use cases or better ways to perform these). The most frequently used bulk actions that were mentioned were host deletion and change of host groups. The other mentioned actions were influenced by plugin usage.

Some design questions didn’t have a straightforward answer so I will open a community thread on that e.g. Do we need to display already selected hosts in the bulk action modal/wizard to allow double-checking of the selection? How do we want to display these hosts - an expandable section with a list of hosts (not scalable for thousands of hosts) or a paginated table in an optional step in the wizard, or even something different?

I would like to also promote Maria Aga’s research about the new job invocation form that she pro actively started in the cfgmgmt camp.
(Do you want to review your job invocation or run it right away?)

Thank you to all of you that found time and came to our booth or talks and spoke with us. Also, great thanks to the ATIX team for having a thorough talk with me about the bulk action redesign.

Anything to add?
In the discussions and also during my talk in the config camp, we promoted community demos and also encouraged the active involvement of the users.

If you don’t agree with any of my findings or have other UX-related issues I haven’t mentioned. Feel free to write it to this open form or ping me.


That was indeed one of the better moments of the conference. I believe it was @Bernhard_Suttner who very excitedly came over to me and said “have you seen what she’s building, it’s amazing”. Or to that extend. And I have to agree: the it makes the wizard a lot better for me.


This made me think that we should cast this upon our own repositories first to see how they stack up. Perhaps it would be able to give us some insight into how our own repositories are either rotting or not following a best practice we could improve upon (e.g. not signing tags).

A bit late to the party :slight_smile:

To somewhat replay my words, it does work if you do the dnf update, but before running foreman-installer you have to do a dnf config-manager --disable <all Katello managed repos> && dnf config-manager --enable <local repos>, if not the config part will fail. (and switch back afterwards)
It’s still a design flaw which really shouldn’t be practiced, still the benefit of hooking the server to its own hosted application (if there is no other node available) is that security updates are centrally managed and you get the package visibility. For sure the best option is hooking the one Katello server into another one.

I’m really glad I managed to walk up to you all, had a really great talk :grinning: