Report from CentOS Connect, Fosdem and Cfgmgmt Camp 2024

Community
1

Hi, while I’m traveling with various public transports across Europe, I thought it would be a good opportunity to use the time and write down all the things, experiences, talks, and information that I gathered during my three conferences journey in Brussels and Ghent, still having them in my mind.

It was an amazing opportunity to meet our colleagues that I haven’t had a chance to meet previously in person, talk to the users and customers who are using Foreman, or thinking about utilizing it for their cases, or just simply talk with anyone who’s involved in the Linux and open source community.

I visited three conferences: CentOS Connect, Fosdem, and Cfgmgmt camp. Prepare yourself coffee & make yourself comfortable, it’s not exactly just a paragraph summary.

CentOS Connect

This year CentOS Connect wasn’t just a conference, but also a celebration as well, CentOS is 20 years old, so happy birthday!

One big heads-up for us is that CentOS 8 hitting the EOL soon, and we need to be ready for this in Foreman. The good news is that we already are working on it, so it’s not something that will bite us later and catch us unprepared.

From the talks, I found the interesting most:

Fosdem

Oh man, Fosdem. When everyone said that “Fosdem is big”, I honestly underestimated the meaning of “big” in this case. From what I remember from the official statistics, there were more than 900 speakers and over 40,000 connected unique devices on the university’s network.

As not a big fan of overcrowded places, this was quite a challenge for me, but I managed it and had a chance to see some interesting talks. Sadly I wasn’t able to meet any of our users at the booth, simply because the crowd and noise were so big that I couldn’t make any meaningful conversation.

For the talks I visited, I can recommend watching the following presentations:

Cfgmgmt camp

Oh man, I enjoyed my time there. Of all of these conferences, I enjoyed the Cfgmgmt camp the most.

Interesting talks, awesome people, less crowded than Fosdem, yet plenty of people from the Linux & Foreman community, eager to talk, share ideas, or just chat about technical stuff.

DEV talks

ATIX
On Monday we met with guys from ATIX and had a chance to talk about provisioning and the current state of Foreman. We discussed possible implementations of the SecureBoot provisioning, but about that later.

Since the teams are working on similar features, I think that both sides would benefit from having regular sync meetings, let’s say bi-weekly, to sync about our team efforts, discuss open PRs, reviews, and so on.

For example, we’ve been told that ATIX is already working on VMware 8 support, which could be an ideal topic for our regular meetups and we could cooperate on it more effectively.

P.I.G. (or F.P.G.)
Foreman devs came up with the idea of having a Provisioning Interest Group, or Foreman Provisioning Group, depending on which name you prefer, focused on meeting with people who are directly involved in the development of features related to the provisioning and computing resources.
PS: Thanks @evgeni for the naming idea :smiley:

Foreman Birthday party
And of course, I cannot forget to mention that we’ve been invited to the Foreman BDay party in Munich, organized by the ATIX team.

I don’t know how about you folks, but I think it’s another great opportunity to meet again.

User stories & talks

VMware exodus
Quite a few people mentioned leaving VMware due to the changes in their licensing after they had been bought by Broadcom company. Simply it’s more expensive than it was before and users are looking for cheaper alternatives.

Terraform
Terraform is not going to be open source anymore after a specific version. There is an open source alternative called OpenTofu which is gaining popularity pretty fast. Several other visitors mentioned Terraform as their go-to tool for daily work. And it seemed to me all of them were pretty happy with it. One user asked if we have any plans to implement Terraform in Foreman, for which I tried to motivate them to write their plugin.

Foreman statistics
I talked to three Foreman users about their willingness to share anonymized data.

  • The first user said, “Absolutely not”. Not that he as the user wasn’t willing to share them, but because of his employer - the Government. “It doesn’t matter if it’s anonymized”, he said, “it will never pass through the management.”
  • The second user said that they might be interested in it, but my feeling was that they would do it only if it would gain something for them.
  • The third user was totally for the idea, and willing to share it without any problems. They also suggest having multiple levels of privacy, so they can choose from what they want to share, and what not.
    For example: 1. level would be just number statistics, 2. level would be HW details, 3. level would be something more private, and so on …

UI
The number of users that don’t use UI at all was (for me) surprisingly high. They just use API or Hammer, and that’s it.

Outdated docs in GitHub repos

One customer was asking about how to install Hammer as a stand-alone package. I googled some stuff and found something in GitHub repo. The only problem was, that it was outdated and didn’t work at all. Which is something the customer pointed out.

Maybe we should do a proper cleanup of publicly available docs and make sure they are up-to-date or at least with a warning about deprecated status.

What’s new since the last year?
That was a question from one of the users, for which I didn’t know what to answer. It took me a while to come up with something. Lessons learned, next time, have a list of all new changes.

Other provisioning stories
Users without access to DHCP, users deciding if Puppet is better than Ansible (and opposite), provisioning and management of network devices (routers, switches), high-CPU computing, and last, but not least, ARM and IOT.

For each of these topics, we had at least one or more users that are working on that and were interested in how Foreman fits into this category and how it can help them solve their problems.

Bootdisk and discovery
A lot of people were not aware of foreman_bootdisk or foreman_discovery features. Like at all. We should do something with it.

RFC for deprecating template
A small feature we discussed with guys from Sweden, see RFC: Deprecating templates

Give users a chance to vote
One interesting idea was to have a system where we could allow users to vote on issues and requested features, something like UserVoice, but open source of course. It could be another valuable input, with a simple and clear message of what the community wants.

Secure Boot

On Wednesday we met to talk about the Secure Boot, and how to tackle & understand the problem. Or feature depends on how you look at it.

There were many great ideas, various approaches, and ideas for solutions that had been posted at the same time by someone (looking at you @lzap) as an RFC.

Preparations

  • Setup UEFI provisioning on local DEV, check the process for RHEL & Debian
  • Setup SecureBoot for RHEL and Debian on the Fedora machine
  • UEFI HTTP provisioning with Foreman+smart-proxy
  • Implement the feature (see below)

The feature

  • Users would have to get their shims for the OS they want to provision manually. The Installer will take care only of shims that are for the smart proxy’s host.
  • When the user creates the host with SecureBoot, we’ll check that the shims are present. If not, raise an error.
  • Smart Proxy will create config files for the grub, and make sure they are pointing to shims that are required for the provisioned OS.
  • Provision the host
  • …
  • Profit

For VMware, guys from Sweden University have a plugin that enables secure boot for VMware, so we’ll take inspiration there for sure.

And just one small note, this is not by any means a complete description and final feature solution, it might change in the upcoming weeks, talking is still in progress, and we need to sync it with current open PRs from Jan (ATIX), and still do some investigations.

Don’t worry, we will share more details when we will have them, I promise.

And that’s all folks for today, I probably (more likely definitely) forgot to mention something, apologizing in advance, but my writing skills are hitting their limits, plus there were simply so many talks and ideas that writing them down would be for a small book.

If you made it here, thank you for your time, and feel free to share your thoughts here or contact me on IRC and Slack.

Cheers

12 Likes
2

To add to the list of interesting or surprising bits and pieces, I’ve heard from some folks running rather old (think 1.24-ish). Made me wonder how many old releases are still running somewhere out there

Could you quantify “surprisingly high”?

Had someone ask the same thing, not sure if it was the same person.

On a similar note, there have been people who used Foreman in the past. Coming up with a list of things that changed since an arbitrary point of time was tricky, especially the further into past we went. The furthest I had to go back when talking with someone was around 8 years.

IIRC I’ve had two people ask about provisioning raspberry pis with foreman.

2 Likes
3

Old versions are a problem I think, had at least some one asking me questions on a 2.x and someone of the workshop attendees picking an old 3.x instead of a supported one.

Talking about Terraform: Some were surprised when I was mentioning a customer using the Foreman Terraform Provider. Perhaps also something to investigate and promote more?

1 Like
4

Could you quantify “surprisingly high”?

I met 4 users who said that. I know, I know, it’s not that high, but still, it was way more than I was expecting.

5

This is for the first time I hear about this plugin TBH

1 Like
6

Oink oink, count me in.

4 Likes
7

Couple of things that are still on my mind:

7 Likes
8

Thanks everyone for sharing! I will also try to share some of my thoughts from the conferences:

Staring at CentOS, @Shimon_Shtein gave a talk about provisioning hosts with Foreman, which I believe was new to many attendees. This highlights the importance of such talks.

At FOSDEM, we shared a booth with the Ansible team. It was definitely more crowded than last year, but we were still able to talk to a lot of people. I think most of the people we spoke to didn’t know about Foreman, but there were also many existing users.

At Config Management Camp, I moderated the Foreman room, so I was always there. @ekohl had the first talk, giving us a brief community update on everything that has happened in the last year. @iballou also gave two great talks about Katello, one for beginners and another for more advanced users. I also had a talk later that day about the foreman_ansible plugin, and I was surprised at how many people wanted to hear about it, even though it’s not new (the same goes for @lstejska’s talk on the next day). I was happy to see that the room remained full during the last talk by @x9c4 from the Pulp team.

On the second day of Config Management Camp, we also had a great talk by Stefan Joosten from AT about managing repositories in Pulp using Ansible. @evgeni’s talk about foreman-ansible-modules - 8 years later was a great reminder of how many users use these modules and for new contributors.

@bastian-src from ATIX introduced a new plugin they’ve been working on called the Foreman Resource Quota Plugin. It sounds like something many of our users would use; however, it’s in the early stages, so we need to review and test it properly.

Leos Stejskal gave a short talk about “Mastering Foreman,” and considering how full the room was and how much more he had to share, I suggest extending it to a 50-minute talk next time :slight_smile:

On the last day, we spent some time talking to our users in the Foreman room, and in the second part of the day, we discussed the Secure Boot feature and came up with a plan on how to implement it.

For me personally, it was a great experience. I learned a lot during this week, both from my teammates and from our users. Thank you @lstejska, @Shimon_Shtein, @ananace, @magnus and @ekohl for taking the time to talk to me about provisioning.

A big thank you to all the participants! It was a pleasure seeing everyone and hearing your feedback. These conferences are a great reminder of how great our community is, they’re a great place to share new use-cases, to hear about new initiatives, and simply to catch up with the community :slight_smile:

And finally, here are some statistics from the people we engaged with:

  • 44 currently use Foreman.
  • 10 have used Foreman in the past.
  • 23 have heard about Foreman.
  • 57 have never heard about Foreman.
10 Likes
9

Other things I forgot to mention:

On the first day, my local environment didn’t work. It turns out that this was due to a new version release of the sqlite3 gem. To fix this, we added gem 'sqlite3', '< 1.7' to my bundler.d/*.rb file in the smart-proxy directory. Thanks to @Shimon_Shtein and @lstejska for the help!

I asked some of our users what they were missing in Foreman’s provisioning process, and these are the answers I got:

  • More deep-dives, example use-cases, documentation, etc.
  • In the host creation form, remove irrelevant configuration options (for example, in the PXE loader selection).
  • Add a console to display the output of the provisioning on the client host (nice to have, not a must).

In another conversation I had with @magnus, I asked if it would be helpful to enable the update of VMware NIC type and network options on the host edit form (currently it’s disabled). I received a positive response, indicating that users actually need it. There is also an open BZ on downstream: link for this issue.

10

How do I sign up? FTR I vote for P.I.G.

Did you get also where they migrate to? Is that still private virtualization or public cloud? Are they abandoning the virtualization completely and switching to containers?

Did they or you come up with some idea, what that something may be?

We do have some provisioning templates for this and ansible integration, that could use the popular ansible roles Use Ansible network roles — Ansible Community Documentation

I wonder, do we have anyone here who played with this or even use this from Foreman? If not, should we clean up the existing provisioning templates?

What about this? Highlighting Popular Use Cases in Foreman Community Demos

We have voting plugin in redmine, but almost no one is using it. Probably because people report issues on discourse.

I guess our liu.se friends, such as @ananace and @magnus. Does that mean that they can create VMware VM with UEFI? Does it set the bootorder? Or something else? Is there a public repo?

Yeah, we need to start doing the use case deepdives

Any more examples that you could remember? I’m starting to have a feeling we need the existing complex form as well as some very simplified version that users can tailor to their needs.

I think we have that, it’s just hard to configure. The only good thread I found about debugging this was Foreman ui noVNC console broke - #11 by lzap

Thanks everyone for detailed trip reports, that’s really useful reading!

1 Like
11

What it does for secureboot is attach a vTPM device to all UEFI VMs on creation (necessary for secureboot, also acts as a great entropy source on Linux), and it also sets the necessary secureboot boot flag - though at the moment only for Windows due to internal setup reasons.

3 Likes
12

Well, we actually decided to drop it: Disable the "Redmine Vote" plugin :wink:

13

We have already the first customers asking about migration support to Proxmox or Openstack, but I have the feeling it is more a panic reaction than them having thought through this. Proxmox has a policy of only communicating by mail so even urgent cases will take very long, the reason why we gave up on partnering. Openstack is still a complex thing and a private cloud is different from virtualization solutions so you need to rethink many small things. My guess is many will land on HyperV not having learned their lesson of vendor lock!

From my experience at customers it is very hard to convince the network team (which is typically someone else then the team responsible for Foreman/Configuration management) to use something else then the vendor provided management tools. So I fear even if it is great to have the option, the user base will be small.

We had it, as it was not used in greater numbers it was deactivated. Disable the "Redmine Vote" plugin

1 Like
14

My old fog/foreman Hyper-V gems are still available for people to build on if that does indeed start happening, though I can’t offer any support - nor guarantee any further development - due to a complete lack of time and resources. I’m more than happy to help push them to upstream namespaces if it comes to that though.
(We used Hyper-V for a bit due to its - at the time - superior support for directly attaching FC drives to VMs, but it turned out to be too fragile and costly to maintain for it to be worth the continued effort)

15

We decided, but the plugin is still there :wink: Just to be clear, I’m not saying we should keep it, just trying to explain that while we have/had the option, people don’t use it much for some reason.

16

Aaand this is something we should handle as well. Do we want to have discourse as an issue tracker? We should force users to use proper tracker (redmine or Jira), and keep this site for community talks.

3 Likes
17

And here is the problem I think. In the community you can ask anything, the trackers are limited scope. Community will provide in most cases an answer while issues not so often. And trackers are scattered with some using redmine, some github and there is also Red Hat’s bugzilla / Jira which brings some issues from a different pipeline, sometimes closed to the public. So before trying to convince the users I thing we should try to provide one convenient workflow here (for users, developers, release team, …).

18
There’s a voting feature in Discourse
  • I knew all along.
  • Thats news to me.
0 voters
19

Any more examples that you could remember? I’m starting to have a feeling we need the existing complex form as well as some very simplified version that users can tailor to their needs.

Nothing comes to mind at the moment. If I recall any later, I’ll be sure to share it.

I think we have that, it’s just hard to configure. The only good thread I found about debugging this was Foreman ui noVNC console broke - #11 by lzap

Thanks! I’ll make sure to suggest that option next time someone asks.
Also, I remembered mentioning this PR - GUI to allow cloning of Ansible roles from VCS to some of our users, and they mentioned it would be helpful for them.