I recently traveled from Boston to Brussels and Ghent to represent Foreman in FOSDEM and Configuration Management Camp. This is my second time visiting the conferences, however the last time was in 2020 when I was brand-new to Foreman. I believe that participation in these conferences presents more opportunities to learn about Foreman than any other experience. Never have I met more people passionate about open source in the timespan of a week.
Part 1: FOSDEM
My trip began explosively with the excitement of FOSDEM. On the tram ride to The Université libre de Bruxelles, I already had a chance to meet excited participants from Red Hat, CERN and elsewhere and discussed anticipations for the day ahead. Arriving at FOSDEM is always breathtaking – booths from recognizable projects are abound and people are everywhere. We crammed our belongings behind the half-table that we shared with Ansible and began immediately talking to interested participants.
The mix of people approaching our booth was scattered, but the incoming flow of people was consistent. We received great feedback on the new navigation bar along with general positive feedback. A number of users expressed interest in managing Debian content or running Foreman + Katello on Debian itself. One user who was particularly fond of Katello mentioned their wish for “container push” support before I even mentioned that it is already in the early stages of development. The user mentioned that they use Foreman only for content management since all of their provisioning is done through Terraform. They said that they would be interested in being able to manage Terraform through Foreman. If Foreman had this capability, it would be able to provision on more machines with less maintenance overhead.
Many people overall were very happy to hear that Foreman integrates well with Ansible. For new users, after describing Foreman’s basic functionality, the first question was usually if we had an integration with Ansible, or if Foreman was “like Ansible”.
Miscellaneous feedback was received from varying sorts of participants:
- A complaint that Proxmox support breaks whenever Proxmox comes out with a new version;
- A complaint that Debian packages are distributed in an inconvenient format. I believe this will be solved by the upcoming “Structured APT” feature that ATIX is working on;
- A complaint that the host create/edit page offers too many impossible configuration combinations. I think the issue here is that Foreman expects the user to know what combinations of inputs should work together, especially with networking;
- A complaint that Katello requires too many actions to send updates to hosts. Perhaps they didn’t know they could avoid content views altogether;
- A request to allow filtering of provisioning boot options. The user has many boot options that are simply broken, and they wish to make the working ones the only selectable options. (I didn’t ask why they can’t clean up the broken ones);
- A request for more plugin development documentation around React modules;
- A request for a new compute resource for Apache Cloudstack;
- A compliment that we support high-availability for smart proxy content distribution;
- A complaint that the Foreman Debian packages are unusable due to the need for old versions of Ruby and NodeJS. I told them that this will be fixed with the Ruby 3 upgrade.
I ended up staying at the booth the whole time during FOSDEM since we had so many people to talk to. I considered going to the Ruby dev room to hear some talks there, but the room filled up and there was a massive line to get in.
Part 2: Configuration Management Camp
While FOSDEM was amazing, Configuration Management Camp was the main event for me. We hitched a ride with some of the organizers from Brussels to Ghent, where Config Camp was located. The first evening was a great kickoff to the event since we had both the Ghent Light Festival to enjoy and the Config Camp speakers’ dinner that is made possible by the generous supporters.
Configuration Management Camp is a conference especially made for systems administrators who want to use open source software. Most of the people that I talked to there knew something about Foreman, which is always a treat. For the folks who don’t know Foreman, their work is often related enough for us to recommend Foreman to them. Config Camp is also a much better space for productivity. FOSDEM is loud and crowded, but Config Camp offers space to quietly hack and design with others. Also, unlike FOSDEM, you can usually join any talk without worrying about fitting in the room.
While the days were understandably spent attending talks and discussing tips and plans with community members, the nights often saw these talks continue. After the conference, groups of people typically stuck together to go to dinner or a pub. This all-day all-night atmosphere really allowed for community members to make great connections, not to mention have a lot of fun.
I gave two talks at Config Camp in a room reserved for Foreman and Pulp talks. One talk was an introduction to Katello, and the other taught about advanced Katello features, like alternate content sources and container management. Both talks featured demos. The first demo was a recorded video that showed how to set up repositories and content views to support deploying tomcat applications via remote execution and Ansible (since it was recorded, I’m considering adding a voice-over and posting it on the Foreman YouTube channel).
The talks were both well-attended, and there was a good spread of new Foreman users and existing users. Questions at the Katello talk were mostly around lifecycle management, which is a complicated subject that people usually need to try before they fully understand it. I always suggest that folks install Katello on a VM and give it a try since our installer is so easy to use. There was some confusion as well around alternate content sources since it’s such a new feature. After explaining how they work in detail people understood the concepts, but the initial confusion makes me wonder if our documentation alone is enough to portray when and how users should deploy ACSs.
One important conversation that I had was with Matthias @x9c4 from Pulp. In Katello, we support syncing smart proxies that are two versions older than the current version (so 4.11 to 4.9, for example). We hope in the future to expand this to every four versions. To do so, however, we need to have a better solution for talking to older Pulp instances. Currently, we use newer Pulp Ruby bindings to talk to older Pulp smart proxies, which is not guaranteed to work. If there are issues, we monkey-patch them in Katello.
To work around this, Matthias suggested that we start using Pulp Glue (https://github.com/pulp/pulp-cli/tree/main/pulp-glue). Pulp Glue was developed alongside the Pulp CLI to allow for a CLI client to talk to any older version of Pulp. The Glue layer keeps track of “quirks” in older versions, like API bugs, so that there is one released version of the Glue that can communicate with any older Pulp version. If Katello starts to use Pulp Glue for smart proxy syncing, we’ll be able to theoretically sync content to any older Pulp. The biggest challenge with adding Glue to Katello will be figuring out how to call Python code from Ruby safely. If anyone has ideas for that, please let me know!
On Wednesday, our workshop day, I spent time with Quirin @quba42 and Markus @m-bucher from ATIX hacking on some Debian content PRs. We got one PR merged and another ACKed. We also discussed the latest in creating development environments in Forklift. At the time, Forklift was still pointing to Koji rather than our Copr staging repositories, which was creating misconfigured environments. Besides Forklift, we also discussed getting Debian errata support into Katello, which will require some thinking about how to fit their separate CVE parsing service into a Katello installation. It’s likely that we’ll just have users set up the service themselves to reduce installation complexity. In general we’d like to synchronize better around merging big Debian features into Katello. That way the time that their PRs spend sitting would be reduced which also means less rebasing on newer versions of Katello.
Earlier in the week, we were also discussing with Bernhard @Bernhard_Suttner about how to make ATIX Katello reviews smoother. We thought that it would be best for ATIX to have someone with merge access to Katello. Following our committer nomination workflow, this in turn would mean that ATIX would participate more in general Katello development, including code reviews. I was happy to have this conversation since it should mean that Katello code in general gets merged faster and our relationship with ATIX developers grows stronger. I think it’s a testament to open source that two companies can work on the same piece of software while collaborating healthily.
Speaking of ATIX, Bastian @bastian-src gave a talk showcasing a resource quota plugin that he is developing. The plugin allows admins to set quotas for things like CPU cores, RAM, and disk utilization for different users. There are still some things to be hashed out, like how to get disk usage reliably from different compute resources, but I thought the plugin looked really promising. Here is the source code: GitHub - ATIX-AG/foreman_resource_quota
I was also very impressed by the talks given by my fellow colleagues. By the end of each day, the Foreman dev room was full and we had plenty of questions.
Here are some miscellaneous interesting things that I discussed with others:
- One user reported managing 8000 machines across a few orgs and locations. His customers use a portal to hook up machines and add ansible variables via the API. This isn’t the first reported case of users essentially creating their own UIs to talk to Foreman.
- Users are interested in seeing more diff information output from Ansible runs. One example is being able to see what changes were made to a file.
- There were some complaints generally about Foreman + Katello being too complicated.
- File repositories should be more easily usable. In my talk demo, I used a file repository to serve WAR files to hosts. The URL needed to be manually discovered via browsing the Pulp content directories. While finding the URL was a one-time operation since it was tied to a specific content view and lifecycle environment, it should be easier for users to find that URL.
- The same could be said for other content types, like Ansible or Container.
- One user wanted to be able to more easily recreate destroyed entities, like hosts.
Overall, I wish all of my colleagues could go to Belgium to experience the open source conferences there. Being in an atmosphere where so many people can relate to the work you do is quite rare for a Foreman developer. The people who participate all have such interesting and unique ideas that show perspectives that are difficult to obtain elsewhere. Beyond this, it is fulfilling hearing how much people appreciate your software when you work daily to address complaints. Such an experience boosts my drive to deliver back to the Foreman community.
I’d like to give a special thanks to my colleagues Nofar, Leos, Maria, Shim, Adam, Ewoud, Evgeni, Lubos, and Matthias for making Foreman and Pulp’s presence at the conferences successful. I’d also like to thank all of the other community members who gave great talks and workshops. Lastly, thank you to the organizers and sponsors of the conferences who allow this special community of open source contributors to meet!
P.S. please look out for posts from other users who are also sharing their Belgian conference experiences.
I think we may have the same need when we start working on the support for the new VMware API, there’s no Ruby SDK for this AFAIK. Google shows there are some libraries that could help. Other option is to shell out 