I’m on my way back to home from Brussels and I thought I’d share some of my notes and observations from the conversations I had with other visitors. I hope my colleagues from Foreman team (@nofaralfasi @ekohl @evgeni @MariaAga @MariSvirik) and our friends from @atix will also add their perspectives.
This may be long posting, brace yourself.
So we started early in the morning at FOSDEM in the K building. This year we had probably the best position, the line of tables people saw at first after entering the ground floor. We were in between Postgresql and Bareos (one of their dev was ex-user of Foreman), in the same line with Ansible. We got our first visitors even before the schedule officially started.
We met existing users, ex-users, but also people who haven’t heard about Foreman before. @MariSvirik will share the statistics in her post, I got the feeling there were a lot of people who didn’t know the project, but using it would make their life much easier. I hope to see them joining our community soon, if you see newcomers on the forum asking questions regarding installation, please help 
The ex-user I spoke with used Foreman years ago (Foreman 1.14) and they loved it for automating their developers setups. They automated web servers installation through Puppet, shortening the deployment from almost a day to 30 minutes. When developers needed to quickly get a new setup for “beta” testing a new app and getting the field response, they didn’t have to wait a month for sysadmin to hand them over to a properly configured and secured environment. Another ex-user (5 years ago, used it mainly for baremetal through discovery) is searching for the engine for the new service they build, that would provision to OpenStack and would be able to trigger Ansible runs. He liked our configuration management capabilities we have around Ansible. Also the discovery rules were received very positively.
When I spoke with existing users, I tried to get feedback on the new UI. Everyone I spoke with liked the new host detail page (5 users at FOSDEM). Users of Katello for debian content complained about the missing Content tab on this page, we hope @atix could look into this soon. Everyone also agreed we can drop the old page in the near future, they didn’t find anything missing on the new page. List of puppet classes and compliance information was the most frequently asked additions, but they were not present on the old page either, so that does not block us from the removal.
Regarding the selectable columns, again, people generally liked it. They are looking forward to the moment we’ll completely merge All hosts and Content hosts page, wink wink @jeremylenz. The last bit that’s missing at this point, is merging and redesign of the bulk actions, again, something that @MariSvirik works on right now and she will comment on in her post. One suggestion was to add the possibility for the admin, to define the default set of columns for all users, who didn’t set their preference. RFE opened. Second commonly asked enhancement was to add all host sub-statuses as separate columns. I’m not sure about the feasibility, perhaps @ofedoren would know. Anyway, I’ve created RFE to track this. One user also asked for a customizable order of the columns, but it seemed only nice to have. Another comment was regarding the installable updates column, it does not have the fixed witdh, which does not look nice in case some host has installable packages and some does not - see this, the Content Hosts page displays that correctly.
I asked people who use Puppet about their experience. Most people very rarely look at facts, they are mainly interested in reports. When I asked about the possibility to drop View Chart button due to various problems with it, no one was concerned since they don’t really see a good use for it. Tracking here.
The team from the Swedish Linköping University pointed out few UX improvements. One was that our avatar support is nice, but we only display the pictures in the users list. It would be great to display this in the top right menu and for the host owner. Avatars for user groups would also be great, however harder to achieve. Similarly it would be nice to display them on the audits page. Note that our avatar support is currently limited just to users, who use LDAP authentication. If the LDAP provides images, Foreman stores them on the filesystem and can serve them. Well… not anymore, this is currently broken, Apache config does not ProxyPass the /images directory. Given how limited the current support is, I’ll open a separate poll on keeping or dropping this, based on that I’d open redmine issues to track addition to more places or the removal. Another topic was the missing DB constraint on host FQDN. I believe that’s due to our taxonomy mode, while it’s perfectly fine to have two hosts with FQDN set to machine.example.com in two different orgs, they end up in the same table and therefore we can’t have the constraint. As it turned out, our validation however does not reflect that and we don’t support this full multitenancy, therefore I opened the redmine issue for this easy fix.
There was a user still running some old 2.x version of Foreman that was concerned with upgrading since we dropped Smart Variables. We explained they can easily migrate to Parameters. The only missing functionality is validations, but they didn’t use that at all. This is rather a feedback for engineers, we need to better communicate our changes and document alternative solutions.
I also met with someone from Luxembourg University running a ~6 months old version with roughly 500 hosts. They experienced some performance issues, some processes consumed RAM a lot. I pointed them to the logrotate issue we had at some point, but it may have also been related to the Puppet’s JVM, for which had added some tuning recently. I asked for a process snapshot from when the problem occurs, so if they report it here, I’d like to ask all Puppet and performance gurus to help.
An interesting user from the credit card issuing company mentioned, they have a setup of 15-20k hosts (mostly VMware) and they use Puppet with Foreman to manage this infrastructure. They have built terraform modules for configuring Foreman (I think they are open source, but sadly don’t have a link, please share if you know where it lives). The main complaint was about the performance, our scoped search is great, but autocompletion takes a very long time. Namely, searching by facts is painful, since they have a lot of facts. As many others, they liked the new UI, but they don’t use it much (terraform)
I performed roughly 5 demos to people who didn’t know Foreman at all. They liked mostly the discovery workflow, our compliance management capabilities (foreman_openscap) and the use of Ansible for configuration management.
Very common questions from both existing users and newcomers were around the existence of a docker image, deployability to the kubernetes and similar. People who’d like to experiment with it would appreciate this greatly.
More people who deal with the content liked the Alternative Content Source functionality. Also the Content View diff was received very well. They didn’t have a hard time to quickly find out how to find only the changed packages in the diff view.
I had rather a longer discussion with the user known as @lumarel. He has been using the project for a year and a half now. He said it’s used as a manual step in the Rocky building pipeline, to verify the repo xml files are generated correctly. That means Katello/Pulp does a great job in validating the repositories it syncs. Reactions to the new host detail page, selectable columns on host index page and the new REX wizard were all positive. As we talked, we came across the possibility of a self-subscribed Foreman. We stopped recommending doing so quite some time ago, but apparently we’re not clear about that in the docs. We should explicitly mention this, probably in the installation guide (@docs team). According to this user, upgrades are also quite smooth. @lumarel is even testing our RC builds and reporting issues he finds, perhaps our release process could get such contributors more involved and make the user testing a formal step. Other UI feedback - the menu is hard to navigate and sadly the menu searching wouldn’t solve this, since he usually doesn’t know the label right away. Also he disliked the extra padding we have in new tables, the more compact the tables are, the better. The example he demonstrated this at was, the packages list on the old content host page and on the new one. Since this is always subjective, it may be interesting to have a settings “compact tables” with yes/no options. That would have to be respected by all tables (… if only we had one way to generate the tables). The last UI feedback I got from him is something I also find quite bad user experience, which is the fact that it takes 3 seconds for the menu to disappear after hovering the cursor off. I will open a separate poll for this with a hope, this could be changed if preferred by the majority. There was one more suggestion, adding one new notification type. Today, the user can subscribe to get notifications about successful repo syncing, but there’s no way to get a notification in case of a failure. Which I agree is even more important. I opened RFE for Katello. @lumarel regularly watches our demos from recordings and would even be willing to demo his use case.
A huge surprise for me was the user telling me about using Foreman for LXD containers deployment. He created a daemon that mimics VMware API, so he can create a VMware compute resource in Foreman. That daemon then performs all actions necessary to create the LXD. It is open source, sadly I don’t have the link nor I have the contact information. Dear user, if you’re reading this, please link your repo so others can take a look.
Overall I met many users who are working in our domain but didn’t know Foreman until now, despite we’re around for ~14 years. That means we need to work more on spreading the word.
Configuration management camp was, as always, very different from FOSDEM. However we still encountered some users not knowing the Foreman there. We had the Foreman room the first day after lunch. The audience was roughly around 20 people the whole day. Ewoud’s community state talk became an interactive session, where he surveyed our users. We realized we don’t really well document our support policy, as some user was still running with 2.x because he thought 3.x would be some major upgrade. At the same time we should be clearer about our “last two releases’’ support policy. @ekohl could you please make sure it’s somewhere in the new docs? The new docs are apparently already used by a lot of users. Similarly, we found out we need instructions for creating the plugin. While we have some developer docs for this, we don’t provide detailed instruction on how they should handle the packaging or how they should add the installer support. Ewoud showed his work in progress on the list of the plugins, which may help in this area. @ekohl can you please link this work if it’s already available somewhere?
We also discussed the project definition or mission statement if you will. The group has generally identified with “Customizable infrastructure automation console”. It was suggested that infrastructure may be a bit limiting term, but we didn’t come up with anything better. Not everyone agreed with saying the Foreman is typically the Source of truth as sometimes it defines the world, sometimes it just reflects it.
In Jan Bundesmann’s talk about deploying Openshift using Foreman, we briefly discussed the existing proposal of performing some steps like calling home after the actual reboot, which seemed would be a welcomed addition by many. Any news regarding this @lstejska?
At the stand I met with @langesmalle who helped with the support for provisioning Ubuntu 22.04.3+. He showed us how the community can be nice, when he brought a bottle of special Belgian beer to @iballou (who was sadly not at the conference), because Ian helped him with some issues during the year. I think the bottle ended up with @ekohl, because he also helped the user with some issues at the conference.
I also spoke with Maxmilian from @atix about the new documentation. He made a good point that we should have all engineers listed in the foreman github org, so they can be pinged easily. I’ll work on this in the following days.
Someone also pointed out, we don’t have a host status widget on the dashboard. We only have that for configuration status. I’d say that would probably be a trivial addition yet probably the most important feature of the dashboard. I opened the RM issue, I hope someone will pick up this extremely easy fix.
Another suggestion was for foreman_webhooks and/or foreman_remote_execution. Users would love to have a webhook event for when the job starts executing. What is meant by that is not the actual creation of the job invocation object, but rather the moment the jobs starts to be executed (I guess it switched to running state?). This would be very helpful when jobs are scheduled to future or repeat regularly. Users could then call the notification system that would inform them via email that the patching has started. Redmine opened at here, @aruzicka or @ofedoren could you please look into this? Feedback from people who migrated or are in the process of migrating from foreman_hooks to foreman_webhooks was only positive. We also need to officially deprecate foreman_hooks. @ekohl, what would be the best progress here?
I met a lot of users interested or using our compliance management functionality. We should add at least a widget to the new host detail page representing the compliance status for SCAP policies. The plugin is currently not maintained, if you have the capacity or will to learn, please let me know, I’d help you get started. @MariSvirik works on the design already.
Also few discovery users mentioned that the customization flow is quite broken. They prefer the short form, where they only specify the host group. If we added the possibility to specify the name and potentially the comment, we may not need that option at all as the rest can be set from the host group. Looking at @lstejska since he was playing with this recently.
During preparing and performing the demo (on nightly) I found the following bugs:
- Hosts cloning broken (likely with katello only)
- The new rhsm command we used in anaconda does not work (at least for centos stream 9) and the machine (without any error) ends up consuming content from upstream mirror. I had to force the registration snippet in the %post even for el9 to make it use the synced content.
- Settings autoceompletion (search) fails with 500
- Some pages have breadcrumbs that don’t have the host name clickable. That makes it hard to navigate back to the previous page (e.g. host detail). More details in the redmine issue
- We no longer need the link to reports from the host detail page and it should be removed
That’s all I have for now as my train slowly arrives to Brno.
Thanks all who are working on this great project and contributed to our great community one way or the other! See you next year!
.