DevConf.us 2019: Notes and reflections by akofink

Community
, , , ,
1

DevConf.us 2019 was August 15-17 in Boston
this year. I helped run the Foreman booth with Ian Ballou (@iballou) and John Mitsch (@John_Mitsch),
demoing Foreman’s capabilities through user workflows depending on what people
were interested in seeing. We saw some industry folks as well as many Boston
University students; The conference was held in the BU Student Union building,
afterall, and many of the speakers were BU students or alumni. There were even
presentations from high school students mentored by Red Hat interns from BU.
Overall, it’s clear there is some really great collaboration between Red Hat and
students in Boston.

Some from the industry were already using Foreman and were excited to see new
features in the recent versions (some were running Foreman as old as 1.10!),
while others jokingly asked if we were demoing the
Foreman gem and found humor in the fact
that Foreman was using the Foreman gem in part for development.

As expected, most users of Foreman were using only some small subset of features available.
Very few users wanted to see increased Puppet support - in fact, some users
wanted to see less of a Puppet mandate. It was suggested that some of the lifecycle
management part of puppet, namely environments, should be first-class objects in
Foreman, not necessarily associated with just Puppet. The ability to have a Dev
→ Test → Production promotion of Ansible rules or Salt recipes, for example,
is desired. In addition, confusion with puppet environments vs. lifecycle
environments in Katello continues to be a pain point for UX in general. As a
first thought, allowing a generic pluggable environment in Foreman would be a great tool
for plugin authors/maintainers to allow lifecycle management of whatever the
resource may be, whether content, configuration, or provisioning related.

Students were mostly interested in learning more about what a company like Red
Hat is contributing to the open source community and how they could get
involved. Many were interested in Foreman’s underlying technologies, most were
happy to hear that we use Rails and React atop Postgres, most quite familiar to
the students we talked to. Of course, I was sure to mention the amazing Foreman
community we’ve all worked so hard to help build.

On Saturday there were lightning talks which anyone could sign up for. The talks
were voted on, and the top 3 were presented. The first talk was about speech recognition
using machine learning by a BU student. The second talk was also a BU student
talking about Women in Open Source. The Pulp team presented third on Pulp 3:
Stop Using rsync to Sync EPEL. My talk about how Red Hat operates in Open
Source communities was voted 4th, so I did not get to present.

While sitting at our booth, we had some discussions amongst ourselves about ways to improve
Foreman as well:

  • The confusion around environments and the ability to have
    lifecycle management in more than just content and Puppet, as already mentioned,
    was one. Think about (i.e.) lifecycle environments for SCAP policies or
    lifecycle for salt states. If this is supported for any given resource (such as
    foreman_salt), it is reinvented each time and not associated with other
    environments from other plugins (or core for Puppet environments). For mixed
    infrastructures with multiple configuration management plugins enabled, it’d be
    much nicer for users to be able to see all the resources that are part of a
    given environment, regardless of which plugin the resources originate.
  • Another was the unification of hosts vs. content hosts - there was some
    effort in the past to unify the underlying models, but the UX in this area could
    still be improved to unify the concept of a host in the UI/CLI.
  • How to test/gate releases upstream better: As we were preparing for this
    conference, we found several bugs in Foreman 1.22 that we reported and/or
    worked around directly on the demo environment. Could we include more end to
    end tests that are automatically run? Existing method to gate a nightly
    release is bats tests, but developers hardly ever update bats to cover new
    features or bugs; is this because bats lives with forklift rather than
    Foreman/the relevant plugin? Perhaps we could add end to end tests in the same
    repository as the code to increase the liklihood that a feature/bug fixer
    updates end to end tests. Other issues with bats: if hammer is broken, bats
    fails. Is this a feature of bats that we’re also testing hammer or just a
    hinderance? I’d argue we should be testing hammer for sure in end to end
    tests, but is bats the best way? Should we have end to end tests for the UI
    using Selenium or a JS based suite?
  • Groups or collections of hosts - hostgroups are only for changing hosts during
    provisioning. There is no way to group hosts outside of provisioning unless
    each plugin provides a grouping mechanism (i.e. host collections from
    Katello). Host groups should be a Foreman primative that is not only for
    provisioning and is also pluggable. Changing certain fields in the group
    should update hosts immediately, while others only affect hosts during the
    provisioning process.

The following are my more raw notes day by day:

2019-08-15 Thursday

  • Datto
    • Foreman 1.10, mostly used as puppet master
    • Many OS types (SUSE, CentOS, Ubuntu)
    • Multi-arch containers in quay? [TODO: follow up]
    • ‘Runbook task’ for allowing a user/role to run only a set of commands, and
      no permissions to change the runbook/command itself [RFE]
    • Need better searchability on community demos [Community RFE, add timestamps to demo descriptions on YouTube]
    • 4000 systems, 3-4 proxies, 2 redundant puppet masters
    • Salt is used to run Ansible at scale
  • Katie Riker UX talk
    • Wizard of Oz UX study, having a real person respond like the system would
      (useful for CLI UX studies without having to implement any code)
    • Knowledge bias, addressing assumptions
    • Clear error messages, tell the user how to solve the problem
  • Ampere Computing
    • Puppet 6 support in Foreman 1.22 is great!
    • ARM provisioning?
      • Foreman discovery image support ARM?
      • Does syslinux work on ARM?
    • They use Beaker in production
    • Follow up with Peter Pouliot from Ampere about ARM provisioning
  • Using Chef with Foreman
  • Lots of people that were BU students that didn’t know about Foreman or many of
    the concepts involved
    • Asked how best to get internships
    • Answer includes: Open Source focus
  • Swag suggestion: Foreman breakfast bowls? :wink:

2019-08-16 Friday

  • Bug when using dnsmasq with NetworkManager on the same box as Foreman
    • Smart Proxy dns doesn’t match /etc/resolv.conf
    • When looking for dns conflicts, Foreman queries the dns host in
      /etc/resolv.conf instead of the DNS server in the proxy

2019-08-17 Saturday

  • Datto again
    • Ansible with Foreman doesn’t support environments?
    • 4000 systems, 35 environments, 300 hostgroups
    • changing the environment for a system changes the use of a system
    • environments should be a foreman primative, as it’s presented in the UI that way
    • Ansible doesn’t scale to 1000s of systems? Need to configure fan out? Is that possible?
    • Expose ansible inventory groups in Foreman (i.e. host collections for Ansible)
7 Likes
2

I’ve been suggesting this in various places. Great to see users agree :slight_smile:

This has been suggested before but I never liked this. It’s very hard to find which source repositories are actually needed matching your install. Repositories can add integration tests (built into Rails, enabled on our CI) which can test this. Is this not sufficient to catch most issues?

We’re just shipping ISC DHCP by default and the installer can configure it, but this is in no way mandatory and I don’t see why you can’t set it up.

I thought we had query_local_nameservers for this though I agree this is very ugly and we need to fix this.

1 Like
3

This is definitely a great sum-up of what occurred during DevConf.us! I’ll add some of my observations and experiences:

On the Wednesday before DevConf started, there was a CentOS Dojo in the same location. The day started off with a brief mention of CentOS 8 and its current status that can be followed here: https://wiki.centos.org/About/Building_8

Talks during the CentOS Dojo weren’t all directly related to CentOS. Many of them were projects that ran in some way on CentOS. Here was the talk rundown:

  1. Keylime, which is an open source project for server attestation using the latest TPM devices.

  2. Terraform, with a basic introduction to its features. It was demoed with AWS EC2 instances.

  3. Buildah, with in-depth information on how it works.

  4. A couple of students and their advisor from NC State gave a talk about their new SuperComputing’19 team. Definitely a great way to get internships as a student!

  5. AppStreams, Fedora Modularity, and how they work right now.

  6. Lightening talks. The Pulp 3 team gave the same talk that @akofink mentioned above.

During DevConf I talked to quite a few students who were wondering the best way to score an internship. My usual advice is pretty much what was mentioned above but also that they should seek research opportunities in their schools. Often times there are programs relating to cloud, big data, security, etc that accept undergrads as software development assistants.

DevConf also helped give me a better perspective on the relation between developers and system administrators. It’s very important to regularly keep track of how folks are using Foreman since it’s easy for us developers to fall behind on the times. Not to mention that many of us don’t have as much system administration experience as we should!

I also want to echo the bug that @akofink mentioned above with Foreman reading from /etc/resolv.conf. For hours we thought we had a networking error when it was Foreman all along. Perhaps we had an unusual setup on our demo machine, but it’s a strange bug nonetheless.

Overall it was a great experience and I’m looking forward to promoting Foreman at more community conferences!

5 Likes
4

I agree having a separate repo isn’t a serious issue for testing, yet putting end to end tests in Foreman or related plugins is dangerous, as the set of plugins loaded isn’t necessarily known; however, I think we could do something like this for tests that only require Foreman and the Plugin in question (not other dependent plugins). This is the environment in which today’s integration tests run. Having the tests (or some) remain separate, we need to do a better job of updating the end to end tests - by updating our contributing docs and leading our community by example on this front.

5

These are both great summaries of devconf and what we experienced while there!

I won’t repeat everything but I’ll add my experience:

There is a great community spirit to devconf.us and it was nice to experience that. This is the second year it has ran and it seems like it is building a great community. Those involved in the conference were very welcoming and the attendees were a mix of sysadmins, developers, designers, and students.

We had our booth set up for the three days of the conference and we got many people stopping by who were interested in what the booth with the big yellow hard hat was. Some were familiar with Foreman, some with Satellite, and others were not as familiar but interested. Many asked us a lot of questions and want to check it out for their own use. Of course, some came for the free hats and we did give all of them out! You could see the bright yellow caps throughout the conference and we even saw a couple on the streets of Boston :eyes:

I think the user feedback was mostly covered, but to add a bit, here are some things that stood out to me:

  • Suse provisioning and management was asked for by a Foreman user, I think this is a known request but I’ll mention it was specifically asked for :slight_smile:
  • Community Demos are hard to search by specific topic. We may want to make more descriptive titles so they show up better in Youtube search and include timestamps in the description.
  • Foreman Ansible - I got to play with this during some of the quiet times of the conference, it is a really great plugin and I was able to run roles on hosts. However, as already mentioned, what are the expectations for running with different parameters on groups of hosts? I get that you can match parameters on hostgroup, os, domain, and fqdn, but this doesn’t seem too helpful if I want to specify a group of hosts after everything has been created and set up. Having a concept like host collection or decoupling environments from puppet to be used in other config tools seems like two approaches worth exploring (though this is a big topic that probably deserves its own discussion). I think its a bit concerning though because the users who were interested in the feature couldn’t figure out how this would fit in their setup and said they probably could not use it. Please let me know if there is anything I am missing, I’m only recently familiarized myself with the plugin.
  • One idea I’ve been toying with and floated by a couple of users is doing a Foreman/Katello install and setup livestream. There would be some end goal, such as setting up web servers running a basic app in various lifecycle environments that could deploy and update to new versions. I mention this because the users really liked that idea and said they would love to attend. Something to keep in mind for the nearish future!

I was able to attend two talks: One on Test Driven Development, which went over the red/green/refactor steps and also talked about using TDD in legacy code. Another on Fedora Silverblue, which talked about how it uses rpm-ostree and flatpak to keep application and system layers separate. It seems like a really interesting technology and I plan try it out.

I had a great time and I enjoyed making connections with existing users and educating others about Foreman, I hope we continue to attend the conference.

Here is a picture from the booth :slight_smile:

picture
picture680×510 69.3 KB

7 Likes