Infrastructure SIG Meeting Notes 2021-10-07

Infra SIG meeting 2021-10-07

These are the notes from today’s meeting. Our living SIG document for all the background.

People present: @evgeni (chair), @ekohl, @Odilhao, Kenny Tordeur

Various updates

Since our last meeting a few updates have been posted as logs:


Redmine migration

No update

Migrate setup to OpenShift

@evgeni has made some progress. By giving the anonymous user permissions it’s now publicly visible as our current CI but we’re blocked on a missing Ansible installation in the nodes. Our pipelines heavily rely on this. The issue has been open for a while.

Action Item: @evgeni will ping the issue limits with testing matrix

No update

Archiving Old Debian Releases

Still needs documentation.

Action Item: @evgeni to document procedure to archive

Auto-building Debian on PR merge

Various code has been written but it has not been tested yet. Right now @ekohl and @evgeni don’t have time to monitor it. We’ll wait for @ehelms to return to decide when to enable it.

Rackspace migration

We’d like to investigate if the OpenShift hosted Jenkins instance that’s intended to replace can serve as a replacement instead. For this we would need to see if we can add our own nodes to that Jenkins instance.

For Foreman/Puppet: we can possibly decommission Jenkins nodes on OSUOSL to free up capacity. This would give us space for the new machines. Decomissioning those is made possible by the new machine as Conova. More on that later.

Rebuilding Koji

No update

Use of Jenkinsfiles

No update

New sponsor

Conova sponsors a dedicated physical machine (node01.conova.t.o). @evgeni has installed CentOS 8 Stream on it together with libvirt. Two new Jenkins nodes (deb-node01.jenkins.conova.t.o and node01.jenkins.conova.t.o) are now running on it and working well. These are much faster than the old ones. We’ve seen about 2x to 3x faster builds on average.

The name node01 is confusing since we use the same name for our Jenkins nodes. The name virt01 is clearer.

Action item: Rename node01.conova.t.o to virt01.conova.t.o

deb-node01.jenkins.conova.t.o is now Debian 11 while deb-node01.jenkins.osuosl.t.o was running Debian 10. Right now the node at OSUOSL is still running, but doesn’t have a label so it’s not used. Decomissioning is needed now that the new node has proven stable and reliable.

node01.jenkins.conova.t.o has the label rpmbuild, but this eats up an executioner slot with basically just waiting for Koji to finish. Our slow nodes can take care of this.

Action item: @ekohl remove the rpmbuild label (done)

@ekohl has also installed and configured foreman-libvirt on our Foreman instance. This allows restarting nodes but provisioning has not yet been set up. We can also set up foreman-openstack for our accounts at Netways and OSUOSL. There is no objection to this, but at least for OSUOSL we’ll need a service accounts so it’s not tied to a personal account.

Action item: Set up compute resources for Netways and OSUOSL

CDN for the Website

No update

RHEL for Open Source Infrastructure

@pcreech sent an email but there hasn’t been a reply yet.

Action Item: @evgeni / @pcreech: ping to get an update

No update

Move prprocessor to shared infra

No update

Testing multiple plugins together

There was a posssible solution but it’s unclear if that has been tested. @Ondrej_Prazak has been pinged if that worked (Testing multiple plugins together - #3 by evgeni).

Space on web01

The existing cleanup for Debian builds has some issues, need to dive into this. For example, foreman-dbgsym is no longer generated but remains in the repository. This needs some attention but right now it’s unclear where to exactly focus. There are no immediate problems now.

CentOS Stream 9

No packaging progress. Puppet modules are blocked on a missing puppet-agent RPM for EL9 which makes continuous integration hard.

Azure credits

Azure gives credits for FOSS projects. This may be interesting to us.

Action Item: @Odilhao look into the possibilities