These are the notes from the meeting. See our living SIG document for all the background.
People present: @ehelms (chair), @ekohl , @evgeni , @mcorr , @odilhao , @pcreech
Various updates
Since our last meeting a few updates have been posted as logs:
Tracking issues on foreman-infra
In the previous meeting’s notes @ekohl created issues for all action items. Based on that @ehelms proposes to move away from the living SIG document on hackmd. Instead, all work will be tracked in GitHub issues (and milestones if needed) in the foreman-infra repository .
Redmine migration
opened 12:38PM - 05 Nov 21 UTC
closed 01:13PM - 21 Sep 23 UTC
Current Redmine version: 3.Y
Current infrastructure: Scaleways
Codebase: https… ://github.com/theforeman/redmine
Migration options:
- OSUOSL
- Conova box
- OSCI Openshift (https://openshift-console.osci.io)
Action Items
- [x] Build migration plan and document here
- [x] Move Redmine to new infrastructure
- [x] Test Redmine upgrade locally
- [x] Upgrade Redmine to 5.Y
No update
Migrate ci.centos.org setup to OpenShift
opened 12:45PM - 05 Oct 20 UTC
closed 10:46AM - 10 May 22 UTC
see https://lists.centos.org/pipermail/ci-users/2020-July/002098.html and https:… //docs.fedoraproject.org/en-US/cpe/day_to_day_centos/#_our_workflow
essentially we get an own jenkins controller running on OpenShift and then can do the same things we've been doing on the old ci.centos.org infra (yes, Duffy works ;))
Migration Was blocked on a lack of Ansible. After a bit of yak shaving, @evgeni and Fabian (from CentOS) this got resolved together with other improvements. @evgeni was able to launch a pipeline but that failed somewhere. Needs further debugging.
ci.centos.org will also be shut down “soon”. This increases the priority.
An open question was whether we can attach external nodes. @evgeni says this should be possible since the storage is non-empheral, but it hasn’t been tried.
We are responsible for backing up the new OpenShift instance. Maybe not needed if we can automatically reprovision.
opened 01:06PM - 05 Nov 21 UTC
closed 03:31PM - 03 Mar 22 UTC
* Jenkins node owned by ci.centos.org
* Request bare metal machines fro… m Duffy
* Limited to 6/8 parallel machines from Duffy
* Each OS - install,upgrade pairing requests a machine from Duffy to run a Vagrant pipeline on
* We end up having 2 jobs rejected when a pipeline runs and it fails
* Will need to scale to additional OSes:
* Ubuntu 20.04
* Debian 11
* CentOS 8 Stream (would eventually replace CentOS 8)
* Proposals
1. Split our release pipelines similar to the nightly split
* Schedule EL pipeline, if that succeeds schedule the Debian pipeline
* Ground work done to enable the split, next step is to split the jobs
~~2. Reduce combinations that are run, only a single Debian~~
~~3. Run all installs first, if they pass, run all upgrade jobs~~
4. Is there other infrastructure we could explore using available to us?
* Exploring what IBM cloud might could give us
* Reasonably priced virt instances
* Able to run vagrant on top of them
* Performance comparable to ci.centos.org
* Exploring Conova supplying HP boxes
* Exploring the idea of splitting across ci.centos.org and IBM cloud: run Foreman on one, Katello on the other
5. Look into throttling inside Jenkins
6. https://plugins.jenkins.io/throttle-concurrents/#example-2-throttling-of-parallel-steps
No update
Document archiving old Debian releases
opened 04:36PM - 07 Oct 21 UTC
The process to move releases from https://deb.theforeman.org/ to https://archive… deb.theforeman.org/ needs to be documented.
No update
Auto-building Debian on PR merge
This is implemented and can be marked resolved.
Rackspace migration
For other tasks we’re running into the problem that our Foreman is too old. Migration would help.
Rebuilding Koji
opened 02:07PM - 29 Aug 20 UTC
closed 12:28PM - 26 Oct 23 UTC
Discourse discussion: https://community.theforeman.org/t/rebuilding-koji/20977
…
Notes:
- Current Koji has server, builder, database all in one machine
- Requires a separate builder to handle EL8 running Fedora
- Is unmanaged by any config management
Server and hostnames layout for new Koji:
- Koji server
- Hub + database
- hostname: koji01.aws.theforeman.org
- service name: koji.theforeman.org
- Koji builders
- hostname: builder0X.koji.aws.theforeman.org
Additional improvements to consider:
* Reduce the amount of swap being used [comment](https://github.com/theforeman/foreman-infra/pull/1415#discussion_r472295326)
* Systemd dependency on mount points [comment](https://github.com/theforeman/foreman-infra/pull/1415#discussion_r472293943)
* Drop use of /etc/hosts due to floating IP [comment](https://github.com/theforeman/foreman-infra/pull/1415#discussion_r472298280)
* Get rid of random hostname [comment](https://github.com/theforeman/foreman-infra/pull/1415#discussion_r472301834)
* Enable SELinux [comment](https://github.com/theforeman/foreman-infra/pull/1415#discussion_r475807780)
* Add ` /usr/local/bin/foreman-root-restrict.sh` to source control
Action Items:
- [ ] Build out Ansible or Puppet to setup and manage Koji infrastructure
- [ ] Build a new environment with config management on new infrastructure
- [ ] Manage Koji within Foreman instance
- [ ] Migrate to new disk format
Andrew was working on this, but had to pull back due to a lack of time
Use of Jenkinsfiles
opened 01:16PM - 05 Nov 21 UTC
closed 12:29PM - 26 Oct 23 UTC
Projects in our organization should be able to use Jenkinsfile to create CI jobs… and control how their projects are tested similar to GH actions.
Discussion link: (https://community.theforeman.org/t/jenkins-github-app/20244
Action Items
- [ ] Isolate secrets through folders for internal Jenkins jobs (https://github.com/theforeman/jenkins-jobs/pull/101)
- [ ] Spin up a test Jenkins to test the workflow on
No update
New Sponsor (Conova)
opened 01:36AM - 05 Nov 21 UTC
closed 03:22PM - 03 Feb 22 UTC
Conova has provided us with a new hardware resource for our use how we want to h… elp the project.
Box:
- HP 325
- 24 core x2 (hyper-threading)
- 64GB RAM
- 2 1TB SSD
- In Austria
Action Items:
- [x] Setup libvirt
- [x] Configure foreman-libvirt in foreman.theforeman.org
- [x] Rename to virt01: https://github.com/theforeman/foreman-infra/issues/1667
- [x] Document libvirt setup: https://github.com/theforeman/foreman-infra/issues/1708
- [x] Provision VMs
- [x] deb-node01 - deb builder
- [x] node01 - CI node
- [x] Update infrastructure docs (https://github.com/theforeman/foreman-infra/blob/master/docs/overview.md)
During the previous meeting we decided to rename the host but haven’t gotten around to it. It’s still running smoothly.
CDN for the website
No update
RHEL for Open Source Infrastructure
opened 04:45PM - 07 Oct 21 UTC
closed 12:30PM - 26 Oct 23 UTC
We should contact Red Hat to see if RHEL for Open Source Infrastructure can be u… sed by the Foreman project to build RPMs in our Koji setup.
Within Red Hat some internal emails were sent and Patrick has received a huge email with steps forward. The short summary is that we can use RHEL for Koji but can’t let anyone retrieve the RHEL bits. This means we need to store them locally on our Koji.
Action Item: @ehelms will set up a dedicated meeting for using RHEL on Koji
opened 01:40AM - 05 Nov 21 UTC
closed 03:03PM - 07 Apr 22 UTC
The https://osci.io/ project provides community hosting resources and Openshift.…
There is an Openshift for the project available at: https://openshift-console.osci.io
Could be used for running some of our infrastructure apps:
- Prprocessor
- Redmine
Action Items
- [x] Figure out what to do with this
Follow ups:
- https://github.com/theforeman/foreman-infra/issues/1679
- https://github.com/theforeman/foreman-infra/issues/1681
No update
Move prprocessor to shared infra
opened 01:48AM - 05 Nov 21 UTC
closed 01:44PM - 14 Apr 22 UTC
The [prprocessor](https://github.com/theforeman/prprocessor) project handles a n… umber of tasks for the project and currently runs on @ekohl own infrastructure.
Options to move to:
- OSCI Openshift: https://openshift-console.osci.io
Action Items
- [x] Move to Python 3.7
- [x] Move to one of the shared infrastructure environments
- [ ] Make a post to discourse introducing Prprocessor to community again since it's been a while and new developers may not be familiar with it [@ekohl]
The prprocessor has gained the ability to set Fixed in version in Redmine for stable branches (cherry picks):
theforeman:app
← ekohl:set-fixed-in-for-stable-branches
opened 01:07PM - 18 Oct 21 UTC
When a PR is merged to a stable branch, the fixed in version must be set to keep… the administration correct. This uses the target branch name with the convention that they're named x.y-stable.
Edit: for now this is untested.
During this it came up that the prprocessor is something few people know about but it can be considered vital to our workflows.
Action Item: @ekohl create a post about what prprocessor is and what it can do
Space on web01
No updates
CentOS 9 Stream
opened 01:42PM - 05 Nov 21 UTC
closed 01:09PM - 05 May 22 UTC
Not released yet, but there are public repositories published as CentOS communit… y is wanting to get SIGs to test against it.
Notes:
* Are there containers yet?
* https://quay.io/repository/centos/centos?tab=tags
* centos:stream9-development
* Are there vagrant boxes yet?
* No
* Are there repositories yet?
* Development repositories exist
Action Items
- [x] Start with Centos 9 Stream pulpcore
Like at our last meeting, it’s not formally announcement but there should be containers and the mirrors are available. There are no vagrant boxes yet.
Azure credits for FOSS
opened 04:53PM - 07 Oct 21 UTC
https://opensource.microsoft.com/azure-credits may be interesting for us.
@odilhao Checked what Azure asks for. That’s
A Code of Conduct: ours suffices
A list of our current sponsors and what they do. Foreman needs to be sent.
Then once it’s approved, sponsorship lasts a year and can be renewed each year. Review takes generally about 2 weeks.
NPM install times
@ekohl suggests that a month ago we improved our node setup (New Jenkins node, removed fast label from OSUOSL nodes ). Perhaps that’s sufficient?
@evgeni thinks we install too much. For example, the storybook pulls in a lot. package.json
lacks the capability to define groups as bundler does. Perhaps yarn does.
There is some talk about switching from NPM to Yarn but it’s not something Infra can do.