Upcoming Koji maintenance

Hello Foreman/Katello!

Recently I have taken over the role of ‘owning’ the Koji server.

There are a couple of upcoming maintenance items that I wanted to share, and to get feedback on a good maintenance window for some possible upcoming downtime.

Fedora 28 support (asap)

This should be able to be done without requiring any downtime. For this, we will be provisioning a new builder for koji with Fedora 28 as the OS. The simple fact is, mock uses the system’s package manager, and we need the system package manager to be a dnf version that understands the new dependency changes with Fedora 28.

Some small configuration changes that will require a httpd service restart on the koji server, and modify what uri we are using for external repos, are all that need to be made on the server. This is to ensure that (initially) only fedora28 build tags will go to this new Fedora 28 builder, and this new builder can access the content it neds. Eventually all Fedora builds will be assigned to this builder.

Koji Upgrades

Our koji system is pretty old at this point (1.11 vs 1.15 being latest). I’m hoping to get us on the latest koji system, and there will need to be some downtime to achieve this. I have already done a mock upgrade with a cloned koji instance, and so far has passed all testing ran against it.

The outage window shouldn’t be too bad (best case of ~1 hour) as long as nothing major happens. I’ve already documented all the issues I ran in to on the cloned koji to ensure minimal downtime. Ideally, this will happen during a period where releases aren’t being actively worked on.

Some feedback on current tentative release schedules would be nice here, to help plan when I can do the upgrade. I don’t want to interrupt current 1.19/3.8 rc’s

5 Likes

Thank you for the effort and for posting the details first! Please make sure to coordinate the downtime with @ekohl, @Jonathon_Turel and @sajha. We are currently working on Foreman 1.19 RC1, and I expect Katello 3.7.0 and 3.8 RC1 will also be building in the near future.

Thank you so much @packaging team to getting involved, I am terrible admin and those planned or unplanned maintenance work was really ruining my dev schedule.

Before I clean swat from my forehead: make sure to edit /etc/aliases because logwatch is still sending reports to root (my email).

There is a PCP monitoring daemon running, if you don’t have plans integrating it or using it, just stop pmcd and pmlogger and delete /var/log/pcp/HOSTNAME directory.

Make sure to cleanup on AWS account, there might be some old EBS volumes or instances - work with @Bryan_Kearney in this case.

I was trying to maintain the wiki pages, they might be slightly out of date now:

I think that’s all.

1 Like

@lzap thanks for the info

Quick update, I’ve got all the prep work squared away for adding a fedora builder for fedora 28 support, without causing service disruption to koji.

I’ll be looking to get that done soon and will send an update in it’s own topic once finished and we have confirmed green fedora 28 builds.

I’m not sure if it’s related or not, but koji seems to be offline for the past couple of days.