Koji maintenance Dec 16, EST midday/afternoon

Development
1

In an effort to help facilitate EL8 builds, we are looking to update koji on manday, Dec 16 during the EST timezone’s midday/afternoon time frame, 11:00 AM EST to 5:00PM EST, or 16:00 UTC to 22:00 UTC

If anyone has any objections, please raise them here

1 Like
2

Reminder, this will occur today, in about 4-5 hours

3

Starting this now

4

Maintenance has been completed, and successful builds have been verified

http://koji.katello.org/koji/taskinfo?taskID=257707
http://koji.katello.org/koji/taskinfo?taskID=257706
http://koji.katello.org/koji/taskinfo?taskID=257705
http://koji.katello.org/koji/taskinfo?taskID=257704

5

Looks like foreman-installer started to fail after this. From foreman-installer-develop-package-release 229 failed

01:08:31      257724 build (foreman-nightly-nonscl-rhel7, foreman-installer-1.25.0-0.2.develop.20191217000301git4838c1a.el7.src.rpm): open (kojibuilder1) -> FAILED: Fault: <Fault 1: "<type 'exceptions.IOError'>: [Errno 2] No such file or directory: '/mnt/koji/work/tasks/7726/257726/foreman-installer-1.25.0-0.2.develop.20191217000301git4838c1a.el7.src.rpm'">
6

this file is a dangling symlink:

lrwxrwxrwx. 1 apache apache  164 Dec 17 03:23 foreman-installer-1.25.0-0.2.develop.20191217031816git4838c1a.el7.src.rpm -> ../../../../packages/foreman-installer/1.25.0/0.2.develop.20191217031816git4838c1a.el7/src/foreman-installer-1.25.0-0.2.develop.20191217031816git4838c1a.el7.src.rpm

it seems it points at the wrong mount?

# readlink -m /mnt/koji/work/tasks/7729/257729/foreman-installer-1.25.0-0.2.develop.20191217031816git4838c1a.el7.src.rpm
/mnt/tmp/packages/foreman-installer/1.25.0/0.2.develop.20191217031816git4838c1a.el7/src/foreman-installer-1.25.0-0.2.develop.20191217031816git4838c1a.el7.src.rpm

# ls /mnt/tmp/packages/foreman-installer/1.25.0/0.2.develop.20191217031816git4838c1a.el7/src/foreman-installer-1.25.0-0.2.develop.20191217031816git4838c1a.el7.src.rpm
ls: cannot access /mnt/tmp/packages/foreman-installer/1.25.0/0.2.develop.20191217031816git4838c1a.el7/src/foreman-installer-1.25.0-0.2.develop.20191217031816git4838c1a.el7.src.rpm: No such file or directory

# ls /mnt/koji/packages/foreman-installer/1.25.0/0.2.develop.20191217031816git4838c1a.el7/src/foreman-installer-1.25.0-0.2.develop.20191217031816git4838c1a.el7.src.rpm 
/mnt/koji/packages/foreman-installer/1.25.0/0.2.develop.20191217031816git4838c1a.el7/src/foreman-installer-1.25.0-0.2.develop.20191217031816git4838c1a.el7.src.rpm
7

Or do we need a symlink from /mnt/tmp/packages to /mnt/koji/packages?

8

fwiw, all release tasks fail like this: http://koji.katello.org/koji/taskinfo?taskID=257732

9

At some point in the upgrade version path, relative instead of absolute paths were used for artifact symlinks. This caused the link to follow the path on the other filesystem, instead of directory heiarchy.

Adding a new symbolic link to /mnt/koji/packages at /mnt/tmp/packages resolved the issue.

10

We seem to be running smoothly, i’ll consider this completed if nothing creeps up in the next couple days

11

No issues have cropped up that need attention, considering this done.