Koji repo sync metadata problem

Hey,

when I initiated mrepo -n (dry run) this morning to see how mrepo
works on our koji in order to test if we are able to add Fedora 27 for
Pulp, I learned that this "dry run" is actually a real run and mrepo
initiated full resync of our content without metadata regeneration.

This rendered our external repositories unusable as all metadata were
deleted. All builds will fail today with missing dependencies.

The plan: we need to let mrepo finish "dry run" downloading of Fedora
27 and then we need to reexecute it with proper flags:

cat /usr/local/bin/koji-sync-external-repos
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk '{print $2}' | sort -u | xargs -I '{}'
koji regen-repo --nowait '{}'

Note the -u -g -f flags for mrepo, this will cause all metadata to be
regenerated. Once this command is done, koji will see missing
dependencies to appear.

The dry run of mrepo is running in screen and it is about to finish
hopefully soon. I will defer this to Eric in case it won't finish by
EMEA EOB.

Sorry for inconvenience, but dry run should be dry run, right.

··· -- Later, Lukas @lzap Zapletal

Looks like everything is back up and working as expected. For packagers,
keep in mind that this updated some of our external repositories to their
latest versions if you see any oddities. This also means that Fedora 27 is
available as an external repository for Pulp and Katello clients.

··· On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal wrote:

Hey,

when I initiated mrepo -n (dry run) this morning to see how mrepo
works on our koji in order to test if we are able to add Fedora 27 for
Pulp, I learned that this “dry run” is actually a real run and mrepo
initiated full resync of our content without metadata regeneration.

This rendered our external repositories unusable as all metadata were
deleted. All builds will fail today with missing dependencies.

The plan: we need to let mrepo finish “dry run” downloading of Fedora
27 and then we need to reexecute it with proper flags:

cat /usr/local/bin/koji-sync-external-repos
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk ‘{print $2}’ | sort -u | xargs -I '{}'
koji regen-repo --nowait ‘{}’

Note the -u -g -f flags for mrepo, this will cause all metadata to be
regenerated. Once this command is done, koji will see missing
dependencies to appear.

The dry run of mrepo is running in screen and it is about to finish
hopefully soon. I will defer this to Eric in case it won’t finish by
EMEA EOB.

Sorry for inconvenience, but dry run should be dry run, right.


Later,
Lukas @lzap Zapletal


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering

I still get errors when building:

http://koji.katello.org/koji/taskinfo?taskID=52363
http://koji.katello.org/kojifiles/work/tasks/2363/52363/root.log

DEBUG util.py:439: Error downloading packages:
DEBUG util.py:439: 1:nodejs-6.10.3-1.el7.x86_64: failed to retrieve nodejs-6.10.3-1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist: /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-6.10.3-1.el7.x86_64.rpm
DEBUG util.py:439: http-parser-2.7.1-3.el7.x86_64: failed to retrieve http-parser-2.7.1-3.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist: /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-parser-2.7.1-3.el7.x86_64.rpm
DEBUG util.py:439: 1:npm-3.10.10-1.6.10.3.1.el7.x86_64: failed to retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist: /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm

··· On Wed, Nov 15, 2017 at 11:44:24AM -0500, Eric D Helms wrote: >Looks like everything is back up and working as expected. For packagers, >keep in mind that this updated some of our external repositories to their >latest versions if you see any oddities. This also means that Fedora 27 is >available as an external repository for Pulp and Katello clients. > >On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal wrote: > >> Hey, >> >> when I initiated mrepo -n (dry run) this morning to see how mrepo >> works on our koji in order to test if we are able to add Fedora 27 for >> Pulp, I learned that this "dry run" is actually a real run and mrepo >> initiated full resync of our content without metadata regeneration. >> >> This rendered our external repositories unusable as all metadata were >> deleted. All builds will fail today with missing dependencies. >> >> The plan: we need to let mrepo finish "dry run" downloading of Fedora >> 27 and then we need to reexecute it with proper flags: >> >> cat /usr/local/bin/koji-sync-external-repos >> #!/bin/bash >> mrepo -qugf >> koji list-targets --quiet | awk '{print $2}' | sort -u | xargs -I '{}' >> koji regen-repo --nowait '{}' >> >> Note the -u -g -f flags for mrepo, this will cause all metadata to be >> regenerated. Once this command is done, koji will see missing >> dependencies to appear. >> >> The dry run of mrepo is running in screen and it is about to finish >> hopefully soon. I will defer this to Eric in case it won't finish by >> EMEA EOB. >> >> Sorry for inconvenience, but dry run should be dry run, right. >> >> -- >> Later, >> Lukas @lzap Zapletal >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > > >-- >Eric D. Helms >Red Hat Engineering > >-- >You received this message because you are subscribed to the Google Groups "foreman-dev" group. >To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. >For more options, visit https://groups.google.com/d/optout.

Thanks Eric, have you also performed the second command from the cron
job (regen-repo)?

We used to run this on a weekly basis on the old koji, but this was
disabled on the new one.

··· On Wed, Nov 15, 2017 at 5:44 PM, Eric D Helms wrote: > Looks like everything is back up and working as expected. For packagers, > keep in mind that this updated some of our external repositories to their > latest versions if you see any oddities. This also means that Fedora 27 is > available as an external repository for Pulp and Katello clients. > > On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal wrote: >> >> Hey, >> >> when I initiated mrepo -n (dry run) this morning to see how mrepo >> works on our koji in order to test if we are able to add Fedora 27 for >> Pulp, I learned that this "dry run" is actually a real run and mrepo >> initiated full resync of our content without metadata regeneration. >> >> This rendered our external repositories unusable as all metadata were >> deleted. All builds will fail today with missing dependencies. >> >> The plan: we need to let mrepo finish "dry run" downloading of Fedora >> 27 and then we need to reexecute it with proper flags: >> >> cat /usr/local/bin/koji-sync-external-repos >> #!/bin/bash >> mrepo -qugf >> koji list-targets --quiet | awk '{print $2}' | sort -u | xargs -I '{}' >> koji regen-repo --nowait '{}' >> >> Note the -u -g -f flags for mrepo, this will cause all metadata to be >> regenerated. Once this command is done, koji will see missing >> dependencies to appear. >> >> The dry run of mrepo is running in screen and it is about to finish >> hopefully soon. I will defer this to Eric in case it won't finish by >> EMEA EOB. >> >> Sorry for inconvenience, but dry run should be dry run, right. >> >> -- >> Later, >> Lukas @lzap Zapletal >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > > > -- > Eric D. Helms > Red Hat Engineering > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal

I do not have access here, can someone take a look? Are repodata
present for EPEL7?

LZ

··· On Wed, Nov 15, 2017 at 6:15 PM, Ewoud Kohl van Wijngaarden wrote: > I still get errors when building: > > http://koji.katello.org/koji/taskinfo?taskID=52363 > http://koji.katello.org/kojifiles/work/tasks/2363/52363/root.log > > DEBUG util.py:439: Error downloading packages: > DEBUG util.py:439: 1:nodejs-6.10.3-1.el7.x86_64: failed to retrieve > nodejs-6.10.3-1.el7.x86_64.rpm from build > DEBUG util.py:439: error was [Errno 2] Local file does not exist: > /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-6.10.3-1.el7.x86_64.rpm > DEBUG util.py:439: http-parser-2.7.1-3.el7.x86_64: failed to retrieve > http-parser-2.7.1-3.el7.x86_64.rpm from build > DEBUG util.py:439: error was [Errno 2] Local file does not exist: > /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-parser-2.7.1-3.el7.x86_64.rpm > DEBUG util.py:439: 1:npm-3.10.10-1.6.10.3.1.el7.x86_64: failed to > retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build > DEBUG util.py:439: error was [Errno 2] Local file does not exist: > /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm > > > On Wed, Nov 15, 2017 at 11:44:24AM -0500, Eric D Helms wrote: >> >> Looks like everything is back up and working as expected. For packagers, >> keep in mind that this updated some of our external repositories to their >> latest versions if you see any oddities. This also means that Fedora 27 is >> available as an external repository for Pulp and Katello clients. >> >> On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal wrote: >> >>> Hey, >>> >>> when I initiated mrepo -n (dry run) this morning to see how mrepo >>> works on our koji in order to test if we are able to add Fedora 27 for >>> Pulp, I learned that this "dry run" is actually a real run and mrepo >>> initiated full resync of our content without metadata regeneration. >>> >>> This rendered our external repositories unusable as all metadata were >>> deleted. All builds will fail today with missing dependencies. >>> >>> The plan: we need to let mrepo finish "dry run" downloading of Fedora >>> 27 and then we need to reexecute it with proper flags: >>> >>> cat /usr/local/bin/koji-sync-external-repos >>> #!/bin/bash >>> mrepo -qugf >>> koji list-targets --quiet | awk '{print $2}' | sort -u | xargs -I '{}' >>> koji regen-repo --nowait '{}' >>> >>> Note the -u -g -f flags for mrepo, this will cause all metadata to be >>> regenerated. Once this command is done, koji will see missing >>> dependencies to appear. >>> >>> The dry run of mrepo is running in screen and it is about to finish >>> hopefully soon. I will defer this to Eric in case it won't finish by >>> EMEA EOB. >>> >>> Sorry for inconvenience, but dry run should be dry run, right. >>> >>> -- >>> Later, >>> Lukas @lzap Zapletal >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "foreman-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to foreman-dev+unsubscribe@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Eric D. Helms >> Red Hat Engineering >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal

> I do not have access here, can someone take a look? Are repodata
> present for EPEL7?

If I have the correct path, it does not appear so:

[root@koji repodata]# pwd
/mnt/koji/external-repos/epel7-x86_64/epel/repodata
[root@koji repodata]# ls
[root@koji repodata]#

··· On Wed, 2017-11-15 at 19:56 +0100, Lukas Zapletal wrote:

LZ

On Wed, Nov 15, 2017 at 6:15 PM, Ewoud Kohl van Wijngaarden > ewoud@kohlvanwijngaarden.nl wrote:

I still get errors when building:

http://koji.katello.org/koji/taskinfo?taskID=52363
http://koji.katello.org/kojifiles/work/tasks/2363/52363/root.log

DEBUG util.py:439: Error downloading packages:
DEBUG util.py:439: 1:nodejs-6.10.3-1.el7.x86_64: failed to retrieve
nodejs-6.10.3-1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-6.10.3-1.el7.x86_64.rpm
DEBUG util.py:439: http-parser-2.7.1-3.el7.x86_64: failed to retrieve
http-parser-2.7.1-3.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-parser-2.7.1-3.el7.x86_64.rpm
DEBUG util.py:439: 1:npm-3.10.10-1.6.10.3.1.el7.x86_64: failed to
retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm

On Wed, Nov 15, 2017 at 11:44:24AM -0500, Eric D Helms wrote:

Looks like everything is back up and working as expected. For packagers,
keep in mind that this updated some of our external repositories to their
latest versions if you see any oddities. This also means that Fedora 27 is
available as an external repository for Pulp and Katello clients.

On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal lzap@redhat.com wrote:

Hey,

when I initiated mrepo -n (dry run) this morning to see how mrepo
works on our koji in order to test if we are able to add Fedora 27 for
Pulp, I learned that this “dry run” is actually a real run and mrepo
initiated full resync of our content without metadata regeneration.

This rendered our external repositories unusable as all metadata were
deleted. All builds will fail today with missing dependencies.

The plan: we need to let mrepo finish “dry run” downloading of Fedora
27 and then we need to reexecute it with proper flags:

cat /usr/local/bin/koji-sync-external-repos
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk ‘{print $2}’ | sort -u | xargs -I '{}'
koji regen-repo --nowait ‘{}’

Note the -u -g -f flags for mrepo, this will cause all metadata to be
regenerated. Once this command is done, koji will see missing
dependencies to appear.

The dry run of mrepo is running in screen and it is about to finish
hopefully soon. I will defer this to Eric in case it won’t finish by
EMEA EOB.

Sorry for inconvenience, but dry run should be dry run, right.


Later,
Lukas @lzap Zapletal


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal

> > I do not have access here, can someone take a look? Are repodata
> > present for EPEL7?
>
> If I have the correct path, it does not appear so:
>
> [root@koji repodata]# pwd
> /mnt/koji/external-repos/epel7-x86_64/epel/repodata
> [root@koji repodata]# ls
> [root@koji repodata]#
>

I might not have, this path DOES have repodata:

/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/repodata

··· On Wed, 2017-11-15 at 14:00 -0500, Patrick Creech wrote: > On Wed, 2017-11-15 at 19:56 +0100, Lukas Zapletal wrote:

LZ

On Wed, Nov 15, 2017 at 6:15 PM, Ewoud Kohl van Wijngaarden > > ewoud@kohlvanwijngaarden.nl wrote:

I still get errors when building:

http://koji.katello.org/koji/taskinfo?taskID=52363
http://koji.katello.org/kojifiles/work/tasks/2363/52363/root.log

DEBUG util.py:439: Error downloading packages:
DEBUG util.py:439: 1:nodejs-6.10.3-1.el7.x86_64: failed to retrieve
nodejs-6.10.3-1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-6.10.3-1.el7.x86_64.rpm
DEBUG util.py:439: http-parser-2.7.1-3.el7.x86_64: failed to retrieve
http-parser-2.7.1-3.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-parser-2.7.1-3.el7.x86_64.rpm
DEBUG util.py:439: 1:npm-3.10.10-1.6.10.3.1.el7.x86_64: failed to
retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm

On Wed, Nov 15, 2017 at 11:44:24AM -0500, Eric D Helms wrote:

Looks like everything is back up and working as expected. For packagers,
keep in mind that this updated some of our external repositories to their
latest versions if you see any oddities. This also means that Fedora 27 is
available as an external repository for Pulp and Katello clients.

On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal lzap@redhat.com wrote:

Hey,

when I initiated mrepo -n (dry run) this morning to see how mrepo
works on our koji in order to test if we are able to add Fedora 27 for
Pulp, I learned that this “dry run” is actually a real run and mrepo
initiated full resync of our content without metadata regeneration.

This rendered our external repositories unusable as all metadata were
deleted. All builds will fail today with missing dependencies.

The plan: we need to let mrepo finish “dry run” downloading of Fedora
27 and then we need to reexecute it with proper flags:

cat /usr/local/bin/koji-sync-external-repos
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk ‘{print $2}’ | sort -u | xargs -I '{}'
koji regen-repo --nowait ‘{}’

Note the -u -g -f flags for mrepo, this will cause all metadata to be
regenerated. Once this command is done, koji will see missing
dependencies to appear.

The dry run of mrepo is running in screen and it is about to finish
hopefully soon. I will defer this to Eric in case it won’t finish by
EMEA EOB.

Sorry for inconvenience, but dry run should be dry run, right.


Later,
Lukas @lzap Zapletal


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal

I had forgotten to run the repo-regen on Koji. That is finishing up now. I
will run another test after and report back.

··· On Wed, Nov 15, 2017 at 2:08 PM, Patrick Creech wrote:

On Wed, 2017-11-15 at 14:00 -0500, Patrick Creech wrote:

On Wed, 2017-11-15 at 19:56 +0100, Lukas Zapletal wrote:

I do not have access here, can someone take a look? Are repodata
present for EPEL7?

If I have the correct path, it does not appear so:

[root@koji repodata]# pwd
/mnt/koji/external-repos/epel7-x86_64/epel/repodata
[root@koji repodata]# ls
[root@koji repodata]#

I might not have, this path DOES have repodata:

/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/repodata

LZ

On Wed, Nov 15, 2017 at 6:15 PM, Ewoud Kohl van Wijngaarden > > > ewoud@kohlvanwijngaarden.nl wrote:

I still get errors when building:

http://koji.katello.org/koji/taskinfo?taskID=52363
http://koji.katello.org/kojifiles/work/tasks/2363/52363/root.log

DEBUG util.py:439: Error downloading packages:
DEBUG util.py:439: 1:nodejs-6.10.3-1.el7.x86_64: failed to
retrieve

nodejs-6.10.3-1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-
6.10.3-1.el7.x86_64.rpm

DEBUG util.py:439: http-parser-2.7.1-3.el7.x86_64: failed to
retrieve

http-parser-2.7.1-3.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-
parser-2.7.1-3.el7.x86_64.rpm

DEBUG util.py:439: 1:npm-3.10.10-1.6.10.3.1.el7.x86_64: failed to
retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.
10.10-1.6.10.3.1.el7.x86_64.rpm

On Wed, Nov 15, 2017 at 11:44:24AM -0500, Eric D Helms wrote:

Looks like everything is back up and working as expected. For
packagers,

keep in mind that this updated some of our external repositories
to their

latest versions if you see any oddities. This also means that
Fedora 27 is

available as an external repository for Pulp and Katello clients.

On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal lzap@redhat.com > wrote:

Hey,

when I initiated mrepo -n (dry run) this morning to see how mrepo
works on our koji in order to test if we are able to add Fedora
27 for

Pulp, I learned that this “dry run” is actually a real run and
mrepo

initiated full resync of our content without metadata
regeneration.

This rendered our external repositories unusable as all metadata
were

deleted. All builds will fail today with missing dependencies.

The plan: we need to let mrepo finish “dry run” downloading of
Fedora

27 and then we need to reexecute it with proper flags:

cat /usr/local/bin/koji-sync-external-repos
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk ‘{print $2}’ | sort -u | xargs
-I ‘{}’

koji regen-repo --nowait ‘{}’

Note the -u -g -f flags for mrepo, this will cause all metadata
to be

regenerated. Once this command is done, koji will see missing
dependencies to appear.

The dry run of mrepo is running in screen and it is about to
finish

hopefully soon. I will defer this to Eric in case it won’t
finish by

EMEA EOB.

Sorry for inconvenience, but dry run should be dry run, right.


Later,
Lukas @lzap Zapletal


You received this message because you are subscribed to the
Google Groups

“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from
it, send an

email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering


You received this message because you are subscribed to the Google
Groups

“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it,
send an

email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups

“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it,
send an

email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering

This now appears to be working again. One out standing question that
affects nodejs- packaging builds.

As part of the RHEL 7.4 release, http-parser was removed from EPEL. With
this latest round of Koji external repositories update we now have a newer
EPEL with this package removed. We did not update our CentOS external
repository that would include this package. Not updating the base OS has
been our policy for a while now it would seem. We have two options:

  1. Sync and update CentOS
  2. Add http-parser to the overrides tag for foreman-nightly
··· On Wed, Nov 15, 2017 at 2:11 PM, Eric D Helms wrote:

I had forgotten to run the repo-regen on Koji. That is finishing up now. I
will run another test after and report back.

On Wed, Nov 15, 2017 at 2:08 PM, Patrick Creech pcreech@redhat.com > wrote:

On Wed, 2017-11-15 at 14:00 -0500, Patrick Creech wrote:

On Wed, 2017-11-15 at 19:56 +0100, Lukas Zapletal wrote:

I do not have access here, can someone take a look? Are repodata
present for EPEL7?

If I have the correct path, it does not appear so:

[root@koji repodata]# pwd
/mnt/koji/external-repos/epel7-x86_64/epel/repodata
[root@koji repodata]# ls
[root@koji repodata]#

I might not have, this path DOES have repodata:

/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/repodata

LZ

On Wed, Nov 15, 2017 at 6:15 PM, Ewoud Kohl van Wijngaarden >> > > ewoud@kohlvanwijngaarden.nl wrote:

I still get errors when building:

http://koji.katello.org/koji/taskinfo?taskID=52363
http://koji.katello.org/kojifiles/work/tasks/2363/52363/root.log

DEBUG util.py:439: Error downloading packages:
DEBUG util.py:439: 1:nodejs-6.10.3-1.el7.x86_64: failed to
retrieve

nodejs-6.10.3-1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-6
.10.3-1.el7.x86_64.rpm

DEBUG util.py:439: http-parser-2.7.1-3.el7.x86_64: failed to
retrieve

http-parser-2.7.1-3.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-par
ser-2.7.1-3.el7.x86_64.rpm

DEBUG util.py:439: 1:npm-3.10.10-1.6.10.3.1.el7.x86_64: failed
to

retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.10
.10-1.6.10.3.1.el7.x86_64.rpm

On Wed, Nov 15, 2017 at 11:44:24AM -0500, Eric D Helms wrote:

Looks like everything is back up and working as expected. For
packagers,

keep in mind that this updated some of our external repositories
to their

latest versions if you see any oddities. This also means that
Fedora 27 is

available as an external repository for Pulp and Katello clients.

On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal lzap@redhat.com >> wrote:

Hey,

when I initiated mrepo -n (dry run) this morning to see how
mrepo

works on our koji in order to test if we are able to add Fedora
27 for

Pulp, I learned that this “dry run” is actually a real run and
mrepo

initiated full resync of our content without metadata
regeneration.

This rendered our external repositories unusable as all
metadata were

deleted. All builds will fail today with missing dependencies.

The plan: we need to let mrepo finish “dry run” downloading of
Fedora

27 and then we need to reexecute it with proper flags:

cat /usr/local/bin/koji-sync-external-repos
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk ‘{print $2}’ | sort -u | xargs
-I ‘{}’

koji regen-repo --nowait ‘{}’

Note the -u -g -f flags for mrepo, this will cause all metadata
to be

regenerated. Once this command is done, koji will see missing
dependencies to appear.

The dry run of mrepo is running in screen and it is about to
finish

hopefully soon. I will defer this to Eric in case it won’t
finish by

EMEA EOB.

Sorry for inconvenience, but dry run should be dry run, right.


Later,
Lukas @lzap Zapletal


You received this message because you are subscribed to the
Google Groups

“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from
it, send an

email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering


You received this message because you are subscribed to the
Google Groups

“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it,
send an

email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups

“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it,
send an

email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering


Eric D. Helms
Red Hat Engineering

I think we should sync & update. CentOS has no concept of supported
minor releases and we should be testing with a supported release.

··· On Wed, Nov 15, 2017 at 03:57:40PM -0500, Eric D Helms wrote: >This now appears to be working again. One out standing question that >affects nodejs- packaging builds. > >As part of the RHEL 7.4 release, http-parser was removed from EPEL. With >this latest round of Koji external repositories update we now have a newer >EPEL with this package removed. We did not update our CentOS external >repository that would include this package. Not updating the base OS has >been our policy for a while now it would seem. We have two options: > > 1) Sync and update CentOS > 2) Add http-parser to the overrides tag for foreman-nightly > >On Wed, Nov 15, 2017 at 2:11 PM, Eric D Helms wrote: > >> I had forgotten to run the repo-regen on Koji. That is finishing up now. I >> will run another test after and report back. >> >> On Wed, Nov 15, 2017 at 2:08 PM, Patrick Creech >> wrote: >> >>> On Wed, 2017-11-15 at 14:00 -0500, Patrick Creech wrote: >>> > On Wed, 2017-11-15 at 19:56 +0100, Lukas Zapletal wrote: >>> > > I do not have access here, can someone take a look? Are repodata >>> > > present for EPEL7? >>> > >>> > If I have the correct path, it does not appear so: >>> > >>> > [root@koji repodata]# pwd >>> > /mnt/koji/external-repos/epel7-x86_64/epel/repodata >>> > [root@koji repodata]# ls >>> > [root@koji repodata]# >>> > >>> >>> I might not have, this path DOES have repodata: >>> >>> /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/repodata >>> >>> > > LZ >>> > > >>> > > On Wed, Nov 15, 2017 at 6:15 PM, Ewoud Kohl van Wijngaarden >>> > > wrote: >>> > > > I still get errors when building: >>> > > > >>> > > > http://koji.katello.org/koji/taskinfo?taskID=52363 >>> > > > http://koji.katello.org/kojifiles/work/tasks/2363/52363/root.log >>> > > > >>> > > > DEBUG util.py:439: Error downloading packages: >>> > > > DEBUG util.py:439: 1:nodejs-6.10.3-1.el7.x86_64: failed to >>> retrieve >>> > > > nodejs-6.10.3-1.el7.x86_64.rpm from build >>> > > > DEBUG util.py:439: error was [Errno 2] Local file does not exist: >>> > > > /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-6 >>> .10.3-1.el7.x86_64.rpm >>> > > > DEBUG util.py:439: http-parser-2.7.1-3.el7.x86_64: failed to >>> retrieve >>> > > > http-parser-2.7.1-3.el7.x86_64.rpm from build >>> > > > DEBUG util.py:439: error was [Errno 2] Local file does not exist: >>> > > > /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-par >>> ser-2.7.1-3.el7.x86_64.rpm >>> > > > DEBUG util.py:439: 1:npm-3.10.10-1.6.10.3.1.el7.x86_64: failed >>> to >>> > > > retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build >>> > > > DEBUG util.py:439: error was [Errno 2] Local file does not exist: >>> > > > /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.10 >>> .10-1.6.10.3.1.el7.x86_64.rpm >>> > > > >>> > > > >>> > > > On Wed, Nov 15, 2017 at 11:44:24AM -0500, Eric D Helms wrote: >>> > > > > >>> > > > > Looks like everything is back up and working as expected. For >>> packagers, >>> > > > > keep in mind that this updated some of our external repositories >>> to their >>> > > > > latest versions if you see any oddities. This also means that >>> Fedora 27 is >>> > > > > available as an external repository for Pulp and Katello clients. >>> > > > > >>> > > > > On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal >>> wrote: >>> > > > > >>> > > > > > Hey, >>> > > > > > >>> > > > > > when I initiated mrepo -n (dry run) this morning to see how >>> mrepo >>> > > > > > works on our koji in order to test if we are able to add Fedora >>> 27 for >>> > > > > > Pulp, I learned that this "dry run" is actually a real run and >>> mrepo >>> > > > > > initiated full resync of our content without metadata >>> regeneration. >>> > > > > > >>> > > > > > This rendered our external repositories unusable as all >>> metadata were >>> > > > > > deleted. All builds will fail today with missing dependencies. >>> > > > > > >>> > > > > > The plan: we need to let mrepo finish "dry run" downloading of >>> Fedora >>> > > > > > 27 and then we need to reexecute it with proper flags: >>> > > > > > >>> > > > > > cat /usr/local/bin/koji-sync-external-repos >>> > > > > > #!/bin/bash >>> > > > > > mrepo -qugf >>> > > > > > koji list-targets --quiet | awk '{print $2}' | sort -u | xargs >>> -I '{}' >>> > > > > > koji regen-repo --nowait '{}' >>> > > > > > >>> > > > > > Note the -u -g -f flags for mrepo, this will cause all metadata >>> to be >>> > > > > > regenerated. Once this command is done, koji will see missing >>> > > > > > dependencies to appear. >>> > > > > > >>> > > > > > The dry run of mrepo is running in screen and it is about to >>> finish >>> > > > > > hopefully soon. I will defer this to Eric in case it won't >>> finish by >>> > > > > > EMEA EOB. >>> > > > > > >>> > > > > > Sorry for inconvenience, but dry run should be dry run, right. >>> > > > > > >>> > > > > > -- >>> > > > > > Later, >>> > > > > > Lukas @lzap Zapletal >>> > > > > > >>> > > > > > -- >>> > > > > > You received this message because you are subscribed to the >>> Google Groups >>> > > > > > "foreman-dev" group. >>> > > > > > To unsubscribe from this group and stop receiving emails from >>> it, send an >>> > > > > > email to foreman-dev+unsubscribe@googlegroups.com. >>> > > > > > For more options, visit https://groups.google.com/d/optout. >>> > > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > -- >>> > > > > Eric D. Helms >>> > > > > Red Hat Engineering >>> > > > > >>> > > > > -- >>> > > > > You received this message because you are subscribed to the >>> Google Groups >>> > > > > "foreman-dev" group. >>> > > > > To unsubscribe from this group and stop receiving emails from it, >>> send an >>> > > > > email to foreman-dev+unsubscribe@googlegroups.com. >>> > > > > For more options, visit https://groups.google.com/d/optout. >>> > > > >>> > > > >>> > > > -- >>> > > > You received this message because you are subscribed to the Google >>> Groups >>> > > > "foreman-dev" group. >>> > > > To unsubscribe from this group and stop receiving emails from it, >>> send an >>> > > > email to foreman-dev+unsubscribe@googlegroups.com. >>> > > > For more options, visit https://groups.google.com/d/optout. >>> > > >>> > > >>> > > >>> > > -- >>> > > Later, >>> > > Lukas @lzap Zapletal >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "foreman-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to foreman-dev+unsubscribe@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Eric D. Helms >> Red Hat Engineering >> > > > >-- >Eric D. Helms >Red Hat Engineering > >-- >You received this message because you are subscribed to the Google Groups "foreman-dev" group. >To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. >For more options, visit https://groups.google.com/d/optout.
Thanks Eric, I added this section about the process for the future:

http://projects.theforeman.org/projects/foreman/wiki/KojiSetup#Adding-external-repository

··· On Wed, Nov 15, 2017 at 9:57 PM, Eric D Helms <ericdhelms@gmail.com> wrote:
This now appears to be working again. One out standing question that affects
nodejs- packaging builds.

As part of the RHEL 7.4 release, http-parser was removed from EPEL. With
this latest round of Koji external repositories update we now have a newer
EPEL with this package removed. We did not update our CentOS external
repository that would include this package. Not updating the base OS has
been our policy for a while now it would seem. We have two options:

1) Sync and update CentOS
2) Add http-parser to the overrides tag for foreman-nightly

On Wed, Nov 15, 2017 at 2:11 PM, Eric D Helms <ericdhelms@gmail.com> wrote:

I had forgotten to run the repo-regen on Koji. That is finishing up now. I
will run another test after and report back.

On Wed, Nov 15, 2017 at 2:08 PM, Patrick Creech <pcreech@redhat.com> >> wrote:

On Wed, 2017-11-15 at 14:00 -0500, Patrick Creech wrote:
On Wed, 2017-11-15 at 19:56 +0100, Lukas Zapletal wrote:
I do not have access here, can someone take a look? Are repodata
present for EPEL7?

If I have the correct path, it does not appear so:

[root@koji repodata]# pwd
/mnt/koji/external-repos/epel7-x86_64/epel/repodata
[root@koji repodata]# ls
[root@koji repodata]#

I might not have, this path DOES have repodata:

/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/repodata

LZ

On Wed, Nov 15, 2017 at 6:15 PM, Ewoud Kohl van Wijngaarden >>> > > <ewoud@kohlvanwijngaarden.nl> wrote:
I still get errors when building:

http://koji.katello.org/koji/taskinfo?taskID=52363
http://koji.katello.org/kojifiles/work/tasks/2363/52363/root.log

DEBUG util.py:439: Error downloading packages:
DEBUG util.py:439: 1:nodejs-6.10.3-1.el7.x86_64: failed to
retrieve
nodejs-6.10.3-1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:

/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-6.10.3-1.el7.x86_64.rpm
DEBUG util.py:439: http-parser-2.7.1-3.el7.x86_64: failed to
retrieve
http-parser-2.7.1-3.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:

/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-parser-2.7.1-3.el7.x86_64.rpm
DEBUG util.py:439: 1:npm-3.10.10-1.6.10.3.1.el7.x86_64: failed
to
retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:

/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm

On Wed, Nov 15, 2017 at 11:44:24AM -0500, Eric D Helms wrote:

Looks like everything is back up and working as expected. For
packagers,
keep in mind that this updated some of our external repositories
to their
latest versions if you see any oddities. This also means that
Fedora 27 is
available as an external repository for Pulp and Katello clients.

On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal <lzap@redhat.com> >>> > > > > wrote:

Hey,

when I initiated mrepo -n (dry run) this morning to see how
mrepo
works on our koji in order to test if we are able to add Fedora
27 for
Pulp, I learned that this "dry run" is actually a real run and
mrepo
initiated full resync of our content without metadata
regeneration.

This rendered our external repositories unusable as all
metadata were
deleted. All builds will fail today with missing dependencies.

The plan: we need to let mrepo finish "dry run" downloading of
Fedora
27 and then we need to reexecute it with proper flags:

cat /usr/local/bin/koji-sync-external-repos
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk '{print $2}' | sort -u | xargs
-I '{}'
koji regen-repo --nowait '{}'

Note the -u -g -f flags for mrepo, this will cause all metadata
to be
regenerated. Once this command is done, koji will see missing
dependencies to appear.

The dry run of mrepo is running in screen and it is about to
finish
hopefully soon. I will defer this to Eric in case it won't
finish by
EMEA EOB.

Sorry for inconvenience, but dry run should be dry run, right.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the
Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from
it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Eric D. Helms
Red Hat Engineering

--
You received this message because you are subscribed to the
Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Eric D. Helms
Red Hat Engineering

--
Eric D. Helms
Red Hat Engineering

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Later,
  Lukas @lzap Zapletal
I would be against syncing regularly but I don't do much packaging so it's up to you. The problem I see is "random" breakage. You come to work on Monday after sync and if you don't realize the package was deleted from EPEL, you can burn hour or something to figure it out sometimes. This was pretty clear case but as you know there are situations (SCL) when it is not that clear.

If we ever deploy new koji, I'd vote to avoid getting OS updates completely, just base channels and explicit updates when we need it. We need reproducible builds, with autosync we are not getting it.

One note, if we sync CentOS or RHEL base to particular version, chances are that SELinux won't load on older releases. So basically we must be ready to do minimum requirements change here.

Thanks for solving the problem.

LZ


··· On Wed, Nov 15, 2017 at 10:05 PM, Ewoud Kohl van Wijngaarden <ewoud@kohlvanwijngaarden.nl> wrote:
I think we should sync & update. CentOS has no concept of supported minor
releases and we should be testing with a supported release.

On Wed, Nov 15, 2017 at 03:57:40PM -0500, Eric D Helms wrote:

This now appears to be working again. One out standing question that
affects nodejs- packaging builds.

As part of the RHEL 7.4 release, http-parser was removed from EPEL. With
this latest round of Koji external repositories update we now have a newer
EPEL with this package removed. We did not update our CentOS external
repository that would include this package. Not updating the base OS has
been our policy for a while now it would seem. We have two options:

1) Sync and update CentOS
2) Add http-parser to the overrides tag for foreman-nightly

On Wed, Nov 15, 2017 at 2:11 PM, Eric D Helms <ericdhelms@gmail.com> >> wrote:

I had forgotten to run the repo-regen on Koji. That is finishing up now.
I
will run another test after and report back.

On Wed, Nov 15, 2017 at 2:08 PM, Patrick Creech <pcreech@redhat.com> >>> wrote:

On Wed, 2017-11-15 at 14:00 -0500, Patrick Creech wrote:
On Wed, 2017-11-15 at 19:56 +0100, Lukas Zapletal wrote:
I do not have access here, can someone take a look? Are repodata
present for EPEL7?

If I have the correct path, it does not appear so:

[root@koji repodata]# pwd
/mnt/koji/external-repos/epel7-x86_64/epel/repodata
[root@koji repodata]# ls
[root@koji repodata]#

I might not have, this path DOES have repodata:

/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/repodata

LZ

On Wed, Nov 15, 2017 at 6:15 PM, Ewoud Kohl van Wijngaarden >>>> > > <ewoud@kohlvanwijngaarden.nl> wrote:
I still get errors when building:

http://koji.katello.org/koji/taskinfo?taskID=52363
http://koji.katello.org/kojifiles/work/tasks/2363/52363/root.log

DEBUG util.py:439: Error downloading packages:
DEBUG util.py:439: 1:nodejs-6.10.3-1.el7.x86_64: failed to
retrieve
nodejs-6.10.3-1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-6
.10.3-1.el7.x86_64.rpm
DEBUG util.py:439: http-parser-2.7.1-3.el7.x86_64: failed to
retrieve
http-parser-2.7.1-3.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-par
ser-2.7.1-3.el7.x86_64.rpm
DEBUG util.py:439: 1:npm-3.10.10-1.6.10.3.1.el7.x86_64: failed
to
retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.10
.10-1.6.10.3.1.el7.x86_64.rpm

On Wed, Nov 15, 2017 at 11:44:24AM -0500, Eric D Helms wrote:

Looks like everything is back up and working as expected. For
packagers,
keep in mind that this updated some of our external repositories
to their
latest versions if you see any oddities. This also means that
Fedora 27 is
available as an external repository for Pulp and Katello
clients.

On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal >>>> > > > > <lzap@redhat.com> >>>> wrote:

Hey,

when I initiated mrepo -n (dry run) this morning to see how
mrepo
works on our koji in order to test if we are able to add
Fedora
27 for
Pulp, I learned that this "dry run" is actually a real run and
mrepo
initiated full resync of our content without metadata
regeneration.

This rendered our external repositories unusable as all
metadata were
deleted. All builds will fail today with missing dependencies.

The plan: we need to let mrepo finish "dry run" downloading of
Fedora
27 and then we need to reexecute it with proper flags:

cat /usr/local/bin/koji-sync-external-repos
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk '{print $2}' | sort -u | xargs
-I '{}'
koji regen-repo --nowait '{}'

Note the -u -g -f flags for mrepo, this will cause all
metadata
to be
regenerated. Once this command is done, koji will see missing
dependencies to appear.

The dry run of mrepo is running in screen and it is about to
finish
hopefully soon. I will defer this to Eric in case it won't
finish by
EMEA EOB.

Sorry for inconvenience, but dry run should be dry run, right.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the
Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from
it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Eric D. Helms
Red Hat Engineering

--
You received this message because you are subscribed to the
Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from
it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Eric D. Helms
Red Hat Engineering

--
Eric D. Helms
Red Hat Engineering

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Later,
  Lukas @lzap Zapletal
Those sound like good arguments, but if we don't test on older releases then we don't know for sure they work either. IMHO it's acceptable to explicitly only support EL 7.4 and up. We don't have the resources to support older versions especially since CentOS doesn't support that either. If users want verified working versions on EL < 7.4 then they should either help us properly support it or buy support from a vendor that does want to promise it works.


··· On Thu, Nov 16, 2017 at 08:56:39AM +0100, Lukas Zapletal wrote:
I would be against syncing regularly but I don't do much packaging so
it's up to you. The problem I see is "random" breakage. You come to
work on Monday after sync and if you don't realize the package was
deleted from EPEL, you can burn hour or something to figure it out
sometimes. This was pretty clear case but as you know there are
situations (SCL) when it is not that clear.

If we ever deploy new koji, I'd vote to avoid getting OS updates
completely, just base channels and explicit updates when we need it.
We need reproducible builds, with autosync we are not getting it.

One note, if we sync CentOS or RHEL base to particular version,
chances are that SELinux won't load on older releases. So basically we
must be ready to do minimum requirements change here.

Thanks for solving the problem.

LZ

On Wed, Nov 15, 2017 at 10:05 PM, Ewoud Kohl van Wijngaarden ><ewoud@kohlvanwijngaarden.nl> wrote:
I think we should sync & update. CentOS has no concept of supported minor
releases and we should be testing with a supported release.

On Wed, Nov 15, 2017 at 03:57:40PM -0500, Eric D Helms wrote:

This now appears to be working again. One out standing question that
affects nodejs- packaging builds.

As part of the RHEL 7.4 release, http-parser was removed from EPEL. With
this latest round of Koji external repositories update we now have a newer
EPEL with this package removed. We did not update our CentOS external
repository that would include this package. Not updating the base OS has
been our policy for a while now it would seem. We have two options:

1) Sync and update CentOS
2) Add http-parser to the overrides tag for foreman-nightly

On Wed, Nov 15, 2017 at 2:11 PM, Eric D Helms <ericdhelms@gmail.com> >>> wrote:

I had forgotten to run the repo-regen on Koji. That is finishing up now.
I
will run another test after and report back.

On Wed, Nov 15, 2017 at 2:08 PM, Patrick Creech <pcreech@redhat.com> >>>> wrote:

On Wed, 2017-11-15 at 14:00 -0500, Patrick Creech wrote:
On Wed, 2017-11-15 at 19:56 +0100, Lukas Zapletal wrote:
I do not have access here, can someone take a look? Are repodata
present for EPEL7?

If I have the correct path, it does not appear so:

[root@koji repodata]# pwd
/mnt/koji/external-repos/epel7-x86_64/epel/repodata
[root@koji repodata]# ls
[root@koji repodata]#

I might not have, this path DOES have repodata:

/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/repodata

LZ

On Wed, Nov 15, 2017 at 6:15 PM, Ewoud Kohl van Wijngaarden >>>>> > > <ewoud@kohlvanwijngaarden.nl> wrote:
I still get errors when building:

http://koji.katello.org/koji/taskinfo?taskID=52363
http://koji.katello.org/kojifiles/work/tasks/2363/52363/root.log

DEBUG util.py:439: Error downloading packages:
DEBUG util.py:439: 1:nodejs-6.10.3-1.el7.x86_64: failed to
retrieve
nodejs-6.10.3-1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-6
.10.3-1.el7.x86_64.rpm
DEBUG util.py:439: http-parser-2.7.1-3.el7.x86_64: failed to
retrieve
http-parser-2.7.1-3.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-par
ser-2.7.1-3.el7.x86_64.rpm
DEBUG util.py:439: 1:npm-3.10.10-1.6.10.3.1.el7.x86_64: failed
to
retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.10
.10-1.6.10.3.1.el7.x86_64.rpm

On Wed, Nov 15, 2017 at 11:44:24AM -0500, Eric D Helms wrote:

Looks like everything is back up and working as expected. For
packagers,
keep in mind that this updated some of our external repositories
to their
latest versions if you see any oddities. This also means that
Fedora 27 is
available as an external repository for Pulp and Katello
clients.

On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal >>>>> > > > > <lzap@redhat.com> >>>>> wrote:

Hey,

when I initiated mrepo -n (dry run) this morning to see how
mrepo
works on our koji in order to test if we are able to add
Fedora
27 for
Pulp, I learned that this "dry run" is actually a real run and
mrepo
initiated full resync of our content without metadata
regeneration.

This rendered our external repositories unusable as all
metadata were
deleted. All builds will fail today with missing dependencies.

The plan: we need to let mrepo finish "dry run" downloading of
Fedora
27 and then we need to reexecute it with proper flags:

cat /usr/local/bin/koji-sync-external-repos
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk '{print $2}' | sort -u | xargs
-I '{}'
koji regen-repo --nowait '{}'

Note the -u -g -f flags for mrepo, this will cause all
metadata
to be
regenerated. Once this command is done, koji will see missing
dependencies to appear.

The dry run of mrepo is running in screen and it is about to
finish
hopefully soon. I will defer this to Eric in case it won't
finish by
EMEA EOB.

Sorry for inconvenience, but dry run should be dry run, right.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the
Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from
it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Eric D. Helms
Red Hat Engineering

--
You received this message because you are subscribed to the
Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from
it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Eric D. Helms
Red Hat Engineering

--
Eric D. Helms
Red Hat Engineering

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google Groups "foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
I started an mrepo run after adding CentOS in a screen session. The run appears to be running a lot longer than previous and it's not clear to me what exactly it's doing at present or whats safe to do. Lukas could you check on it in your morning and see if you spot anything odd about why it seems to have stalled or slowed down?


··· On Thu, Nov 16, 2017 at 6:01 AM, Ewoud Kohl van Wijngaarden < ewoud@kohlvanwijngaarden.nl> wrote:

Those sound like good arguments, but if we don't test on older releases
then we don't know for sure they work either. IMHO it's acceptable to
explicitly only support EL 7.4 and up. We don't have the resources to
support older versions especially since CentOS doesn't support that either.
If users want verified working versions on EL < 7.4 then they should either
help us properly support it or buy support from a vendor that does want to
promise it works.

On Thu, Nov 16, 2017 at 08:56:39AM +0100, Lukas Zapletal wrote:

I would be against syncing regularly but I don't do much packaging so
it's up to you. The problem I see is "random" breakage. You come to
work on Monday after sync and if you don't realize the package was
deleted from EPEL, you can burn hour or something to figure it out
sometimes. This was pretty clear case but as you know there are
situations (SCL) when it is not that clear.

If we ever deploy new koji, I'd vote to avoid getting OS updates
completely, just base channels and explicit updates when we need it.
We need reproducible builds, with autosync we are not getting it.

One note, if we sync CentOS or RHEL base to particular version,
chances are that SELinux won't load on older releases. So basically we
must be ready to do minimum requirements change here.

Thanks for solving the problem.

LZ

On Wed, Nov 15, 2017 at 10:05 PM, Ewoud Kohl van Wijngaarden >> <ewoud@kohlvanwijngaarden.nl> wrote:

I think we should sync & update. CentOS has no concept of supported minor
releases and we should be testing with a supported release.

On Wed, Nov 15, 2017 at 03:57:40PM -0500, Eric D Helms wrote:

This now appears to be working again. One out standing question that
affects nodejs- packaging builds.

As part of the RHEL 7.4 release, http-parser was removed from EPEL. With
this latest round of Koji external repositories update we now have a
newer
EPEL with this package removed. We did not update our CentOS external
repository that would include this package. Not updating the base OS has
been our policy for a while now it would seem. We have two options:

1) Sync and update CentOS
2) Add http-parser to the overrides tag for foreman-nightly

On Wed, Nov 15, 2017 at 2:11 PM, Eric D Helms <ericdhelms@gmail.com> >>>> wrote:

I had forgotten to run the repo-regen on Koji. That is finishing up now.
I
will run another test after and report back.

On Wed, Nov 15, 2017 at 2:08 PM, Patrick Creech <pcreech@redhat.com> >>>>> wrote:

On Wed, 2017-11-15 at 14:00 -0500, Patrick Creech wrote:
On Wed, 2017-11-15 at 19:56 +0100, Lukas Zapletal wrote:
I do not have access here, can someone take a look? Are repodata
present for EPEL7?

If I have the correct path, it does not appear so:

[root@koji repodata]# pwd
/mnt/koji/external-repos/epel7-x86_64/epel/repodata
[root@koji repodata]# ls
[root@koji repodata]#

I might not have, this path DOES have repodata:

/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/repodata

LZ

On Wed, Nov 15, 2017 at 6:15 PM, Ewoud Kohl van Wijngaarden >>>>>> > > <ewoud@kohlvanwijngaarden.nl> wrote:
I still get errors when building:

http://koji.katello.org/koji/taskinfo?taskID=52363
http://koji.katello.org/kojifiles/work/tasks/2363/52363/
root.log

DEBUG util.py:439: Error downloading packages:
DEBUG util.py:439: 1:nodejs-6.10.3-1.el7.x86_64: failed to
retrieve
nodejs-6.10.3-1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not
exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-6
.10.3-1.el7.x86_64.rpm
DEBUG util.py:439: http-parser-2.7.1-3.el7.x86_64: failed to
retrieve
http-parser-2.7.1-3.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not
exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-par
ser-2.7.1-3.el7.x86_64.rpm
DEBUG util.py:439: 1:npm-3.10.10-1.6.10.3.1.el7.x86_64:
failed
to
retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not
exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.10
.10-1.6.10.3.1.el7.x86_64.rpm

On Wed, Nov 15, 2017 at 11:44:24AM -0500, Eric D Helms wrote:

Looks like everything is back up and working as expected. For
packagers,
keep in mind that this updated some of our external
repositories
to their
latest versions if you see any oddities. This also means that
Fedora 27 is
available as an external repository for Pulp and Katello
clients.

On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal >>>>>> > > > > <lzap@redhat.com> >>>>>> wrote:

Hey,

when I initiated mrepo -n (dry run) this morning to see how
mrepo
works on our koji in order to test if we are able to add
Fedora
27 for
Pulp, I learned that this "dry run" is actually a real run
and
mrepo
initiated full resync of our content without metadata
regeneration.

This rendered our external repositories unusable as all
metadata were
deleted. All builds will fail today with missing
dependencies.

The plan: we need to let mrepo finish "dry run" downloading
of
Fedora
27 and then we need to reexecute it with proper flags:

cat /usr/local/bin/koji-sync-external-repos
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk '{print $2}' | sort -u |
xargs
-I '{}'
koji regen-repo --nowait '{}'

Note the -u -g -f flags for mrepo, this will cause all
metadata
to be
regenerated. Once this command is done, koji will see
missing
dependencies to appear.

The dry run of mrepo is running in screen and it is about to
finish
hopefully soon. I will defer this to Eric in case it won't
finish by
EMEA EOB.

Sorry for inconvenience, but dry run should be dry run,
right.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the
Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails
from
it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Eric D. Helms
Red Hat Engineering

--
You received this message because you are subscribed to the
Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from
it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from
it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Eric D. Helms
Red Hat Engineering

--
Eric D. Helms
Red Hat Engineering

--
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Eric D. Helms
Red Hat Engineering
The update seems to have completed, with the new CentOS repository added to Koji and repositories regenerated. For the moment, the
foreman-nightly*-rhel7-build tags have been updated to remove rhel as an external repository and have been replaced with CentOS (7.4). If you notice any oddities building, please let us know so we can work through them.

Eric


··· On Thu, Nov 16, 2017 at 7:14 PM, Eric D Helms <ericdhelms@gmail.com> wrote:

I started an mrepo run after adding CentOS in a screen session. The run
appears to be running a lot longer than previous and it's not clear to me
what exactly it's doing at present or whats safe to do. Lukas could you
check on it in your morning and see if you spot anything odd about why it
seems to have stalled or slowed down?

On Thu, Nov 16, 2017 at 6:01 AM, Ewoud Kohl van Wijngaarden < > ewoud@kohlvanwijngaarden.nl> wrote:

Those sound like good arguments, but if we don't test on older releases
then we don't know for sure they work either. IMHO it's acceptable to
explicitly only support EL 7.4 and up. We don't have the resources to
support older versions especially since CentOS doesn't support that either.
If users want verified working versions on EL < 7.4 then they should either
help us properly support it or buy support from a vendor that does want to
promise it works.

On Thu, Nov 16, 2017 at 08:56:39AM +0100, Lukas Zapletal wrote:

I would be against syncing regularly but I don't do much packaging so
it's up to you. The problem I see is "random" breakage. You come to
work on Monday after sync and if you don't realize the package was
deleted from EPEL, you can burn hour or something to figure it out
sometimes. This was pretty clear case but as you know there are
situations (SCL) when it is not that clear.

If we ever deploy new koji, I'd vote to avoid getting OS updates
completely, just base channels and explicit updates when we need it.
We need reproducible builds, with autosync we are not getting it.

One note, if we sync CentOS or RHEL base to particular version,
chances are that SELinux won't load on older releases. So basically we
must be ready to do minimum requirements change here.

Thanks for solving the problem.

LZ

On Wed, Nov 15, 2017 at 10:05 PM, Ewoud Kohl van Wijngaarden >>> <ewoud@kohlvanwijngaarden.nl> wrote:

I think we should sync & update. CentOS has no concept of supported
minor
releases and we should be testing with a supported release.

On Wed, Nov 15, 2017 at 03:57:40PM -0500, Eric D Helms wrote:

This now appears to be working again. One out standing question that
affects nodejs- packaging builds.

As part of the RHEL 7.4 release, http-parser was removed from EPEL.
With
this latest round of Koji external repositories update we now have a
newer
EPEL with this package removed. We did not update our CentOS external
repository that would include this package. Not updating the base OS
has
been our policy for a while now it would seem. We have two options:

1) Sync and update CentOS
2) Add http-parser to the overrides tag for foreman-nightly

On Wed, Nov 15, 2017 at 2:11 PM, Eric D Helms <ericdhelms@gmail.com> >>>>> wrote:

I had forgotten to run the repo-regen on Koji. That is finishing up
now.
I
will run another test after and report back.

On Wed, Nov 15, 2017 at 2:08 PM, Patrick Creech <pcreech@redhat.com> >>>>>> wrote:

On Wed, 2017-11-15 at 14:00 -0500, Patrick Creech wrote:
On Wed, 2017-11-15 at 19:56 +0100, Lukas Zapletal wrote:
I do not have access here, can someone take a look? Are repodata
present for EPEL7?

If I have the correct path, it does not appear so:

[root@koji repodata]# pwd
/mnt/koji/external-repos/epel7-x86_64/epel/repodata
[root@koji repodata]# ls
[root@koji repodata]#

I might not have, this path DOES have repodata:

/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/repodata

LZ

On Wed, Nov 15, 2017 at 6:15 PM, Ewoud Kohl van Wijngaarden >>>>>>> > > <ewoud@kohlvanwijngaarden.nl> wrote:
I still get errors when building:

http://koji.katello.org/koji/taskinfo?taskID=52363
http://koji.katello.org/kojifiles/work/tasks/2363/52363/root
.log

DEBUG util.py:439: Error downloading packages:
DEBUG util.py:439: 1:nodejs-6.10.3-1.el7.x86_64: failed to
retrieve
nodejs-6.10.3-1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not
exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-6
.10.3-1.el7.x86_64.rpm
DEBUG util.py:439: http-parser-2.7.1-3.el7.x86_64: failed
to
retrieve
http-parser-2.7.1-3.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not
exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-par
ser-2.7.1-3.el7.x86_64.rpm
DEBUG util.py:439: 1:npm-3.10.10-1.6.10.3.1.el7.x86_64:
failed
to
retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not
exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.10
.10-1.6.10.3.1.el7.x86_64.rpm

On Wed, Nov 15, 2017 at 11:44:24AM -0500, Eric D Helms wrote:

Looks like everything is back up and working as expected. For
packagers,
keep in mind that this updated some of our external
repositories
to their
latest versions if you see any oddities. This also means that
Fedora 27 is
available as an external repository for Pulp and Katello
clients.

On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal >>>>>>> > > > > <lzap@redhat.com> >>>>>>> wrote:

Hey,

when I initiated mrepo -n (dry run) this morning to see how
mrepo
works on our koji in order to test if we are able to add
Fedora
27 for
Pulp, I learned that this "dry run" is actually a real run
and
mrepo
initiated full resync of our content without metadata
regeneration.

This rendered our external repositories unusable as all
metadata were
deleted. All builds will fail today with missing
dependencies.

The plan: we need to let mrepo finish "dry run"
downloading of
Fedora
27 and then we need to reexecute it with proper flags:

cat /usr/local/bin/koji-sync-external-repos
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk '{print $2}' | sort -u |
xargs
-I '{}'
koji regen-repo --nowait '{}'

Note the -u -g -f flags for mrepo, this will cause all
metadata
to be
regenerated. Once this command is done, koji will see
missing
dependencies to appear.

The dry run of mrepo is running in screen and it is about
to
finish
hopefully soon. I will defer this to Eric in case it won't
finish by
EMEA EOB.

Sorry for inconvenience, but dry run should be dry run,
right.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the
Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails
from
it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout
.

--
Eric D. Helms
Red Hat Engineering

--
You received this message because you are subscribed to the
Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from
it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from
it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it,
send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Eric D. Helms
Red Hat Engineering

--
Eric D. Helms
Red Hat Engineering

--
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google
Groups "foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Eric D. Helms
Red Hat Engineering



--
Eric D. Helms
Red Hat Engineering
Sorry we had public holiday here.

We use mirrorselect with mrepo, sometimes it can pick slower mirror. Would be safer to pick particular mirrors if we plan to sync regularly, ideally something within AWS datacenters if there's any.

LZ


··· On Fri, Nov 17, 2017 at 4:53 PM, Eric D Helms <ericdhelms@gmail.com> wrote:
The update seems to have completed, with the new CentOS repository added to
Koji and repositories regenerated. For the moment, the
foreman-nightly*-rhel7-build tags have been updated to remove rhel as an
external repository and have been replaced with CentOS (7.4). If you notice
any oddities building, please let us know so we can work through them.

Eric

On Thu, Nov 16, 2017 at 7:14 PM, Eric D Helms <ericdhelms@gmail.com> wrote:

I started an mrepo run after adding CentOS in a screen session. The run
appears to be running a lot longer than previous and it's not clear to me
what exactly it's doing at present or whats safe to do. Lukas could you
check on it in your morning and see if you spot anything odd about why it
seems to have stalled or slowed down?

On Thu, Nov 16, 2017 at 6:01 AM, Ewoud Kohl van Wijngaarden >> <ewoud@kohlvanwijngaarden.nl> wrote:

Those sound like good arguments, but if we don't test on older releases
then we don't know for sure they work either. IMHO it's acceptable to
explicitly only support EL 7.4 and up. We don't have the resources to
support older versions especially since CentOS doesn't support that either.
If users want verified working versions on EL < 7.4 then they should either
help us properly support it or buy support from a vendor that does want to
promise it works.

On Thu, Nov 16, 2017 at 08:56:39AM +0100, Lukas Zapletal wrote:

I would be against syncing regularly but I don't do much packaging so
it's up to you. The problem I see is "random" breakage. You come to
work on Monday after sync and if you don't realize the package was
deleted from EPEL, you can burn hour or something to figure it out
sometimes. This was pretty clear case but as you know there are
situations (SCL) when it is not that clear.

If we ever deploy new koji, I'd vote to avoid getting OS updates
completely, just base channels and explicit updates when we need it.
We need reproducible builds, with autosync we are not getting it.

One note, if we sync CentOS or RHEL base to particular version,
chances are that SELinux won't load on older releases. So basically we
must be ready to do minimum requirements change here.

Thanks for solving the problem.

LZ

On Wed, Nov 15, 2017 at 10:05 PM, Ewoud Kohl van Wijngaarden >>>> <ewoud@kohlvanwijngaarden.nl> wrote:

I think we should sync & update. CentOS has no concept of supported
minor
releases and we should be testing with a supported release.

On Wed, Nov 15, 2017 at 03:57:40PM -0500, Eric D Helms wrote:

This now appears to be working again. One out standing question that
affects nodejs- packaging builds.

As part of the RHEL 7.4 release, http-parser was removed from EPEL.
With
this latest round of Koji external repositories update we now have a
newer
EPEL with this package removed. We did not update our CentOS external
repository that would include this package. Not updating the base OS
has
been our policy for a while now it would seem. We have two options:

1) Sync and update CentOS
2) Add http-parser to the overrides tag for foreman-nightly

On Wed, Nov 15, 2017 at 2:11 PM, Eric D Helms <ericdhelms@gmail.com> >>>>>> wrote:

I had forgotten to run the repo-regen on Koji. That is finishing up
now.
I
will run another test after and report back.

On Wed, Nov 15, 2017 at 2:08 PM, Patrick Creech <pcreech@redhat.com> >>>>>>> wrote:

On Wed, 2017-11-15 at 14:00 -0500, Patrick Creech wrote:
On Wed, 2017-11-15 at 19:56 +0100, Lukas Zapletal wrote:
I do not have access here, can someone take a look? Are repodata
present for EPEL7?

If I have the correct path, it does not appear so:

[root@koji repodata]# pwd
/mnt/koji/external-repos/epel7-x86_64/epel/repodata
[root@koji repodata]# ls
[root@koji repodata]#

I might not have, this path DOES have repodata:

/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/repodata

LZ

On Wed, Nov 15, 2017 at 6:15 PM, Ewoud Kohl van Wijngaarden >>>>>>>> > > <ewoud@kohlvanwijngaarden.nl> wrote:
I still get errors when building:

http://koji.katello.org/koji/taskinfo?taskID=52363

http://koji.katello.org/kojifiles/work/tasks/2363/52363/root.log

DEBUG util.py:439: Error downloading packages:
DEBUG util.py:439: 1:nodejs-6.10.3-1.el7.x86_64: failed to
retrieve
nodejs-6.10.3-1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not
exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-6
.10.3-1.el7.x86_64.rpm
DEBUG util.py:439: http-parser-2.7.1-3.el7.x86_64: failed
to
retrieve
http-parser-2.7.1-3.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not
exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-par
ser-2.7.1-3.el7.x86_64.rpm
DEBUG util.py:439: 1:npm-3.10.10-1.6.10.3.1.el7.x86_64:
failed
to
retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build
DEBUG util.py:439: error was [Errno 2] Local file does not
exist:
/mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.10
.10-1.6.10.3.1.el7.x86_64.rpm

On Wed, Nov 15, 2017 at 11:44:24AM -0500, Eric D Helms wrote:

Looks like everything is back up and working as expected.
For
packagers,
keep in mind that this updated some of our external
repositories
to their
latest versions if you see any oddities. This also means
that
Fedora 27 is
available as an external repository for Pulp and Katello
clients.

On Wed, Nov 15, 2017 at 9:32 AM, Lukas Zapletal >>>>>>>> > > > > <lzap@redhat.com> >>>>>>>> wrote:

Hey,

when I initiated mrepo -n (dry run) this morning to see
how
mrepo
works on our koji in order to test if we are able to add
Fedora
27 for
Pulp, I learned that this "dry run" is actually a real run
and
mrepo
initiated full resync of our content without metadata
regeneration.

This rendered our external repositories unusable as all
metadata were
deleted. All builds will fail today with missing
dependencies.

The plan: we need to let mrepo finish "dry run"
downloading of
Fedora
27 and then we need to reexecute it with proper flags:

cat /usr/local/bin/koji-sync-external-repos
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk '{print $2}' | sort -u |
xargs
-I '{}'
koji regen-repo --nowait '{}'

Note the -u -g -f flags for mrepo, this will cause all
metadata
to be
regenerated. Once this command is done, koji will see
missing
dependencies to appear.

The dry run of mrepo is running in screen and it is about
to
finish
hopefully soon. I will defer this to Eric in case it won't
finish by
EMEA EOB.

Sorry for inconvenience, but dry run should be dry run,
right.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to
the
Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails
from
it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit
https://groups.google.com/d/optout.

--
Eric D. Helms
Red Hat Engineering

--
You received this message because you are subscribed to the
Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails
from
it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from
it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it,
send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Eric D. Helms
Red Hat Engineering

--
Eric D. Helms
Red Hat Engineering

--
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google
Groups "foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Eric D. Helms
Red Hat Engineering

--
Eric D. Helms
Red Hat Engineering

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Later,
  Lukas @lzap Zapletal