Koji is back online

Hello,

our koji is back online, it's fresh new instance with foreman, katello
(nightly, 1.15/3.4) and pulp tags related (nightly, 2.x series).

http://koji.katello.org/koji/

There should not be any changes necessary in your CLI configuration.
This is slightly newer version of Koji running on CentOS 7 but CLI is
compatible enough to do builds or tags.

We are now running one VM with 4 CPUs and 32 GB RAM and NVMe local
volume for faster build times, no NFS involved. Configuration was also
improved so all the important things like postgresql are on EBS
volume, we are working on setting up better backup and restore
procedures.

I apologise for inconvenience with this outage and I would like to
thank to everybody involved across Foreman, Katello and Pulp teams.

Please report all building issues and we will promptly solve them as
we go. It still might take some time for the DNS to propagate, but it
looks good from several locations.

··· -- Later, Lukas @lzap Zapletal

Lukas and Bryan big thanks for the time and effort getting this back
online. As well, the openness and status reports have been greatly
appreciated.

Eric

··· On Jun 13, 2017 9:44 AM, "Lukas Zapletal" wrote:

Hello,

our koji is back online, it’s fresh new instance with foreman, katello
(nightly, 1.15/3.4) and pulp tags related (nightly, 2.x series).

http://koji.katello.org/koji/

There should not be any changes necessary in your CLI configuration.
This is slightly newer version of Koji running on CentOS 7 but CLI is
compatible enough to do builds or tags.

We are now running one VM with 4 CPUs and 32 GB RAM and NVMe local
volume for faster build times, no NFS involved. Configuration was also
improved so all the important things like postgresql are on EBS
volume, we are working on setting up better backup and restore
procedures.

I apologise for inconvenience with this outage and I would like to
thank to everybody involved across Foreman, Katello and Pulp teams.

Please report all building issues and we will promptly solve them as
we go. It still might take some time for the DNS to propagate, but it
looks good from several locations.


Later,
Lukas @lzap Zapletal


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

+1 from me, on both counts. Thank you Lukas and Bryan!

··· On Tue, 2017-06-13 at 10:21 -0400, Eric D Helms wrote: > Lukas and Bryan big thanks for the time and effort getting this back online. As well, the openness > and status reports have been greatly appreciated. > > > Eric

> Lukas and Bryan big thanks for the time and effort getting this back
> online. As well, the openness and status reports have been greatly
> appreciated.
>

+1

Thanks Lukas and Bryan!

··· On Tue, Jun 13, 2017 at 5:21 PM, Eric D Helms wrote:

Eric

On Jun 13, 2017 9:44 AM, “Lukas Zapletal” lzap@redhat.com wrote:

Hello,

our koji is back online, it’s fresh new instance with foreman, katello
(nightly, 1.15/3.4) and pulp tags related (nightly, 2.x series).

http://koji.katello.org/koji/

There should not be any changes necessary in your CLI configuration.
This is slightly newer version of Koji running on CentOS 7 but CLI is
compatible enough to do builds or tags.

We are now running one VM with 4 CPUs and 32 GB RAM and NVMe local
volume for faster build times, no NFS involved. Configuration was also
improved so all the important things like postgresql are on EBS
volume, we are working on setting up better backup and restore
procedures.

I apologise for inconvenience with this outage and I would like to
thank to everybody involved across Foreman, Katello and Pulp teams.

Please report all building issues and we will promptly solve them as
we go. It still might take some time for the DNS to propagate, but it
looks good from several locations.


Later,
Lukas @lzap Zapletal


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Are the builds imported from the existing binaries?

There appears to be a large issue with many builds that have multiple
tags rather than a single tag that's inherited, which will cause
problems with older packages masking new ones. Can you re-check that
builds are only in one tag per version?

For example, foreman-1.15.0-1.el7 is in three tags: foreman-1.15,
plugins-1.15-nonscl and plugins-1.15, but should only be in the
foreman-1.15 tag. Ditto for many dependencies, e.g.
http://koji.katello.org/koji/buildinfo?buildID=15059 or
http://koji.katello.org/koji/buildinfo?buildID=15741.

The tags for 1.14 and 1.13 releases are necessary too for plugin builds.

Have tags been re-created from scratch? i.e. have blocked package lists
etc been lost? I see lots of very strange packages listed in the tags,
e.g. doc and other sub-binary packages, packages with hyphen suffixes,
non-SCL packages in SCL tags.

··· On 13/06/17 14:44, Lukas Zapletal wrote: > our koji is back online, it's fresh new instance with foreman [..]


Dominic Cleal
dominic@cleal.org

Dominic,

we took the simplest approach we could figure out during the weekend
of importing binary packages into new koji. Our primary goal was to
unblock nightly and stable builds. Here are notes for katello 3.4
rhel7 nightly.

  • Grab the set of packages in the most recent build
    (http://34.226.218.207/kojifiles/old_repos/katello-3.4-rhel7-build/latest/x86_64/pkglist)

  • Create a script to load the packages. I did this via:
    #cat katello_34_rhel7.txt | sed 's/.///' | sed
    's/-[^-]
    -[^-].[^.].[^.]*$//' | sort >
    add_packages_katello_34_rhel7.sh

  • Edit that file, and prepend the following to each line:
    koji add-pkg --owner kojiadm katello-3.4-rhel7

  • Create a script to load the builds. I did this via:
    #cat katello_34_rhel7.txt | sed 's/.*///' || sort >
    add_builds_katello_34_rhel7.sh

  • Edit the script to remove all the strings ".x86_64.rpm" and ".noarch.rpm"

  • Edit the script to prepend the following to each line:
    koji tag-build --nowait katello-3.4-rhel7

  • Run the first script, which will add the package mappings

  • Run the second script, which will ahve some failures due to nested
    binary rpms. These can be ignored.

Rinse and repeat for each repo.

··· ~~~

I haven’t noticed builds in multiple (incorrect) tags yet. We will
need to look into that. We do have a koji running from backup (summer
2016) with the last tags from 1.12 release, do you think comparing
these to the new structure and deleting all extra builds would do the
trick?

Tags have been recreated from scratch, yes.

I don’t know what blocked package list is, don’t know.

We have identified that subpackages are tagged which was not necessary
(e.g. -doc packages), we will fix this later but my understanding is
that it won’t hurt the build process. On TODO.

On Tue, Jun 13, 2017 at 10:41 PM, Dominic Cleal dominic@cleal.org wrote:

On 13/06/17 14:44, Lukas Zapletal wrote:

our koji is back online, it’s fresh new instance with foreman […]

Are the builds imported from the existing binaries?

There appears to be a large issue with many builds that have multiple tags
rather than a single tag that’s inherited, which will cause problems with
older packages masking new ones. Can you re-check that builds are only in
one tag per version?

For example, foreman-1.15.0-1.el7 is in three tags: foreman-1.15,
plugins-1.15-nonscl and plugins-1.15, but should only be in the foreman-1.15
tag. Ditto for many dependencies, e.g.
http://koji.katello.org/koji/buildinfo?buildID=15059 or
http://koji.katello.org/koji/buildinfo?buildID=15741.

The tags for 1.14 and 1.13 releases are necessary too for plugin builds.

Have tags been re-created from scratch? i.e. have blocked package lists etc
been lost? I see lots of very strange packages listed in the tags, e.g. doc
and other sub-binary packages, packages with hyphen suffixes, non-SCL
packages in SCL tags.


Dominic Cleal
dominic@cleal.org


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Later,
Lukas @lzap Zapletal

> Dominic,
>
> we took the simplest approach we could figure out during the weekend
> of importing binary packages into new koji. Our primary goal was to
> unblock nightly and stable builds. Here are notes for katello 3.4
> rhel7 nightly.
>

The approach below was taken for Foreman nightly and 1.15 and not Katello.
I realized the pkglist below shows the entire inherited list of packages
not just those specific to Katello and did not follow the procedure.
Furthermore, similar to Foreman we put some packages in different tags
(e.g. katello-thirdparty-rhel7 and katello-nightly-rhel7 have different
packages). In the case of the Foreman rebuild everything got tagged to
every tag. For example, foreman-installer exists in foreman-nightly-rhel7
and foreman-nightly-nonscl-rhel7 where it should only live in the latter
according to the prior breakdown. I grant you, Katello is a lot less
packages so deriving all of this was less painful but the add-pkg step had
to be done manually (the tag-build was easy once everything was added.)

E

··· On Wed, Jun 14, 2017 at 6:20 AM, Lukas Zapletal wrote:
  • Grab the set of packages in the most recent build
    (http://34.226.218.207/kojifiles/old_repos/katello-3.
    4-rhel7-build/latest/x86_64/pkglist)

  • Create a script to load the packages. I did this via:
    #cat katello_34_rhel7.txt | sed ‘s/.///’ | sed
    ’s/-[^-]
    -[^-].[^.].[^.]*$//’ | sort >
    add_packages_katello_34_rhel7.sh

  • Edit that file, and prepend the following to each line:
    koji add-pkg --owner kojiadm katello-3.4-rhel7

  • Create a script to load the builds. I did this via:
    #cat katello_34_rhel7.txt | sed ‘s/.*///’ || sort >
    add_builds_katello_34_rhel7.sh

  • Edit the script to remove all the strings “.x86_64.rpm” and “.noarch.rpm”

  • Edit the script to prepend the following to each line:
    koji tag-build --nowait katello-3.4-rhel7

  • Run the first script, which will add the package mappings

  • Run the second script, which will ahve some failures due to nested
    binary rpms. These can be ignored.

Rinse and repeat for each repo.


I haven't noticed builds in multiple (incorrect) tags yet. We will
need to look into that. We do have a koji running from backup (summer
2016) with the last tags from 1.12 release, do you think comparing
these to the new structure and deleting all extra builds would do the
trick?

Tags have been recreated from scratch, yes.

I don't know what blocked package list is, don't know.

We have identified that subpackages are tagged which was not necessary
(e.g. -doc packages), we will fix this later but my understanding is
that it won't hurt the build process. On TODO.

On Tue, Jun 13, 2017 at 10:41 PM, Dominic Cleal <dominic@cleal.org> wrote:
> On 13/06/17 14:44, Lukas Zapletal wrote:
>>
>> our koji is back online, it's fresh new instance with foreman [..]
>
>
> Are the builds imported from the existing binaries?
>
> There appears to be a large issue with many builds that have multiple
tags
> rather than a single tag that's inherited, which will cause problems with
> older packages masking new ones. Can you re-check that builds are only in
> one tag per version?
>
> For example, foreman-1.15.0-1.el7 is in three tags: foreman-1.15,
> plugins-1.15-nonscl and plugins-1.15, but should only be in the
foreman-1.15
> tag. Ditto for many dependencies, e.g.
> http://koji.katello.org/koji/buildinfo?buildID=15059 or
> http://koji.katello.org/koji/buildinfo?buildID=15741.
>
> The tags for 1.14 and 1.13 releases are necessary too for plugin builds.
>
> Have tags been re-created from scratch? i.e. have blocked package lists
etc
> been lost? I see lots of very strange packages listed in the tags, e.g.
doc
> and other sub-binary packages, packages with hyphen suffixes, non-SCL
> packages in SCL tags.
>
> --
> Dominic Cleal
> dominic@cleal.org
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Later,
  Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering

I assume the fix here is to go look at any package which is tagged into

foreman-nightly-$OS and then remove it from
foreman-plugins-nightly-$OS. I am not sure about nonscl vs regular rhel
7. It looks like those are peers so need to be double tagged?

– bk

··· On 06/14/2017 06:20 AM, Lukas Zapletal wrote: > I haven't noticed builds in multiple (incorrect) tags yet. We will > need to look into that. We do have a koji running from backup (summer > 2016) with the last tags from 1.12 release, do you think comparing > these to the new structure and deleting all extra builds would do the > trick?

Could you please check the failing foreman-installer builds? Some of the
external repositories appear to be missing, probably the Puppet PC1 repo
which IIRC was configured on foreman-nightly-fedora24 and
foreman-nightly-nonscl-rhel7. Older versions used different
configurations though.

> I haven't noticed builds in multiple (incorrect) tags yet. We will
> need to look into that. We do have a koji running from backup (summer
> 2016) with the last tags from 1.12 release, do you think comparing
> these to the new structure and deleting all extra builds would do the
> trick?

If you're trying to determine the correct tag, use the whitelists in
rel-eng/tito.props, they are correct on recent branches. I don't think
that comparing to an old release is a good idea, packages occasionally move.

··· On 14/06/17 11:20, Lukas Zapletal wrote:

On 06/14/2017 06:20 AM, Lukas Zapletal wrote:
I assume the fix here is to go look at any package which is tagged into

foreman-nightly-$OS and then remove it from foreman-plugins-nightly-$OS. I am not sure about nonscl vs regular rhel 7. > It looks like those are peers so need to be double tagged?

I don’t think any build should be tagged twice within one version. An
EL7 build should be tagged to foreman-nightly-rhel7 or
foreman-nightly-nonscl-rhel7.

There was tag inheritance set up between them, IIRC with foreman-release
in the build/srpm-build groups in both SCL and non-SCL tags. This looks
to be missing though.


Dominic Cleal
dominic@cleal.org

Hey

> Could you please check the failing foreman-installer builds? Some of the
> external repositories appear to be missing, probably the Puppet PC1 repo
> which IIRC was configured on foreman-nightly-fedora24 and
> foreman-nightly-nonscl-rhel7. Older versions used different configurations
> though.

Sure, that's for the heads up.

koji add-external-repo puppetlabs-pc1-rhel-7

file:///mnt/koji/external-repos/www/puppetlabs7-$arch/RPMS.pc1/
Created external repo 54

koji add-external-repo puppetlabs-pc1-fedora-24

file:///mnt/koji/external-repos/www/puppetlabsf24-$arch/RPMS.pc1/
Created external repo 55

koji add-external-repo -t foreman-nightly-nonscl-rhel7-build

puppetlabs-pc1-rhel-7
Added external repo puppetlabs-pc1-rhel-7 to tag
foreman-nightly-nonscl-rhel7-build (priority 20)

koji add-external-repo -t foreman-nightly-fedora24-build

puppetlabs-pc1-fedora-24
Added external repo puppetlabs-pc1-fedora-24 to tag
foreman-nightly-fedora24-build (priority 15)

koji add-external-repo -t foreman-1.15-nonscl-rhel7-build

puppetlabs-pc1-rhel-7
Added external repo puppetlabs-pc1-rhel-7 to tag
foreman-1.15-nonscl-rhel7-build (priority 20)

koji add-external-repo -t foreman-1.15-fedora24-build puppetlabs-pc1-fedora-24

Added external repo puppetlabs-pc1-fedora-24 to tag
foreman-1.15-fedora24-build (priority 15)

> If you're trying to determine the correct tag, use the whitelists in
> rel-eng/tito.props, they are correct on recent branches. I don't think that
> comparing to an old release is a good idea, packages occasionally move.

I am going to create a script that will compare koji tags against
whitelists and prepare statements, will send it out for review here
and then we can make the change. I would like to do this once 1.15.1
and 3.4.1 is out just for case.

> I don't think any build should be tagged twice within one version. An EL7
> build should be tagged to foreman-nightly-rhel7 or
> foreman-nightly-nonscl-rhel7.

Ok thanks.

> There was tag inheritance set up between them, IIRC with foreman-release in
> the build/srpm-build groups in both SCL and non-SCL tags. This looks to be
> missing though.

Not sure I get this one.

LZ