Review of koji packages/tags cleanup

lzap · June 15, 2017, 2:16pm

Hello,

I created a small script

gist.github.com

https://gist.github.com/lzap/cc6c100feb0a558f961eaa07e01bfa59

compare-with-tags.rb

#!/usr/bin/env ruby
require 'configparser'

cfg = ConfigParser.new('tito.props')
cfg.each do |entry|
  tag = entry.first
  scl = entry.last['scl']
  next unless tag =~ /^foreman/
  comp_pkgs = entry.last['whitelist'].split
  comp_pkgs.map!{ |x| (scl && x =~ /^rubygem-/) ? "#{scl}-#{x}" : x }

This file has been truncated. show original

which compares our whitelists agains koji. The first phase is packages
added to tags. It was used to generate this wiki page where I want to
track review process. The same script can be modified to generate koji
commands to remove/add packages. I am currently starting with review:

If you spot a package name that should not be added or removed, please
uncheck it and make a comment in the gist or here:

gist.github.com

https://gist.github.com/lzap/0e0ee1a7f11ccd9eaba45282391862a6

koji-cleanup-packages.md


# Packages missing in tag foreman-1.15-nonscl-rhel7 ###

# Packages not expected in foreman-1.15-nonscl-rhel7 ###
 * [x] foreman-proxy-selinux
 * [ ] mod_passenger
 * [x] nodejs-babel-core-doc
 * [x] nodejs-babel-loader-doc
 * [x] nodejs-babel-preset-es2015-doc
 * [x] nodejs-datatables.net-bs-doc

This file has been truncated. show original

Most of the packages are simply either subpackage which should not be
present (we add only source package names) or packages that should not
be both in nonscl-scl or core-plugin. And then there's the rest.

For example I found mod_passenger to be removed but I believe this is
correctly tagged (we carry this dependency in our repos), so I
unchecked. I will carry on tomorrow with review.

If you reviewed all the packages please let me know here or in the gist. Thanks

This is the first stage, the next would be review of tags, I don't
know yet how to do that one. If anyone have an idea how to automate
that, let me know.

···

-- Later, Lukas @lzap Zapletal

lzap · June 16, 2017, 9:24am

Hey,

I deleted about 1800 packages which were obvious to me (subpackages,
wrong packages tfm vs nonscl), added few missing. Here is full log:

gist.github.com

https://gist.github.com/lzap/9074e85da2875a542ef81f338df75c77

executed.shell

koji remove-pkg foreman-nightly-nonscl-rhel7 nodejs-babel-core-doc
koji remove-pkg foreman-nightly-nonscl-rhel7 nodejs-babel-loader-doc
koji remove-pkg foreman-nightly-nonscl-rhel7 nodejs-babel-preset-es2015-doc
koji remove-pkg foreman-nightly-nonscl-rhel7 nodejs-datatables.net-bs-doc
koji remove-pkg foreman-nightly-nonscl-rhel7 nodejs-datatables.net-doc
koji remove-pkg foreman-nightly-nonscl-rhel7 nodejs-diff-doc
koji remove-pkg foreman-nightly-nonscl-rhel7 nodejs-expose-loader-doc
koji remove-pkg foreman-nightly-nonscl-rhel7 nodejs-file-loader-doc
koji remove-pkg foreman-nightly-nonscl-rhel7 nodejs-ipaddr.js-doc
koji remove-pkg foreman-nightly-nonscl-rhel7 nodejs-jquery-doc

This file has been truncated. show original

I am still in doubt with few of others (mostly passenger and
tfm/tfm-build packages). Do you see something incorrect on this list?

gist.github.com

https://gist.github.com/lzap/6a0fcc774bd14a15a30daf702827c5db

compare-with-tags.md


# Packages missing in tag foreman-nightly-nonscl-rhel7
```shell
```

# Packages not expected in foreman-nightly-nonscl-rhel7
 * [x] mod_passenger
```shell
koji remove-pkg foreman-nightly-nonscl-rhel7 mod_passenger
```

This file has been truncated. show original

If not, I will go ahead and execute the rest on Monday.

Updated version of my script is at

gist.github.com

https://gist.github.com/lzap/cc6c100feb0a558f961eaa07e01bfa59

compare-with-tags.rb

#!/usr/bin/env ruby
require 'configparser'

cfg = ConfigParser.new('tito.props')
cfg.each do |entry|
  tag = entry.first
  scl = entry.last['scl']
  next unless tag =~ /^foreman/
  comp_pkgs = entry.last['whitelist'].split
  comp_pkgs.map!{ |x| (scl && x =~ /^rubygem-/) ? "#{scl}-#{x}" : x }

This file has been truncated. show original

···

On Thu, Jun 15, 2017 at 4:16 PM, Lukas Zapletal wrote: > Hello, > > I created a small script > > https://gist.github.com/lzap/cc6c100feb0a558f961eaa07e01bfa59 > > which compares our whitelists agains koji. The first phase is packages > added to tags. It was used to generate this wiki page where I want to > track review process. The same script can be modified to generate koji > commands to remove/add packages. I am currently starting with review: > > If you spot a package name that should not be added or removed, please > uncheck it and make a comment in the gist or here: > > https://gist.github.com/lzap/0e0ee1a7f11ccd9eaba45282391862a6 > > Most of the packages are simply either subpackage which should not be > present (we add only source package names) or packages that should not > be both in nonscl-scl or core-plugin. And then there's the rest. > > For example I found mod_passenger to be removed but I believe this is > correctly tagged (we carry this dependency in our repos), so I > unchecked. I will carry on tomorrow with review. > > If you reviewed all the packages please let me know here or in the gist. Thanks > > This is the first stage, the next would be review of tags, I don't > know yet how to do that one. If anyone have an idea how to automate > that, let me know. > > > -- > Later, > Lukas @lzap Zapletal

–
Later,
Lukas @lzap Zapletal

lzap · June 19, 2017, 12:26pm

I cleaned the rest, nightly should be all clean. Two packages were not
deleted according to our whitelists tho, correct me if I am wrong:

koji remove-pkg foreman-nightly-nonscl-rhel7 mod_passenger

This was not performed as I think we retag mod_passenger from EPEL,
therefore it's not in our whitelists.

koji remove-pkg foreman-plugins-nightly-rhel7 tfm

Also not performed, I think this package does belong into
foreman-plugins-nightly-rhel7 as this is supposed to be scl tag.

All commands executed and logs are in:

/root/tags-cleanup/
/root/tags-cleanup/round02.bash
/root/tags-cleanup/round01.bash
/root/tags-cleanup/round01.log
/root/tags-cleanup/round02.log
/root/tags-cleanup/round03.bash
/root/tags-cleanup/round03.log
/root/tags-cleanup/round04.bash
/root/tags-cleanup/round04.log
/root/tags-cleanup/round05.bash
/root/tags-cleanup/round05.log

The mash log still lists few warnings, fedora24:

2017-06-19 12:20:56 [WARNING ] rubygem-fog-libvirt-doc not in
comps-foreman-fedora24.xml
2017-06-19 12:20:56 [WARNING ] No build for package
rubygem-librarian-puppet in package listing for tag
foreman-nightly-fedora24-dist
2017-06-19 12:20:56 [WARNING ] No build for package
rubygem-therubyracer in package listing for tag
foreman-nightly-fedora24-dist
2017-06-19 12:20:56 [WARNING ] No build for package
rubygem-strong_parameters in package listing for tag
foreman-nightly-fedora24-dist

And el7:

2017-06-19 12:20:27 [WARNING ] In comps but not in
foreman-nightly/RHEL/7 tree: rubygem-rdoc
2017-06-19 12:20:27 [WARNING ] rubygem-passenger-devel not in
comps-foreman-rhel7.xml
2017-06-19 12:20:27 [WARNING ] tfm-rubygem-passenger-devel not in
comps-foreman-rhel7.xml
2017-06-19 12:20:27 [WARNING ] No build for package mod_passenger in
package listing for tag foreman-nightly-rhel7-dist

Any advice on these would be appreciated.

···

On Fri, Jun 16, 2017 at 11:24 AM, Lukas Zapletal wrote: > Hey, > > I deleted about 1800 packages which were obvious to me (subpackages, > wrong packages tfm vs nonscl), added few missing. Here is full log: > > https://gist.github.com/lzap/9074e85da2875a542ef81f338df75c77 > > I am still in doubt with few of others (mostly passenger and > tfm/tfm-build packages). Do you see something incorrect on this list? > > https://gist.github.com/lzap/6a0fcc774bd14a15a30daf702827c5db > > If not, I will go ahead and execute the rest on Monday. > > Updated version of my script is at > > https://gist.github.com/lzap/cc6c100feb0a558f961eaa07e01bfa59 > > On Thu, Jun 15, 2017 at 4:16 PM, Lukas Zapletal wrote: >> Hello, >> >> I created a small script >> >> https://gist.github.com/lzap/cc6c100feb0a558f961eaa07e01bfa59 >> >> which compares our whitelists agains koji. The first phase is packages >> added to tags. It was used to generate this wiki page where I want to >> track review process. The same script can be modified to generate koji >> commands to remove/add packages. I am currently starting with review: >> >> If you spot a package name that should not be added or removed, please >> uncheck it and make a comment in the gist or here: >> >> https://gist.github.com/lzap/0e0ee1a7f11ccd9eaba45282391862a6 >> >> Most of the packages are simply either subpackage which should not be >> present (we add only source package names) or packages that should not >> be both in nonscl-scl or core-plugin. And then there's the rest. >> >> For example I found mod_passenger to be removed but I believe this is >> correctly tagged (we carry this dependency in our repos), so I >> unchecked. I will carry on tomorrow with review. >> >> If you reviewed all the packages please let me know here or in the gist. Thanks >> >> This is the first stage, the next would be review of tags, I don't >> know yet how to do that one. If anyone have an idea how to automate >> that, let me know. >> >> >> -- >> Later, >> Lukas @lzap Zapletal > > > > -- > Later, > Lukas @lzap Zapletal

–
Later,
Lukas @lzap Zapletal