Error in katello-repos-3.17.1-1.el7.noarch

gvde · December 9, 2020, 7:28pm

The latest rpm katello-repos-3.17.1-1.el7.noarch contains a new section for [qpid] which looks like this:

[qpid]
name=Copr repo for qpid owned by @qpid
baseurl=https://download.copr.fedorainfracloud.org/results/@qpid/qpid/epel-7-$basearch/
gpgcheck=1
gpgkey=https://download.copr.fedorainfracloud.org/results/@qpid/qpid/pubkey.gpg
enabled=@PULP_ENABLED@

That causes a syntax error:

Repository 'qpid': Error parsing config: Error parsing "enabled = '@PULP_ENABLED@'": invalid boolean value

tjyang · December 9, 2020, 10:13pm

I am having same exact error message. Any pointer ?

tjyang · December 9, 2020, 10:27pm

I think it was failed M4 variable @PULP_ENABLED@ replacement.
I set it to 1 manually in katello.repo file.

[qpid]
name=Copr repo for qpid owned by @qpid
baseurl=https://download.copr.fedorainfracloud.org/results/@qpid/qpid/epel-7-$basearch/
gpgcheck=1
gpgkey=https://download.copr.fedorainfracloud.org/results/@qpid/qpid/pubkey.gpg
#enabled=@PULP_ENABLED@
enabled=1

tjyang · December 9, 2020, 10:55pm

Here is the update result following the 3.17 upgrade note.

[root@foreman01 yum.repos.d]# foreman-installer
Preparing installation Done
Something went wrong! Check the log for ERROR-level output
The full log is at /var/log/foreman-installer/katello.log
[root@ilmlforeman01 yum.repos.d]# rpm -qa |grep foreman-maintain
[root@foreman01 yum.repos.d]#

This is error message

[ INFO 2020-12-09T16:40:08 main] All hooks in group post finished
[DEBUG 2020-12-09T16:40:08 main] Exit with status code: 6 (signal was 6)
[ERROR 2020-12-09T16:40:08 main] Errors encountered during run:
[ERROR 2020-12-09T16:40:08 main] foreman-maintain packages is-locked --assumeyes failed! Check the output for error!
[ERROR 2020-12-09T16:40:08 main] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Failed to call refresh: ‘pulpcore-manager collectstatic --noinput’ returned 1 instead of one of [0]
[ERROR 2020-12-09T16:40:08 main] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: ‘pulpcore-manager collectstatic --noinput’ returned 1 instead of one of [0]
[DEBUG 2020-12-09T16:40:08 main] Cleaning /tmp/kafo_installation20201209-11901-13oq2xl
[DEBUG 2020-12-09T16:40:08 main] Cleaning /tmp/kafo_installation20201209-11901-1q203a4
[DEBUG 2020-12-09T16:40:08 main] Cleaning /tmp/default_values.yaml
[ INFO 2020-12-09T16:40:08 main] Installer finished in 494.564245779 seconds

gvde · December 10, 2020, 6:15am

I have tested the update and foreman-installer run and noticed the same error. In the log right before I found these messages which I guess are the reason for the failure of pulpcore-manager collectstatic:

[ INFO 2020-12-10T06:27:18 main]  Pulpcore::Admin[collectstatic --noinput]: Starting to evaluate the resource (1183 of 2352)
[ INFO 2020-12-10T06:27:18 main]  Pulpcore::Admin[collectstatic --noinput]: Scheduling refresh of Exec[pulpcore-manager collectstatic --noinput]
[ INFO 2020-12-10T06:27:18 main]  Pulpcore::Admin[collectstatic --noinput]: Evaluated in 0.00 seconds
[ INFO 2020-12-10T06:27:18 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Starting to evaluate the resource (1184 of 2352)
[DEBUG 2020-12-10T06:27:18 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: 'pulpcore-manager collectstatic --noinput' won't be executed because of failed check 'refreshonly'
[DEBUG 2020-12-10T06:27:18 main]  Exec[pulpcore-manager collectstatic --noinput](provider=posix): Executing 'pulpcore-manager collectstatic --noinput'
[DEBUG 2020-12-10T06:27:18 main]  Executing with uid=pulp: 'pulpcore-manager collectstatic --noinput'
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns: Traceback (most recent call last):
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/bin/pulpcore-manager", line 11, in <module>
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     load_entry_point('pulpcore==3.7.3', 'console_scripts', 'pulpcore-manager')()
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/pulpcore/app/manage.py", line 11, in manage
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     execute_from_command_line(sys.argv)
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     utility.execute()
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     self.fetch_command(subcommand).run_from_argv(self.argv)
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     self.execute(*args, **cmd_options)
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     output = self.handle(*args, **options)
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 188, in handle
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     collected = self.collect()
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     handler(path, prefixed_path, storage)
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 342, in copy_file
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     if not self.delete_file(path, prefixed_path, source_storage):
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 292, in delete_file
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     self.storage.delete(prefixed_path)
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 303, in delete
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     os.remove(name)
[ WARN 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns: PermissionError: [Errno 13] Permission denied: '/var/lib/pulp/assets/admin/css/base.css'
[ERROR 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Failed to call refresh: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
[ERROR 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
[ INFO 2020-12-10T06:27:20 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Evaluated in 1.85 seconds

Interesting enough, if I shutdown all foreman services (foreman-maintain service stop) and run the installer again, it doesn’t fail.

There are several other topics regarding pulpcore-manager in the last couple of days. I guess I postpone the update until the dust settles.

I reverted my vm to my snapshot before the update for the time being.

gvde · December 10, 2020, 6:40am

I have noticed that the copr repo provides version python2-qpid-1.37.0-4.el7.noarch while there is 1.37.0-5.el7 already available via epel7. Thus technically, it shouldn’t pull anything from that repositories and it should be an issue. (unless of course the version from copr and epel are different…)

evgeni · December 10, 2020, 9:35am

python2-qpid was gone from epel for a few days, so we deployed that fix (and apparently botched it)

luckytroll · December 10, 2020, 3:00pm

This was affecting one of my attempted installs as well, but if I shorted out the enabled=1 as was done above, I still got the failures seen. I am beginning to suspect it is a result of using a newer centos. The previous build I based my install on was 19xx and the newer one was 20xx . I noticed that python3 upgrades feature in these, and could affect the django parts of the install and DB processes in upgrade. I dont have time to dig, so may just revert my vm as well.

evgeni · December 11, 2020, 2:12pm

A fixed build of katello-repos for 3.16 and 3.17 should be available in the next hour.

stostja · December 11, 2020, 3:51pm

I belive i had same output when foreman-installer was runned after upgrade to 3.18 RC1.

evgeni · December 11, 2020, 4:32pm

For 3.18 and later, you shouldn’t see that exact error, as there the variable substitution actually works (and I didn’t butcher the backport of that change by only including half of it).

stostja · December 11, 2020, 4:49pm

I can simulate it … in 3.18 RC1

i could not change certificates in 3.17 for valid ones, due to bug Bug #31135

        [ INFO 2020-11-28T00:13:13 main] Running installer with args [["--scenario", "katello", "--certs-server-cert", "/root/foreman_cert_20/foreman.cer", "--certs-server-key", "/root/foreman_cert_20/serverkey.key", "
   *    --certs-server-ca-cert", "/root/foreman_cert_20/foreman.pem", "--certs-update-server", "--certs-update-server-ca"]]
   *    [ INFO 2020-11-28T00:13:13 main] Executing hooks in group pre_validations
   *    [DEBUG 2020-11-28T00:13:13 main] Hook /usr/share/foreman-installer/hooks/pre_validations/01-reset_data.rb returned nil
   *    [DEBUG 2020-11-28T00:13:13 main] Hook /usr/share/foreman-installer/hooks/pre_validations/30-el7_upgrade_postgresql.rb returned nil
   *    [DEBUG 2020-11-28T00:13:13 main] Hook /usr/share/foreman-installer/katello/hooks/pre_validations/10-check_foreman_proxy_pulp.rb returned nil
   *    [DEBUG 2020-11-28T00:13:13 main] Hook /usr/share/foreman-installer/katello/hooks/pre_validations/12-check_capsule_tar.rb returned nil
   *    [ INFO 2020-11-28T00:13:13 main] All hooks in group pre_validations finished
   *    [ INFO 2020-11-28T00:13:13 main] Running validation checks
   *    [ INFO 2020-11-28T00:13:13 main] Executing hooks in group pre_commit
   *    [DEBUG 2020-11-28T00:13:13 main] Hook /usr/share/foreman-installer/katello/hooks/pre_commit/05-puppet_certs_exist.rb returned nil
   *    [DEBUG 2020-11-28T00:13:13 main] Executing: foreman-maintain packages is-locked --assumeyes
   *    [DEBUG 2020-11-28T00:13:16 main] Packages are not locked
   *    [ERROR 2020-11-28T00:13:16 main] foreman-maintain packages is-locked --assumeyes failed! Check the output for error!

which is fixed in 2.3.0 (which means Katello 3.18RC1) And when i run foreman-installer after update to 2.3.0 and 3.18RC1 i got:

[root@vmsvlforeman ~]# foreman-installer
2020-12-11 17:44:17 [NOTICE] [pre_migrations] Executing hooks in group pre_migrations
2020-12-11 17:44:17 [NOTICE] [pre_migrations] All hooks in group pre_migrations finished
2020-12-11 17:44:17 [NOTICE] [boot] Executing hooks in group boot
2020-12-11 17:44:17 [NOTICE] [boot] All hooks in group boot finished
2020-12-11 17:44:17 [NOTICE] [init] Executing hooks in group init
2020-12-11 17:44:17 [NOTICE] [init] All hooks in group init finished
2020-12-11 17:44:17 [NOTICE] [root] Loading default values from puppet modules...
2020-12-11 17:44:21 [NOTICE] [root] ... finished
2020-12-11 17:44:21 [NOTICE] [pre_values] Executing hooks in group pre_values
2020-12-11 17:44:21 [NOTICE] [pre_values] All hooks in group pre_values finished
2020-12-11 17:44:24 [NOTICE] [pre_validations] Executing hooks in group pre_validations
2020-12-11 17:44:24 [NOTICE] [pre_validations] All hooks in group pre_validations finished
2020-12-11 17:44:24 [NOTICE] [root] Running validation checks
2020-12-11 17:44:24 [NOTICE] [pre_commit] Executing hooks in group pre_commit
Executing: katello-certs-check -c "/root/foreman_cert_20/foreman.cer" -k     "/root/foreman_cert_20/serverkey.key" -b "/root/foreman_cert_20/foreman.pem"
Checking CA bundle size: 3 [OK]
Checking Subject Alt Name on certificate [OK]
Checking if any Subject Alt Name on certificate matches the Subject CN [OK]
Checking Key Usage extension on certificate for Key Encipherment [OK]
Checking for use of shortname as CN [OK]
Validation succeeded


To install the Katello server with the custom certificates, run:

    foreman-installer --scenario katello \
                      --certs-server-cert "/root/foreman_cert_20/foreman.cer" \
                      --certs-server-key "/root/foreman_cert_20/serverkey.key" \
                      --certs-server-ca-cert "/root/foreman_cert_20/foreman.pem"

To update the certificates on a currently running Katello installation, run:

    foreman-installer --scenario katello \
                      --certs-server-cert "/root/foreman_cert_20/foreman.cer" \
                      --certs-server-key "/root/foreman_cert_20/serverkey.key" \
                      --certs-server-ca-cert "/root/foreman_cert_20/foreman.pem" \
                      --certs-update-server --certs-update-server-ca

To use them inside a NEW $FOREMAN_PROXY, rerun this command with -t foreman-proxy
2020-12-11 17:44:27 [NOTICE] [pre_commit] All hooks in group pre_commit finished
2020-12-11 17:44:27 [NOTICE] [pre] Executing hooks in group pre
2020-12-11 17:44:38 [NOTICE] [pre] All hooks in group pre finished
2020-12-11 17:44:38 [NOTICE] [configure] Starting system configuration.
  The total number of configuration tasks may increase during the run.
  Observe logs or specify --verbose-log-level to see individual configuration tasks.
2020-12-11 17:45:54 [ERROR ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Failed to call refresh: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
2020-12-11 17:45:54 [ERROR ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]

I cannot tell if this happening if i do not finish replacement certificates on 3.17 - but there they did not finish, and after upgrade to 2.3.0 and 3.18Rc1 i got this.

evgeni · December 11, 2020, 6:43pm

I was talking about the Repository 'qpid': Error parsing config: Error parsing "enabled = '@PULP_ENABLED@'": invalid boolean value error.