Katello nightlies not publishing

Development
1

Hello,

Katello nightlies are not passing bats tests and failing to publish rpms because of it. There hasn’t been a nightly Katello build published since May 26th, over a month ago. I know a lot of effort has been put into this recently but this is still failing.

The current error, during starting bats tests, is:

Error: Package: python-gofer-qpid-2.7.6-1.el7.noarch (pulp-koji)
       Requires: python-gofer = 2.7.6
       Available: python-gofer-2.7.6-1.el7.noarch (pulp-koji)
           python-gofer = 2.7.6-1.el7
       Installing: python-gofer-2.12.0-1.el7.noarch (katello-client-koji)
           python-gofer = 2.12.0-1.el7

@katello and @packaging teams - please look into this.

3 Likes
2

Looking into it.

1 Like
3

Looks like some improvement has been made and now bats manage to execute. However, they fail currently on a few of the test cases:

not ok 24 check available errata
# (in test file fb-content-katello.bats, line 179)
#   `hammer host errata list --host $(hostname -f) | grep 'RHEA-2012:0055'' failed

<snip>

not ok 27 install package remotely (katello-agent)
# (from function `tPackageExists' in file os_helper.bash, line 80,
#  in test file fb-content-katello.bats, line 197)
#   `tPackageExists gorilla' failed
# Task 1294aafd-34fd-4469-a09b-346643d9bec9 planned: 0.0/1, 0%, elapsed: 00:00:00
# Task 1294aafd-34fd-4469-a09b-346643d9bec9 running: 0.0/1, 0%, 0.0/s, elapsed: 00:00:02
# Task 1294aafd-34fd-4469-a09b-346643d9bec9 running: 0.25/1, 25%, 0.1/s, elapsed: 00:00:04, ETA: 00:00:12
# Task 1294aafd-34fd-4469-a09b-346643d9bec9 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:06, ETA: 00:00:18
# Task 1294aafd-34fd-4469-a09b-346643d9bec9 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:08, ETA: 00:00:24
# Task 1294aafd-34fd-4469-a09b-346643d9bec9 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:10, ETA: 00:00:31
# Task 1294aafd-34fd-4469-a09b-346643d9bec9 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:12, ETA: 00:00:37
# Task 1294aafd-34fd-4469-a09b-346643d9bec9 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:14, ETA: 00:00:37
# Task 1294aafd-34fd-4469-a09b-346643d9bec9 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:16
# Task 1294aafd-34fd-4469-a09b-346643d9bec9 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:18
# Task 1294aafd-34fd-4469-a09b-346643d9bec9 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:20
# Task 1294aafd-34fd-4469-a09b-346643d9bec9 error: 0.75/1, 75%, 0.0/s, elapsed: 00:00:22, ETA: 00:00:06
# Task 1294aafd-34fd-4469-a09b-346643d9bec9 error: 0.75/1, 75%, 0.0/s, elapsed: 00:00:22, ETA: 00:00:06
# Error: Host did not respond within 20 seconds. The task has been cancelled. Is katello-agent installed and goferd running on the Host?
not ok 28 install errata remotely (katello-agent)
# (from function `tPackageExists' in file os_helper.bash, line 80,
#  in test file fb-content-katello.bats, line 203)
#   `tPackageExists walrus-5.21' failed
# Task 313e7caf-1053-4bdc-b6b4-b02b72729c72 planned: 0.0/1, 0%, elapsed: 00:00:00
# Task 313e7caf-1053-4bdc-b6b4-b02b72729c72 running: 0.25/1, 25%, 0.1/s, elapsed: 00:00:02, ETA: 00:00:06
# Task 313e7caf-1053-4bdc-b6b4-b02b72729c72 running: 0.25/1, 25%, 0.1/s, elapsed: 00:00:04, ETA: 00:00:12
# Task 313e7caf-1053-4bdc-b6b4-b02b72729c72 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:06, ETA: 00:00:18
# Task 313e7caf-1053-4bdc-b6b4-b02b72729c72 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:08, ETA: 00:00:24
# Task 313e7caf-1053-4bdc-b6b4-b02b72729c72 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:10, ETA: 00:00:31
# Task 313e7caf-1053-4bdc-b6b4-b02b72729c72 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:12, ETA: 00:00:37
# Task 313e7caf-1053-4bdc-b6b4-b02b72729c72 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:14
# Task 313e7caf-1053-4bdc-b6b4-b02b72729c72 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:16
# Task 313e7caf-1053-4bdc-b6b4-b02b72729c72 running: 0.25/1, 25%, 0.0/s, elapsed: 00:00:18
# Task 313e7caf-1053-4bdc-b6b4-b02b72729c72 warning: 1.0/1, 100%, 0.1/s, elapsed: 00:00:20
# Task 313e7caf-1053-4bdc-b6b4-b02b72729c72 warning: 1.0/1, 100%, 0.1/s, elapsed: 00:00:20
# Error: Host did not respond within 20 seconds. The task has been cancelled. Is katello-agent installed and goferd running on the Host?

full log for the last run is at: https://ci.theforeman.org/blue/rest/organizations/jenkins/pipelines/katello-nightly-release/runs/312/nodes/123/steps/130/log/?start=0

4

This is known and being addressed by Pulp. I anticipate a 2.16.3 Monday that will fix it.

1 Like
5

Thanks to @pcreech for getting Pulp 2.16.3 released with a fix from Tanya on the Pulp team the failing test is now resolved. Katello nightly has gone full green [1]. Not only that, but with the work @Zhunting did merged, this pipeline used ci.centos.org to run the installation and bats testing. Thus, we got a full green of this new workflow, using their compute and reducing load on Rackspace in addition this test included a full external smart proxy with content install and bats test to test content sync for the first time in nightlies history [2]. Kudos to all!

[1] http://ci.theforeman.org/job/katello-nightly-release/316/console
[2] https://github.com/theforeman/forklift/blob/master/bats/fb-proxy.bats

3 Likes
6

Great job, thank you very much everyone involved! now let’s keep them nightlies green :stuck_out_tongue_winking_eye:

1 Like
7

This is awesome! Thank you guys for all the hard work. Bring on dev box testing!