katello-nightly-rpm-pipeline 1463 failed

[jenkins]

Katello nightly pipeline failed:

https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/1463/

foreman-pipeline-katello-rpm-nightly (failed) (remote job)

[evgeni]

Upgrades are failing with:

[2022-10-26T23:41:23.739Z]     not ok 4 sync proxy
[2022-10-26T23:41:23.739Z]     # (in test file fb-katello-proxy.bats, line 30)
[2022-10-26T23:41:23.739Z]     #   `hammer capsule content synchronize --id=$PROXY_ID' failed with status 70
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 planned: 0.0/1, 0%, elapsed: 00:00:00
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 running: 0.02/1, 2%, 0.0/s, elapsed: 00:00:02, ETA: 00:01:46
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 running: 0.02/1, 2%, 0.0/s, elapsed: 00:00:04, ETA: 00:03:33
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 running: 0.03/1, 3%, 0.0/s, elapsed: 00:00:06, ETA: 00:03:32
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 running: 0.18/1, 18%, 0.0/s, elapsed: 00:00:08, ETA: 00:00:40
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 running: 0.3/1, 30%, 0.0/s, elapsed: 00:00:11, ETA: 00:00:26
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 running: 0.39/1, 39%, 0.0/s, elapsed: 00:00:13, ETA: 00:00:21
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 running: 0.51/1, 51%, 0.0/s, elapsed: 00:00:15, ETA: 00:00:13
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 running: 0.58/1, 57%, 0.0/s, elapsed: 00:00:18, ETA: 00:00:10
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 running: 0.68/1, 68%, 0.0/s, elapsed: 00:00:20, ETA: 00:00:06
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 running: 0.78/1, 78%, 0.0/s, elapsed: 00:00:22, ETA: 00:00:04
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 running: 0.84/1, 84%, 0.0/s, elapsed: 00:00:24, ETA: 00:00:03
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 running: 0.91/1, 91%, 0.0/s, elapsed: 00:00:26, ETA: 00:00:02
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 running: 0.97/1, 97%, 0.0/s, elapsed: 00:00:28
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 warning: 1.0/1, 100%, 0.0/s, elapsed: 00:00:31
[2022-10-26T23:41:23.739Z]     # Task 943e6fdd-8481-4a9c-b98e-1dfd93e3b232 warning: 1.0/1, 100%, 0.0/s, elapsed: 00:00:31
[2022-10-26T23:41:23.739Z]     # Error: Error message: the server returns an error
[2022-10-26T23:41:23.739Z]     # HTTP status code: 400
[2022-10-26T23:41:23.739Z]     # Response headers: {"date"=>"Wed, 26 Oct 2022 23:38:48 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"DENY", "content-length"=>"36", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "correlation-id"=>"423a16fd-23cf-42a4-a02b-a6e0c2a7150c", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 pipe-up-katello-proxy-nightly-almalinux8.example.com", "connection"=>"close"}
[2022-10-26T23:41:23.739Z]     # Response body: {"signed_only":["Unexpected field"]}

This happens during the “N-1” stage, where a nightly Katello (with pulpcore 3.21?) is talking to a 4.6 Proxy (with pulpcore 3.18?)

[evgeni]

cc @sajha

[sajha]

I can reproduce this with container and ansible_collection repos locally. A new field was added in pulp 3.21 to the remote sync which it sends to pulp 3.18 and 3.18 doesn’t like it…Reached out to pulp team to discuss solutions.

[sajha]

Correction: Ansible collection isn’t complaining…Only container repos.