Katello-nightly-rpm-pipeline 2155 failed

Infra & CI
1

Katello nightly pipeline failed:

https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/2155/

foreman-pipeline-katello-rpm-nightly (failed) (remote job)

2 
[2024-10-02T23:50:27.440Z]         RuntimeError: Incompatible versions detected (core >= 3.50.0 needed):
[2024-10-02T23:50:27.440Z]           - 'core'='3.49.19' with api server '45696@pipe-up-katello-proxy-nightly-almalinux8.example.com'
[2024-10-02T23:50:27.440Z]           - 'core'='3.49.19' with api server '45673@pipe-up-katello-proxy-nightly-almalinux8.example.com'
[2024-10-02T23:50:27.440Z]           - 'core'='3.49.19' with api server '45699@pipe-up-katello-proxy-nightly-almalinux8.example.com'
[2024-10-02T23:50:27.440Z]         Please shutdown or upgrade the outdated components before you continue the migration.

Seems to be fallout from the new pulpcore (:eyes: @iballou @Odilhao).

Today we don’t orchestrate worker/api shutdown before doing migrations.
Should we?

Edit:
But we do stop all services :thinking:

Interestingly, this only happens on the proxy, not on the main server :thinking:

3

We need to update the bindings on nightly AFAIK

4

@Odilhao I didn’t realize Katello nightly was going to get the bump yet, we were waiting to merge https://github.com/Katello/katello/pull/11058 before updating the bindings and rerecording VCRs.

But anyway, I don’t think the error above is related to the bindings. It’s possible that the new Pulp “just works” with the older bindings if we’re lucky.

5

Yeah, it’s not.

I have a possible fix in make pulpcore-(api|content) same priority as workers by evgeni · Pull Request #934 · theforeman/foreman_maintain · GitHub

The TL;DR is: we’re shutting down PostgreSQL and Pulpcore at the same time, so there is a chance that PostgreSQL is actually shut down earlier and the Pulpcore processes didn’t write their “done/exit” to the DB. When now the DB starts back up again and we run “migrate”, the tooling things “bah, you still have old versions running” because those entries aren’t cleaned up yet.

6

Welp, releasing that fix to nightly of course doesn’t help, as the previous’ release foreman-maintain is used to stop services :woman_facepalming:

7
8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.