Katello nightly pipeline failed:
https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/449/
foreman-katello-nightly-test (passed)
foreman-katello-upgrade-nightly-test (failed)
[ERROR 2020-03-15T23:59:57 verbose] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[pipeline-upgrade-katello-nightly-centos7.n5.example.com]: Could not evaluate: Proxy pipeline-upgrade-katello-nightly-centos7.n5.example.com cannot be retrieved: unknown error (response 503)Taking a look today
The exact same thing can be seen if you run the vanilla Foreman upgrade pipeline. I believe this is due to Puma which starts up and returns before it’s really running. Passenger opened the socket and blocked till it could process requests.
I looked at implementing pum-plugin-systemd but the plugin needs to be marked compatible with Puma 4. It would only be a limited solution; the installer would work since systemctl restart foreman would block until it was ready. However, during the restart it would still drop connections. Now I’m going to look into Feature #29144: Run Puma using systemd socket activation - Foreman.
If not, we should revert https://github.com/theforeman/foreman-installer/commit/0b244f5623889585eb2b1a16a20ef659ebd2195f.
I’ve also opened https://github.com/theforeman/forklift/pull/1134 & https://github.com/theforeman/forklift/pull/1135 which allows running upgrade pipelines with vanilla Foreman. That would have prevented 2 broken RCs for 2.0.0.
If we end up reverting the puma enablement is that going to have its own set of impacts on nightly? I’m thinking primarily of https://github.com/theforeman/foreman/commit/29fe60a89b47dfb4292f7a4d28d5eb6f5576b488 but it looks backward-compatible
Initial testing showed that the SCL wrapper as it’s used now break socket activation. I have a minimal reproducer that I’m experimenting with and will continue tomorrow.
I still find it odd that it happens on upgrade but not on a fresh install. Perhaps it’s more of a timing thing. It’s about a slow startup so essentially it’s a race condition now. Guess we might see it in installs too if the timing was slightly different.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.