katello-nightly-rpm-pipeline 769 failed

Infra & CI
#1

Katello nightly pipeline failed:

foreman-pipeline-katello-nightly-centos8-install (passed)
foreman-pipeline-katello-nightly-centos7-install (failed)
foreman-pipeline-katello-nightly-centos7-upgrade (failed)

#2

bats fails during sync proxy:

# Error: PLP0000: The Manifest digest does not match the expected value. The remote feed announced a digest of sha256:13280b5914050853a87d662c3229d42b61544e36dd4515f06e188835f3407468, but the downloaded digest was sha256:fc5dcad1472b8d34fe1bd242d160f1a337f3c5a2f22c28169656d13e3a14acb4.
#3

investigating this, it seems that requests to /v2/ against apache on
the main server are going to the pulpcore-api app instead of the rails app.

Commenting out the proxypass lines here:

<Location “/pulpcore_registry/v2/”>
RequestHeader unset REMOTE_USER
RequestHeader set REMOTE_USER “%{SSL_CLIENT_S_DN_CN}s”
env=SSL_CLIENT_S_DN_CN
ProxyPass
unix:///run/pulpcore-api.sock|http://pp-katello-ser-nightly-centos7.windhelm.example.com/v2/
ProxyPassReverse
unix:///run/pulpcore-api.sock|http://pp-katello-ser-nightly-centos7.windhelm.example.com/v2/

seems to get those requests to go back to katello, but i can’t seem to
figure out why this is broken (or when it broke).

Will keep digging

#4

There are two issues uncovered here. The first is a Pulp bug related to manifest calculation that the switch to quay.io revealed.

The second is that tools like podman do not have to authenticate to get content because they are being routed directly to Pulp rather than to Katello for authentication. The expected workflow is:

podman pull ->  katello at /v2/ -> auth and redirect to /pulpcore_registry/v2/ -> proxy to pulpcore-api at /v2/

The problem is that whats currently happening is:

podman pull -> pulpcore-api at /v2/

The entirety of Katello and the auth mechanism is being skirted around (note: we will add a negative test to ensure that container content cannot be accessed without providing credentials).

The reason this is happening appears to stem from our Apache configuration, where we have quite a lot of proxy pass directives going on and what might be a bug in Apache.

In the Apache configuration, to handle the redirect there is:

  <Location "/pulpcore_registry/v2/">
    RequestHeader unset REMOTE_USER
    RequestHeader set REMOTE_USER "%{SSL_CLIENT_S_DN_CN}s" env=SSL_CLIENT_S_DN_CN
    ProxyPass unix:///run/pulpcore-api.sock|http://pipe-katello-server-nightly-centos7.woodford.example.com/v2/
    ProxyPassReverse unix:///run/pulpcore-api.sock|http://pipe-katello-server-nightly-centos7.woodford.example.com/v2/
  </Location>

And to handle requests getting to Foreman/Katello:

  ProxyPass /pulp !
  ProxyPass /pulp2 !
  ProxyPass /streamer !
  ProxyPass /pub !
  ProxyPass /icons !
  ProxyPass / unix:///run/foreman.sock|http://pipe-katello-server-nightly-centos7.woodford.example.com/ retry=0 timeout=900
  ProxyPassReverse / unix:///run/foreman.sock|http://pipe-katello-server-nightly-centos7.woodford.example.com/

While that Location declaration ought to be scoped on purely /pulpcore_registry/v2/ and thereby fall through to the Foreman/Katello directive this is not the case. When the request to /v2/ is made it hits the inside of the Location block. Now, I attempted lots of fanagling, and no matter what you set that Location to, it appears to land inside of it every time. The only thing I could get to work was to change the reference host in the ProxyPass directive to something different than the Foreman directive, i.e.:

  <Location "/pulpcore_registry/v2/">
    RequestHeader unset REMOTE_USER
    RequestHeader set REMOTE_USER "%{SSL_CLIENT_S_DN_CN}s" env=SSL_CLIENT_S_DN_CN
    ProxyPass unix:///run/pulpcore-api.sock|http://localhost/v2/
    ProxyPassReverse unix:///run/pulpcore-api.sock|http://localhost/v2/
  </Location>

This new configuration works as expected. That host, with unix sockets, is supposed to just set the Host: header but otherwise be irrelevant according to the Apache docs.

My proposal then is to change both Foreman and Pulpcore’s deployment here to their service names to fix this issue and keep a relevant Host: header. For example, the above snippets would change to:

  <Location "/pulpcore_registry/v2/">
    RequestHeader unset REMOTE_USER
    RequestHeader set REMOTE_USER "%{SSL_CLIENT_S_DN_CN}s" env=SSL_CLIENT_S_DN_CN
    ProxyPass unix:///run/pulpcore-api.sock|http://pulpcore-api/v2/
    ProxyPassReverse unix:///run/pulpcore-api.sock|http://pulpcore-api/v2/
  </Location>

  ProxyPass /pulp !
  ProxyPass /pulp2 !
  ProxyPass /streamer !
  ProxyPass /pub !
  ProxyPass /icons !
  ProxyPass / unix:///run/foreman.sock|http://foreman/ retry=0 timeout=900
  ProxyPassReverse / unix:///run/foreman.sock|http://foreman/

Open to other suggestions or expert knowledge in this area. If you were wondering if we could set them all to localhost, no we cannot, that results in the same problem. They all need their own host declaration unique to the service.

#5

I have opened two PRs to address this:

#6

The tests don’t fail on EL8. Could it be that it’s an Apache bug where the fix wasn’t cherry picked to EL7 or do we simply not run the content test there?

#7

We do not run these content tests there yet as we do not have a content proxy on EL8 being installed.

#8

Does this affect the HTTP hostname that the application receives? I don’t think we’ve set up middleware the host from a trusted reverse proxy so I worry that that may be a problem but honestly don’t know if it’s used that much.

#9

According to the docs, it affects the Host: header, but I think if you look at previously we were proxying to http://localhost:3000 anyway.

#10

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.