Katello-nightly-rpm-pipeline 1027 failed

jenkins · July 13, 2021, 12:36am

Katello nightly pipeline failed:

https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/1027/

foreman-pipeline-katello-nightly-centos7-upgrade (failed) (remote job)
foreman-pipeline-katello-nightly-centos7-install (failed) (remote job)
foreman-pipeline-katello-nightly-centos8-stream-install (failed) (remote job)
foreman-pipeline-katello-nightly-centos8-install (failed) (remote job)
foreman-pipeline-katello-nightly-centos8-upgrade (passed) (remote job)

evgeni · July 13, 2021, 8:54am

seems this

https://github.com/theforeman/puppet-pulpcore/commit/7a5543b1a47de8d76a1cad1fb76e12db7bbf8e45

broke the proxy, as I see in the logs (on the proxy) now:

pulpcore-api[27707]: pulp [2140b308e4e846deb48c5f130b5ed405]: pulpcore.app.views.status:ERROR: Connection to Redis failed during status check!

which is translated to this error in Katello:

2021-07-13T00:22:34 [I|bac|1fffc985] Task {label: Actions::Katello::CapsuleContent::Sync, id: 007c16b7-20e4-4ca6-8190-4cecf00500e1, execution_plan_id: 5391d69e-d01c-4bc0-bf8e-7a3e52de328c} state changed: stopped  result: error
2021-07-13T00:22:34 [E|app|1fffc985] RuntimeError: There was an issue with the backend service pulp3: Pulp redis connection issue at https://pipe-katello-proxy-nightly-centos7.n12.example.com/pulp/api/v3.

ekohl · July 13, 2021, 9:13am

I didn’t read the logs carefully, but looking at the code this may be an issue:
https://github.com/theforeman/puppet-pulpcore/pull/216

ekohl · July 13, 2021, 9:56am

This suggests Pulp doesn’t consider Redis as optional:

github.com

pulp/pulpcore/blob/0c3b69d90bcccbcd9aa5999e8bac10ac3222dca6/pulpcore/app/views/status.py#L60


      
           * known workers
           * known content apps
           * database connection status
           * redis connection status
           * disk usage information
          """
          versions = []
          for app in pulp_plugin_configs():
              versions.append({"component": app.label, "version": app.version})
          
          redis_status = {"connected": self._get_redis_conn_status()}
          db_status = {"connected": self._get_db_conn_status()}
          
          try:
              online_workers = Worker.objects.online_workers()
          except Exception:
              online_workers = None
          
          try:
              online_content_apps = ContentAppStatus.objects.online()
          except Exception:

I’ve submitted a PR and will reach out to upstream to see if we can change this:
https://github.com/pulp/pulpcore/pull/1484

It also looks like we need to update our local_redis? definition here:

github.com

theforeman/foreman-installer/blob/e7b5cdd7398ed2bedf567657190d2b6f70781861/hooks/boot/01-kafo-hook-extensions.rb#L76-L78


      
          def local_redis?
            (foreman_server? && !param_value('foreman', 'jobs_sidekiq_redis_url')) || pulpcore_enabled? || devel_scenario?
          end

evgeni · July 13, 2021, 1:08pm

The “optional redis” was now reverted by Eric.

evgeni · July 13, 2021, 5:27pm

And to close the loop, after the revert, pipeline is green again

system · July 20, 2021, 5:27pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.