Katello nightly pipeline failed:
https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/1965/
foreman-pipeline-katello-rpm-nightly (failed) (remote job)
There are two (consistent) failures from the Pulp 3.49 upgrade.
First (common) failure
not ok 52: clean-up modules-rpms and rpm-deps
That step calls
hammer repository delete --organization="${ORGANIZATION}" --product="${PRODUCT}" --name="${YUM_REPOSITORY_2}" --remove-from-content-view-versions=true
And fails with
Task a61623de-b1c2-4bc2-bd2f-8a948d592527: Dynflow::Action::V2::WithSubPlans::SubtaskFailedException: A sub task failed; RuntimeError: Repository cannot be deleted since it has already been included in a published Content View. Please delete all Content View versions containing this repository before attempting to delete it or use --remove-from-content-view-versions flag to automatically remove the repository from all published versions.
When we look at the production log, it seems --remove-from-content-view-versions is passed in correctly
2024-04-17T06:43:44 [I|app|baecebc0] Started DELETE "/katello/api/repositories/30?remove_from_content_view_versions=true" for 192.168.122.105 at 2024-04-17 06:43:44 +0000
2024-04-17T06:43:44 [I|app|baecebc0] Processing by Katello::Api::V2::RepositoriesController#destroy as JSON
2024-04-17T06:43:44 [I|app|baecebc0] Parameters: {"remove_from_content_view_versions"=>"true", "api_version"=>"v2", "id"=>"30", "repository"=>{}}
2024-04-17T06:43:44 [I|app|baecebc0] Authorized user admin(Admin User)
But Pulp refuses the deletion:
2024-04-17T06:43:46 [E|bac|baecebc0] Error message: the server returns an error
baecebc0 | HTTP status code: 400
baecebc0 | Response headers: {"date"=>"Wed, 17 Apr 2024 06:43:46 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, DELETE, HEAD, OPTIONS", "x-frame-options"=>"DENY", "content-length"=>"174", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "cross-origin-opener-policy"=>"same-origin", "correlation-id"=>"baecebc0-956d-4c36-8558-24ddaa3f6ce4", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 pipe-katello-server-nightly-centos8-stream.example.com", "connection"=>"close"}
baecebc0 | Response body: ["The repository version cannot be deleted because it (or its publications) are currently being used to distribute content. Please update the necessary distributions first."] (PulpRpmClient::ApiError)
baecebc0 | /usr/share/gems/gems/pulp_rpm_client-3.25.2/lib/pulp_rpm_client/api_client.rb:83:in `call_api'
baecebc0 | /usr/share/gems/gems/pulp_rpm_client-3.25.2/lib/pulp_rpm_client/api/repositories_rpm_versions_api.rb:77:in `delete_with_http_info'
baecebc0 | /usr/share/gems/gems/pulp_rpm_client-3.25.2/lib/pulp_rpm_client/api/repositories_rpm_versions_api.rb:28:in `delete'
baecebc0 | /usr/share/gems/gems/katello-4.13.0.pre.master/app/services/katello/pulp3/repository.rb:335:in `block in delete_version'
the code calling this is
def delete_version
ignore_404_exception { api.repository_versions_api.delete(repo.version_href) } unless version_zero?
end
but the exception is a 400, not a 404, so it’s not swallowed.
Interestingly, this error does not always (but like 95% of the time) happen, so it might be some race condition while we delete stuff in Pulp?
The second failure
not ok 4: sync proxy
That step calls
hammer capsule content synchronize --id=$PROXY_ID
production.log:
2024-04-17T07:02:13 [E|bac|4c33ccc9] Error message: the server returns an error
4c33ccc9 | HTTP status code: 500
4c33ccc9 | Response headers: {"date"=>"Wed, 17 Apr 2024 07:02:13 GMT", "server"=>"gunicorn", "content-type"=>"text/html; charset=utf-8", "x-frame-options"=>"DENY", "content-length"=>"145", "vary"=>"Cookie", "x-content-type-options
"=>"nosniff", "referrer-policy"=>"same-origin", "cross-origin-opener-policy"=>"same-origin", "correlation-id"=>"4c33ccc9-00e2-413f-a9cc-fefa20bcc212", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 pipe-katello-prox
y-nightly-centos8-stream.example.com", "connection"=>"close"}
4c33ccc9 | Response body:
4c33ccc9 | <!doctype html>
4c33ccc9 | <html lang="en">
4c33ccc9 | <head>
4c33ccc9 | <title>Server Error (500)</title>
4c33ccc9 | </head>
4c33ccc9 | <body>
4c33ccc9 | <h1>Server Error (500)</h1><p></p>
4c33ccc9 | </body>
4c33ccc9 | </html>
4c33ccc9 | (PulpCertguardClient::ApiError)
4c33ccc9 | /usr/share/gems/gems/pulp_certguard_client-3.49.3/lib/pulp_certguard_client/api_client.rb:83:in `call_api'
4c33ccc9 | /usr/share/gems/gems/pulp_certguard_client-3.49.3/lib/pulp_certguard_client/api/contentguards_rhsm_api.rb:79:in `create_with_http_info'
4c33ccc9 | /usr/share/gems/gems/pulp_certguard_client-3.49.3/lib/pulp_certguard_client/api/contentguards_rhsm_api.rb:28:in `create'
4c33ccc9 | /usr/share/gems/gems/katello-4.13.0.pre.master/app/services/katello/pulp3/api/content_guard.rb:55:in `create'
And looking at the pulp logs on the proxy, we see
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39047]: pulp [4c33ccc9-00e2-413f-a9cc-fefa20bcc212]: - - [17/Apr/2024:07:02:13 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4691 "-" "rest-client/2.1.0 (linux x86_64) ruby/2.7.8p225"
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39047]: pulp [4c33ccc9-00e2-413f-a9cc-fefa20bcc212]: - - [17/Apr/2024:07:02:13 +0000] "GET /pulp/api/v3/contentguards/certguard/rhsm/?name=RHSMCertGuard HTTP/1.1" 200 52 "-" "OpenAPI-Generator/3.49.3/ruby"
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: pulp [4c33ccc9-00e2-413f-a9cc-fefa20bcc212]: django.request:ERROR: Internal Server Error: /pulp/api/v3/contentguards/certguard/rhsm/
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: Traceback (most recent call last):
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: response = get_response(request)
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: ^^^^^^^^^^^^^^^^^^^^^
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: return view_func(*args, **kwargs)
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/rest_framework/viewsets.py", line 125, in view
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: return self.dispatch(request, *args, **kwargs)
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: response = self.handle_exception(exc)
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: self.raise_uncaught_exception(exc)
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: raise exc
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: response = handler(request, *args, **kwargs)
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/rest_framework/mixins.py", line 18, in create
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: serializer.is_valid(raise_exception=True)
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/rest_framework/serializers.py", line 227, in is_valid
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: self._validated_data = self.run_validation(self.initial_data)
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/rest_framework/serializers.py", line 426, in run_validation
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: value = self.to_internal_value(data)
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/rest_framework/serializers.py", line 485, in to_internal_value
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: validated_value = validate_method(validated_value)
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/pulp_certguard/app/serializers.py", line 48, in validate_ca_certificate
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: get_rhsm() # Validate that rhsm is installed
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: ^^^^^^^^^^
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: File "/usr/lib/python3.11/site-packages/pulp_certguard/app/utils.py", line 27, in get_rhsm
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: raise ImproperlyConfigured(error_msg.format(rhsm_import_error))
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: django.core.exceptions.ImproperlyConfigured: RHSMCertGuard requires the Python package 'rhsm' to be installed (No module named 'rhsm').
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39054]: pulp [4c33ccc9-00e2-413f-a9cc-fefa20bcc212]: - - [17/Apr/2024:07:02:13 +0000] "POST /pulp/api/v3/contentguards/certguard/rhsm/ HTTP/1.1" 500 145 "-" "OpenAPI-Generator/3.49.3/ruby"
Apr 17 07:02:13 ip-192-168-122-85 pulpcore-api[39047]: pulp [4c33ccc9-00e2-413f-a9cc-fefa20bcc212]: - - [17/Apr/2024:07:02:13 +0000] "GET /pulp/api/v3/contentguards/certguard/rhsm/ HTTP/1.1" 200 52 "-" "OpenAPI-Generator/3.49.3/ruby"
Which seems like a packaging issue to me. I’ll dig into that.
1 Like
1 Like
For the first failure, I think what we’re running into is this behaviour change in Pulpcore (since 3.44):
Looking at the pulp logs, it seems we delete the distribution too late:
[38963d1d-33da-4bf1-8f3a-8153c556e7b5]: - - [18/Apr/2024:23:24:00 +0000] "DELETE /pulp/api/v3/remotes/rpm/rpm/018ef383-6af8-76b2-9c66-29312edd724e/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.25.2/ruby"
[38963d1d-33da-4bf1-8f3a-8153c556e7b5]: - - [18/Apr/2024:23:24:00 +0000] "DELETE /pulp/api/v3/distributions/rpm/rpm/018ef383-7495-7ca7-8f9b-1b13df94fb5c/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.25.2/ruby"
[38963d1d-33da-4bf1-8f3a-8153c556e7b5]: - - [18/Apr/2024:23:24:00 +0000] "DELETE /pulp/api/v3/repositories/rpm/rpm/018ef383-6c76-7526-8fc3-3c8f9f905db4/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.25.2/ruby"
[38963d1d-33da-4bf1-8f3a-8153c556e7b5]: - - [18/Apr/2024:23:24:01 +0000] "DELETE /pulp/api/v3/distributions/rpm/rpm/018ef385-64ef-7c2f-82de-f20343c2d739/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.25.2/ruby"
[38963d1d-33da-4bf1-8f3a-8153c556e7b5]: - - [18/Apr/2024:23:24:01 +0000] "DELETE /pulp/api/v3/distributions/rpm/rpm/018ef384-63e7-706b-960e-c882dea30078/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.25.2/ruby"
[38963d1d-33da-4bf1-8f3a-8153c556e7b5]: - - [18/Apr/2024:23:24:01 +0000] "DELETE /pulp/api/v3/distributions/rpm/rpm/018ef384-af18-7398-9a3e-287c189be303/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.25.2/ruby"
[38963d1d-33da-4bf1-8f3a-8153c556e7b5]: - - [18/Apr/2024:23:24:02 +0000] "DELETE /pulp/api/v3/distributions/rpm/rpm/018ef385-31c0-7dce-bffb-587824151a31/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.25.2/ruby"
[38963d1d-33da-4bf1-8f3a-8153c556e7b5]: - - [18/Apr/2024:23:24:02 +0000] "DELETE /pulp/api/v3/repositories/rpm/rpm/018ef384-471e-72c9-8a53-1d250b00b6ea/versions/2/ HTTP/1.1" 400 174 "-" "OpenAPI-Generator/3.25.2/ruby"
[38963d1d-33da-4bf1-8f3a-8153c556e7b5]: - - [18/Apr/2024:23:24:02 +0000] "DELETE /pulp/api/v3/distributions/rpm/rpm/018ef384-7422-7e74-aded-363d050d5b66/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.25.2/ruby"
[38963d1d-33da-4bf1-8f3a-8153c556e7b5]: - - [18/Apr/2024:23:24:03 +0000] "DELETE /pulp/api/v3/distributions/rpm/rpm/018ef384-bb85-7586-bef3-aabd2a5195ce/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.25.2/ruby"
[38963d1d-33da-4bf1-8f3a-8153c556e7b5]: - - [18/Apr/2024:23:24:03 +0000] "DELETE /pulp/api/v3/distributions/rpm/rpm/018ef384-ecf7-7f75-b188-c2eb679f2e69/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.25.2/ruby"
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.