Cannot publish new version of content view

Problem:
I tried to publish a new version of a content view consisting of several repositories. The publish task gets stuck and in the log I see several entries like these:
pulpcore-api: pulp [f408d1bc-6dc7-4c92-83ae-327f3a47e11d]: - - [05/May/2023:15:31:26 +0000] “GET /pulp/api/v3/repositories/rpm/rpm/68b48011-6a0c-494b-b80e-b1d1e674225b/ HTTP/1.1” 404 23 “-” “OpenAPI-Generator/3.17.4/ruby”
There are 8 lines with different UUIDs which apparently correspond to the 8 RPM repos the content view contains. So far I could not find out what the UUIDs are referring to.

Expected outcome:
New version of content view is created

Foreman and Proxy versions:
3.2.1, Katello 4.4.2.2

Foreman and Proxy plugin versions:

Distribution and version:
CentOS 7.9

Other relevant data:
There were similar problems reported, but some affected Katello before 4.4.2.2, or only package groups. The fixes described there didn’t help. One solution I haven’t tried so far is a full repo sync. Will do that soon.

Hi @hjb,

If you open up the foreman task for the publish and then open the Dynflow Console (by hitting the Dynflow Console button), which action does it appear to be stuck on? Feel free to send screen shots or paste the contents of the page.

Hi @iballou,

I was just investigating that. It fails in Actions::Pulp3::Repository::SaveVersion. When I skip all the failed actions, the content view is created, but I don’t know yet if it is usable.

Here is all the info about this step from DynFlow.

9: Actions::Pulp3::Repository::SaveVersion (skipped) [ 401.59s / 0.40s ]

Queue: default

Started at: 2023-05-09 11:56:20 UTC

Ended at: 2023-05-09 12:03:01 UTC

Real time: 401.59s

Execution time (excluding suspended state): 0.40s

Input:


repository_id: 1337
tasks:
repository_details:
force_fetch_version: false
remote_user: admin
remote_cp_user: admin
current_request_id: b160a3ba-7d46-4c91-8618-1e31766f8e54
current_timezone: Berlin
current_organization_id:
current_location_id:
current_user_id: 4

Output:

— {}

Chunked output:

Error:

PulpRpmClient::ApiError

Error message: the server returns an error HTTP status code: 404 Response headers: {“date”=>“Tue, 09 May 2023 11:56:21 GMT”, “server”=>“gunicorn”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“GET, PUT, PATCH, DELETE, HEAD, OPTIONS”, “x-frame-options”=>“DENY”, “content-length”=>“23”, “x-content-type-options”=>“nosniff”, “referrer-policy”=>“same-origin”, “correlation-id”=>“b160a3ba-7d46-4c91-8618-1e31766f8e54”, “access-control-expose-headers”=>“Correlation-ID”, “via”=>“1.1 operator.aqua.cjt”, “connection”=>“close”} Response body: {“detail”:“Not found.”}


  • “/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.17.4/lib/pulp_rpm_client/api_client.rb:83:in
    `call_api’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.17.4/lib/pulp_rpm_client/api/repositories_rpm_api.rb:438:in
    `read_with_http_info’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.17.4/lib/pulp_rpm_client/api/repositories_rpm_api.rb:385:in
    `read’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.2.2/app/lib/actions/pulp3/repository/save_version.rb:45:in
    `fetch_version_href’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.2.2/app/lib/actions/pulp3/repository/save_version.rb:19:in
    `run’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:582:in
    `block (3 levels) in execute_run’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in
    `pass’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in
    `pass’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in
    `run’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in
    `call’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in
    `pass’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in
    `pass’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.2.2/app/lib/actions/middleware/remote_action.rb:16:in
    `block in run’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.2.2/app/lib/actions/middleware/remote_action.rb:40:in
    `block in as_remote_user’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.2.2/app/models/katello/concerns/user_extensions.rb:21:in
    `cp_config’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.2.2/app/lib/actions/middleware/remote_action.rb:27:in
    `as_cp_user’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.2.2/app/lib/actions/middleware/remote_action.rb:39:in
    `as_remote_user’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.2.2/app/lib/actions/middleware/remote_action.rb:16:in
    `run’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in
    `call’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in
    `pass’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in
    `pass’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
    `block in run’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in
    `wrap’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
    `run’”
  • “/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in
    `call’”

I looked up the repository_id in PostgreSQL:

select * from katello_repositories where id = 1337;
id | pulp_id | library_instance_id | content_view_version_id | relative_pa
th | environment_id | saved_checksum_type | distribution_version | distribution_arch | distribution_bootable | distribution_family
| distribution_variant | container_repository_name | root_id | remote_href | publication_href |
version_href | last_contents_changed | last_applicability_regen | last_indexed
------±--------------------------------------------------------------±--------------------±------------------------±-------------------------------------------
-----------------------------------±---------------±--------------------±---------------------±------------------±----------------------±--------------------
±---------------------±--------------------------±--------±------------±------------------------------------------------------------------------±------------
-----------------------------------------------------------------------±----------------------±-------------------------±---------------------------
1337 | 1-AQuA_Server_Base-v11_8-4e0e86f1-0670-4a8e-80b1-bfef3fb313f7 | 4 | 90 | AQuA/content_views/AQuA_Server_Base/11.8/cu
stom/CentOS7/centos7_extras_x86_64 | | | | | |
| | | 4 | | /pulp/api/v3/publications/rpm/rpm/6fee8e0d-74cd-4792-ac25-a8d07da21a7a/ | /pulp/api/v3
/repositories/rpm/rpm/8caf4c27-f1df-4619-ac3b-996c5f3d69f0/versions/6/ | 1970-01-01 00:00:00 | 1970-01-01 00:00:00 | 2023-05-09 12:03:05.538785

And here’s the command I used to create the content view:

hammer content-view publish --id 2 --major 11 --minor 8 --description “CentOS 7.9/Foreman Client 3.3”

Thanks for your help.

Best regards,
Hans-Joachim

Thanks for the info, do you mind opening up the foreman console foreman-rake console and show us the output of:

::Katello::Repository.find(1337).backend_service(SmartProxy.pulp_primary).repository_reference.repository_href

That should return /pulp/api/v3 /repositories/rpm/rpm/8caf4c27-f1df-4619-ac3b-996c5f3d69f0/

Also, Katello 4.4 is quite old, after we get your issues here settled I’d recommend upgrading if you can. We consider our last two releases to be “supported” in that we backport fixes to them. Currently those two are Katello 4.7 and Katello 4.8.

The actual result is

“/pulp/api/v3/repositories/rpm/rpm/6ef763de-9cea-4825-8364-9a77e3a77033/”

If a SmartProxy is involved here, then I should mention that we recently added another SmartProxy for testing purposes. Could this cause the problem if it is out of sync or otherwise?

I tested on a different instance with essentially the same config, but without an additional smartproxy. Same error. So it is unlikely that the smartproxy is the problem here.

You’re right, the smart proxy isn’t the issue. We just have to mention the related smart proxy in the console so our code knows which Pulp API instance to hit.

If the issue is reproducible across two systems, are you able to give reproducer steps? For example, how are you creating the repositories and content views? Are you using content view filters? Have you run an incremental update?

Sorry, I don’t know how to reproduce it. I have read of problems with content view filters but they don’t apply because we don’t use them.

We don’t know when exactly it started to go wrong. Half a year ago I deleted some old versions of content views and I suspect it could have to do with that.

What we found out: If we make copies of the affected content views, we are able to publish new versions without problems. Also when we go to https://server/pulp/content/, all RPM repos of the affected content views are missing. In contrast, the repos of the new copies of the content views do all appear. Is that a candlepin problem? We had something similar before.

Hey, sorry for the late reply, my draft never sent!


It looks like you might have some Katello DB corruption, perhaps due to an old bug.

The first thing that I would try would be to correct the repository reference that I mentioned earlier:

::Katello::Repository.find(1337).backend_service(SmartProxy.pulp_primary).repository_reference.update(repository_href: '/pulp/api/v3 /repositories/rpm/rpm/8caf4c27-f1df-4619-ac3b-996c5f3d69f0/')

I’m assuming here that you’re still seeing an error that mentions “repository_id: 1337”.

In Katello we have these references to Pulp repositories that tell content view repos how to line up with their Pulp backend repos.

If you’re uncomfortable with messing around with the DB from the console, a workaround should be to delete the content view and recreate it.

Hi!

Thank you for the hint, I will try it out. I share your suspicion that the problem comes from an older bug. On the two instances where I see problems I had deleted some outdated content views and I think this caused the problems.

In the meantime I worked around the problem:

  1. Renamed the affected content views
  2. Renamed also the labels via database commands
  3. Made copies of the content views with their original name and also the original label

The new copies could be published and promoted without an error. The problematic content views are still available for investigating.

1 Like

Great to hear that the bug didn’t follow the new content views! Appreciate the workaround steps too, for anyone else hitting this it should be helpful.

The console command didn’t work and I don’t have the time to investigate further. At least as long as the new content views are working :wink: Thank you very much for your help.