Strange pulp3 error

mhjacks · October 2, 2021, 1:14am

Problem: One of my content views is failing consistently on what appears to be a pulpapi error.

Expected outcome: Content view publishes or errors cleanly

Foreman and Proxy versions: 3.0.0 / Katello 4.2.0rc1

Foreman and Proxy plugin versions:

Distribution and version: CentOS 8-stream

Other relevant data:

Message from task output:

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Sat, 02 Oct 2021 01:07:55 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"104", "correlation-id"=>"99365825-65f2-4bdd-9315-26a9d00618f2", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 srv-katello.imladris.lan", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/content/rpm/advisories/09d8fae6-4043-4809-9624-a19e6a440332/ not found for content."]

pulpcore-api log:

Oct 01 20:07:54 srv-katello.imladris.lan pulpcore-api[959]: pulp [99365825-65f2-4bdd-9315-26a9d00618f2]:  - - [02/Oct/2021:01:07:54 +0000] "GET /pulp/api/v3/content/rpm/repo_metadata_files/?repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Frpm%2Frpm%2F86fd3813-8d1b-424e-a436-fe426cb37531%2Fversions%2F16%2F HTTP/1.1" 200 52 "-" "OpenAPI-Generator/3.13.3/ruby"   Oct 01 20:07:54 srv-katello.imladris.lan pulpcore-api[959]: pulp [99365825-65f2-4bdd-9315-26a9d00618f2]:  - - [02/Oct/2021:01:07:54 +0000] "GET /pulp/api/v3/content/rpm/distribution_trees/?repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Frpm%2Frpm%2F86fd3813-8d1b-424e-a436-fe426cb37531%2Fversions%2F16%2F HTTP/1.1" 200 52 "-" "OpenAPI-Generator/3.13.3/ruby"    Oct 01 20:07:54 srv-katello.imladris.lan pulpcore-api[959]: pulp [99365825-65f2-4bdd-9315-26a9d00618f2]:  - - [02/Oct/2021:01:07:54 +0000] "GET /pulp/api/v3/content/rpm/packageenvironments/?limit=2000&offset=0&repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Frpm%2Frpm%2F86fd3813-8d1b-424e-a436-fe426cb37531%2Fversions%2F16%2F HTTP/1.1" 200 9002 "-" "OpenAPI-Generator/3.13.3/ruby"
Oct 01 20:07:54 srv-katello.imladris.lan pulpcore-api[959]: pulp [99365825-65f2-4bdd-9315-26a9d00618f2]:  - - [02/Oct/2021:01:07:54 +0000] "POST /pulp/api/v3/repositories/rpm/rpm/70ae0f2e-de76-4b60-86b2-df69ba2da8e7/modify/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.13.3/ruby"
Oct 01 20:07:55 srv-katello.imladris.lan pulpcore-api[959]: pulp [99365825-65f2-4bdd-9315-26a9d00618f2]: django.request:WARNING: Bad Request: /pulp/api/v3/repositories/rpm/rpm/70ae0f2e-de76-4b60-86b2-df69ba2da8e7/modify/
Oct 01 20:07:55 srv-katello.imladris.lan pulpcore-api[959]: pulp [99365825-65f2-4bdd-9315-26a9d00618f2]:  - - [02/Oct/2021:01:07:55 +0000] "POST /pulp/api/v3/repositories/rpm/rpm/70ae0f2e-de76-4b60-86b2-df69ba2da8e7/modify/ HTTP/1.1" 400 104 "-" "OpenAPI-Generator/3.13.3/ruby"

mhjacks · October 2, 2021, 1:35am

The repository in question is EPEL 8 with some filters on it to prevent conflicts with packages from pulpcore/katello/foreman repos. Specifically:

||Exclude python3-tablib||Sep 13, 08:33 PM|RPM|Exclude|
||Exclude rubygem-gssapi||Sep 13, 08:33 PM|RPM|Exclude|
||Exclude rubygem-net_ssh||Sep 13, 08:33 PM|RPM|Exclude|
||Exclude rubygem-qpid_proton*||Sep 13, 10:46 PM|RPM|Exclude|
||Exclude rubygem-sqlite3*||Sep 13, 08:33 PM|RPM|Exclude|

mhjacks · October 2, 2021, 2:02am

Correction: This is occurring on katello 4.2.0rc3

Justin_Sherrill · October 5, 2021, 2:46pm

I think we’'ll need some more information. could you run the following and provide the output?:

from the foreman/katello server run as root:

foreman-rake console

Katello::RepositoryErratum.where(:erratum_pulp3_href => "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-4809-9624-a19e6a440332/")

from the foreman/katello server run as root:

curl -vvv https://`hostname`/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-4809-9624-a19e6a440332/  --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key

mhjacks · October 5, 2021, 2:54pm

Sure, here they are in order. Hope this helps!

irb(main):001:0> Katello::RepositoryErratum.where(:erratum_pulp3_href => "/pulp/api/v3/content/rpm/a
dvisories/09d8fae6-4043-4809-9624-a19e6a440332/")
=> #<ActiveRecord::Relation [#<Katello::RepositoryErratum id: 2835568, erratum_id: 10000, repository_id: 10894, created_at: "2021-10-04 04:06:24", updated_at: "2021-10-04 04:06:24", erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-...">, #<Katello::RepositoryErratum id: 2838697, erratum_id: 10000, repository_id: 10896, created_at: "2021-10-04 04:07:29", updated_at: "2021-10-04 04:07:29", erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-...">, #<Katello::RepositoryErratum id: 2729606, erratum_id: 10000, repository_id: 10510, created_at: "2021-10-03 10:06:19", updated_at: "2021-10-03 10:06:19", erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-...">, #<Katello::RepositoryErratum id: 2889834, erratum_id: 10000, repository_id: 11078, created_at: "2021-10-04 15:55:49", updated_at: "2021-10-04 15:55:49", erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-...">, #<Katello::RepositoryErratum id: 2780700, erratum_id: 10000, repository_id: 10674, created_at: "2021-10-03 21:55:48", updated_at: "2021-10-03 21:55:48", erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-...">, #<Katello::RepositoryErratum id: 2896347, erratum_id: 10000, repository_id: 11112, created_at: "2021-10-04 16:00:33", updated_at: "2021-10-04 16:00:33", erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-...">, #<Katello::RepositoryErratum id: 1129612, erratum_id: 10000, repository_id: 149, created_at: "2021-09-17 22:10:36", updated_at: "2021-09-17 22:10:36", erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-...">, #<Katello::RepositoryErratum id: 2694273, erratum_id: 10000, repository_id: 10382, created_at: "2021-10-03 04:05:31", updated_at: "2021-10-03 04:05:31", erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-...">, #<Katello::RepositoryErratum id: 2844998, erratum_id: 10000, repository_id: 10912, created_at: "2021-10-04 04:10:26", updated_at: "2021-10-04 04:10:26", erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-...">, #<Katello::RepositoryErratum id: 2944722, erratum_id: 10000, repository_id: 11302, created_at: "2021-10-05 03:55:49", updated_at: "2021-10-05 03:55:49", erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-...">, ...]>

*   Trying 192.168.5.40...
* TCP_NODELAY set
* Connected to srv-katello.imladris.lan (192.168.5.40) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=North Carolina; O=Katello; OU=SomeOrgUnit; CN=srv-katello.imladris.lan
*  start date: Sep  4 03:31:11 2021 GMT
*  expire date: Jan 18 03:31:12 2038 GMT
*  subjectAltName: host "srv-katello.imladris.lan" matched cert's "srv-katello.imladris.lan"
*  issuer: C=US; ST=North Carolina; L=Raleigh; O=Katello; OU=SomeOrgUnit; CN=srv-katello.imladris.lan
*  SSL certificate verify ok.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /pulp/api/v3/content/rpm/advisories/09d8fae6-4043-4809-9624-a19e6a440332/ HTTP/1.1
> Host: srv-katello.imladris.lan
> User-Agent: curl/7.61.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 404 Not Found
< Date: Tue, 05 Oct 2021 14:53:08 GMT
< Server: gunicorn
< Content-Type: application/json
< Vary: Accept,Cookie
< Allow: GET, HEAD, OPTIONS
< X-Frame-Options: SAMEORIGIN
< Content-Length: 23
< Correlation-ID: f3425357c61043bcb7dd9e9730136b8e
< Access-Control-Expose-Headers: Correlation-ID
< Via: 1.1 srv-katello.imladris.lan
< 
* Connection #0 to host srv-katello.imladris.lan left intact
{"detail":"Not found."}

iballou · October 5, 2021, 4:08pm

@mhjacks when were you able to publish this content view last without any errors? Was it on a previous version?

mhjacks · October 5, 2021, 5:15pm

Last good CV publish I have is from Oct 01, 08:07 PM. I’m only having this problem with the filters applied - I have another CV of EPEL 8 x86_64 that doesn’t have the filters and consistently publishes just fine.

iballou · October 5, 2021, 5:43pm

One thing to try would be doing a full re-sync of EPEL 8 before trying another publish. It looks like the erratum no longer exists in Pulp for some reason, but it is still indexed into Katello.

Also, if re-syncing doesn’t work, could you please show us the result of the following in the console:

::Katello::Erratum.find(10000)

I’m curious about which specific erratum is causing the trouble.

If we know the erratum’s ID, we could search for it in Pulp directly to check if the :erratum_pulp3_href changed somehow.

mhjacks · October 5, 2021, 5:48pm

=> #<Katello::Erratum id: 10000, pulp_id: "FEDORA-EPEL-2021-f177872061", errata_id: "FEDORA-EPEL-2021-f177872061", created_at: "2021-09-17 17:12:15", updated_at: "2021-09-19 05:11:37", issued: "2021-09-17", updated: "2021-09-10", errata_type: "bugfix", severity: "None", title: "singularity-3.8.3-1.el8", solution: "", description: "Upgrade to upstream 3.8.3\n\n----\n\nUpgrade to upstre...", summary: "singularity-3.8.3-1.el8 bugfix update", reboot_suggested: false, migrated_pulp3_href: nil>

Why would this happen in the filtered content view but not the unfiltered one? Aren’t they tied to the same repository?

iballou · October 5, 2021, 8:14pm

For content views without filters, we simply copy over the Pulp repository’s version href. That essentially means we’re pointing to the Library repo from a different published URL. If there are filters, we actually copy the units between Pulp repositories. Katello is telling Pulp in your case to copy an erratum that doesn’t exist in Pulp anymore, which is causing the error.

Okay, so that erratum is definitely still in EPEL. What does the following show?

curl -vvv https://`hostname`/pulp/api/v3/content/rpm/advisories/?id=FEDORA-EPEL-2021-f177872061  --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key

mhjacks · October 5, 2021, 8:17pm

I see, thanks for the explanation.

Here’s what I get for that command:

# curl -vvv https://`hostname`/pulp/api/v3/content/rpm/advisories/?id=FEDORA-EPEL-2021-f177872061  --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key
*   Trying 192.168.5.40...
* TCP_NODELAY set
* Connected to srv-katello.imladris.lan (192.168.5.40) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=North Carolina; O=Katello; OU=SomeOrgUnit; CN=srv-katello.imladris.lan
*  start date: Sep  4 03:31:11 2021 GMT
*  expire date: Jan 18 03:31:12 2038 GMT
*  subjectAltName: host "srv-katello.imladris.lan" matched cert's "srv-katello.imladris.lan"
*  issuer: C=US; ST=North Carolina; L=Raleigh; O=Katello; OU=SomeOrgUnit; CN=srv-katello.imladris.lan
*  SSL certificate verify ok.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /pulp/api/v3/content/rpm/advisories/?id=FEDORA-EPEL-2021-f177872061 HTTP/1.1
> Host: srv-katello.imladris.lan
> User-Agent: curl/7.61.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 200 OK
< Date: Tue, 05 Oct 2021 20:15:53 GMT
< Server: gunicorn
< Content-Type: application/json
< Vary: Accept,Cookie,Accept-Encoding
< Allow: GET, POST, HEAD, OPTIONS
< X-Frame-Options: SAMEORIGIN
< Content-Length: 4336
< Correlation-ID: f92508588849438990c77693b6f0f180
< Access-Control-Expose-Headers: Correlation-ID
< Via: 1.1 srv-katello.imladris.lan
< 
* TLSv1.3 (IN), TLS app data, [no content] (0):
{"count":1,"next":null,"previous":null,"results":[{"pulp_href":"/pulp/api/v3/content/rpm/advisories/7cf1120b-8a5a-4c2b-bb93-7b82e529c5a2/","pulp_created":"2021-09-19T05:09:39.063025Z","id":"FEDORA-EPEL-2021-f177872061","updated_date":"2021-09-10 13:40:15","description":"Upgrade to upstream 3.8.3\n\n----\n\nUpgrade to upstream 3.8.2\n\n","issued_date":"2021-09-17 15:53:29","fromstr":"updates@fedoraproject.org","status":"stable","title":"singularity-3.8.3-1.el8","summary":"singularity-3.8.3-1.el8 bugfix update","version":"2.0","type":"bugfix","severity":"None","solution":"","release":"Fedora EPEL 8","rights":"Copyright (C) 2021 Red Hat, Inc. and others.","pushcount":"","pkglist":[{"name":"Fedora EPEL 8","shortname":"EPEL-8","module":null,"packages":[{"arch":"src","epoch":"0","filename":"singularity-3.8.3-1.el8.src.rpm","name":"singularity","reboot_suggested":false,"relogin_suggested":false,"restart_suggested":false,"release":"1.el8","src":"https://download.fedoraproject.org/pub/fedora/linux/updates/8/SRPMS/s/singularity-3.8.3-1.el8.src.rpm","sum":"","sum_type":"","version":"3.8.3"},{"arch":"aarch64","epoch":"0","filename":"singularity-3.8.3-1.el8.aarch64.rpm","name":"singularity","reboot_suggested":false,"relogin_suggested":false,"restart_suggested":false,"release":"1.el8","src":"https://download.fedoraproject.org/pub/fedora/linux/updates/8/aarch64/s/singularity-3.8.3-1.el8.aarch64.rpm","sum":"","sum_type":"","version":"3.8.3"},{"arch":"aarch64","epoch":"0","filename":"singularity-debuginfo-3.8.3-1.el8.aarch64.rpm","name":"singularity-debuginfo","reboot_suggested":false,"relogin_suggested":false,"restart_suggested":false,"release":"1.el8","src":"https://download.fedoraproject.org/pub/fedora/linux/updates/8/aarch64/s/singularity-debuginfo-3.8.3-1.el8.aarch64.rpm","sum":"","sum_type":"","version":"3.8.3"},{"arch":"ppc64le","epoch":"0","filename":"singularity-3.8.3-1.el8.ppc64le.rpm","name":"singularity","reboot_suggested":false,"relogin_suggested":false,"restart_suggested":false,"release":"1.el8","src":"https://download.fedoraproject.org/pub/fedora/linux/updates/8/ppc64le/s/singularity-3.8.3-1.el8.ppc64le.rpm","sum":"","sum_type":"","version":"3.8.3"},{"arch":"ppc64le","epoch":"0","filename":"singularity-debuginfo-3.8.3-1.el8.ppc64le.rpm","name":"singularity-debuginfo","reboot_suggested":false,"relogin_suggested":false,"restart_suggested":false,"release":"1.el8","src":"https://download.fedoraproject.org/pub/fedora/linux/updates/8/ppc64le/s/singularity-debuginfo-3.8.3-1.el8.ppc64le.rpm","sum":"","sum_type":"","version":"3.8.3"},{"arch":"s390x","epoch":"0","filename":"singularity-3.8.3-1.el8.s390x.rpm","name":"singularity","reboot_suggested":false,"relogin_suggested":false,"restart_suggested":false,"release":"1.el8","src":"https://download.fedoraproject.org/pub/fedora/linux/updates/8/s390x/s/singularity-3.8.3-1.el8.s390x.rpm","sum":"","sum_type":"","version":"3.8.3"},{"arch":"s390x","epoch":"0","filename":"singularity-debuginfo-3.8.3-1.el8.s390x.rpm","name":"singularity-debuginfo","reboot_suggested":false,"relogin_suggested":false,"restart_suggested":false,"release":"1.el8","src":"https://download.fedoraproject.org/pub/fedora/linux/updates/8/s390x/s/singularity-debuginfo-3.8.3-1.el8.s390x.rpm","sum":"","sum_type":"","version":"3.8.3"},{"arch":"x86_64","epoch":"0","filename":"singularity-3.8.3-1.el8.x86_64.rpm","name":"singularity","reboot_suggested":false,"relogin_suggested":false,"restart_suggested":false,"release":"1.el8","src":"https://download.fedoraproject.org/pub/fedora/linux/updates/8/x86_64/s/singularity-3.8.3-1.el8.x86_64.rpm","sum":"","sum_type":"","version":"3.8.3"},{"arch":"x86_64","epoch":"0","filename":"singularity-debuginfo-3.8.3-1.el8.x86_64.rpm","name":"singularity-debuginfo","reboot_suggested":false,"relogin_suggested":false,"restart_suggested":false,"release":"1.el8","src":"https://download.fedoraproject.org/pub/fedora/linux/updates/8/x86_64/s/singularity-debuginfo-3.8.3-1.el8.x86_64.rpm","sum":"","sum_type":"","version":"3.8.3"}]}],"references":[{"href":"https://bugzilla.redhat.com/show_bug.cgi?id=2000194",* Connection #0 to host srv-katello.imladris.lan left intact
"id":"2000194","title":"singularity-3.8.2 is available","type":"bugzilla"},{"href":"https://bugzilla.redhat.com/show_bug.cgi?id=2002195","id":"2002195","title":"singularity-3.8.3 is available","type":"bugzilla"}],"reboot_suggested":false}]}

iballou · October 6, 2021, 2:52pm

Thanks. There’s a chance that you’re experiencing a bug in our content indexing code. I’m going to try reproducing the issue myself.

mhjacks · October 6, 2021, 9:57pm

Thanks so much for looking into this!

mhjacks · October 8, 2021, 1:13pm

I saw there was an update in the pulp repos today, and I updated to the new versions and tried to generate and promote this content view again. I get different looking results now, but what seems to be the same error (this is after deleting and re-creating the content view):

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Fri, 08 Oct 2021 13:09:29 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"6054", "correlation-id"=>"39ae80d3-8d11-41ff-85d0-3500ca7260f5", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 srv-katello.imladris.lan", "connection"=>"close"}
Response body: ["Could not find the following content units: ['/pulp/api/v3/content/rpm/advisories/3d4d0595-f6d9-48fb-9148-bfbcf40d4a72/', '/pulp/api/v3/content/rpm/advisories/a6be09c7-8fae-49e1-80dc-1d5f9165e94b/', '/pulp/api/v3/content/rpm/advisories/418fcc4c-4cba-4962-84e9-beade6d87297/', '/pulp/api/v3/content/rpm/advisories/6de43375-558d-42b1-9a18-69a56c382bd6/', '/pulp/api/v3/content/rpm/advisories/7dc21dae-7b88-4998-b60f-b4d12cc0e32e/', '/pulp/api/v3/content/rpm/advisories/f7206432-61b2-4da6-85e3-be2058f03840/', '/pulp/api/v3/content/rpm/advisories/763658ee-c224-43fe-8026-cfdd7b372d8b/', '/pulp/api/v3/content/rpm/advisories/1a60b4ae-03b1-4519-b5f6-98ac1d55a844/', '/pulp/api/v3/content/rpm/advisories/0cbfbba5-5edd-4376-9bce-0bf38b9e4aee/', '/pulp/api/v3/content/rpm/advisories/a9f59032-3061-47f5-b7b8-b2af1c541385/', '/pulp/api/v3/content/rpm/advisories/a06a5caf-0244-4738-8c01-5246199128d4/', '/pulp/api/v3/content/rpm/advisories/9528cf63-3f7f-441f-83a1-755611555486/', '/pulp/api/v3/content/rpm/advisories/283205ec-12f9-4b16-9830-e954b999c538/', '/pulp/api/v3/content/rpm/advisories/5d83c08b-cb72-4e6f-9116-0a8c43a3e72e/', '/pulp/api/v3/content/rpm/advisories/af38624a-eb98-42a0-8f04-e1fe2153bb45/', '/pulp/api/v3/content/rpm/advisories/dae01b92-f090-4815-929f-8bcb07ee4637/', '/pulp/api/v3/content/rpm/advisories/b7bbeed9-fb5b-4816-8c82-13bbaf8e4e55/', '/pulp/api/v3/content/rpm/advisories/81a396a0-c69e-4990-955e-e23f2654516a/', '/pulp/api/v3/content/rpm/advisories/8de22076-e4f2-4d95-be64-5f0a6dc967e8/', '/pulp/api/v3/content/rpm/advisories/df9a469d-8925-4777-a9e2-17fde145d2fc/', '/pulp/api/v3/content/rpm/advisories/f568a850-e715-420b-be5d-5d373dd36c8e/', '/pulp/api/v3/content/rpm/advisories/5ab1679c-e4a7-46a4-9869-15a4230a09f0/', '/pulp/api/v3/content/rpm/advisories/e64024f6-188e-46b9-85ab-c52ece9ed146/', '/pulp/api/v3/content/rpm/advisories/4260432d-44a9-4c46-b323-53431d2d9494/', '/pulp/api/v3/content/rpm/advisories/4c7043a2-a93f-4320-87cc-fa4e3cb62141/', '/pulp/api/v3/content/rpm/advisories/6de042b9-7eb3-485c-a49b-b7511d742ec6/', '/pulp/api/v3/content/rpm/advisories/6a447d71-4b80-4faa-b614-3cb6ba79a114/', '/pulp/api/v3/content/rpm/advisories/882d3c55-a3ef-4bcc-8748-030e697e0d65/', '/pulp/api/v3/content/rpm/advisories/47507859-8076-4ff1-ba4d-cdff8c7c53b7/', '/pulp/api/v3/content/rpm/advisories/91225f5c-7f7f-4d40-9319-6cf45b1ae814/', '/pulp/api/v3/content/rpm/advisories/4127eca6-9cc6-4486-910d-f1a61c26c62d/', '/pulp/api/v3/content/rpm/advisories/6238690f-1b3c-4b67-a1af-62a7016c10ab/', '/pulp/api/v3/content/rpm/advisories/b88a9079-b5bc-4407-bb32-18aa429a04fb/', '/pulp/api/v3/content/rpm/advisories/b54b6138-06ee-423e-aff6-0bafb4c1c977/', '/pulp/api/v3/content/rpm/advisories/9ec9e087-eb98-4052-bba3-cdcaceabe509/', '/pulp/api/v3/content/rpm/advisories/d743e1de-3da3-4b49-ab53-d7fa3b74e3b5/', '/pulp/api/v3/content/rpm/advisories/70b1f79a-4d8f-4844-a23d-b330a15618a0/', '/pulp/api/v3/content/rpm/advisories/194331bd-70fe-4390-881d-65326e999c72/', '/pulp/api/v3/content/rpm/advisories/695f2959-9e44-44d7-98f9-df4ab3173331/', '/pulp/api/v3/content/rpm/advisories/ff7897ef-bffd-488e-8d09-5b9233c47bb2/', '/pulp/api/v3/content/rpm/advisories/3562ca40-e2ff-41c7-8abb-fe7a64231b13/', '/pulp/api/v3/content/rpm/advisories/2c19a66a-0f11-4618-9948-f6fb3a28c584/', '/pulp/api/v3/content/rpm/advisories/899a8b6c-dd0e-4318-92b3-cb5307e0d5f0/', '/pulp/api/v3/content/rpm/advisories/5816e295-b425-40da-bce2-996e0590f572/', '/pulp/api/v3/content/rpm/advisories/943057b0-76ac-4509-9499-ec779029d133/', '/pulp/api/v3/content/rpm/advisories/5daed36c-2698-41af-9b47-4b4a6def4094/', '/pulp/api/v3/content/rpm/advisories/4470e762-2852-47cd-96a3-0ecbdb66a482/', '/pulp/api/v3/content/rpm/advisories/8477b0d4-8fbc-4d75-bf0f-f869a0183ea2/', '/pulp/api/v3/content/rpm/advisories/a54274fa-3925-4ebd-a880-ccf167e64497/', '/pulp/api/v3/content/rpm/advisories/1d38b53e-ef44-4216-9e45-888355e932ad/', '/pulp/api/v3/content/rpm/advisories/803791f0-7891-41c6-8315-f4965ff4aa05/', '/pulp/api/v3/content/rpm/advisories/66cee998-68af-4d2b-972d-a8bc88395cec/', '/pulp/api/v3/content/rpm/advisories/af8e55c6-0064-4df5-a871-878270215ac0/', '/pulp/api/v3/content/rpm/advisories/ff259aac-f67e-4288-8ece-39ac4b63c618/', '/pulp/api/v3/content/rpm/advisories/33a95496-e251-466c-9660-b6eecf128d50/', '/pulp/api/v3/content/rpm/advisories/41bb5bdc-617c-4a06-bf97-dca09f20914b/', '/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-4809-9624-a19e6a440332/', '/pulp/api/v3/content/rpm/advisories/a42f9b4c-af16-401d-988b-935b4acdf920/', '/pulp/api/v3/content/rpm/advisories/38f44bdf-934e-4930-97f5-012b43ce28ca/', '/pulp/api/v3/content/rpm/advisories/b56f140f-5376-4122-89a8-240645361cd9/', '/pulp/api/v3/content/rpm/advisories/57914258-eab1-44fa-b052-d6a1ad0185ff/', '/pulp/api/v3/content/rpm/advisories/22bce75c-3f3a-4cc2-a1e5-cca2bfd7ef48/', '/pulp/api/v3/content/rpm/advisories/1a305a7f-3291-42ec-894a-f286b0a1dd77/', '/pulp/api/v3/content/rpm/advisories/b2334511-27a5-46e3-890e-9ddca1c6a974/', '/pulp/api/v3/content/rpm/advisories/a8d14775-5a1a-4176-bb91-101397dacf72/', '/pulp/api/v3/content/rpm/advisories/58584f97-b618-44b8-b08c-62b47d5ae41c/', '/pulp/api/v3/content/rpm/advisories/6e558276-f451-4866-a4b3-c340408e5d48/', '/pulp/api/v3/content/rpm/advisories/8238cc1b-6f4b-428f-8d5e-37bca6969624/', '/pulp/api/v3/content/rpm/advisories/342b8eca-6404-43f3-b7f8-090f5af9eb93/', '/pulp/api/v3/content/rpm/advisories/f9506c16-e315-4687-87ee-3bfc55eeb9f3/', '/pulp/api/v3/content/rpm/advisories/a6836381-9054-4a0c-8e83-e64e97decbc4/', '/pulp/api/v3/content/rpm/advisories/e25ac705-c7ea-4865-a80b-e4581fafa660/', '/pulp/api/v3/content/rpm/advisories/4566ee58-c015-4551-823b-37069f2ebfa9/', '/pulp/api/v3/content/rpm/advisories/5c9b45bb-5bf5-4cc0-9fcf-0aba9202f3c7/', '/pulp/api/v3/content/rpm/advisories/36cc757a-f903-458c-8836-c000bd7e7716/', '/pulp/api/v3/content/rpm/advisories/154511ee-0219-420e-b7f0-92bef6c3b9a3/', '/pulp/api/v3/content/rpm/advisories/3db0905e-99a3-4015-b23f-e01a3ca02116/', '/pulp/api/v3/content/rpm/advisories/7c84ba49-8f67-406c-af58-e50fd2e12d3e/']"]

I hope this might help. If you need more info, I’m happy to provide it - just let me know.

Thanks again for looking into this!

iballou · October 11, 2021, 1:52pm

@mhjacks,

Thanks for the info, that helps to confirm that the issue is with errata. I was able to reproduce it and I have a PR here: https://github.com/Katello/katello/pull/9703

It looks like errata that were edited in the upstream repo are not being re-indexed correctly into Katello.

mhjacks · October 11, 2021, 3:30pm

Awesome! Glad I could help

John_Beranek · November 8, 2021, 12:24pm

So, I’m getting something very similar looking on 2 Oracle Linux 8 repositories, with Katello 4.2.0/pulp-core 3.14.8/pulp-rpm 3.14.6.

The ‘baseos’ and ‘appstream’ repositories won’t sync, and I can’t publish new CVVs for the CV containing these repositories either.

Which version of Katello should fix this?

John_Beranek · November 8, 2021, 1:29pm

So, I found https://github.com/Katello/katello/commit/a90925ac3dae880a0cd8eb52e2584babbaab1b30 - I presume that means the answer is Katello 4.2.1?

mhjacks · November 8, 2021, 1:57pm

I’ve been watching this too. 4.2.1 is in the release process now (Katello 4.2.1 release process). I am hopeful this fix will be included…

mhjacks · November 9, 2021, 3:03pm

Running 4,2,1, I seem to be seeing the same error:

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Tue, 09 Nov 2021 03:54:19 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"12599", "correlation-id"=>"0912d652-dcb0-46f6-b68e-6f221f94477f", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 srv-katello.imladris.lan", "connection"=>"close"}
Response body: ["Could not find the following content units: ['/pulp/api/v3/content/rpm/advisories/ceaa4290-488c-42cb-9f3a-b9d2da2ffc40/', '/pulp/api/v3/content/rpm/advisories/a6d3a3fb-bc29-4f55-b7a9-e38ba7fa8515/', '/pulp/api/v3/content/rpm/advisories/61443fa5-34a6-4775-a64a-4cad51b4ffcf/', '/pulp/api/v3/content/rpm/advisories/763658ee-c224-43fe-8026-cfdd7b372d8b/', '/pulp/api/v3/content/rpm/advisories/800d94b3-f516-409f-a662-71c3d1dd4ed1/', '/pulp/api/v3/content/rpm/advisories/6a8f8ed6-7b5e-455e-a1f4-56c2155442c7/', '/pulp/api/v3/content/rpm/advisories/a42f9b4c-af16-401d-988b-935b4acdf920/', '/pulp/api/v3/content/rpm/advisories/91225f5c-7f7f-4d40-9319-6cf45b1ae814/', '/pulp/api/v3/content/rpm/advisories/e4492df0-eeaf-441d-a63e-c4aa663f95f9/', '/pulp/api/v3/content/rpm/advisories/36afe4bc-d04c-4a5d-91fa-ba400ce6a0ac/', '/pulp/api/v3/content/rpm/advisories/283205ec-12f9-4b16-9830-e954b999c538/', '/pulp/api/v3/content/rpm/advisories/5975bb24-9b6a-4076-9dce-ce98aa210c40/', '/pulp/api/v3/content/rpm/advisories/b54b6138-06ee-423e-aff6-0bafb4c1c977/', '/pulp/api/v3/content/rpm/advisories/239a2bff-6c62-4a2c-b306-84322d9c0060/', '/pulp/api/v3/content/rpm/advisories/f7206432-61b2-4da6-85e3-be2058f03840/', '/pulp/api/v3/content/rpm/advisories/1a60b4ae-03b1-4519-b5f6-98ac1d55a844/', '/pulp/api/v3/content/rpm/advisories/e64024f6-188e-46b9-85ab-c52ece9ed146/', '/pulp/api/v3/content/rpm/advisories/aae4a3cd-2301-4955-a7be-9ae78177ad7e/', '/pulp/api/v3/content/rpm/advisories/57914258-eab1-44fa-b052-d6a1ad0185ff/', '/pulp/api/v3/content/rpm/advisories/0990ab7c-09e4-46b6-8988-77bb6ae15751/', '/pulp/api/v3/content/rpm/advisories/695f2959-9e44-44d7-98f9-df4ab3173331/', '/pulp/api/v3/content/rpm/advisories/806e22b9-7619-4b22-9daf-e80f0cbf25b1/', '/pulp/api/v3/content/rpm/advisories/33a95496-e251-466c-9660-b6eecf128d50/', '/pulp/api/v3/content/rpm/advisories/f568a850-e715-420b-be5d-5d373dd36c8e/', '/pulp/api/v3/content/rpm/advisories/e6612c11-07e3-418c-bc49-3d9d36b55fc0/', '/pulp/api/v3/content/rpm/advisories/d6e087bc-eb6e-4807-b9c5-bc31adb93a5a/', '/pulp/api/v3/content/rpm/advisories/5bf1b4e1-6ade-42d5-95c8-72744920c93a/', '/pulp/api/v3/content/rpm/advisories/fb3d2f44-488c-4b2c-8372-808951d51030/', '/pulp/api/v3/content/rpm/advisories/a54274fa-3925-4ebd-a880-ccf167e64497/', '/pulp/api/v3/content/rpm/advisories/fd30be41-b6f1-4d6a-872b-ccfa36e5ff7a/', '/pulp/api/v3/content/rpm/advisories/724d4d15-2fae-48b9-a28c-441e8dc46f15/', '/pulp/api/v3/content/rpm/advisories/ac8f4922-b292-4c46-80c3-ab7ba3a7da2a/', '/pulp/api/v3/content/rpm/advisories/d8ae7cb3-327a-4464-a8ae-8ec312e75c65/', '/pulp/api/v3/content/rpm/advisories/ff7897ef-bffd-488e-8d09-5b9233c47bb2/', '/pulp/api/v3/content/rpm/advisories/6fa64241-1fba-490d-9d20-b979d9a70beb/', '/pulp/api/v3/content/rpm/advisories/6d4a65da-082f-4887-aed7-7570142d89df/', '/pulp/api/v3/content/rpm/advisories/51a26cb2-5905-409b-9dec-3c494483c524/', '/pulp/api/v3/content/rpm/advisories/85306bd3-943e-4c74-9f83-ce83801b2dc8/', '/pulp/api/v3/content/rpm/advisories/532630a0-8511-4a82-8ae9-b03ccd891b7a/', '/pulp/api/v3/content/rpm/advisories/018753b1-320a-423b-a4eb-e4394ecd1d60/', '/pulp/api/v3/content/rpm/advisories/a6836381-9054-4a0c-8e83-e64e97decbc4/', '/pulp/api/v3/content/rpm/advisories/a229daa5-dcc9-415b-8766-d90d376cc424/', '/pulp/api/v3/content/rpm/advisories/8de22076-e4f2-4d95-be64-5f0a6dc967e8/', '/pulp/api/v3/content/rpm/advisories/bb8f04ae-b5e9-4a2a-83a8-e72c9dfe523f/', '/pulp/api/v3/content/rpm/advisories/8a1aa5dc-74f9-48e7-8100-b34e0f3a7605/', '/pulp/api/v3/content/rpm/advisories/edaa5a7e-c373-412b-9d1c-9341c2dff1dd/', '/pulp/api/v3/content/rpm/advisories/340d1d11-e52f-43d4-97e9-e45fb181e7f4/', '/pulp/api/v3/content/rpm/advisories/66858017-bcf5-436e-9ea8-f84b5556e90c/', '/pulp/api/v3/content/rpm/advisories/40365fcf-ce5b-41a6-bec0-97278baeaccc/', '/pulp/api/v3/content/rpm/advisories/add5d26d-458a-41a0-9fc4-311682ed04c9/', '/pulp/api/v3/content/rpm/advisories/5c9b45bb-5bf5-4cc0-9fcf-0aba9202f3c7/', '/pulp/api/v3/content/rpm/advisories/48cd6129-10ab-40e7-ab56-1e040d18e879/', '/pulp/api/v3/content/rpm/advisories/194331bd-70fe-4390-881d-65326e999c72/', '/pulp/api/v3/content/rpm/advisories/6a447d71-4b80-4faa-b614-3cb6ba79a114/', '/pulp/api/v3/content/rpm/advisories/8dd072b5-1ae5-4671-b059-ffe401511b3f/', '/pulp/api/v3/content/rpm/advisories/3a7c9c38-26e5-401b-a770-eebe9348bd47/', '/pulp/api/v3/content/rpm/advisories/17d5b4ec-4e35-49b0-8d23-e794336bc820/', '/pulp/api/v3/content/rpm/advisories/ba8ea5d5-7b2f-47e5-ab54-6aaac47d4414/', '/pulp/api/v3/content/rpm/advisories/c97ef57e-b0bc-4868-8db6-91cf0ef51289/', '/pulp/api/v3/content/rpm/advisories/6de042b9-7eb3-485c-a49b-b7511d742ec6/', '/pulp/api/v3/content/rpm/advisories/38f44bdf-934e-4930-97f5-012b43ce28ca/', '/pulp/api/v3/content/rpm/advisories/b99ac40e-ce64-4738-a14e-f496866257d1/', '/pulp/api/v3/content/rpm/advisories/ad06de0c-b376-4005-a3d6-2fae2d98a916/', '/pulp/api/v3/content/rpm/advisories/5340e7c6-2b3e-4a89-9376-bd3117555222/', '/pulp/api/v3/content/rpm/advisories/a6513de5-7158-4aaa-aa40-60ddb9047810/', '/pulp/api/v3/content/rpm/advisories/899a8b6c-dd0e-4318-92b3-cb5307e0d5f0/', '/pulp/api/v3/content/rpm/advisories/8924917f-4a32-41a0-8b85-7bd7f3a481aa/', '/pulp/api/v3/content/rpm/advisories/8b5196fe-2846-4250-9bec-449306fd3efd/', '/pulp/api/v3/content/rpm/advisories/943057b0-76ac-4509-9499-ec779029d133/', '/pulp/api/v3/content/rpm/advisories/4d7b6910-df4b-4876-bf43-f3c66af01fe8/', '/pulp/api/v3/content/rpm/advisories/65f5048e-b040-4372-8e56-afe31a6031a4/', '/pulp/api/v3/content/rpm/advisories/09d8fae6-4043-4809-9624-a19e6a440332/', '/pulp/api/v3/content/rpm/advisories/4470e762-2852-47cd-96a3-0ecbdb66a482/', '/pulp/api/v3/content/rpm/advisories/154511ee-0219-420e-b7f0-92bef6c3b9a3/', '/pulp/api/v3/content/rpm/advisories/ec33b0be-64b8-4530-888b-950aab0d9b3c/', '/pulp/api/v3/content/rpm/advisories/cf8b2756-6c9e-466f-9c34-c8a7f33cee05/', '/pulp/api/v3/content/rpm/advisories/91a14008-544d-4ca7-9901-10caaa5c7fb6/', '/pulp/api/v3/content/rpm/advisories/2419b15b-2fd2-4fb9-b9f9-574eb08877c0/', '/pulp/api/v3/content/rpm/advisories/3ff0539f-93aa-4f77-ada6-b49c0decdae8/', '/pulp/api/v3/content/rpm/advisories/8bdbe6a3-2402-40f5-90a5-a768acf20489/', '/pulp/api/v3/content/rpm/advisories/df9a469d-8925-4777-a9e2-17fde145d2fc/', '/pulp/api/v3/content/rpm/advisories/79b0cb71-bad1-42d9-9f2e-d2bb4370a193/', '/pulp/api/v3/content/rpm/advisories/3e515cbb-15a7-4886-a5b5-d06b79bc6fda/', '/pulp/api/v3/content/rpm/advisories/a7105213-a985-48bf-a30c-06782b99edae/', '/pulp/api/v3/content/rpm/advisories/c884c08b-2f57-489c-98ae-441e989f491c/', '/pulp/api/v3/content/rpm/advisories/24a3d212-0ba7-495e-92f0-d24ec18c26c4/', '/pulp/api/v3/content/rpm/advisories/c9da9b79-d831-480c-b23c-7d955ed7d1de/', '/pulp/api/v3/content/rpm/advisories/5816e295-b425-40da-bce2-996e0590f572/', '/pulp/api/v3/content/rpm/advisories/66cee998-68af-4d2b-972d-a8bc88395cec/', '/pulp/api/v3/content/rpm/advisories/2d57e016-8d87-4cd3-ae2d-40597e67a1e3/', '/pulp/api/v3/content/rpm/advisories/418fcc4c-4cba-4962-84e9-beade6d87297/', '/pulp/api/v3/content/rpm/advisories/11ded541-d6ad-4312-8d12-3746a0b5849a/', '/pulp/api/v3/content/rpm/advisories/9cfdb9a3-ba08-47dd-b538-023c65e9795c/', '/pulp/api/v3/content/rpm/advisories/b2334511-27a5-46e3-890e-9ddca1c6a974/', '/pulp/api/v3/content/rpm/advisories/be19e56e-d1ab-443a-9479-972d832d33d4/', '/pulp/api/v3/content/rpm/advisories/633f337e-a0f0-4c6b-ad0d-b6fdbfe470f7/', '/pulp/api/v3/content/rpm/advisories/8238cc1b-6f4b-428f-8d5e-37bca6969624/', '/pulp/api/v3/content/rpm/advisories/4566ee58-c015-4551-823b-37069f2ebfa9/', '/pulp/api/v3/content/rpm/advisories/0cbfbba5-5edd-4376-9bce-0bf38b9e4aee/', '/pulp/api/v3/content/rpm/advisories/d743e1de-3da3-4b49-ab53-d7fa3b74e3b5/', '/pulp/api/v3/content/rpm/advisories/f7378c53-b1cb-473b-9be6-73ba20341788/', '/pulp/api/v3/content/rpm/advisories/6ada50f6-24b5-45aa-bbd1-4bc82abb6752/', '/pulp/api/v3/content/rpm/advisories/7027ca2d-3dd8-439e-8d6e-66bf47de46e2/', '/pulp/api/v3/content/rpm/advisories/5ab1679c-e4a7-46a4-9869-15a4230a09f0/', '/pulp/api/v3/content/rpm/advisories/81a396a0-c69e-4990-955e-e23f2654516a/', '/pulp/api/v3/content/rpm/advisories/a3b907b1-3446-4a46-9399-2e6e02d65b9d/', '/pulp/api/v3/content/rpm/advisories/12f53e42-bcd5-45c4-b66c-36beea997280/', '/pulp/api/v3/content/rpm/advisories/0dc867a9-60f6-4f7e-b928-bc5171776120/', '/pulp/api/v3/content/rpm/advisories/e7eebc32-7ad2-4121-85db-4e4ff76b4c5c/', '/pulp/api/v3/content/rpm/advisories/a9f59032-3061-47f5-b7b8-b2af1c541385/', '/pulp/api/v3/content/rpm/advisories/803791f0-7891-41c6-8315-f4965ff4aa05/', '/pulp/api/v3/content/rpm/advisories/ff259aac-f67e-4288-8ece-39ac4b63c618/', '/pulp/api/v3/content/rpm/advisories/c747ef5c-d21c-4c4d-b0df-6048ad046d31/', '/pulp/api/v3/content/rpm/advisories/f6dbe489-d5f1-4a5b-a09e-958222aa02e8/', '/pulp/api/v3/content/rpm/advisories/85b49054-83d8-40ac-b4c6-85a85c53ae7b/', '/pulp/api/v3/content/rpm/advisories/d2d63b60-3ddb-4793-acb9-0bc2d8e94de7/', '/pulp/api/v3/content/rpm/advisories/f96c30ed-ab9f-43af-bc18-2096d87a94bb/', '/pulp/api/v3/content/rpm/advisories/47507859-8076-4ff1-ba4d-cdff8c7c53b7/', '/pulp/api/v3/content/rpm/advisories/043bd7f6-cba9-46a5-91f4-9f22a1effe90/', '/pulp/api/v3/content/rpm/advisories/a6be09c7-8fae-49e1-80dc-1d5f9165e94b/', '/pulp/api/v3/content/rpm/advisories/e25ac705-c7ea-4865-a80b-e4581fafa660/', '/pulp/api/v3/content/rpm/advisories/5daed36c-2698-41af-9b47-4b4a6def4094/', '/pulp/api/v3/content/rpm/advisories/882d3c55-a3ef-4bcc-8748-030e697e0d65/', '/pulp/api/v3/content/rpm/advisories/a8d14775-5a1a-4176-bb91-101397dacf72/', '/pulp/api/v3/content/rpm/advisories/c9fde76e-3def-4c0b-bdcf-949dceb52479/', '/pulp/api/v3/content/rpm/advisories/04d69f3e-9a3c-4845-a2ac-24a8b350eb38/', '/pulp/api/v3/content/rpm/advisories/207d5736-126f-45bf-939b-4dc89d5b57fd/', '/pulp/api/v3/content/rpm/advisories/2fcc7fa1-d14c-4261-8b70-c3d1bdc313b4/', '/pulp/api/v3/content/rpm/advisories/8d1f38f5-3b0d-48d5-a4e1-75957d7d64be/', '/pulp/api/v3/content/rpm/advisories/a06a5caf-0244-4738-8c01-5246199128d4/', '/pulp/api/v3/content/rpm/advisories/3db0905e-99a3-4015-b23f-e01a3ca02116/', '/pulp/api/v3/content/rpm/advisories/c9924d6a-587d-45ce-a5e4-e9ebac238f8b/', '/pulp/api/v3/content/rpm/advisories/705c5624-1848-40fc-b5c4-5da6120d0fbd/', '/pulp/api/v3/content/rpm/advisories/a037f797-6754-47c8-ac79-b299a46a1305/', '/pulp/api/v3/content/rpm/advisories/3d55da91-a081-449c-b8d2-4a539a505c4c/', '/pulp/api/v3/content/rpm/advisories/8477b0d4-8fbc-4d75-bf0f-f869a0183ea2/', '/pulp/api/v3/content/rpm/advisories/035d4617-b796-40d6-af41-ad19556bc17c/', '/pulp/api/v3/content/rpm/advisories/a59dcb2b-1b0e-4b10-811b-00a94fc7d4c3/', '/pulp/api/v3/content/rpm/advisories/22bce75c-3f3a-4cc2-a1e5-cca2bfd7ef48/', '/pulp/api/v3/content/rpm/advisories/aeb89d15-10ab-40e7-ac9b-d72816949fe5/', '/pulp/api/v3/content/rpm/advisories/b1d37fbc-2db2-4d3b-9ddf-2fc3b56f4500/', '/pulp/api/v3/content/rpm/advisories/5cf0a49e-f276-46f7-9f81-4ecef4b84f23/', '/pulp/api/v3/content/rpm/advisories/7dc21dae-7b88-4998-b60f-b4d12cc0e32e/', '/pulp/api/v3/content/rpm/advisories/9ec9e087-eb98-4052-bba3-cdcaceabe509/', '/pulp/api/v3/content/rpm/advisories/242f4ac3-2e69-42da-99be-75544cc7b546/', '/pulp/api/v3/content/rpm/advisories/4c7043a2-a93f-4320-87cc-fa4e3cb62141/', '/pulp/api/v3/content/rpm/advisories/36cc757a-f903-458c-8836-c000bd7e7716/', '/pulp/api/v3/content/rpm/advisories/bdae9e29-2f2c-4018-8d2c-4017983f9536/', '/pulp/api/v3/content/rpm/advisories/b48aaea6-89cc-4a97-9b90-92820766aad9/', '/pulp/api/v3/content/rpm/advisories/56a80ec2-d7f9-458c-9db2-110bee0d8c96/', '/pulp/api/v3/content/rpm/advisories/d0050502-44fd-4985-b62f-beadc11ae21c/', '/pulp/api/v3/content/rpm/advisories/599e04cf-116b-4836-b6cd-c8fc582de3bb/', '/pulp/api/v3/content/rpm/advisories/b88a9079-b5bc-4407-bb32-18aa429a04fb/', '/pulp/api/v3/content/rpm/advisories/4a954fd4-4537-43dd-abdc-80596ae97387/', '/pulp/api/v3/content/rpm/advisories/5caacf44-d4b6-482e-bcc6-56cf7005f7db/', '/pulp/api/v3/content/rpm/advisories/ee53dcb5-6e2c-4ba8-8014-d12c40571000/', '/pulp/api/v3/content/rpm/advisories/c350e7ea-726c-4332-900a-14f990c26b7b/', '/pulp/api/v3/content/rpm/advisories/0596f80f-6a7d-4e24-bd00-2875a9e9b5ae/', '/pulp/api/v3/content/rpm/advisories/d767cbab-b708-43c1-973e-5a3fad0140f6/', '/pulp/api/v3/content/rpm/advisories/f5244a1b-dc24-40eb-8adf-ba7dc5f9d5c8/', '/pulp/api/v3/content/rpm/advisories/c7af66e7-f80f-4454-9776-4cd411646194/', '/pulp/api/v3/content/rpm/advisories/a9449704-2ee1-4d81-9131-791c6abda6c7/', '/pulp/api/v3/content/rpm/advisories/590b37a6-9f3f-42dc-be26-5f762b32e30a/']"]