Katello 4.8.0 rc1 release process

lfu · February 21, 2023, 9:20pm

Make this post a wiki

Roles

Release Owner: @lfu
Release Engineer: @zhunting

When Ready to Release

Release Owner

Request Hammer CLI Katello release from maintainers
Request Virt Who Configure release from maintainers
Request any other releases as needed from the release owners (consult the :repos: section of configs/katello/4.8.yaml)
Do Cherry-picks: Clone tool_belt
- If any minor versions of Katello have been added to Redmine since the last cherry-pick, make sure to include them in prior_releases in configs/katello/4.8.yaml
- Run ./tools.rb setup-environment configs/katello/4.8.yaml
- Run GITHUB_ACCESS_TOKEN=<secret> ./tools.rb cherry-picks --version 4.8.0.rc1 configs/katello/4.8.yaml
- Open a PR in Katello release branch. Make sure the PR name starts with [CP] to prevent our automations from adding it to Redmine issues.
- Using git cherry-pick -x as needed, verify tickets in the cherry_picks_4.8.0.rc1 file are accounted for, or additionally cherry-pick them. Recommended: Do this in tool_belt’s checkout of Katello, in repos/katello/4.8.0.rc1/katello. This way when you run cherry-picks again, tool_belt will be aware of any picks already completed.
- For any cherry-picks that are not needed (including Redmine trackers) you can add them to the :ignores: section of tool_belt in configs/katello/4.8.yaml
Change Redmine version 4.8.0.rc1 state to Closed
Check for outdated deprecation warnings in the current and next release with ./tools check-deprecation-warnings configs/katello/4.8.yaml. Follow the instructions in the output of the command. Don’t forget to create any Redmine issues needed!
Update docs.theforeman.org
- Using redmine_release_notes script (see README as well): ./guides/doc-Release_Notes/redmine_release_notes katello 4.8.0.rc1 > ./guides/doc-Release_Notes/topics/katello-4.8.0.rc1.adoc
- Add topics/katello-4.8.0.rc1.adoc to guides/doc-Release_Notes/master.adoc: sed '/x.y.z releases here/a include::topics/katello-4.8.0.rc1.adoc[leveloffset=+1]' master.adoc
- Make sure katello-contributors.adoc is updated
- Submit this as a PR
Open a PR (or use cherry-pick PR) against the release branch which updates lib/katello/version.rb to 4.8.0.rc1:
- git pull to make sure you have the latest changes
- sed '/VERSION/ s/".\+"/"4.8.0.rc1"/' lib/katello/version.rb
- Update/add the CHANGELOG.md file: GITHUB_ACCESS_TOKEN=<secret> ./tools changelog configs/katello/4.8.yaml
- Commit: git commit -m "Release 4.8.0.rc1"
- Ensure that the commit above is the last commit and there are no commits after it. This is the commit that will get tagged. (Rearrange commits with git rebase -i if needed.)
Once the PR is merged, perform the following in the Katello release branch (the real one, not your fork):
- Tag: git tag -s -m "Release 4.8.0.rc1" 4.8.0.rc1
- Push: git push --follow-tags (Must be pushed directly to the release branch, as pull request merges will not preserve tags.)
- Generate source gem: gem build katello.gemspec
- Ensure you have a working login and password at rubygems.org
- Push gem: gem push katello-4.8.0.rc1.gem
Inform the delivery team that the gem is published

Once Source is Available

Note it is considered good practice to release on a day when the next day is a working day. This means no releases on Fridays or on the day before a holiday.

Release Engineer

Update release version similar to here
Update katello, katello-repos and rubygem-katello
- PROJECT=katello VERSION=4.8 ./bump_rpm_packaging
Merge packaging PR once job is green
Wait for Jenkins to build the packages (replace 4.8 with the matching Foreman version)
Download, sign, upload RPM signatures and upload RPMs
Kick off the release pipeline by calling release_pipeline

Once release is out

Release Owner

Confirm response that the build succeeded (or if necessary, do more cherry-picks and version bumps repeating the steps above)
Test the install and upgrade documentation for both the Katello server and smart proxy
Post a release announcement

cintrix84 · February 27, 2023, 2:18pm

@ekohl

For the docs step, do you know what happened to the redmine script?

evgeni · March 3, 2023, 8:34am

The installer needs support for the new HStore stuff for Pulpcore, which we forgot for Foreman’s RC1.

I’ve opened a PR for pulpcore to release Release 7.2.0 by evgeni · Pull Request #276 · theforeman/puppet-pulpcore · GitHub

And then we need a RC2 of the Foreman installer to include that, Katello won’t work before this change is in.

evgeni · March 14, 2023, 8:17am

Foreman 3.6 RC2 has been released and a new Katello 4.8 RC1 pipe has run, but it still has errors.

See katello-4.8-rpm-pipeline #3 [Jenkins]

I think the first place that starts falling apart is the incremental update test:

not ok 45 incremental update first component cv with composite propagation
# (in test file fb-katello-content.bats, line 361)
#   `hammer content-view version incremental-update --organization="${ORGANIZATION}" \' failed with status 70
# Task 30b57539-e224-4fa8-8f55-8b6af006127d planned: 0.0/1, 0%, elapsed: 00:00:00
# Task 30b57539-e224-4fa8-8f55-8b6af006127d running: 0.11/1, 11%, 0.1/s, elapsed: 00:00:02, ETA: 00:00:17
# Task 30b57539-e224-4fa8-8f55-8b6af006127d running: 0.17/1, 17%, 0.0/s, elapsed: 00:00:04, ETA: 00:00:21
# Task 30b57539-e224-4fa8-8f55-8b6af006127d error: 0.2/1, 20%, 0.0/s, elapsed: 00:00:06, ETA: 00:00:26
# Task 30b57539-e224-4fa8-8f55-8b6af006127d error: 0.2/1, 20%, 0.0/s, elapsed: 00:00:06, ETA: 00:00:26
# Error: failed to parse nevra 'rpms' not a valid nevra
# Incremental Update incomplete.

Which is logged by Katello as:

2023-03-10T19:46:15 [I|app|057e21dd] Started POST "/katello/api/content_view_versions/incremental_update" for 192.168.122.60 at 2023-03-10 19:46:15 +0000
2023-03-10T19:46:15 [I|app|057e21dd] Processing by Katello::Api::V2::ContentViewVersionsController#incremental_update as JSON
2023-03-10T19:46:15 [I|app|057e21dd]   Parameters: {"propagate_all_composites"=>true, "add_content"=>{"errata_ids"=>["WALRUS-2013:0002"]}, "content_view_version_environments"=>[{"content_view_version_id"=>12, "environment_ids"=>[3]}], "api_version"=>"v2", "content_view_version"=>{}}
2023-03-10T19:46:15 [I|app|057e21dd] Authorized user admin(Admin User)
2023-03-10T19:46:15 [I|bac|057e21dd] Task {label: , execution_plan_id: 90c305fa-d547-4a29-a749-2d8c9a0dcf1e} state changed: pending 
2023-03-10T19:46:15 [I|bac|057e21dd] Task {label: Actions::Katello::ContentView::IncrementalUpdates, id: e50e04e0-01b8-4d6f-8eb5-06a2513c69c7, execution_plan_id: 90c305fa-d547-4a29-a749-2d8c9a0dcf1e} state changed: planning 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::ContentViewVersion (14) create event on content_view_id 10
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::ContentViewVersion (14) create event on major 1
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::ContentViewVersion (14) create event on definition_archive_id 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::ContentViewVersion (14) create event on minor 1
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::ContentViewVersion (14) create event on content_counts 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::ContentViewVersion (14) create event on repository_ids []
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::ContentViewVersion (14) create event on environment_ids []
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on pulp_id 4-component_1-v1_1-80f5f113-5268-4207-944a-af9ec376f239
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on library_instance_id 3
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on content_view_version_id 14
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on relative_path test_organization-Component_1-1.1-Test_Product-foremanbusybox
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on environment_id 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on saved_checksum_type 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on distribution_version 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on distribution_arch 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on distribution_bootable 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on distribution_family 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on distribution_variant 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on container_repository_name test_organization-component_1-1_1-test_product-foremanbusybox
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on root_id 3
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on remote_href 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on publication_href 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on version_href 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on last_contents_changed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on last_applicability_regen 1970-01-01 00:00:00 UTC
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) create event on last_indexed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on pulp_id 4-Component_1-v1_1-19674a3b-adf7-4102-ba46-dcd1f6ec55d1
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on library_instance_id 31
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on content_view_version_id 14
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on relative_path Test_Organization/content_views/Component_1/1.1/custom/Test_Product/rpm-deps
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on environment_id 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on saved_checksum_type 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on distribution_version 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on distribution_arch 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on distribution_bootable 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on distribution_family 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on distribution_variant 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on container_repository_name 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on root_id 10
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on remote_href 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on publication_href 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on version_href 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on last_contents_changed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on last_applicability_regen 1970-01-01 00:00:00 UTC
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) create event on last_indexed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on pulp_id 4-Component_1-v1_1-07f87d7f-9656-4592-93e3-a8e39d3b1a9c
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on library_instance_id 1
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on content_view_version_id 14
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on relative_path Test_Organization/content_views/Component_1/1.1/custom/Test_Product/Zoo
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on environment_id 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on saved_checksum_type 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on distribution_version 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on distribution_arch 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on distribution_bootable 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on distribution_family 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on distribution_variant 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on container_repository_name 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on root_id 1
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on remote_href 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on publication_href 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on version_href 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on last_contents_changed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on last_applicability_regen 1970-01-01 00:00:00 UTC
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) create event on last_indexed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on pulp_id 4-Component_1-v1_1-2d76ca86-f721-4d0d-a026-d4230bc0dee1
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on library_instance_id 30
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on content_view_version_id 14
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on relative_path Test_Organization/content_views/Component_1/1.1/custom/Test_Product/modules-rpms
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on environment_id 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on saved_checksum_type 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on distribution_version 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on distribution_arch 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on distribution_bootable 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on distribution_family 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on distribution_variant 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on container_repository_name 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on root_id 9
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on remote_href 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on publication_href 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on version_href 
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on last_contents_changed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on last_applicability_regen 1970-01-01 00:00:00 UTC
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) create event on last_indexed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (52) update event on version_href , /pulp/api/v3/repositories/container/container/87ed1909-27cf-403c-a850-afe8db3ea31b/versions/1/
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (53) update event on version_href , /pulp/api/v3/repositories/rpm/rpm/bc20cf5d-15bb-42e6-9689-fd79a572114d/versions/1/
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (54) update event on version_href , /pulp/api/v3/repositories/rpm/rpm/983db18e-5642-4703-a06e-05a50072581f/versions/1/
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (55) update event on version_href , /pulp/api/v3/repositories/rpm/rpm/01ebf037-5876-4604-8ffc-4e76f2dbe8da/versions/1/
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::ContentViewEnvironment (11) update event on content_view_version_id 12, 14
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (36) update event on content_view_version_id 12, 14
2023-03-10T19:46:16 [I|aud|057e21dd] Katello::Repository (39) update event on content_view_version_id 12, 14
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (37) update event on content_view_version_id 12, 14
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (38) update event on content_view_version_id 12, 14
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::ContentViewVersion (15) create event on content_view_id 11
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::ContentViewVersion (15) create event on major 1
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::ContentViewVersion (15) create event on definition_archive_id 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::ContentViewVersion (15) create event on minor 1
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::ContentViewVersion (15) create event on content_counts 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::ContentViewVersion (15) create event on repository_ids []
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::ContentViewVersion (15) create event on environment_ids []
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on pulp_id 4-Composite-v1_1-07f87d7f-9656-4592-93e3-a8e39d3b1a9c
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on library_instance_id 1
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on content_view_version_id 15
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on relative_path Test_Organization/content_views/Composite/1.1/custom/Test_Product/Zoo
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on environment_id 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on saved_checksum_type 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on distribution_version 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on distribution_arch 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on distribution_bootable 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on distribution_family 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on distribution_variant 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on container_repository_name 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on root_id 1
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on remote_href 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on publication_href 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on version_href 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on last_contents_changed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on last_applicability_regen 1970-01-01 00:00:00 UTC
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (56) create event on last_indexed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on pulp_id 4-composite-v1_1-80f5f113-5268-4207-944a-af9ec376f239
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on library_instance_id 3
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on content_view_version_id 15
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on relative_path test_organization-Composite-1.1-Test_Product-foremanbusybox
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on environment_id 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on saved_checksum_type 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on distribution_version 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on distribution_arch 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on distribution_bootable 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on distribution_family 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on distribution_variant 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on container_repository_name test_organization-composite-1_1-test_product-foremanbusybox
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on root_id 3
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on remote_href 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on publication_href 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on version_href 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on last_contents_changed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on last_applicability_regen 1970-01-01 00:00:00 UTC
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) create event on last_indexed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on pulp_id 4-Composite-v1_1-19674a3b-adf7-4102-ba46-dcd1f6ec55d1
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on library_instance_id 31
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on content_view_version_id 15
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on relative_path Test_Organization/content_views/Composite/1.1/custom/Test_Product/rpm-deps
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on environment_id 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on saved_checksum_type 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on distribution_version 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on distribution_arch 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on distribution_bootable 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on distribution_family 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on distribution_variant 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on container_repository_name 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on root_id 10
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on remote_href 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on publication_href 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on version_href 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on last_contents_changed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on last_applicability_regen 1970-01-01 00:00:00 UTC
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (58) create event on last_indexed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on pulp_id 4-Composite-v1_1-2d76ca86-f721-4d0d-a026-d4230bc0dee1
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on library_instance_id 30
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on content_view_version_id 15
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on relative_path Test_Organization/content_views/Composite/1.1/custom/Test_Product/modules-rpms
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on environment_id 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on saved_checksum_type 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on distribution_version 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on distribution_arch 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on distribution_bootable 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on distribution_family 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on distribution_variant 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on container_repository_name 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on root_id 9
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on remote_href 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on publication_href 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on version_href 
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on last_contents_changed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on last_applicability_regen 1970-01-01 00:00:00 UTC
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (59) create event on last_indexed 1970-01-01 00:00:00 UTC
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (57) update event on version_href , /pulp/api/v3/repositories/container/container/87ed1909-27cf-403c-a850-afe8db3ea31b/versions/1/
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::ContentViewEnvironment (12) update event on content_view_version_id 13, 15
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (46) update event on content_view_version_id 13, 15
2023-03-10T19:46:17 [I|aud|057e21dd] Katello::Repository (47) update event on content_view_version_id 13, 15
2023-03-10T19:46:18 [I|aud|057e21dd] Katello::Repository (45) update event on content_view_version_id 13, 15
2023-03-10T19:46:18 [I|aud|057e21dd] Katello::Repository (44) update event on content_view_version_id 13, 15
2023-03-10T19:46:18 [I|aud|057e21dd] Katello::ContentViewEnvironment (13) update event on content_view_version_id 13, 15
2023-03-10T19:46:18 [I|aud|057e21dd] Katello::Repository (48) update event on content_view_version_id 13, 15
2023-03-10T19:46:18 [I|aud|057e21dd] Katello::Repository (49) update event on content_view_version_id 13, 15
2023-03-10T19:46:18 [I|aud|057e21dd] Katello::Repository (50) update event on content_view_version_id 13, 15
2023-03-10T19:46:18 [I|aud|057e21dd] Katello::Repository (51) update event on content_view_version_id 13, 15
2023-03-10T19:46:19 [I|bac|057e21dd] Task {label: Actions::Katello::ContentView::IncrementalUpdates, id: e50e04e0-01b8-4d6f-8eb5-06a2513c69c7, execution_plan_id: 90c305fa-d547-4a29-a749-2d8c9a0dcf1e} state changed: planned 
2023-03-10T19:46:19 [I|app|057e21dd]   Rendered /usr/share/gems/gems/katello-4.8.0.rc1/app/views/katello/api/v2/common/async.json.rabl within katello/api/v2/layouts/resource (Duration: 44.7ms | Allocations: 83550)
2023-03-10T19:46:19 [I|app|057e21dd]   Rendered layout /usr/share/gems/gems/katello-4.8.0.rc1/app/views/katello/api/v2/layouts/resource.json.erb (Duration: 46.7ms | Allocations: 86075)
2023-03-10T19:46:19 [I|app|057e21dd] Completed 202 Accepted in 3518ms (Views: 46.7ms | ActiveRecord: 210.0ms | Allocations: 1507427)
2023-03-10T19:46:19 [I|bac|057e21dd] Task {label: Actions::Katello::ContentView::IncrementalUpdates, id: e50e04e0-01b8-4d6f-8eb5-06a2513c69c7, execution_plan_id: 90c305fa-d547-4a29-a749-2d8c9a0dcf1e} state changed: running 
2023-03-10T19:46:21 [I|app|057e21dd] CONTENT_INDEX type=Katello::DockerManifest success=true duration=80.42
2023-03-10T19:46:21 [I|app|057e21dd] CONTENT_INDEX type=Katello::DockerManifestList success=true duration=62.35
2023-03-10T19:46:21 [I|app|057e21dd] CONTENT_INDEX type=Katello::DockerTag success=true duration=73.16
2023-03-10T19:46:21 [I|aud|057e21dd] Katello::Repository (52) update event on last_indexed 1970-01-01 00:00:00 UTC, 2023-03-10 19:46:21 UTC
2023-03-10T19:46:21 [I|aud|057e21dd] Katello::Repository (53) update event on publication_href , /pulp/api/v3/publications/rpm/rpm/b7bb3dbb-5b07-4f8c-8d64-4a16646604d3/
2023-03-10T19:46:22 [I|aud|057e21dd] Katello::Repository (53) update event on last_indexed 1970-01-01 00:00:00 UTC, 2023-03-10 19:46:22 UTC
2023-03-10T19:46:22 [I|aud|057e21dd] Katello::Repository (54) update event on publication_href , /pulp/api/v3/publications/rpm/rpm/8823b9fe-7202-4e21-9972-7d385e8a1fc0/
2023-03-10T19:46:23 [I|aud|057e21dd] Katello::Repository (54) update event on last_indexed 1970-01-01 00:00:00 UTC, 2023-03-10 19:46:23 UTC
2023-03-10T19:46:23 [I|aud|057e21dd] Katello::Repository (55) update event on publication_href , /pulp/api/v3/publications/rpm/rpm/fb72df28-0a45-46c4-81bb-b773511f37a8/
2023-03-10T19:46:24 [I|aud|057e21dd] Katello::Repository (55) update event on last_indexed 1970-01-01 00:00:00 UTC, 2023-03-10 19:46:24 UTC
2023-03-10T19:46:25 [E|bac|057e21dd] failed to parse nevra 'rpms' not a valid nevra (Katello::Errors::Pulp3Error)
 057e21dd | /usr/share/gems/gems/katello-4.8.0.rc1/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
 057e21dd | /usr/share/gems/gems/katello-4.8.0.rc1/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
 057e21dd | /usr/share/gems/gems/katello-4.8.0.rc1/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
 057e21dd | /usr/share/gems/gems/katello-4.8.0.rc1/app/lib/actions/pulp3/abstract_async_task.rb:160:in `poll_external_task'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:22:in `run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/cancellable.rb:14:in `run'
 057e21dd | /usr/share/gems/gems/katello-4.8.0.rc1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 057e21dd | /usr/share/gems/gems/katello-4.8.0.rc1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 057e21dd | /usr/share/gems/gems/katello-4.8.0.rc1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 057e21dd | /usr/share/gems/gems/katello-4.8.0.rc1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 057e21dd | /usr/share/gems/gems/katello-4.8.0.rc1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 057e21dd | /usr/share/gems/gems/katello-4.8.0.rc1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 057e21dd | /usr/share/gems/gems/katello-4.8.0.rc1/app/lib/actions/middleware/remote_action.rb:16:in `run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 057e21dd | /usr/share/gems/gems/activesupport-6.1.7.2/lib/active_support/execution_wrapper.rb:91:in `wrap'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:17:in `run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 057e21dd | /usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `catch'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `block in execute_run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `block in with_error_handling'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `catch'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `with_error_handling'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:575:in `execute_run'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:296:in `execute'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director.rb:94:in `execute'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors.rb:18:in `run_user_code'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 057e21dd | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 057e21dd | [ sidekiq ]
 057e21dd | [ concurrent-ruby ]
2023-03-10T19:46:25 [I|bac|057e21dd] Task {label: Actions::Katello::ContentView::IncrementalUpdates, id: e50e04e0-01b8-4d6f-8eb5-06a2513c69c7, execution_plan_id: 90c305fa-d547-4a29-a749-2d8c9a0dcf1e} state changed: paused  result: error

And in pulp as:

Mar 10 19:46:16 pipe-katello-server-4-8-centos8-stream pulpcore-api[56798]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:16 +0000] "GET /pulp/api/v3/status HTTP/1.1" 301 0 "-" "rest-client/2.1.0 (linux x86_64) ruby/2.7.6p219"
Mar 10 19:46:16 pipe-katello-server-4-8-centos8-stream pulpcore-api[56798]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:16 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 2010 "-" "rest-client/2.1.0 (linux x86_64) ruby/2.7.6p219"
Mar 10 19:46:20 pipe-katello-server-4-8-centos8-stream pulpcore-api[56798]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:20 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/01ebf037-5876-4604-8ffc-4e76f2dbe8da/ HTTP/1.1" 200 639 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:20 pipe-katello-server-4-8-centos8-stream pulpcore-api[56800]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:20 +0000] "GET /pulp/api/v3/repositories/container/container/87ed1909-27cf-403c-a850-afe8db3ea31b/ HTTP/1.1" 200 515 "-" "OpenAPI-Generator/2.14.3/ruby"
Mar 10 19:46:20 pipe-katello-server-4-8-centos8-stream pulpcore-api[56798]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:20 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/983db18e-5642-4703-a06e-05a50072581f/ HTTP/1.1" 200 630 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:20 pipe-katello-server-4-8-centos8-stream pulpcore-api[56800]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:20 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/bc20cf5d-15bb-42e6-9689-fd79a572114d/ HTTP/1.1" 200 635 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:20 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:20 +0000] "GET /pulp/api/v3/distributions/container/container/?base_path=test_organization-component_1-1_1-test_product-foremanbusybox HTTP/1.1" 200 52 "-" "OpenAPI-Generator/2.14.3/ruby"
Mar 10 19:46:20 pipe-katello-server-4-8-centos8-stream pulpcore-api[56800]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:20 +0000] "POST /pulp/api/v3/distributions/container/container/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/2.14.3/ruby"
Mar 10 19:46:20 pipe-katello-server-4-8-centos8-stream pulpcore-worker-1[67089]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Starting task b78de528-41f0-45cc-8f4d-d892f5a7ab16
Mar 10 19:46:20 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:20 +0000] "GET /pulp/api/v3/tasks/b78de528-41f0-45cc-8f4d-d892f5a7ab16/ HTTP/1.1" 200 535 "-" "OpenAPI-Generator/3.22.2/ruby"
Mar 10 19:46:20 pipe-katello-server-4-8-centos8-stream pulpcore-worker-1[67089]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Task completed b78de528-41f0-45cc-8f4d-d892f5a7ab16
Mar 10 19:46:20 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:20 +0000] "GET /pulp/api/v3/tasks/b78de528-41f0-45cc-8f4d-d892f5a7ab16/ HTTP/1.1" 200 648 "-" "OpenAPI-Generator/3.22.2/ruby"
Mar 10 19:46:20 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:20 +0000] "GET /pulp/api/v3/distributions/container/container/4eb390ab-3c7f-4859-b2d8-66e4752a1775/ HTTP/1.1" 200 767 "-" "OpenAPI-Generator/2.14.3/ruby"
Mar 10 19:46:21 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:21 +0000] "GET /pulp/api/v3/content/container/manifests/?limit=2000&media_type=application%2Fvnd.docker.distribution.manifest.v1%2Bjson&media_type=application%2Fvnd.docker.distribution.manifest.v2%2Bjson&media_type=application%2Fvnd.oci.image.manifest.v1%2Bjson&offset=0&repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Fcontainer%2Fcontainer%2F87ed1909-27cf-403c-a850-afe8db3ea31b%2Fversions%2F1%2F HTTP/1.1" 200 635 "-" "OpenAPI-Generator/2.14.3/ruby"
Mar 10 19:46:21 pipe-katello-server-4-8-centos8-stream pulpcore-api[56800]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:21 +0000] "GET /pulp/api/v3/content/container/manifests/?limit=2000&media_type=application%2Fvnd.docker.distribution.manifest.list.v2%2Bjson&media_type=application%2Fvnd.oci.image.index.v1%2Bjson&offset=0&repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Fcontainer%2Fcontainer%2F87ed1909-27cf-403c-a850-afe8db3ea31b%2Fversions%2F1%2F HTTP/1.1" 200 52 "-" "OpenAPI-Generator/2.14.3/ruby"
Mar 10 19:46:21 pipe-katello-server-4-8-centos8-stream pulpcore-api[56800]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:21 +0000] "GET /pulp/api/v3/content/container/tags/?limit=2000&offset=0&repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Fcontainer%2Fcontainer%2F87ed1909-27cf-403c-a850-afe8db3ea31b%2Fversions%2F1%2F HTTP/1.1" 200 301 "-" "OpenAPI-Generator/2.14.3/ruby"
Mar 10 19:46:21 pipe-katello-server-4-8-centos8-stream pulpcore-api[56800]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:21 +0000] "GET /pulp/api/v3/distributions/rpm/rpm/?base_path=Test_Organization%2Fcontent_views%2FComponent_1%2F1.1%2Fcustom%2FTest_Product%2Frpm-deps HTTP/1.1" 200 52 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:21 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:21 +0000] "GET /pulp/api/v3/publications/rpm/rpm/b7bb3dbb-5b07-4f8c-8d64-4a16646604d3/ HTTP/1.1" 200 447 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:21 pipe-katello-server-4-8-centos8-stream pulpcore-api[56800]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:21 +0000] "POST /pulp/api/v3/distributions/rpm/rpm/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:21 pipe-katello-server-4-8-centos8-stream pulpcore-worker-1[67100]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 76653716-146c-4cc3-a541-50a096c23ff5
Mar 10 19:46:21 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:21 +0000] "GET /pulp/api/v3/tasks/76653716-146c-4cc3-a541-50a096c23ff5/ HTTP/1.1" 200 535 "-" "OpenAPI-Generator/3.22.2/ruby"
Mar 10 19:46:21 pipe-katello-server-4-8-centos8-stream pulpcore-worker-1[67100]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 76653716-146c-4cc3-a541-50a096c23ff5
Mar 10 19:46:22 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:22 +0000] "GET /pulp/api/v3/tasks/76653716-146c-4cc3-a541-50a096c23ff5/ HTTP/1.1" 200 636 "-" "OpenAPI-Generator/3.22.2/ruby"
Mar 10 19:46:22 pipe-katello-server-4-8-centos8-stream pulpcore-api[56800]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:22 +0000] "GET /pulp/api/v3/distributions/rpm/rpm/c8dde705-8caf-4878-a408-d65546158a9d/ HTTP/1.1" 200 556 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:22 pipe-katello-server-4-8-centos8-stream pulpcore-api[56800]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:22 +0000] "GET /pulp/api/v3/distributions/rpm/rpm/?base_path=Test_Organization%2Fcontent_views%2FComponent_1%2F1.1%2Fcustom%2FTest_Product%2FZoo HTTP/1.1" 200 52 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:22 pipe-katello-server-4-8-centos8-stream pulpcore-api[56800]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:22 +0000] "GET /pulp/api/v3/publications/rpm/rpm/8823b9fe-7202-4e21-9972-7d385e8a1fc0/ HTTP/1.1" 200 447 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:22 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:22 +0000] "POST /pulp/api/v3/distributions/rpm/rpm/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:22 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67108]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 31c351e5-ca3d-4367-ac4e-58aeb954ca0a
Mar 10 19:46:22 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:22 +0000] "GET /pulp/api/v3/tasks/31c351e5-ca3d-4367-ac4e-58aeb954ca0a/ HTTP/1.1" 200 535 "-" "OpenAPI-Generator/3.22.2/ruby"
Mar 10 19:46:22 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67108]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 31c351e5-ca3d-4367-ac4e-58aeb954ca0a
Mar 10 19:46:23 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:23 +0000] "GET /pulp/api/v3/tasks/31c351e5-ca3d-4367-ac4e-58aeb954ca0a/ HTTP/1.1" 200 636 "-" "OpenAPI-Generator/3.22.2/ruby"
Mar 10 19:46:23 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:23 +0000] "GET /pulp/api/v3/distributions/rpm/rpm/731b17a7-ad8e-41ba-b8f2-8180d81066c9/ HTTP/1.1" 200 541 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:23 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:23 +0000] "GET /pulp/api/v3/distributions/rpm/rpm/?base_path=Test_Organization%2Fcontent_views%2FComponent_1%2F1.1%2Fcustom%2FTest_Product%2Fmodules-rpms HTTP/1.1" 200 52 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:23 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:23 +0000] "GET /pulp/api/v3/publications/rpm/rpm/fb72df28-0a45-46c4-81bb-b773511f37a8/ HTTP/1.1" 200 447 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:23 pipe-katello-server-4-8-centos8-stream pulpcore-api[56800]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:23 +0000] "POST /pulp/api/v3/distributions/rpm/rpm/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:23 pipe-katello-server-4-8-centos8-stream pulpcore-worker-1[67116]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Starting task c6d7ded4-84ec-42fd-8eb2-9fdd4dcfeab2
Mar 10 19:46:23 pipe-katello-server-4-8-centos8-stream pulpcore-api[56798]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:23 +0000] "GET /pulp/api/v3/tasks/c6d7ded4-84ec-42fd-8eb2-9fdd4dcfeab2/ HTTP/1.1" 200 535 "-" "OpenAPI-Generator/3.22.2/ruby"
Mar 10 19:46:23 pipe-katello-server-4-8-centos8-stream pulpcore-worker-1[67116]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Task completed c6d7ded4-84ec-42fd-8eb2-9fdd4dcfeab2
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:24 +0000] "GET /pulp/api/v3/tasks/c6d7ded4-84ec-42fd-8eb2-9fdd4dcfeab2/ HTTP/1.1" 200 636 "-" "OpenAPI-Generator/3.22.2/ruby"
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:24 +0000] "GET /pulp/api/v3/distributions/rpm/rpm/fbd658a6-5f2e-4666-968d-e99842fa4f95/ HTTP/1.1" 200 568 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:24 +0000] "GET /pulp/api/v3/distributions/container/container/?base_path=test_organization-component_1-1_1-test_product-foremanbusybox HTTP/1.1" 200 819 "-" "OpenAPI-Generator/2.14.3/ruby"
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-api[56798]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:24 +0000] "POST /pulp/api/v3/rpm/copy/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.19.0/ruby"
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:24 +0000] "PATCH /pulp/api/v3/distributions/container/container/4eb390ab-3c7f-4859-b2d8-66e4752a1775/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/2.14.3/ruby"
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 72516459-d17b-419a-87eb-b294419e587e
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-1[67126]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 45f2ef42-26ce-45a4-a13a-a550d7ef6dbc
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-api[56802]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:24 +0000] "GET /pulp/api/v3/tasks/45f2ef42-26ce-45a4-a13a-a550d7ef6dbc/ HTTP/1.1" 200 535 "-" "OpenAPI-Generator/3.22.2/ruby"
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-api[56798]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:24 +0000] "GET /pulp/api/v3/tasks/72516459-d17b-419a-87eb-b294419e587e/ HTTP/1.1" 200 973 "-" "OpenAPI-Generator/3.22.2/ruby"
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Task 72516459-d17b-419a-87eb-b294419e587e failed (failed to parse nevra 'rpms' not a valid nevra)
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 444, in _perform_task
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:    result = func(*args, **kwargs)
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:  File "/usr/lib64/python3.9/contextlib.py", line 79, in inner
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:    return func(*args, **kwds)
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/tasks/copy.py", line 200, in copy_content
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:    solver.load_target_repo(dest_repo_version)
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/depsolving.py", line 622, in load_target_repo
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:    libsolv_repo_name = self._load_from_version(repo_version, as_target=True)
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/depsolving.py", line 689, in _load_from_version
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:    self._add_unit_to_solver(module_to_solvable, module, repo, libsolv_repo_name)
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/depsolving.py", line 713, in _add_unit_to_solver
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:    solvable = conversion_func(repo, unit)
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/depsolving.py", line 349, in module_to_solvable
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:    nevra_tuple = parse_nevra(artifact)
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/depsolving.py", line 61, in parse_nevra
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:    raise ValueError("failed to parse nevra '%s' not a valid nevra" % name)
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-1[67126]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 45f2ef42-26ce-45a4-a13a-a550d7ef6dbc
Mar 10 19:46:25 pipe-katello-server-4-8-centos8-stream pulpcore-api[56800]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:25 +0000] "GET /pulp/api/v3/tasks/45f2ef42-26ce-45a4-a13a-a550d7ef6dbc/ HTTP/1.1" 200 562 "-" "OpenAPI-Generator/3.22.2/ruby"
Mar 10 19:46:25 pipe-katello-server-4-8-centos8-stream pulpcore-api[56798]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:25 +0000] "GET /pulp/api/v3/tasks/72516459-d17b-419a-87eb-b294419e587e/ HTTP/1.1" 200 2304 "-" "OpenAPI-Generator/3.22.2/ruby"
Mar 10 19:46:25 pipe-katello-server-4-8-centos8-stream pulpcore-api[56800]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]:  - - [10/Mar/2023:19:46:25 +0000] "PATCH /pulp/api/v3/distributions/container/container/4eb390ab-3c7f-4859-b2d8-66e4752a1775/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/2.14.3/ruby"
Mar 10 19:46:25 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67133]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 9b964e40-b2f3-4c75-982d-608b74f99a63
Mar 10 19:46:25 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67133]: pulp [057e21dd-6557-4cd7-be67-0e6303d2a64e]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 9b964e40-b2f3-4c75-982d-608b74f99a63

jeremylenz · March 14, 2023, 8:57pm

Seems like we’re getting the same thing in nightly: katello-nightly-rpm-pipeline #1614 Console [Jenkins]

iballou · March 14, 2023, 10:04pm

@dralley do you have an idea about that error we’re seeing?

Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/depsolving.py", line 61, in parse_nevra
Mar 10 19:46:24 pipe-katello-server-4-8-centos8-stream pulpcore-worker-2[67124]:    raise ValueError("failed to parse nevra '%s' not a valid nevra" % name)

We run dep solving during incremental update, so that all makes sense. I’m guessing this might have to do with the Pulp-RPM 3.19 upgrade that we recently did?

evgeni · March 15, 2023, 8:02am

the last green nightly is katello-nightly-rpm-pipeline #1595 [Jenkins]

did it already have rpm 3.19? sosreport suggests no:

% grep python39-pulp sosreport-katello-81149-2023-02-22-tdcdbmn/installed-rpms 
python39-pulpcore-3.22.2-4.el8.noarch                       Wed Feb 22 12:17:06 2023
python39-pulp-ansible-0.16.0-1.el8.noarch                   Wed Feb 22 12:21:09 2023
python39-pulp-certguard-1.5.6-1.el8.noarch                  Wed Feb 22 12:21:19 2023
python39-pulp-cli-0.14.0-4.el8.noarch                       Wed Feb 22 12:20:42 2023
python39-pulp-container-2.14.3-2.el8.noarch                 Wed Feb 22 12:20:50 2023
python39-pulp-deb-2.20.1-1.el8.noarch                       Wed Feb 22 12:21:00 2023
python39-pulp-file-1.11.3-1.el8.noarch                      Wed Feb 22 12:20:53 2023
python39-pulp-ostree-2.0.0-0.8.a6.el8.noarch                Wed Feb 22 12:21:16 2023
python39-pulp-python-3.8.0-1.el8.noarch                     Wed Feb 22 12:21:13 2023
python39-pulp-rpm-3.18.10-1.el8.noarch                      Wed Feb 22 12:20:57 2023

evgeni · March 15, 2023, 8:15am

digging a bit further (due to branching etc the later nightlies aren’t “green” as updates fail, but the install part was actually green for a while):

katello-nightly-rpm-pipeline #1603 [Jenkins] has the incremental tests passing
katello-nightly-rpm-pipeline #1604 [Jenkins] is the first one failing.

1603 is exactly the one that lists python39-pulp-rpm-3.18.10-1.el8.noarch as being installed, while 1604 has python39-pulp-rpm-3.19.0-3.el8.noarch

Otherwise, the diff of installed packages looks unsuspicious:

% diff -Nrwu <(cat sosreport-katello-81110-2023-02-28-ybavspe/installed-rpms| awk '{print $1}' | sort) <(cat sosreport-katello-80549-2023-03-01-comxier/installed-rpms| awk '{print $1}' | sort) 
--- /proc/self/fd/11	2023-03-15 09:15:03.405153886 +0100
+++ /proc/self/fd/12	2023-03-15 09:15:03.406153887 +0100
@@ -114,16 +114,16 @@
 firewalld-filesystem-0.9.3-13.el8.noarch
 fontconfig-2.13.1-4.el8.x86_64
 fontpackages-filesystem-1.44-22.el8.noarch
-foreman-3.7.0-0.1.develop.20230228142608giteb06bd0.el8.noarch
-foreman-cli-3.7.0-0.1.develop.20230228142608giteb06bd0.el8.noarch
-foreman-debug-3.7.0-0.1.develop.20230228142608giteb06bd0.el8.noarch
-foreman-dynflow-sidekiq-3.7.0-0.1.develop.20230228142608giteb06bd0.el8.noarch
-foreman-installer-3.7.0-0.1.develop.20230228031607git3757e22.el8.noarch
-foreman-installer-katello-3.7.0-0.1.develop.20230228031607git3757e22.el8.noarch
-foreman-postgresql-3.7.0-0.1.develop.20230228142608giteb06bd0.el8.noarch
+foreman-3.7.0-0.1.develop.20230301150116git51b8d8e.el8.noarch
+foreman-cli-3.7.0-0.1.develop.20230301150116git51b8d8e.el8.noarch
+foreman-debug-3.7.0-0.1.develop.20230301150116git51b8d8e.el8.noarch
+foreman-dynflow-sidekiq-3.7.0-0.1.develop.20230301150116git51b8d8e.el8.noarch
+foreman-installer-3.7.0-0.1.develop.20230301031403git3757e22.el8.noarch
+foreman-installer-katello-3.7.0-0.1.develop.20230301031403git3757e22.el8.noarch
+foreman-postgresql-3.7.0-0.1.develop.20230301150116git51b8d8e.el8.noarch
 foreman-proxy-3.7.0-0.1.develop.20230227161212gitd8b56bf.el8.noarch
 foreman-selinux-3.7.0-0.1.develop.20230227150222gite9fd90e.el8.noarch
-foreman-service-3.7.0-0.1.develop.20230228142608giteb06bd0.el8.noarch
+foreman-service-3.7.0-0.1.develop.20230301150116git51b8d8e.el8.noarch
 freetype-2.9.1-9.el8.x86_64
 fribidi-1.0.4-9.el8.x86_64
 fuse-2.9.7-16.el8.x86_64
@@ -648,10 +648,10 @@
 python39-pulp-container-2.14.3-2.el8.noarch
 python39-pulpcore-3.22.2-4.el8.noarch
 python39-pulp-deb-2.20.1-1.el8.noarch
-python39-pulp-file-1.11.3-1.el8.noarch
+python39-pulp-file-1.12.0-1.el8.noarch
 python39-pulp-ostree-2.0.0-0.8.a6.el8.noarch
 python39-pulp-python-3.8.0-1.el8.noarch
-python39-pulp-rpm-3.18.10-1.el8.noarch
+python39-pulp-rpm-3.19.0-3.el8.noarch
 python39-pycairo-1.20.1-3.el8.x86_64
 python39-pycares-4.1.2-2.el8.x86_64
 python39-pycodestyle-2.7.0-5.el8.noarch
@@ -876,7 +876,7 @@
 rubygem-kafo-6.5.0-1.el8.noarch
 rubygem-kafo_parsers-1.2.1-1.el8.noarch
 rubygem-kafo_wizards-0.0.2-2.el8.noarch
-rubygem-katello-4.9.0-0.1.pre.master.20230228203747git6e8274f.el8.noarch
+rubygem-katello-4.9.0-0.1.pre.master.20230301001732git22e1dbf.el8.noarch
 rubygem-ldap_fluff-0.6.0-1.el8.noarch
 rubygem-little-plugger-1.1.4-3.el8.noarch
 rubygem-locale-2.1.3-1.el8.noarch

iballou · March 15, 2023, 2:00pm

It makes sense that it would be something in Pulp since the error happened in the dependency solving code. That’s a black box for Katello.

iballou · March 15, 2023, 7:21pm

Pulp issue to watch: Error during dependency solving · Issue #3012 · pulp/pulp_rpm · GitHub

dralley · March 16, 2023, 3:53am

I’ll have a PR up and a new build shortly

evgeni · March 21, 2023, 6:38am

a new pulp_rpm release is out and was packaged.
nightly pipe passes install tests
4.8 does too, at least for pulp things. f-maintain is broken in rc2 (fix is already staged) so that doesn’t pass right now, but the fixed f-maintain will be made available with the release of 3.6 later today and then 4.8 rc1 should also fully pass.

Zhunting · March 21, 2023, 7:46pm

The rpm pipeline passed katello-4.8-rpm-pipeline #5 [Jenkins]