Foreman 2.4 / Katello 4 unable to sync repos

trendyandy · May 19, 2021, 6:19am

Problem:
After migrating from Foreman2.3 /Katello 3.18 to Foreman2.4 /Katello 4.0 some of my repos do not sync and fail with:
Katello::Errors::Pulp3Error: No declared artifact with relative path "pgdg-redhat-repo-42.0-11.noarch.rpm" for content "<Package: pgdg-redhat-repo>"

My Repos which do not sync:

Index of /pub/repos/yum/12/redhat/rhel-7-x86_64/ -> on demand
https://packages.grafana.com/oss/rpm -> on demand

I also tried to generate a new Repo, which also fails with the same error on sync
Running foreman-rake katello:delete_orphaned_content RAILS_ENV=production did not help.

May anyone has a solution or a workaround for this issue.

The full is attached below:

Expected outcome:
Repo sync success

Foreman and Proxy versions:

candlepin-3.2.11-1.el7.noarch
candlepin-selinux-3.2.11-1.el7.noarch
foreman-2.4.0-1.el7.noarch
foreman-cli-2.4.0-1.el7.noarch
foreman-debug-2.4.0-1.el7.noarch
foreman-dynflow-sidekiq-2.4.0-1.el7.noarch
foreman-installer-2.4.0-1.el7.noarch
foreman-installer-katello-2.4.0-1.el7.noarch
foreman-postgresql-2.4.0-1.el7.noarch
foreman-proxy-2.4.0-1.el7.noarch
foreman-release-2.4.0-1.el7.noarch
foreman-selinux-2.4.0-1.el7.noarch
foreman-service-2.4.0-1.el7.noarch
foreman-vmware-2.4.0-1.el7.noarch
katello-4.0.0-1.el7.noarch
katello-ca-consumer-ssibol-foreman.ssi.bfc.bolcom.net-1.0-2.noarch
katello-certs-tools-2.7.3-1.el7.noarch
katello-client-bootstrap-1.7.6-1.el7.noarch
katello-common-4.0.0-1.el7.noarch
katello-debug-4.0.0-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-host-tools-3.5.1-2.el7sat.noarch
katello-host-tools-fact-plugin-3.5.1-2.el7sat.noarch
katello-repos-4.0.0-1.el7.noarch
katello-selinux-4.0.0-1.el7.noarch
katello-server-ca-1.0-8.noarch
katello-service-3.14.1-1.el7.noarch
pulp-client-1.0-3.noarch
pulpcore-selinux-1.2.4-1.el7.x86_64
python-gofer-qpid-2.12.5-3.el7.noarch
python2-qpid-1.37.0-5.el7.noarch
python2-qpid-proton-0.33.0-1.el7.x86_64
python2-qpid-qmf-1.39.0-1.el7.x86_64
python3-pulp-2to3-migration-0.11.0-1.el7.noarch
python3-pulp-certguard-1.1.0-1.el7.noarch
python3-pulp-container-2.2.1-1.el7.noarch
python3-pulp-file-1.5.0-1.el7.noarch
python3-pulp-rpm-3.10.0-1.el7.noarch
python3-pulpcore-3.9.1-2.el7.noarch
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-client-devel-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.15.0-1.el7.x86_64
qpid-proton-c-0.33.0-1.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
qpid_router_katello_agent-qpid-router-client-1.0-1.noarch
rubygem-foreman_maintain-0.7.9-1.el7.noarch
ssibol-foreman.ssi.bfc.bolcom.net-apache-1.0-8.noarch
ssibol-foreman.ssi.bfc.bolcom.net-foreman-client-1.0-3.noarch
ssibol-foreman.ssi.bfc.bolcom.net-foreman-proxy-1.0-8.noarch
ssibol-foreman.ssi.bfc.bolcom.net-foreman-proxy-client-1.0-3.noarch
ssibol-foreman.ssi.bfc.bolcom.net-puppet-client-1.0-1.noarch
ssibol-foreman.ssi.bfc.bolcom.net-qpid-broker-1.0-3.noarch
ssibol-foreman.ssi.bfc.bolcom.net-qpid-client-cert-1.0-1.noarch
ssibol-foreman.ssi.bfc.bolcom.net-qpid-router-client-1.0-1.noarch
ssibol-foreman.ssi.bfc.bolcom.net-qpid-router-server-1.0-3.noarch
ssibol-foreman.ssi.bfc.bolcom.net-tomcat-1.0-1.noarch
tfm-rubygem-actioncable-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailbox-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailer-6.0.3.4-1.el7.noarch
tfm-rubygem-actionpack-6.0.3.4-1.el7.noarch
tfm-rubygem-actiontext-6.0.3.4-1.el7.noarch
tfm-rubygem-actionview-6.0.3.4-1.el7.noarch
tfm-rubygem-activejob-6.0.3.4-1.el7.noarch
tfm-rubygem-activemodel-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-import-1.0.0-2.el7.noarch
tfm-rubygem-activerecord-session_store-1.1.1-4.el7.noarch
tfm-rubygem-activestorage-6.0.3.4-1.el7.noarch
tfm-rubygem-activesupport-6.0.3.4-1.el7.noarch
tfm-rubygem-addressable-2.6.0-2.el7.noarch
tfm-rubygem-algebrick-0.7.3-7.el7.noarch
tfm-rubygem-amazing_print-1.1.0-1.el7.noarch
tfm-rubygem-ancestry-3.0.7-1.el7.noarch
tfm-rubygem-anemone-0.7.2-17.el7.noarch
tfm-rubygem-angular-rails-templates-1.1.0-1.el7.noarch
tfm-rubygem-ansi-1.5.0-2.el7.noarch
tfm-rubygem-apipie-bindings-0.4.0-1.el7.noarch
tfm-rubygem-apipie-dsl-2.3.0-1.el7.noarch
tfm-rubygem-apipie-params-0.0.5-4.el7.noarch
tfm-rubygem-apipie-rails-0.5.17-3.el7.noarch
tfm-rubygem-arel-9.0.0-4.el7.noarch
tfm-rubygem-audited-4.9.0-3.el7.noarch
tfm-rubygem-awesome_print-1.8.0-5.el7.noarch
tfm-rubygem-bcrypt-3.1.12-3.el7.x86_64
tfm-rubygem-builder-3.2.4-1.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-5.el7.noarch
tfm-rubygem-clamp-1.1.2-6.el7.noarch
tfm-rubygem-coffee-rails-5.0.0-1.el7.noarch
tfm-rubygem-coffee-script-2.4.1-4.el7.noarch
tfm-rubygem-coffee-script-source-1.12.2-4.el7.noarch
tfm-rubygem-concurrent-ruby-1.1.6-2.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.6.0-2.fm2_1.el7.noarch
tfm-rubygem-connection_pool-2.2.2-2.el7.noarch
tfm-rubygem-crass-1.0.6-1.el7.noarch
tfm-rubygem-css_parser-1.4.7-4.el7.noarch
tfm-rubygem-daemons-1.2.3-6.el7.noarch
tfm-rubygem-deacon-1.0.0-4.el7.noarch
tfm-rubygem-deep_cloneable-3.0.0-3.el7.noarch
tfm-rubygem-deface-1.5.3-2.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-4.el7sat.noarch
tfm-rubygem-dynflow-1.4.7-1.fm2_3.el7.noarch
tfm-rubygem-erubi-1.9.0-1.el7.noarch
tfm-rubygem-ethon-0.12.0-1.el7.noarch
tfm-rubygem-excon-0.76.0-1.el7.noarch
tfm-rubygem-execjs-2.7.0-4.el7.noarch
tfm-rubygem-facter-4.0.44-1.el7.x86_64
tfm-rubygem-faraday-0.17.3-1.el7.noarch
tfm-rubygem-fast_gettext-1.4.1-4.el7.noarch
tfm-rubygem-ffi-1.12.2-1.el7.x86_64
tfm-rubygem-fog-core-2.1.0-3.el7.noarch
tfm-rubygem-fog-vsphere-3.5.0-1.el7.noarch
tfm-rubygem-foreman-tasks-4.0.1-1.fm2_4.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-foreman_ansible-6.2.0-1.fm2_4.el7.noarch
tfm-rubygem-foreman_ansible_core-4.0.0-1.fm2_4.el7.noarch
tfm-rubygem-foreman_remote_execution-4.3.0-1.fm2_4.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.4.0-1.el7.noarch
tfm-rubygem-foreman_virt_who_configure-0.5.5-1.fm2_4.el7.noarch
tfm-rubygem-formatador-0.2.1-12.el7.noarch
tfm-rubygem-friendly_id-5.3.0-1.el7.noarch
tfm-rubygem-fx-0.5.0-1.el7.noarch
tfm-rubygem-get_process_mem-0.2.1-4.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.8.0-2.el7.noarch
tfm-rubygem-gitlab-sidekiq-fetcher-0.6.0-1.el7.noarch
tfm-rubygem-globalid-0.4.2-1.el7.noarch
tfm-rubygem-graphql-1.8.14-2.el7.noarch
tfm-rubygem-graphql-batch-0.3.10-2.el7.noarch
tfm-rubygem-gssapi-1.2.0-7.el7.noarch
tfm-rubygem-hammer_cli-2.4.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.4.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_ansible-0.3.3-1.fm2_4.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.7-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.2.1-1.fm2_4.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.15-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_katello-1.0-1.el7.noarch
tfm-rubygem-hashie-3.6.0-2.el7.noarch
tfm-rubygem-highline-2.0.3-1.el7.noarch
tfm-rubygem-hocon-1.3.1-1.el7.noarch
tfm-rubygem-http-cookie-1.0.2-5.el7sat.noarch
tfm-rubygem-i18n-1.8.2-1.el7.noarch
tfm-rubygem-ipaddress-0.8.0-12.el7.noarch
tfm-rubygem-jwt-2.2.1-2.el7.noarch
tfm-rubygem-kafo-6.2.1-1.el7.noarch
tfm-rubygem-kafo_parsers-1.1.0-3.el7.noarch
tfm-rubygem-kafo_wizards-0.0.2-1.el7.noarch
tfm-rubygem-katello-4.0.0-1.el7.noarch
tfm-rubygem-ldap_fluff-0.5.0-1.el7.noarch
tfm-rubygem-little-plugger-1.1.4-2.el7.noarch
tfm-rubygem-locale-2.0.9-14.el7.noarch
tfm-rubygem-logging-2.3.0-1.el7.noarch
tfm-rubygem-loofah-2.4.0-1.el7.noarch
tfm-rubygem-mail-2.7.1-1.el7.noarch
tfm-rubygem-marcel-0.3.3-1.el7.noarch
tfm-rubygem-method_source-0.9.2-2.el7.noarch
tfm-rubygem-mime-types-3.2.2-4.el7.noarch
tfm-rubygem-mime-types-data-3.2018.0812-4.el7.noarch
tfm-rubygem-mimemagic-0.3.5-1.el7.noarch
tfm-rubygem-mini_mime-1.0.2-1.el7.noarch
tfm-rubygem-mini_portile2-2.4.0-1.el7.noarch
tfm-rubygem-multi_json-1.14.1-2.el7.noarch
tfm-rubygem-multipart-post-2.0.0-2.el7.noarch
tfm-rubygem-mustermann-1.0.2-4.el7.noarch
tfm-rubygem-net-ldap-0.17.0-1.el7.noarch
tfm-rubygem-net-ping-2.0.1-4.el7.noarch
tfm-rubygem-net-scp-1.2.1-4.el7.noarch
tfm-rubygem-net-ssh-4.2.0-2.el7.noarch
tfm-rubygem-netrc-0.11.0-5.el7.noarch
tfm-rubygem-nio4r-2.5.4-1.el7.x86_64
tfm-rubygem-nokogiri-1.10.9-1.el7.x86_64
tfm-rubygem-oauth-0.5.4-4.el7.noarch
tfm-rubygem-optimist-3.0.0-2.el7.noarch
tfm-rubygem-parse-cron-0.1.4-4.fm2_1.el7.noarch
tfm-rubygem-passenger-4.0.18-10.13.el7.x86_64
tfm-rubygem-passenger-native-4.0.18-10.13.el7.x86_64
tfm-rubygem-passenger-native-libs-4.0.18-10.13.el7.x86_64
tfm-rubygem-pg-1.1.4-3.el7.x86_64
tfm-rubygem-polyglot-0.3.5-2.el7.noarch
tfm-rubygem-powerbar-2.0.1-2.el7sat.noarch
tfm-rubygem-promise.rb-0.7.4-2.el7.noarch
tfm-rubygem-public_suffix-3.0.3-2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.7.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.6.0-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.3-1.el7.noarch
tfm-rubygem-pulp_container_client-2.2.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.8.0-1.el7.noarch
tfm-rubygem-pulp_docker_client-4.0.0b6.dev01562331743-1.el7.noarch
tfm-rubygem-pulp_file_client-1.5.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.10.0-1.el7.noarch
tfm-rubygem-pulpcore_client-3.9.0-1.el7.noarch
tfm-rubygem-puma-5.1.1-1.el7.x86_64
tfm-rubygem-qpid_messaging-1.38.0-1.el7.x86_64
tfm-rubygem-qpid_proton-0.33.0-1.el7.x86_64
tfm-rubygem-rabl-0.14.3-1.el7.noarch
tfm-rubygem-rack-2.2.3-1.el7.noarch
tfm-rubygem-rack-cors-1.0.2-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-9.el7.noarch
tfm-rubygem-rack-protection-2.1.0-1.el7.noarch
tfm-rubygem-rack-test-1.1.0-4.el7.noarch
tfm-rubygem-rails-6.0.3.4-1.el7.noarch
tfm-rubygem-rails-dom-testing-2.0.3-6.el7.noarch
tfm-rubygem-rails-html-sanitizer-1.3.0-1.el7.noarch
tfm-rubygem-rails-i18n-6.0.0-2.el7.noarch
tfm-rubygem-railties-6.0.3.4-1.el7.noarch
tfm-rubygem-rainbow-2.2.1-3.el7.noarch
tfm-rubygem-rake-compiler-1.0.7-3.el7.noarch
tfm-rubygem-rb-inotify-0.9.7-5.el7.noarch
tfm-rubygem-rbvmomi-2.2.0-3.el7.noarch
tfm-rubygem-record_tag_helper-1.0.1-3.el7.noarch
tfm-rubygem-redfish_client-0.5.2-1.el7.noarch
tfm-rubygem-redis-4.1.2-2.el7.noarch
tfm-rubygem-responders-3.0.0-3.el7.noarch
tfm-rubygem-rest-client-2.0.2-3.el7.noarch
tfm-rubygem-rkerberos-0.1.5-19.el7.x86_64
tfm-rubygem-roadie-3.4.0-3.el7.noarch
tfm-rubygem-roadie-rails-2.1.1-2.el7.noarch
tfm-rubygem-robotex-1.0.0-21.el7.noarch
tfm-rubygem-rsec-0.4.3-4.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.1-1.el7.x86_64
tfm-rubygem-ruby2ruby-2.4.2-3.el7.noarch
tfm-rubygem-ruby_parser-3.10.1-3.el7.noarch
tfm-rubygem-rubyipmi-0.10.0-6.el7.noarch
tfm-rubygem-runcible-2.13.1-1.el7.noarch
tfm-rubygem-safemode-1.3.6-1.el7.noarch
tfm-rubygem-scoped_search-4.1.9-1.el7.noarch
tfm-rubygem-sd_notify-0.1.0-1.el7.noarch
tfm-rubygem-secure_headers-6.3.0-2.el7.noarch
tfm-rubygem-sequel-5.7.1-3.el7.noarch
tfm-rubygem-server_sent_events-0.1.2-1.el7.noarch
tfm-rubygem-sexp_processor-4.10.0-6.el7.noarch
tfm-rubygem-sidekiq-5.2.7-3.el7.noarch
tfm-rubygem-sinatra-2.1.0-1.el7.noarch
tfm-rubygem-smart_proxy_ansible-3.0.1-6.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_dynflow-0.3.0-2.fm2_4.el7.noarch
tfm-rubygem-smart_proxy_dynflow_core-0.3.2-1.fm2_4.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_remote_execution_ssh-0.3.1-1.fm2_4.el7.noarch
tfm-rubygem-sprockets-4.0.2-1.el7.noarch
tfm-rubygem-sprockets-rails-3.2.1-6.el7.noarch
tfm-rubygem-sqlite3-1.3.13-6.el7.x86_64
tfm-rubygem-sshkey-1.9.0-4.el7.noarch
tfm-rubygem-statsd-instrument-2.1.4-3.el7.noarch
tfm-rubygem-stomp-1.4.9-1.el7.noarch
tfm-rubygem-thor-1.0.1-2.el7.noarch
tfm-rubygem-thread_safe-0.3.6-5.el7.noarch
tfm-rubygem-tilt-2.0.8-4.el7.noarch
tfm-rubygem-typhoeus-1.3.1-1.el7.noarch
tfm-rubygem-tzinfo-1.2.6-1.el7.noarch
tfm-rubygem-unf-0.1.3-8.el7.noarch
tfm-rubygem-unf_ext-0.0.7.2-3.el7.x86_64
tfm-rubygem-unicode-0.4.4.4-3.el7.x86_64
tfm-rubygem-unicode-display_width-1.0.5-5.el7sat.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-7.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-5.el7.noarch
tfm-rubygem-websocket-driver-0.7.1-1.el7.x86_64
tfm-rubygem-websocket-extensions-0.1.5-1.el7.noarch
tfm-rubygem-will_paginate-3.1.7-3.el7.noarch
tfm-rubygem-xmlrpc-0.3.0-2.el7.noarch
tfm-rubygem-zeitwerk-2.2.2-1.el7.noarch
tfm-runtime-6.1-4.el7.x86_64

Foreman and Proxy plugin versions:

Distribution and version:
Rhel 7.9
Other relevant data:

Foreman task sync log:

{“pulp_tasks”=>
[{“pulp_href”=>"/pulp/api/v3/tasks/7ef900f6-6e4e-482d-ae0c-f393aacb32bf/",
“pulp_created”=>“2021-05-18T14:08:29.674+00:00”,
“state”=>“failed”,
“name”=>“pulp_rpm.app.tasks.synchronizing.synchronize”,
“logging_cid”=>“c1f1e51359a24ca6b4855b084c0cd079”,
“started_at”=>“2021-05-18T14:08:30.718+00:00”,
“finished_at”=>“2021-05-18T14:08:40.640+00:00”,
“error”=>
{“traceback”=>
" File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 975, in perform_job\n" +
" rv = job.perform()\n" +
" File “/usr/lib/python3.6/site-packages/rq/job.py”, line 696, in perform\n" +
" self._result = self._execute()\n" +
" File “/usr/lib/python3.6/site-packages/rq/job.py”, line 719, in _execute\n" +
" return self.func(*self.args, **self.kwargs)\n" +
" File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 269, in synchronize\n" +
" dv.create()\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create\n" +
" loop.run_until_complete(pipeline)\n" +
" File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete\n" +
" return future.result()\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline\n" +
" await asyncio.gather(*futures)\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in **call**\n" +
" await self.run()\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 263, in run\n" +
" RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 316, in _needed_remote_artifacts\n" +
" msg.format(rp=content_artifact.relative_path, c=d_content.content)\n",
“description”=>
“No declared artifact with relative path “pgdg-redhat-repo-42.0-11.noarch.rpm” for content “<Package: pgdg-redhat-repo>””},
“worker”=>"/pulp/api/v3/workers/25ce6979-f5d5-4a05-9e21-160934741f44/",
“child_tasks”=>,
“progress_reports”=>
[{“message”=>“Parsed Packages”,
“code”=>“parsing.packages”,
“state”=>“completed”,
“total”=>540,
“done”=>540},
{“message”=>“Downloading Metadata Files”,
“code”=>“downloading.metadata”,
“state”=>“completed”,
“done”=>4},
{“message”=>“Downloading Artifacts”,
“code”=>“downloading.artifacts”,
“state”=>“canceled”,
“done”=>0},
{“message”=>“Associating Content”,
“code”=>“associating.content”,
“state”=>“canceled”,
“done”=>0},
{“message”=>“Un-Associating Content”,
“code”=>“unassociating.content”,
“state”=>“canceled”,
“done”=>0},
{“message”=>“Parsed Comps”,
“code”=>“parsing.comps”,
“state”=>“completed”,
“total”=>1,
“done”=>1}],
“created_resources”=>,
“reserved_resources_record”=>
["/pulp/api/v3/repositories/rpm/rpm/8ea73b81-63f3-4064-a9af-c9d19d083060/",
“/pulp/api/v3/remotes/rpm/rpm/01f78dd7-200f-460a-a47f-23510c99ade0/”]}],
“create_version”=>true,
“task_groups”=>,
“poll_attempts”=>{“total”=>11, “failed”=>1}}

/var/log/messages:

May 19 07:55:56 foreman pulpcore-api: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: d2f0c95d79fa4fbfb225c3c93c491967
May 19 07:55:56 foreman pulpcore-api: - - [19/May/2021:05:55:56 +0000] "GET /pulp/api/v3/status HTTP/1.1" 301 0 "-" "rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157"
May 19 07:55:56 foreman pulpcore-api: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 32b730f1a8754e3da09faf68c1cfacef
May 19 07:55:56 foreman pulpcore-api: - - [19/May/2021:05:55:56 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4054 "-" "rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157"
May 19 07:55:57 foreman pulpcore-api: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 4727215826544302a4a291fc6a87118c
May 19 07:55:57 foreman pulpcore-resource-manager: pulp [None]: rq.worker:INFO: resource-manager: 2c0e4c35-3ff9-4562-b7d4-b6ed3efc92fa
May 19 07:55:57 foreman pulpcore-api: - - [19/May/2021:05:55:57 +0000] "POST /pulp/api/v3/repositories/rpm/rpm/8ea73b81-63f3-4064-a9af-c9d19d083060/sync/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.10.0/ruby"
May 19 07:55:57 foreman pulpcore-api: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 57372fff3f824bc9a1172e95f158ac57
May 19 07:55:57 foreman pulpcore-api: - - [19/May/2021:05:55:57 +0000] "GET /pulp/api/v3/tasks/acbf6a85-a2b1-43fa-b211-314b3d8e6ed9/ HTTP/1.1" 200 574 "-" "OpenAPI-Generator/3.9.0/ruby"
May 19 07:55:57 foreman pulpcore-resource-manager: pulp [4727215826544302a4a291fc6a87118c]: rq.worker:INFO: resource-manager: Job OK (2c0e4c35-3ff9-4562-b7d4-b6ed3efc92fa)
May 19 07:55:57 foreman pulpcore-worker-8: pulp [None]: rq.worker:INFO: 1361@foreman.example.com: acbf6a85-a2b1-43fa-b211-314b3d8e6ed9
May 19 07:55:57 foreman pulpcore-worker-8: pulp [4727215826544302a4a291fc6a87118c]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=75d31a7e-28c8-41bc-be6b-c63180040824 remote=5f358220f1028544342eb26f-75d31a7e-28c8-41bc-be6b-c63180040824
May 19 07:55:57 foreman pulpcore-api: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: f04c873e78e345229725c7db9b7fcc29
May 19 07:55:57 foreman pulpcore-api: - - [19/May/2021:05:55:57 +0000] "GET /pulp/api/v3/tasks/acbf6a85-a2b1-43fa-b211-314b3d8e6ed9/ HTTP/1.1" 200 655 "-" "OpenAPI-Generator/3.9.0/ruby"
May 19 07:55:58 foreman pulpcore-api: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 06daa573d4c14ba3913b8fb8ab4abd8b
May 19 07:55:58 foreman pulpcore-api: - - [19/May/2021:05:55:58 +0000] "GET /pulp/api/v3/tasks/acbf6a85-a2b1-43fa-b211-314b3d8e6ed9/ HTTP/1.1" 200 655 "-" "OpenAPI-Generator/3.9.0/ruby"
May 19 07:55:59 foreman pulpcore-api: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: d622f13dd9954b25934ddec3a0759509
May 19 07:55:59 foreman pulpcore-api: - - [19/May/2021:05:55:59 +0000] "GET /pulp/api/v3/tasks/acbf6a85-a2b1-43fa-b211-314b3d8e6ed9/ HTTP/1.1" 200 1246 "-" "OpenAPI-Generator/3.9.0/ruby"
May 19 07:55:59 foreman systemd: serial-getty@ttyS0.service holdoff time over, scheduling restart.
May 19 07:55:59 foreman systemd: Stopped Serial Getty on ttyS0.
May 19 07:55:59 foreman systemd: Started Serial Getty on ttyS0.
May 19 07:55:59 foreman pulpcore-api: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: a03aae47c9b04e6385efb9f8ec714334
May 19 07:55:59 foreman pulpcore-api: - - [19/May/2021:05:55:59 +0000] "GET /pulp/api/v3/tasks/acbf6a85-a2b1-43fa-b211-314b3d8e6ed9/ HTTP/1.1" 200 1361 "-" "OpenAPI-Generator/3.9.0/ruby"
May 19 07:56:00 foreman pulpcore-worker-8: pulp [4727215826544302a4a291fc6a87118c]: rq.worker:ERROR: Traceback (most recent call last):
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
May 19 07:56:00 foreman pulpcore-worker-8: rv = job.perform()
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
May 19 07:56:00 foreman pulpcore-worker-8: self._result = self._execute()
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
May 19 07:56:00 foreman pulpcore-worker-8: return self.func(*self.args, **self.kwargs)
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 269, in synchronize
May 19 07:56:00 foreman pulpcore-worker-8: dv.create()
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create
May 19 07:56:00 foreman pulpcore-worker-8: loop.run_until_complete(pipeline)
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
May 19 07:56:00 foreman pulpcore-worker-8: return future.result()
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
May 19 07:56:00 foreman pulpcore-worker-8: await asyncio.gather(*futures)
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
May 19 07:56:00 foreman pulpcore-worker-8: await self.run()
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 263, in run
May 19 07:56:00 foreman pulpcore-worker-8: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 316, in _needed_remote_artifacts
May 19 07:56:00 foreman pulpcore-worker-8: msg.format(rp=content_artifact.relative_path, c=d_content.content)
May 19 07:56:00 foreman pulpcore-worker-8: ValueError: No declared artifact with relative path "pgdg-redhat-repo-42.0-11.noarch.rpm" for content "<Package: pgdg-redhat-repo>"
May 19 07:56:00 foreman pulpcore-worker-8: Traceback (most recent call last):
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
May 19 07:56:00 foreman pulpcore-worker-8: rv = job.perform()
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
May 19 07:56:00 foreman pulpcore-worker-8: self._result = self._execute()
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
May 19 07:56:00 foreman pulpcore-worker-8: return self.func(*self.args, **self.kwargs)
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 269, in synchronize
May 19 07:56:00 foreman pulpcore-worker-8: dv.create()
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create
May 19 07:56:00 foreman pulpcore-worker-8: loop.run_until_complete(pipeline)
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
May 19 07:56:00 foreman pulpcore-worker-8: return future.result()
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
May 19 07:56:00 foreman pulpcore-worker-8: await asyncio.gather(*futures)
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
May 19 07:56:00 foreman pulpcore-worker-8: await self.run()
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 263, in run
May 19 07:56:00 foreman pulpcore-worker-8: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
May 19 07:56:00 foreman pulpcore-worker-8: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 316, in _needed_remote_artifacts
May 19 07:56:00 foreman pulpcore-worker-8: msg.format(rp=content_artifact.relative_path, c=d_content.content)
May 19 07:56:00 foreman pulpcore-worker-8: ValueError: No declared artifact with relative path "pgdg-redhat-repo-42.0-11.noarch.rpm" for content "<Package: pgdg-redhat-repo>"
May 19 07:56:00 foreman pulpcore-worker-8: pulp [None]: rq.worker:INFO: 1361@foreman.example.com: 61856726-c4c7-4274-b637-32c7730b5d1c
May 19 07:56:00 foreman pulpcore-worker-8: pulp [None]: rq.worker:INFO: 1361@foreman.example.com: Job OK (61856726-c4c7-4274-b637-32c7730b5d1c)
May 19 07:56:00 foreman pulpcore-api: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 2d734bdab816443399919046656e58d7
May 19 07:56:00 foreman pulpcore-api: - - [19/May/2021:05:56:00 +0000] "GET /pulp/api/v3/tasks/acbf6a85-a2b1-43fa-b211-314b3d8e6ed9/ HTTP/1.1" 200 2924 "-" "OpenAPI-Generator/3.9.0/ruby"
May 19 07:56:00 foreman pulpcore-api: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 074eb41994b443f9b3a43880a23e90c5
May 19 07:56:00 foreman pulpcore-api: - - [19/May/2021:05:56:00 +0000] "PATCH /pulp/api/v3/distributions/rpm/rpm/da7b128d-0e47-40b2-9bc5-298dad3f435c/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.10.0/ruby"
May 19 07:56:00 foreman pulpcore-resource-manager: pulp [None]: rq.worker:INFO: resource-manager: eb31b108-b98e-4b8b-bea7-31076d38381e
May 19 07:56:00 foreman pulpcore-api: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: b7bcd65fc82b48cba495467d4e0de8a7
May 19 07:56:00 foreman pulpcore-api: - - [19/May/2021:05:56:00 +0000] "GET /pulp/api/v3/tasks/0d9fb0a4-9758-4af5-81a6-e73255ef6c0f/ HTTP/1.1" 200 450 "-" "OpenAPI-Generator/3.9.0/ruby"
May 19 07:56:00 foreman pulpcore-resource-manager: pulp [074eb41994b443f9b3a43880a23e90c5]: rq.worker:INFO: resource-manager: Job OK (eb31b108-b98e-4b8b-bea7-31076d38381e)
May 19 07:56:00 foreman pulpcore-worker-2: pulp [None]: rq.worker:INFO: 1346@foreman.example.com: 0d9fb0a4-9758-4af5-81a6-e73255ef6c0f
May 19 07:56:00 foreman pulpcore-worker-2: pulp [074eb41994b443f9b3a43880a23e90c5]: rq.worker:INFO: 1346@foreman.example.com: Job OK (0d9fb0a4-9758-4af5-81a6-e73255ef6c0f)
May 19 07:56:00 foreman pulpcore-worker-2: pulp [None]: rq.worker:INFO: 1346@foreman.example.com: d002f572-8432-43a2-839f-0fbfbcaa6bf8
May 19 07:56:00 foreman pulpcore-worker-2: pulp [None]: rq.worker:INFO: 1346@foreman.example.com: Job OK (d002f572-8432-43a2-839f-0fbfbcaa6bf8)
May 19 07:56:01 foreman pulpcore-api: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 558bb4d5dca84f7ea4c67f54cb7449e3
May 19 07:56:01 foreman pulpcore-api: - - [19/May/2021:05:56:01 +0000] "GET /pulp/api/v3/tasks/0d9fb0a4-9758-4af5-81a6-e73255ef6c0f/ HTTP/1.1" 200 558 "-" "OpenAPI-Generator/3.9.0/ruby"

jeremylenz · May 19, 2021, 1:45pm

Hi @trendyandy, thanks for posting the info!

Can you try a validate content sync and see if that helps?

Products > (product name) > Repositories > (repository name) > Select Action button > Advanced Sync > Validate Content Sync

trendyandy · May 19, 2021, 2:21pm

Hi, @jeremylenz thanks for yor reply.
I actually wanted to avoid that, because of the on demand policy you can end up with a full disk.
Nevertheless i tried the Validate Content Sync. It ended with success, but it obviously did nothing.

Task output:

Total steps: 0/0
--------------------------------
Identify corrupted units: 0/0
Identify missing units: 0/0
Repair corrupted units: 0/0

dynflow output

---
response:
  task: "/pulp/api/v3/tasks/34820d85-abc7-4e2e-bd47-90645c8a2dd0/"
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/34820d85-abc7-4e2e-bd47-90645c8a2dd0/"
  pulp_created: '2021-05-19T14:12:08.756+00:00'
  state: completed
  name: pulpcore.app.tasks.repository.repair_version
  logging_cid: 13ced881f72f4f4885e4784c1ceec113
  started_at: '2021-05-19T14:12:08.871+00:00'
  finished_at: '2021-05-19T14:12:09.072+00:00'
  worker: "/pulp/api/v3/workers/12aadcd6-ec2b-4c8b-a640-a394abe19026/"
  child_tasks: []
  progress_reports:
  - message: Identify missing units
    code: repair.missing
    state: completed
    done: 0
  - message: Identify corrupted units
    code: repair.corrupted
    state: completed
    done: 0
  - message: Repair corrupted units
    code: repair.repaired
    state: completed
    done: 0
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/8ea73b81-63f3-4064-a9af-c9d19d083060/"
task_groups: []
poll_attempts:
  total: 1
  failed: 0

trendyandy · May 19, 2021, 2:50pm

I also tried Republish Repository Metadata
I always end up with the error message from the first post. May something went wrong while migrating the content from pulp2.

gvde · June 4, 2021, 5:59am

I have the same issue. Validate content sync doesn’t help… With pulp2/3.18 everything was fine.

gvde · June 4, 2021, 8:52am

Even if I set up a new repository, I’ll get the same issue, e.g. syncing against grafana repository https://packages.grafana.com/oss/rpm

No declared artifact with relative path "grafana-2.6.0-1.x86_64.rpm" for content "<Package: grafana>"

gvde · June 6, 2021, 12:11pm

Can anyone explain what exactly is missing with this error?

I went through the pulpcore database but that seems all to be O.K. I can find an entry in core_contentartifact with relative_path “grafana-2.6.0-1.x86_64.rpm”. The referenced core_artifact with artifcat_id and core_content with content_id have the entries. Via core_artifact I can find the rpm file in the pulp media directory and it seems to contain the correct rpm file. I can also find the package in rpm_package via the content_id.

So what exactly is that ‘declared artifact’ which is missing?

trendyandy · June 10, 2021, 6:44am

The grafana repo contains 2 packages with the same pkgid, which is causing the issue.

<package type="rpm">
  <name>grafana</name>
  <arch>x86_64</arch>
  <version epoch="0" ver="2.6.0" rel="1"/>
  <checksum type="sha256" pkgid="YES">e9cf75c0fe1695c68fb7b5004164787364393f3d9049e41711d0f831288875fb</checksum>
  <summary>Grafana</summary>
  <description>Grafana</description>
  <packager>contact@grafana.org</packager>
  <url>http://grafana.org</url>
  <time file="1545209942" build="1450103218"/>
  <size package="23587232" installed="69583665" archive="69714708"/>
  <location href="grafana-2.6.0-1.x86_64.rpm"/>
  <format>
    <rpm:license>Apache 2.0</rpm:license>
    <rpm:vendor>Grafana</rpm:vendor>
    <rpm:group>default</rpm:group>
    <rpm:buildhost>a7dabff35e56</rpm:buildhost>
    <rpm:sourcerpm>grafana-2.6.0-1.src.rpm</rpm:sourcerpm>
    <rpm:header-range start="880" end="79852"/>
    <rpm:provides>
      <rpm:entry name="grafana" flags="EQ" epoch="0" ver="2.6.0" rel="1"/>
      <rpm:entry name="grafana(x86-64)" flags="EQ" epoch="0" ver="2.6.0" rel="1"/>
    </rpm:provides>
    <rpm:requires>
      <rpm:entry name="initscripts"/>
      <rpm:entry name="fontconfig"/>
      <rpm:entry name="/bin/sh" pre="1"/>
    </rpm:requires>
    <file>/etc/grafana/grafana.ini</file>
    <file>/etc/grafana/ldap.toml</file>
    <file>/etc/init.d/grafana-server</file>
    <file>/etc/sysconfig/grafana-server</file>
    <file>/usr/sbin/grafana-server</file>
  </format>
</package>
<package type="rpm">
  <name>grafana</name>
  <arch>x86_64</arch>
  <version epoch="0" ver="2.6.0" rel="1"/>
  <checksum type="sha256" pkgid="YES">e9cf75c0fe1695c68fb7b5004164787364393f3d9049e41711d0f831288875fb</checksum>
  <summary>Grafana</summary>
  <description>Grafana</description>
  <packager>contact@grafana.org</packager>
  <url>http://grafana.org</url>
  <time file="1545209942" build="1450103218"/>
  <size package="23587232" installed="69583665" archive="69714708"/>
  <location href="grafana-2.6.0.x86_64.rpm"/>
  <format>
    <rpm:license>Apache 2.0</rpm:license>
    <rpm:vendor>Grafana</rpm:vendor>
    <rpm:group>default</rpm:group>
    <rpm:buildhost>a7dabff35e56</rpm:buildhost>
    <rpm:sourcerpm>grafana-2.6.0-1.src.rpm</rpm:sourcerpm>
    <rpm:header-range start="880" end="79852"/>
    <rpm:provides>
      <rpm:entry name="grafana" flags="EQ" epoch="0" ver="2.6.0" rel="1"/>
      <rpm:entry name="grafana(x86-64)" flags="EQ" epoch="0" ver="2.6.0" rel="1"/>
    </rpm:provides>
    <rpm:requires>
      <rpm:entry name="initscripts"/>
      <rpm:entry name="fontconfig"/>
      <rpm:entry name="/bin/sh" pre="1"/>
    </rpm:requires>
    <file>/etc/grafana/grafana.ini</file>
    <file>/etc/grafana/ldap.toml</file>
    <file>/etc/init.d/grafana-server</file>
    <file>/etc/sysconfig/grafana-server</file>
    <file>/usr/sbin/grafana-server</file>
  </format>
</package>

this is somehow related to: Repo metadata relative path · pulp/pulp_rpm@dcf82ef · GitHub

gvde · June 10, 2021, 8:38am

Interesting. Well, I guess that error message could be better then to help point the user into the right direction.

I get the same error message also with most of the postgresql repositories for el7 and el8 taken from https://download.postgresql.org/pub/repos/yum/

No declared artifact with relative path "pgdg-redhat-repo-42.0-9.noarch.rpm" for content "<Package: pgdg-redhat-repo>"

Some with version 42.0-11. I think it’s because pgdg-redhat-repo-latest.noarch.rpm is identical to the versioned one…

sajha · June 11, 2021, 2:20pm

@trendyandy , @gvde : Nice debugging! Did you happen to find any open pulp issue around this?

gvde · June 11, 2021, 2:27pm

I don’t think there is a pulp issue for. I cannot find anything with either search on Issues - Migration Plugin - Pulp or with a plain google search.

I have found some issues related to .treeinfo or to boot/distribution files. But nothing closely resembling this. So I guess there isn’t a issue with pulp regarding this, yet.

trendyandy · June 14, 2021, 6:21am

There was Issue #8133: Same package from different repos yields error on sync - RPM Support - Pulp which had the same error message.
marked as (not reproducable)

gvde · June 14, 2021, 9:29am

Strange. I didn’t find that. I have just added my comment to pulp issue 8133 that I still have this issue.

trendyandy · June 29, 2021, 8:59am

Are there any news about this issue?

gvde · July 6, 2021, 2:02pm

It seems as if pulp issue #8133 doesn’t get any attention. Maybe we have to open a new issue as 8133 originally couldn’t be replicated?

gvde · July 8, 2021, 6:45am

@iballou I don’t know who is the right person to ask who knows the pulp people…

iballou · July 8, 2021, 1:12pm

I’ll try to bring some attention to the bug. They may need extra information from your environments to reproduce and fix.

gvde · August 21, 2021, 12:09pm

@iballou Today I have installed pulpcore 3.14.4 into my katello 4.1.2.1 on CentOS 7, as it contains Issue #9130: Backport #8133 "Same package from different repos yields error on sync" to 3.14.z - Pulp which backports Issue #8133: Same package from different repos yields error on sync - Pulp which should fix this problem.

However, my two repositories with sync errors

https://packages.grafana.com/oss/rpm
https://repo.letsconnect-vpn.org/2/rpm/epel-7-x86_64/

still throw errors.

Reading the note in #9130:

There are RPM repositories “in the wild” that violate Pulp’s assumptions about what is ‘legal’ in an incoming repository version. This commit changes some fatal errors into log-warnings, along with some heuristics around how to get a ‘reasonable’ repository in the face of suboptimal data.

To control activation of this behavior, RemoteArtifactSaver recognizes a ctor-keyword “fix_mismatched_remote_artifacts”. This defaults to False; if provided as True, invokes the new behavior.

it seems this is intentionally. It’s unclear to me how to set this options to activate it.

ggainey · August 23, 2021, 1:04pm

You need pulp_rpm 3.14.2 as well, which takes advantage of the work in pulpcore/3.14.4 for 8133

Justin_Sherrill · August 23, 2021, 1:21pm

we’ll work on getting it packaged and updated in our yum repos