RPM1008: Checksum type "sha256" is not available for all units in the repository

Hello John,
thanks for reply!
Yes, I ran foreman-installer --upgrade before installing katello rpm.
These are the two answers for your request:

=> ["arch", "checksum_type", "container_repository_name", "content_id", "content_type", "content_view_version_id", "cp_label", "created_at", "deb_architectures", "deb_components", "deb_releases", "description", "distribution_arch", "distribution_bootable", "distribution_family", "distribution_uuid", "distribution_variant", "distribution_version", "docker_upstream_name", "download_policy", "environment_id", "gpg_key_id", "id", "ignorable_content", "ignore_global_proxy", "label", "library_instance_id", "major", "minor", "mirror_on_sync", "name", "ostree_upstream_sync_depth", "ostree_upstream_sync_policy", "product_id", "pulp_id", "relative_path", "source_repo_checksum_type", "ssl_ca_cert_id", "ssl_client_cert_id", "ssl_client_key_id", "unprotected", "updated_at", "upstream_password", "upstream_username", "url", "verify_ssl_on_sync"]

and

MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017/pulp_database
MongoDB server version: 3.4.9
65330

The first one differ from your after “relative path” :

(mine)
"source_repo_checksum_type", "ssl_ca_cert_id", "ssl_client_cert_id", "ssl_client_key_id", "unprotected", "updated_at", "upstream_password", "upstream_username", "url", "verify_ssl_on_sync"]

(your)
"ssl_ca_cert_id", "ssl_client_cert_id", "ssl_client_key_id", "unprotected", "updated_at", "upstream_password", "upstream_username", "url", "verify_ssl_on_sync"]

It seems "source_repo_checksum_type" remained!

From those two outputs, it looks like the upgrade tasks didn’t run

I think we found the issue, you have to run the upgrade command after installing the katello rpm. This ensures that the new database migration, upgrade tasks, etc… that the new rpm provide are correctly applied.

Try running ‘foreman-installer --upgrade’ now and see where that gets you :slight_smile:

Thanks John, I ran the upgrade the output is the same of before upgrade (source_repo_checksum_type is present):

=> ["arch", "checksum_type", "container_repository_name", "content_id", "content_type", "content_view_version_id", "cp_label", "created_at", "deb_architectures", "deb_components", "deb_releases", "description", "distribution_arch", "distribution_bootable", "distribution_family", "distribution_uuid", "distribution_variant", "distribution_version", "docker_upstream_name", "download_policy", "environment_id", "gpg_key_id", "id", "ignorable_content", "ignore_global_proxy", "label", "library_instance_id", "major", "minor", "mirror_on_sync", "name", "ostree_upstream_sync_depth", "ostree_upstream_sync_policy", "product_id", "pulp_id", "relative_path", "source_repo_checksum_type", "ssl_ca_cert_id", "ssl_client_cert_id", "ssl_client_key_id", "unprotected", "updated_at", "upstream_password", "upstream_username", "url", "verify_ssl_on_sync"]
irb(main):002:0> quit


MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017/pulp_database
MongoDB server version: 3.4.9
65330

This is the --upgrade log (i attach also katello.log)

Upgrading, to monitor the progress on all related services, please do:
  foreman-tail | tee upgrade-$(date +%Y-%m-%d-%H%M).log
Upgrade Step: stop_services...
Redirecting to 'foreman-maintain service'
Running Stop Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Stop applicable services: Stopping the following service(s):

postgresql, rh-mongodb34-mongod, qdrouterd, qpidd, squid, pulp_resource_manager, pulp_streamer, smart_proxy_dynflow_core, foreman-proxy, pulp_celerybeat, pulp_workers, tomcat, goferd, dynflowd, httpd, puppetserver

/ All services stopped                                                [OK]
--------------------------------------------------------------------------------

katello-service stop finished successfully!
Upgrade Step: start_postgresql...
Redirecting to 'foreman-maintain service'
Running Start Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Start applicable services: Starting the following service(s):

postgresql

\ All services started                                                [OK]
--------------------------------------------------------------------------------

katello-service start --only postgresql finished successfully!
Upgrade Step: migrate_pulp...
systemctl start rh-mongodb34-mongod finished successfully!
Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors [deb.json]
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest_list = pulp_docker.plugins.models:ManifestList
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Loading unit model: deb_release = pulp_deb.plugins.db.models:DebRelease
Loading unit model: deb = pulp_deb.plugins.db.models:DebPackage
Loading unit model: deb_component = pulp_deb.plugins.db.models:DebComponent
Loading auxiliary model: erratum_pkglists = pulp_rpm.plugins.db.models:ErratumPkglist
Updating the database with types [deb, deb_component, deb_release]
Found the following type definitions that were not present in the update collection [puppet_module, drpm, package_langpacks, docker_manifest, docker_blob, erratum, yum_repo_metadata_file, package_group, package_category, iso, package_e  nvironment, docker_tag, srpm, rpm, distribution, docker_image, docker_manifest_list]
Updating the database with types [puppet_module, drpm, deb_component, package_langpacks, docker_manifest, docker_blob, erratum, yum_repo_metadata_file, deb_release, package_group, package_category, iso, package_environment, docker_tag,   deb, distribution, rpm, srpm, docker_image, docker_manifest_list]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 28
Migration package pulp_deb.plugins.migrations is up to date at version 1
Migration package pulp_docker.plugins.migrations is up to date at version 5
Migration package pulp_puppet.plugins.migrations is up to date at version 5
Migration package pulp_rpm.plugins.migrations is up to date at version 42
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest_list = pulp_docker.plugins.models:ManifestList
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Loading unit model: deb_release = pulp_deb.plugins.db.models:DebRelease
Loading unit model: deb = pulp_deb.plugins.db.models:DebPackage
Loading unit model: deb_component = pulp_deb.plugins.db.models:DebComponent
Loading auxiliary model: erratum_pkglists = pulp_rpm.plugins.db.models:ErratumPkglist
Database migrations complete.
su - apache -s /bin/bash -c pulp-manage-db finished successfully!
Upgrade Step: migrate_candlepin...
Configuring PostgreSQL with JDBC URL: jdbc:postgresql://localhost/candlepin
Migrating Candlepin database
--driver=org.postgresql.Driver --classpath=/usr/share/java/postgresql-jdbc.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ --changeLogFile=db/changelog/changelog-update.xml --url=jdbc:postgresql://localhost/candlepin --username=  candlepin --password=UuzUMnK3XgDb9Heu2FXa4jZHwYUky7mR --logLevel=severe
Liquibase Update Successful
/usr/share/candlepin/cpdb --update --database '//localhost/caandlepin' --user 'candlepin' --password 'UuzUMnK3XgDb9Heu2FXa4jZHwYUky7mR' finished successfully!
Upgrade Step: migrate_foreman...
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
foreman-rake db:migrate finished successfully!
Upgrade Step: Running installer...
Installing             Done                                               [100%] [........................................................................................................................................................]
  The full log is at /var/log/foreman-installer/katello.log
Upgrade Step: restart_services...
Redirecting to 'foreman-maintain service'
Running Restart Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Restart applicable services: Stopping the following service(s):

postgresql, rh-mongodb34-mongod, qdrouterd, qpidd, squid, pulp_resource_manager, pulp_streamer, smart_proxy_dynflow_core, foreman-proxy, pulp_celerybeat, pulp_workers, tomcat, goferd, dynflowd, httpd, puppetserver

| All services stopped
Starting the following service(s):

postgresql, rh-mongodb34-mongod, qdrouterd, qpidd, squid, pulp_resource_manager, pulp_streamer, smart_proxy_dynflow_core, foreman-proxy, pulp_celerybeat, pulp_workers, tomcat, goferd, dynflowd, httpd, puppetserver

\ All services started                                                [OK]
--------------------------------------------------------------------------------

katello-service restart finished successfully!
Upgrade Step: db_seed...
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
Seeding /usr/share/foreman/db/seeds.d/020-partition_tables_list.rb
Seeding /usr/share/foreman/db/seeds.d/020-permissions_list.rb
Seeding /usr/share/foreman/db/seeds.d/020-provisioning_templates_list.rb
Seeding /usr/share/foreman/db/seeds.d/020-roles_list.rb
Seeding /usr/share/foreman/db/seeds.d/030-auth_sources.rb
Seeding /usr/share/foreman/db/seeds.d/030-permissions.rb
Seeding /usr/share/foreman/db/seeds.d/035-admin.rb
Seeding /usr/share/foreman/db/seeds.d/040-roles.rb
Seeding /usr/share/foreman/db/seeds.d/050-taxonomies.rb
Seeding /usr/share/foreman/db/seeds.d/060-architectures.rb
Seeding /usr/share/foreman/db/seeds.d/070-provisioning_templates.rb
Seeding /usr/share/foreman/db/seeds.d/080-partition_tables.rb
Seeding /usr/share/foreman/db/seeds.d/100-installation_media.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/db/seeds.d/101-locations.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/db/seeds.d/102-organizations.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/db/seeds.d/104-proxy.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/db/seeds.d/106-mail_notifications.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/db/seeds.d/107-enable_dynflow.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/db/seeds.d/108-subcription-bookmarks.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/db/seeds.d/109-katello-notification-blueprints.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/db/seeds.d/110-content-view-autopublish.rb
Seeding /usr/share/foreman/db/seeds.d/110-smart_proxy_features.rb
Seeding /usr/share/foreman/db/seeds.d/130-compute_profiles.rb
Seeding /usr/share/foreman/db/seeds.d/150-bookmarks.rb
Seeding /usr/share/foreman/db/seeds.d/160-mail_notifications.rb
Seeding /usr/share/foreman/db/seeds.d/170-notification_blueprints.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/db/seeds.d/20-foreman_tasks_permissions.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_bootdisk-12.0.0/db/seeds.d/50-bootdisk_templates.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.5.6/db/seeds.d/50-notification_blueprints.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/db/seeds.d/60-dynflow_proxy_feature.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.5.6/db/seeds.d/60-ssh_proxy_feature.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/db/seeds.d/61-foreman_tasks_bookmarks.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-2.2.9/db/seeds.d/62_ansible_proxy_feature.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.5.6/db/seeds.d/70-job_templates.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/db/seeds.d/75-job_templates.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-2.2.9/db/seeds.d/75_job_templates.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.5.6/db/seeds.d/90-bookmarks.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-2.2.9/db/seeds.d/90_notification_blueprints.rb
All seed files executed
foreman-rake db:seed finished successfully!
Upgrade Step: correct_repositories (this may take a while) ...
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
Processing Repository 1/51: 
.......<a class="attachment" href="/uploads/default/original/2X/6/6b09ba73aa957e4fbf53be6806a068b946b1cdb3.log">katello.log</a> (2.3 MB)
 
[cut]
.......
foreman-rake katello:correct_repositories COMMIT=true finished successfully!
Upgrade Step: correct_puppet_environments (this may take a while) ...
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
Processing Puppet Environment 1/1: 1-RHEL_Views-v18_0-puppet-c8f546ea-4d31-48e4-8371-76010ff1d687 (34)
foreman-rake katello:correct_puppet_environments COMMIT=true finished successfully!
Upgrade Step: clean_backend_objects (this may take a while) ...
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []
0 orphaned consumer id(s) found in pulp.
Pulp orphaned consumers: []
foreman-rake katello:clean_backend_objects COMMIT=true finished successfully!
Upgrade completed!

Just to confirm you are on the katello-3.8.1 rpm?

If so, I think the upgrade is skipping the migration as it thinks it already has ran and then that is affecting the upgrade task.

Try foreman-rake db:migrate

Check the Repository column names again to see if they have updated properly

then run the foreman-installer --upgrade again

actually a better way would be to run foreman-installer --upgrade --force-upgrade-steps, so the steps are forcibly re-run

Try running things manually
foreman-rake db:migrate
then source_repo_checksum_type should be gone from the column names

and then
foreman-rake katello:upgrades:3.8:clear_checksum_type
after this task runs, the mongo query should return 0

then for good measure foreman-installer --upgrade

it return the same values:

foreman-rake db:migrate
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here

foreman-rake console
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
Loading production environment (Rails 5.1.6)
Failed to load console gems, starting anyway
irb(main):001:0> Katello::Repository.column_names.sort
=> ["arch", "checksum_type", "container_repository_name", "content_id", "content_type", "content_view_version_id", "cp_label", "created_at", "deb_architectures", "deb_components", "deb_releases", "description", "distribution_arch", "distribution_bootable", "distribution_family", "distribution_uuid", "distribution_variant", "distribution_version", "docker_upstream_name", "download_policy", "environment_id", "gpg_key_id", "id", "ignorable_content", "ignore_global_proxy", "label", "library_instance_id", "major", "minor", "mirror_on_sync", "name", "ostree_upstream_sync_depth", "ostree_upstream_sync_policy", "product_id", "pulp_id", "relative_path", "source_repo_checksum_type", "ssl_ca_cert_id", "ssl_client_cert_id", "ssl_client_key_id", "unprotected", "updated_at", "upstream_password", "upstream_username", "url", "verify_ssl_on_sync"]
irb(main):002:0> quit

foreman-rake katello:upgrades:3.8:clear_checksum_type
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here

mongo pulp_database --eval 'db.units_rpm.find({checksumtype: { $ne: "sha256" }}).count()'
MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017/pulp_database
MongoDB server version: 3.4.9
65330

I ran also the upgrade but nothing changed (same results from query after that).
If it’s useful, my mongo db packages are:

python-mongoengine.noarch              0.10.5-1.el7          @pulp
python-pymongo.x86_64                  3.2-2.el7             @pulp
python-pymongo-gridfs.x86_64           3.2-2.el7             @pulp
rh-mongodb34.x86_64                    3.0-14.el7            @centos-sclo-rh
rh-mongodb34-boost-chrono.x86_64       1.60.0-2.el7          @centos-sclo-rh
rh-mongodb34-boost-filesystem.x86_64   1.60.0-2.el7          @centos-sclo-rh
rh-mongodb34-boost-iostreams.x86_64    1.60.0-2.el7          @centos-sclo-rh
rh-mongodb34-boost-program-options.x86_64
rh-mongodb34-boost-regex.x86_64        1.60.0-2.el7          @centos-sclo-rh
rh-mongodb34-boost-system.x86_64       1.60.0-2.el7          @centos-sclo-rh
rh-mongodb34-boost-thread.x86_64       1.60.0-2.el7          @centos-sclo-rh
rh-mongodb34-gperftools-libs.x86_64    2.5-4.el7             @centos-sclo-rh
rh-mongodb34-libstemmer.x86_64         0-5.585svn.el7        @centos-sclo-rh
rh-mongodb34-libunwind.x86_64          1.2-3.el7             @centos-sclo-rh
rh-mongodb34-mongo-tools.x86_64        3.4.7-1.el7           @centos-sclo-rh
rh-mongodb34-mongo-tools-syspaths.x86_64
rh-mongodb34-mongodb.x86_64            3.4.9-1.el7           @centos-sclo-rh
rh-mongodb34-mongodb-server.x86_64     3.4.9-1.el7           @centos-sclo-rh
rh-mongodb34-mongodb-server-syspaths.x86_64
rh-mongodb34-mongodb-syspaths.x86_64   3.4.9-1.el7           @centos-sclo-rh
rh-mongodb34-runtime.x86_64            3.0-14.el7            @centos-sclo-rh
rh-mongodb34-syspaths.x86_64           3.0-14.el7            @centos-sclo-rh
rh-mongodb34-yaml-cpp.x86_64           0.5.3-1.el7           @centos-sclo-rh

Just to check, you are definitely on Katello 3.8.1? There were no errors during the db:migrate command?

Yes, I’m on Katello 3.8.1 and no errors about db:migrate, my production.log during db:migrate wrote only:

2018-11-28T13:42:33 [W|app|] Creating scope :path_of. Overwriting existing method Taxonomy.path_of.
2018-11-28T13:42:35 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2018-11-28T13:42:35 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2018-11-28T13:42:38 [W|app|] Creating scope :path_of. Overwriting existing method Hostgroup.path_of.
2018-11-28T13:42:48 [W|app|] ignoring associations organization_ids, location_ids audit definition for Container, the resource is not audited
2018-11-28T13:42:48 [W|app|] Creating scope :path_of. Overwriting existing method FactName.path_of.
2018-11-28T13:42:49 [I|dyn|] start terminating throttle_limiter...
2018-11-28T13:42:49 [I|dyn|] start terminating executor...
2018-11-28T13:42:49 [I|dyn|] shutting down Core ...
2018-11-28T13:42:49 [E|dyn|] ... core terminated.
2018-11-28T13:42:49 [I|dyn|] start terminating executor dispatcher...
2018-11-28T13:42:49 [I|dyn|] start terminating client dispatcher...
2018-11-28T13:42:49 [I|dyn|] stop listening for new events...
2018-11-28T13:42:49 [I|dyn|] start terminating clock...

Can you share the exact katello rpm version? I just want to compare to my instance to rule any difference out

Sure:

foreman-installer-katello.noarch       3.8.1-1.el7  
katello.noarch                         3.8.1-1.el7         
katello-certs-tools.noarch             2.4.0-1.el7       
katello-client-bootstrap.noarch        1.4.2-1.el7           
katello-common.noarch                  3.8.1-1.el7           
katello-debug.noarch                   3.8.1-1.el7                       
katello-host-tools.noarch              3.3.5-3.el7           
katello-host-tools-fact-plugin.noarch  3.3.5-3.el7           
katello-installer-base.noarch          3.8.1-1.el7           
katello-repos.noarch                   3.8.1-1.el7          
katello-selinux.noarch                 3.0.3-1.el7           
katello-service.noarch                 3.8.1-1.el7               

What version of tfm-rubygem-katello do you have?

Also, can you check if the original issue is still present? It actually looks like your database is fully migrated now, I compared to an updated katello 3.8.1, the instance I was on had an older tfm-rubygem-katello package, which is what ultimately contains the migrations. Comparing now, the column names look correct.

Looking more closely at the task to change the checksum_types, it only updates on-demand repositories, so the count doesn’t actually have to be zero for that mongo query.

Please double check the original issue is present, its possible that some of the above things we tried actually worked. If not, we can proceed with more debugging

I tried to publish an incremental update but the error is still present (it fail at 69%)

The installed tfm is:

tfm-rubygem-katello.noarch 3.8.1-1.el7

and the others from katello repo are:

tfm-rubygem-anemone.noarch             0.7.2-15.el7          
tfm-rubygem-hammer_cli_csv.noarch      2.3.1-1.el7          
tfm-rubygem-hammer_cli_katello.noarch  0.14.1-1.el7         
tfm-rubygem-qpid_messaging.x86_64      1.36.0-2.el7          
tfm-rubygem-robotex.noarch             1.0.0-20.el7         

@kraba Thanks for the info, this may be an actual bug on an incremental update. Pulp changed the way they handle checksum types and we accounted for this in katello with the upgrade task, but its possible we missed cases dealing with an incremental update.

I’m trying to reproduce myself and will get back to you on any related issues or fixes.

For now, if you want to work around this, you can set the download policies of the repos that are giving you trouble to ‘immediate’ and resyncing.

Hope this helps!

Thanks a lot John, if you need logs/test/something feel free to ask that!

@kraba what repository(ies) were triggering the incremental update?

Our repositories are the follows:

---|-----------------------------------------------------------------------------|
ID | NAME                                                           REPOSITORIES |
---|-----------------------------------------------------------------------------|
9  | Red Hat Developer Toolset (for RHEL Server)                    2            |
6  | Red Hat Developer Tools (for RHEL Server)                      1            |
2  | Red Hat Enterprise Linux Server                                7            |
11 | Red Hat Enterprise Linux Server - Extended Life Cycle Support  1            |
10 | Red Hat Software Collections (for RHEL Server)                 2            |
---|-----------------------------------------------------------------------------|

and they are configured as the follow:

SHA1
=> #<Katello::Repository id: 1142, name: "Red Hat Enterprise Linux 7 Server RPMs x86_64 7Ser...", pulp_id: "d8946303-7782-4197-a43d-7e60be8a11a6", created_at: "2018-10-16 15:29:44", updated_at: "2018-11-28 13:21:55", major: 7, minor: "7Server", gpg_key_id: nil, cp_label: "rhel-7-server-rpms", library_instance_id: nil, content_id: "2456", arch: "x86_64", label: "Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Ser...", content_view_version_id: 1, relative_path: "our-organization/Library/content/dist/rhel/server/7/7Se...", url: "https://cdn.redhat.com/content/dist/rhel/server/7/...", unprotected: false, content_type: "yum", product_id: 2, environment_id: 1, checksum_type: nil, docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, container_repository_name: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignore_global_proxy: false, ignorable_content: nil, description: nil, source_repo_checksum_type: "sha1">
=> #<Katello::Repository id: 1154, name: "Red Hat Software Collections RPMs for Red Hat Ente...", pulp_id: "92304746-5061-436a-abb6-67ae17fda5d3", created_at: "2018-10-16 15:41:28", updated_at: "2018-11-28 13:22:03", major: 6, minor: "6Server", gpg_key_id: nil, cp_label: "rhel-server-rhscl-6-rpms", library_instance_id: nil, content_id: "1997", arch: "x86_64", label: "Red_Hat_Software_Collections_RPMs_for_Red_Hat_Ente...", content_view_version_id: 1, relative_path: "our-organization/Library/content/dist/rhel/server/6/6Se...", url: "https://cdn.redhat.com/content/dist/rhel/server/6/...", unprotected: false, content_type: "yum", product_id: 10, environment_id: 1, checksum_type: nil, docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, container_repository_name: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignore_global_proxy: false, ignorable_content: nil, description: nil, source_repo_checksum_type: "sha1">
=> #<Katello::Repository id: 1220, name: "Red Hat Software Collections RPMs for Red Hat Ente...", pulp_id: "afe43e60-de92-4047-9314-4b1e2203cb00", created_at: "2018-10-17 11:24:34", updated_at: "2018-11-28 13:22:04", major: 7, minor: "7Server", gpg_key_id: nil, cp_label: "rhel-server-rhscl-7-rpms", library_instance_id: nil, content_id: "2808", arch: "x86_64", label: "Red_Hat_Software_Collections_RPMs_for_Red_Hat_Ente...", content_view_version_id: 1, relative_path: "our-organization/Library/content/dist/rhel/server/7/7Se...", url: "https://cdn.redhat.com/content/dist/rhel/server/7/...", unprotected: false, content_type: "yum", product_id: 10, environment_id: 1, checksum_type: nil, docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, container_repository_name: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignore_global_proxy: false, ignorable_content: nil, description: nil, source_repo_checksum_type: "sha1">
=> #<Katello::Repository id: 1146, name: "Red Hat Enterprise Linux 6 Server RPMs x86_64 6Ser...", pulp_id: "4bd34ff5-3f67-44b0-94ac-56c5a921c3b1", created_at: "2018-10-16 15:31:09", updated_at: "2018-11-28 13:21:58", major: 6, minor: "6Server", gpg_key_id: nil, cp_label: "rhel-6-server-rpms", library_instance_id: nil, content_id: "168", arch: "x86_64", label: "Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Ser...", content_view_version_id: 1, relative_path: "our-organization/Library/content/dist/rhel/server/6/6Se...", url: "https://cdn.redhat.com/content/dist/rhel/server/6/...", unprotected: false, content_type: "yum", product_id: 2, environment_id: 1, checksum_type: nil, docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, container_repository_name: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignore_global_proxy: false, ignorable_content: nil, description: nil, source_repo_checksum_type: "sha1">
=> #<Katello::Repository id: 1145, name: "Red Hat Enterprise Linux 7 Server - Extras RPMs x8...", pulp_id: "b669c18e-0c25-43a2-82ee-93cad0db2d14", created_at: "2018-10-16 15:30:32", updated_at: "2018-11-28 13:21:58", major: nil, minor: nil, gpg_key_id: nil, cp_label: "rhel-7-server-extras-rpms", library_instance_id: nil, content_id: "3030", arch: "x86_64", label: "Red_Hat_Enterprise_Linux_7_Server_-_Extras_RPMs_x8...", content_view_version_id: 1, relative_path: "our-organization/Library/content/dist/rhel/server/7/7Se...", url: "https://cdn.redhat.com/content/dist/rhel/server/7/...", unprotected: false, content_type: "yum", product_id: 2, environment_id: 1, checksum_type: nil, docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, container_repository_name: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignore_global_proxy: false, ignorable_content: nil, description: nil, source_repo_checksum_type: "sha1">
=> #<Katello::Repository id: 1151, name: "Red Hat Enterprise Linux 6 Server - Extras RPMs x8...", pulp_id: "ea2ab144-f6e8-4f43-8537-a31c7faf724d", created_at: "2018-10-16 15:40:01", updated_at: "2018-11-28 13:22:02", major: nil, minor: nil, gpg_key_id: nil, cp_label: "rhel-6-server-extras-rpms", library_instance_id: nil, content_id: "3026", arch: "x86_64", label: "Red_Hat_Enterprise_Linux_6_Server_-_Extras_RPMs_x8...", content_view_version_id: 1, relative_path: "our-organization/Library/content/dist/rhel/server/6/6Se...", url: "https://cdn.redhat.com/content/dist/rhel/server/6/...", unprotected: false, content_type: "yum", product_id: 2, environment_id: 1, checksum_type: nil, docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, container_repository_name: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignore_global_proxy: false, ignorable_content: nil, description: nil, source_repo_checksum_type: "sha1">
=> #<Katello::Repository id: 1153, name: "Red Hat Developer Toolset 2 RPMs for Red Hat Enter...", pulp_id: "41b4c3c0-c519-4c9a-a2ab-2e63484fdb85", created_at: "2018-10-16 15:41:13", updated_at: "2018-11-28 13:22:02", major: 6, minor: "6Server", gpg_key_id: nil, cp_label: "rhel-server-dts2-6-rpms", library_instance_id: nil, content_id: "1979", arch: "x86_64", label: "Red_Hat_Developer_Toolset_2_RPMs_for_Red_Hat_Enter...", content_view_version_id: 1, relative_path: "our-organization/Library/content/dist/rhel/server/6/6Se...", url: "https://cdn.redhat.com/content/dist/rhel/server/6/...", unprotected: false, content_type: "yum", product_id: 9, environment_id: 1, checksum_type: nil, docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, container_repository_name: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignore_global_proxy: false, ignorable_content: nil, description: nil, source_repo_checksum_type: "sha1">
=> #<Katello::Repository id: 1152, name: "Red Hat Developer Toolset RPMs for Red Hat Enterpr...", pulp_id: "deee1c9a-d43e-4a75-8897-1e57bd90631f", created_at: "2018-10-16 15:40:52", updated_at: "2018-11-28 13:22:02", major: 6, minor: "6Server", gpg_key_id: nil, cp_label: "rhel-server-dts-6-rpms", library_instance_id: nil, content_id: "1449", arch: "x86_64", label: "Red_Hat_Developer_Toolset_RPMs_for_Red_Hat_Enterpr...", content_view_version_id: 1, relative_path: "our-organization/Library/content/dist/rhel/server/6/6Se...", url: "https://cdn.redhat.com/content/dist/rhel/server/6/...", unprotected: false, content_type: "yum", product_id: 9, environment_id: 1, checksum_type: nil, docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, container_repository_name: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignore_global_proxy: false, ignorable_content: nil, description: nil, source_repo_checksum_type: "sha1">

SHA256
=> #<Katello::Repository id: 1150, name: "Red Hat Developer Tools RPMs for Red Hat Enterpris...", pulp_id: "143b5b70-43bb-41cf-b7d0-49b0ae5914b6", created_at: "2018-10-16 15:35:30", updated_at: "2018-11-17 13:00:44", major: nil, minor: nil, gpg_key_id: nil, cp_label: "rhel-7-server-devtools-rpms", library_instance_id: nil, content_id: "6054", arch: "x86_64", label: "Red_Hat_Developer_Tools_RPMs_for_Red_Hat_Enterpris...", content_view_version_id: 1, relative_path: "our-organization/Library/content/dist/rhel/server/7/7Se...", url: "https://cdn.redhat.com/content/dist/rhel/server/7/...", unprotected: false, content_type: "yum", product_id: 6, environment_id: 1, checksum_type: nil, docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, container_repository_name: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignore_global_proxy: false, ignorable_content: nil, description: nil, source_repo_checksum_type: "sha256">
=> #<Katello::Repository id: 1149, name: "Red Hat Enterprise Linux 6 Server - RH Common RPMs...", pulp_id: "1ea42c3f-b364-4fde-8768-26e11939b55a", created_at: "2018-10-16 15:32:11", updated_at: "2018-11-17 12:58:08", major: 6, minor: "6Server", gpg_key_id: nil, cp_label: "rhel-6-server-rh-common-rpms", library_instance_id: nil, content_id: "2396", arch: "x86_64", label: "Red_Hat_Enterprise_Linux_6_Server_-_RH_Common_RPMs...", content_view_version_id: 1, relative_path: "our-organization/Library/content/dist/rhel/server/6/6Se...", url: "https://cdn.redhat.com/content/dist/rhel/server/6/...", unprotected: false, content_type: "yum", product_id: 2, environment_id: 1, checksum_type: nil, docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, container_repository_name: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignore_global_proxy: false, ignorable_content: nil, description: nil, source_repo_checksum_type: "sha256">
=> #<Katello::Repository id: 1143, name: "Red Hat Enterprise Linux 7 Server - RH Common RPMs...", pulp_id: "6ade5a8b-cb88-40bd-b1b8-1441c92e7a8c", created_at: "2018-10-16 15:30:02", updated_at: "2018-11-19 09:16:40", major: 7, minor: "7Server", gpg_key_id: nil, cp_label: "rhel-7-server-rh-common-rpms", library_instance_id: nil, content_id: "2472", arch: "x86_64", label: "Red_Hat_Enterprise_Linux_7_Server_-_RH_Common_RPMs...", content_view_version_id: 1, relative_path: "our-organization/Library/content/dist/rhel/server/7/7Se...", url: "https://cdn.redhat.com/content/dist/rhel/server/7/...", unprotected: false, content_type: "yum", product_id: 2, environment_id: 1, checksum_type: nil, docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, container_repository_name: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignore_global_proxy: false, ignorable_content: nil, description: nil, source_repo_checksum_type: "sha256">
=> #<Katello::Repository id: 1217, name: "Red Hat Enterprise Linux 6 Server - Extended Life ...", pulp_id: "3fed9433-1140-47a2-a0c4-4782663149b7", created_at: "2018-10-17 11:14:23", updated_at: "2018-11-17 12:57:10", major: nil, minor: nil, gpg_key_id: nil, cp_label: "rhel-6-server-els-optional-rpms", library_instance_id: nil, content_id: "7408", arch: "x86_64", label: "Red_Hat_Enterprise_Linux_6_Server_-_Extended_Life_...", content_view_version_id: 1, relative_path: "our-organization/Library/content/els/rhel/server/6/6Ser...", url: "https://cdn.redhat.com/content/els/rhel/server/6/6...", unprotected: false, content_type: "yum", product_id: 11, environment_id: 1, checksum_type: nil, docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, container_repository_name: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignore_global_proxy: false, ignorable_content: nil, description: nil, source_repo_checksum_type: "sha256">

Our new errata are present now into:

Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server

who have SHA1 on source_repo_checksum_type.
What is not weird is we have 8 errors as checksum type sha256 and the SHA1 is setted on 8 repos.

Previously I tried also (w/out success) to force the repos to SHA256:

   repo = Katello::Repository.find_by_pulp_id("$pulp_id")
   repo.checksum_type = "sha256"
   repo.save!

If you want I can retry this and check.

I wrote the wrong command, the correct one is repo.source_repo_checksum_type = "sha256". The result was the same (just tried). Into /var/log/messages I’ve 8 errors like this:

pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[c28bbb28-df15-4022-a713-9d82710e2d69]
pulp: qpid.messaging:WARNING: [c28bbb28] process forked, child must not use parent qpid.messaging
pulp: kombu.transport.qpid:INFO: [c28bbb28] Connected to qpid with SASL mechanism ANONYMOUS
pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[a9f737da-8b98-4eff-b039-9b079c6f782b]
pulp: celery.app.trace:INFO: [c28bbb28] Task pulp.server.controllers.repository.queue_download_deferred[c28bbb28-df15-4022-a713-9d82710e2d69] succeeded in 0.0689762160182s: None
pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
pulp: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
mongod.27017[22924]: [conn3973] command pulp_database.repo_content_units command: getMore { getMore: 79648581097, collection: "repo_content_units" } originatingCommand: { find: "repo_content_units", filter: { repo_id: "1-RHEL_Views-v18_1-d8946303-7782-4197-a43d-7e60be8a11a6", unit_type_id: "rpm" }, noCursorTimeout: true } planSummary: IXSCAN { repo_id: -1, unit_type_id: -1 } cursorid:79648581097 keysExamined:23105 docsExamined:23105 cursorExhausted:1 numYields:180 nreturned:23105 reslen:5487999 locks:{ Global: { acquireCount: { r: 362 } }, Database: { acquireCount: { r: 181 } }, Collection: { acquireCount: { r: 181 } } } protocol:op_query 144ms
mongod.27017[22924]: [conn3973] command pulp_database.repo_content_units command: getMore { getMore: 77459390620, collection: "repo_content_units" } originatingCommand: { find: "repo_content_units", filter: { repo_id: "d8946303-7782-4197-a43d-7e60be8a11a6", unit_type_id: "rpm" }, noCursorTimeout: true } planSummary: IXSCAN { repo_id: -1, unit_type_id: -1 } cursorid:77459390620 keysExamined:23105 docsExamined:23105 cursorExhausted:1 numYields:180 nreturned:23105 reslen:5049004 locks:{ Global: { acquireCount: { r: 362 } }, Database: { acquireCount: { r: 181 } }, Collection: { acquireCount: { r: 181 } } } protocol:op_query 112ms
mongod.27017[22924]: [conn3973] command pulp_database.repo_content_units command: aggregate { aggregate: "repo_content_units", pipeline: [ { $match: { repo_id: "1-RHEL_Views-v18_1-d8946303-7782-4197-a43d-7e60be8a11a6" } }, { $group: { sum: { $sum: 1 }, _id: "$unit_type_id" } } ], cursor: {} } planSummary: IXSCAN { repo_id: 1, unit_type_id: 1, unit_id: 1 } keysExamined:26704 docsExamined:0 cursorExhausted:1 numYields:209 nreturned:5 reslen:304 locks:{ Global: { acquireCount: { r: 428 } }, Database: { acquireCount: { r: 214 } }, Collection: { acquireCount: { r: 213 } } } protocol:op_query 105ms
pulp: celery.app.trace:INFO: [5d312311] Task pulp.server.managers.repo.unit_association.associate_from_repo[5d312311-155c-4e63-8f1a-b335e37291bd] succeeded in 847.000600081s: {'units_successful': [{'signing_key': None, 'unit_key': {...}, 'type_id': 'rpm'}, {'signing_key': None, 'unit_key': {...}, 'type_id': 'rpm'}, {'signing_key': None, 'unit_key': {...}, 'type_id': 'rpm'}, {'signing_key': None, 'unit_key': {...}, 'type_id': 'rpm'}, {'signing_key': None, 'unit_key': {...}, 'type_id': 'rpm'}, {'unit_key': {...}, 'type_id': 'erratum'}, {'signing_key': None, 'unit_key': {...}, 'type_id': 'rpm'}, {'signing_key': None, 'unit_key': {...}, 'type_id': 'rpm'}, {'unit_key': {...}, 'type_id': 'erratum'}, {'signing_key': None, 'unit_key': {...}, 'type_id': 'rpm'}, {'signing_key': None, 'unit_key': {...}, 'type_id': 'rpm'}, {'signing_key': None, 'unit_key': {...}, 'type_id': 'rpm'}, {'signing_key': None, 'unit_key': {...}, 'type_id': 'rpm'}], 'units_failed_signature_filter': []}
pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
pulp: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
pulp: py.warnings:WARNING: [441fc1f0] (4171-49664) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
pulp: py.warnings:WARNING: [441fc1f0] (4171-49664)   "MongoClient opened before fork. Create MongoClient "
pulp: py.warnings:WARNING: [441fc1f0] (4171-49664)
pulp: celery.app.trace:INFO: [441fc1f0] Task pulp.server.async.tasks._release_resource[441fc1f0-5b01-48e6-91d2-3f1921d2382d] succeeded in 0.0193273266777s: None
pulp: celery.app.trace:INFO: [a9f737da] Task pulp.server.controllers.repository.download_deferred[a9f737da-8b98-4eff-b039-9b079c6f782b] succeeded in 0.0482160113752s: None
pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
pulp: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
mongod.27017[22924]: [conn3229] command pulp_database.repo_content_units command: getMore { getMore: 81311342213, collection: "repo_content_units" } originatingCommand: { find: "repo_content_units", filter: { repo_id: "1-RHEL_Views-v18_1-d8946303-7782-4197-a43d-7e60be8a11a6", unit_type_id: "rpm" }, projection: { unit_id: 1 }, noCursorTimeout: true } planSummary: IXSCAN { repo_id: -1, unit_type_id: -1 } cursorid:81311342213 keysExamined:23105 docsExamined:23105 cursorExhausted:1 numYields:181 nreturned:23105 reslen:1814304 locks:{ Global: { acquireCount: { r: 364 } }, Database: { acquireCount: { r: 182 } }, Collection: { acquireCount: { r: 182 } } } protocol:op_query 164ms
pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[651a5785-4102-4870-ad0d-a3d890b52ee6]
pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[85394fd7-69aa-43db-9f61-a472776d02d1]
pulp: celery.app.trace:INFO: [651a5785] Task pulp.server.async.tasks._queue_reserved_task[651a5785-4102-4870-ad0d-a3d890b52ee6] succeeded in 0.0481885029003s: None
pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[616da798-b48f-44fb-9317-1cba9440fb6b]
pulp: py.warnings:WARNING: [85394fd7] (4188-49664) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
pulp: py.warnings:WARNING: [85394fd7] (4188-49664)   "MongoClient opened before fork. Create MongoClient "
pulp: py.warnings:WARNING: [85394fd7] (4188-49664)
mongod.27017[22924]: [conn4029] command pulp_database.repo_content_units command: getMore { getMore: 77345549584, collection: "repo_content_units" } originatingCommand: { find: "repo_content_units", filter: { repo_id: "1-RHEL_Views-v18_1-d8946303-7782-4197-a43d-7e60be8a11a6", unit_type_id: "rpm" }, projection: { unit_id: 1 }, noCursorTimeout: true } planSummary: IXSCAN { repo_id: -1, unit_type_id: -1 } cursorid:77345549584 keysExamined:23105 docsExamined:23105 cursorExhausted:1 numYields:181 nreturned:23105 reslen:1814304 locks:{ Global: { acquireCount: { r: 364 } }, Database: { acquireCount: { r: 182 } }, Collection: { acquireCount: { r: 182 } } } protocol:op_query 159ms
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664) Exception caught from plugin during publish for repo [1-RHEL_Views-v18_1-d8946303-7782-4197-a43d-7e60be8a11a6]
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664) Traceback (most recent call last):
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1258, in _do_publish
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)     publish_report = publish_repo(transfer_repo, conduit, call_config)
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 737, in wrap_f
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)     return f(*args, **kwargs)
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 174, in publish_repo
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)     return self._publisher.process_lifecycle()
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 572, in process_lifecycle
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)     super(PluginStep, self).process_lifecycle()
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 163, in process_lifecycle
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)     step.process()
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 239, in process
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)     self._process_block(item=item)
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 301, in _process_block
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)     self.process_main(item=item)
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 496, in process_main
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)     context.add_unit_metadata(unit)
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/filelists.py", line 42, in add_unit_metadata
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)     self.metadata_file_handle.write(unit.render_filelists(self.checksum_type))
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 863, in render_filelists
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)     context = Context({'pkgid': self.get_or_calculate_and_save_checksum(checksumtype)})
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 258, in get_or_calculate_and_save_checksum
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664)     checksumtype=checksumtype)
pulp: pulp.server.controllers.repository:ERROR: [85394fd7] (4188-49664) PulpCodedException: Checksum type "sha256" is not available for all units in the repository. Make sure those units have been downloaded.
pulp: pulp.server.async.tasks:INFO: [85394fd7] Task failed : [85394fd7-69aa-43db-9f61-a472776d02d1] : Checksum type "sha256" is not available for all units in the repository. Make sure those units have been downloaded.
pulp: celery.app.trace:INFO: [85394fd7] Task pulp.server.managers.repo.publish.publish[85394fd7-69aa-43db-9f61-a472776d02d1] raised expected: PulpCodedException()
pulp: celery.app.trace:INFO: [616da798] Task pulp.server.async.tasks._release_resource[616da798-b48f-44fb-9317-1cba9440fb6b] succeeded in 0.0048699202016s: None
pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
pulp: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}

and the all repository are sha256:


=> #<Katello::Repository id: 1151, name: "Red Hat Enterprise Linux 6 Server - Extras RPMs x8...", pulp_id: "ea2ab144-f6e8-4f43-8537-a31c7faf724d", created_at: "2018-10-16 15:40:01", updated_at: "2018-11-30 15:23:27", major: nil, minor: nil, gpg_key_id: nil, cp_label: "rhel-6-server-extras-rpms", library_instance_id: nil, content_id: "3026", arch: "x86_64", label: "Red_Hat_Enterprise_Linux_6_Server_-_Extras_RPMs_x8...", content_view_version_id: 1, relative_path: "ATS-vCentre/Library/content/dist/rhel/server/6/6Se...", url: "https://cdn.redhat.com/content/dist/rhel/server/6/...", unprotected: false, content_type: "yum", product_id: 2, environment_id: 1, checksum_type: "sha256", docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, container_repository_name: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignore_global_proxy: false, ignorable_content: nil, description: nil, source_repo_checksum_type: "sha256">

I solved the issue (maybe). I deleted all repos in my CV, I readd they and resync all. Now the incremental update are up & running…
I think something gone wrong or misconfiogurations on repos were present.

1 Like