Foreman 2.4 / Katello 4.0: Issue sync'ing repos for CentOS Stream

Support
1

Problem:
Created a product for CentOS Stream and added 4 repos: BaseOS, AppStream, Extras and PowerTools.
Extras is ok.
BaseOS, AppStream and PowerTools all end up with Warnings:
'‘utf-8’ codec can’t decode byte 0xfd in position 0: invalid start byte"

I’m using:
Bytemark - Mirror
Bytemark - Mirror
https://mirror.bytemark.co.uk/centos/8-stream/Extras/x86_64/os/
Bytemark - Mirror

Expected outcome:
BaseOS, AppStream and PowerTools repo sync without warning.

Foreman and Proxy versions:
2.4 + 4.0

Foreman and Proxy plugin versions:
candlepin-3.2.11-1.el7.noarch
candlepin-selinux-3.2.11-1.el7.noarch
foreman-2.4.0-1.el7.noarch
foreman-cli-2.4.0-1.el7.noarch
foreman-debug-2.4.0-1.el7.noarch
foreman-dynflow-sidekiq-2.4.0-1.el7.noarch
foreman-ec2-2.4.0-1.el7.noarch
foreman-gce-2.4.0-1.el7.noarch
foreman-installer-2.4.0-1.el7.noarch
foreman-installer-katello-2.4.0-1.el7.noarch
foreman-libvirt-2.4.0-1.el7.noarch
foreman-openstack-2.4.0-1.el7.noarch
foreman-ovirt-2.4.0-1.el7.noarch
foreman-postgresql-2.4.0-1.el7.noarch
foreman-proxy-2.4.0-1.el7.noarch
foreman-release-2.4.0-1.el7.noarch
foreman-selinux-2.4.0-1.el7.noarch
foreman-service-2.4.0-1.el7.noarch
foreman-vmware-2.4.0-1.el7.noarch
iomforemankatello01.devops.csr.pstars-apache-1.0-2.noarch
iomforemankatello01.devops.csr.pstars-foreman-client-1.0-1.noarch
iomforemankatello01.devops.csr.pstars-foreman-proxy-1.0-2.noarch
iomforemankatello01.devops.csr.pstars-foreman-proxy-client-1.0-1.noarch
iomforemankatello01.devops.csr.pstars-puppet-client-1.0-1.noarch
iomforemankatello01.devops.csr.pstars-qpid-broker-1.0-1.noarch
iomforemankatello01.devops.csr.pstars-qpid-router-server-1.0-1.noarch
katello-4.0.0-1.el7.noarch
katello-certs-tools-2.7.3-1.el7.noarch
katello-client-bootstrap-1.7.6-1.el7.noarch
katello-common-4.0.0-1.el7.noarch
katello-debug-4.0.0-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-repos-4.0.0-1.el7.noarch
katello-selinux-4.0.0-1.el7.noarch
katello-server-ca-1.0-2.noarch
pulp-client-1.0-1.noarch
pulpcore-selinux-1.2.4-1.el7.x86_64
python2-qpid-1.37.0-5.el7.noarch
python2-qpid-proton-0.33.0-1.el7.x86_64
python2-qpid-qmf-1.39.0-1.el7.x86_64
python3-pulp-certguard-1.1.0-1.el7.noarch
python3-pulp-container-2.2.1-1.el7.noarch
python3-pulp-deb-2.9.1-1.el7.noarch
python3-pulp-file-1.5.0-1.el7.noarch
python3-pulp-rpm-3.10.0-1.el7.noarch
python3-pulpcore-3.9.1-2.el7.noarch
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.15.0-1.el7.x86_64
qpid-proton-c-0.33.0-1.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
qpid_router_katello_agent-qpid-router-client-1.0-1.noarch
rubygem-foreman_maintain-0.7.9-1.el7.noarch
tfm-rubygem-actioncable-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailbox-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailer-6.0.3.4-1.el7.noarch
tfm-rubygem-actionpack-6.0.3.4-1.el7.noarch
tfm-rubygem-actiontext-6.0.3.4-1.el7.noarch
tfm-rubygem-actionview-6.0.3.4-1.el7.noarch
tfm-rubygem-activejob-6.0.3.4-1.el7.noarch
tfm-rubygem-activemodel-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-import-1.0.0-2.el7.noarch
tfm-rubygem-activerecord-session_store-1.1.1-4.el7.noarch
tfm-rubygem-activestorage-6.0.3.4-1.el7.noarch
tfm-rubygem-activesupport-6.0.3.4-1.el7.noarch
tfm-rubygem-addressable-2.6.0-2.el7.noarch
tfm-rubygem-algebrick-0.7.3-7.el7.noarch
tfm-rubygem-amazing_print-1.1.0-1.el7.noarch
tfm-rubygem-ancestry-3.0.7-1.el7.noarch
tfm-rubygem-anemone-0.7.2-17.el7.noarch
tfm-rubygem-angular-rails-templates-1.1.0-1.el7.noarch
tfm-rubygem-ansi-1.5.0-2.el7.noarch
tfm-rubygem-apipie-bindings-0.4.0-1.el7.noarch
tfm-rubygem-apipie-dsl-2.3.0-1.el7.noarch
tfm-rubygem-apipie-params-0.0.5-4.el7.noarch
tfm-rubygem-apipie-rails-0.5.17-3.el7.noarch
tfm-rubygem-audited-4.9.0-3.el7.noarch
tfm-rubygem-bcrypt-3.1.12-3.el7.x86_64
tfm-rubygem-builder-3.2.4-1.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-5.el7.noarch
tfm-rubygem-clamp-1.1.2-6.el7.noarch
tfm-rubygem-concurrent-ruby-1.1.6-2.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.6.0-2.fm2_1.el7.noarch
tfm-rubygem-connection_pool-2.2.2-2.el7.noarch
tfm-rubygem-crass-1.0.6-1.el7.noarch
tfm-rubygem-css_parser-1.4.7-4.el7.noarch
tfm-rubygem-daemons-1.2.3-6.el7.noarch
tfm-rubygem-deacon-1.0.0-4.el7.noarch
tfm-rubygem-declarative-0.0.10-2.el7.noarch
tfm-rubygem-declarative-option-0.1.0-2.el7.noarch
tfm-rubygem-deep_cloneable-3.0.0-3.el7.noarch
tfm-rubygem-deface-1.5.3-2.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-4.el7.noarch
tfm-rubygem-dynflow-1.4.7-1.fm2_3.el7.noarch
tfm-rubygem-erubi-1.9.0-1.el7.noarch
tfm-rubygem-excon-0.76.0-1.el7.noarch
tfm-rubygem-facter-4.0.44-1.el7.x86_64
tfm-rubygem-faraday-0.17.3-1.el7.noarch
tfm-rubygem-fast_gettext-1.4.1-4.el7.noarch
tfm-rubygem-ffi-1.12.2-1.el7.x86_64
tfm-rubygem-fog-aws-3.6.5-1.el7.noarch
tfm-rubygem-fog-core-2.1.0-3.el7.noarch
tfm-rubygem-fog-google-1.11.0-1.el7.noarch
tfm-rubygem-fog-json-1.2.0-3.el7.noarch
tfm-rubygem-fog-libvirt-0.8.0-1.el7.noarch
tfm-rubygem-fog-openstack-1.0.8-3.el7.noarch
tfm-rubygem-fog-ovirt-2.0.1-1.el7.noarch
tfm-rubygem-fog-vsphere-3.5.0-1.el7.noarch
tfm-rubygem-fog-xml-0.1.2-8.el7.noarch
tfm-rubygem-foreman-tasks-4.0.1-1.fm2_4.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-foreman_remote_execution-4.3.0-1.fm2_4.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.4.0-1.el7.noarch
tfm-rubygem-formatador-0.2.1-12.el7.noarch
tfm-rubygem-friendly_id-5.3.0-1.el7.noarch
tfm-rubygem-fx-0.5.0-1.el7.noarch
tfm-rubygem-get_process_mem-0.2.1-4.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.8.0-2.el7.noarch
tfm-rubygem-gitlab-sidekiq-fetcher-0.6.0-1.el7.noarch
tfm-rubygem-globalid-0.4.2-1.el7.noarch
tfm-rubygem-google-api-client-0.33.2-1.el7.noarch
tfm-rubygem-google-cloud-env-1.3.3-1.el7.noarch
tfm-rubygem-googleauth-0.13.1-1.el7.noarch
tfm-rubygem-graphql-1.8.14-2.el7.noarch
tfm-rubygem-graphql-batch-0.3.10-2.el7.noarch
tfm-rubygem-gssapi-1.2.0-7.el7.noarch
tfm-rubygem-hammer_cli-2.4.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.4.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.7-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.2.1-1.fm2_4.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.15-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_katello-1.0-1.el7.noarch
tfm-rubygem-hashie-3.6.0-2.el7.noarch
tfm-rubygem-highline-2.0.3-1.el7.noarch
tfm-rubygem-hocon-1.3.1-1.el7.noarch
tfm-rubygem-http-cookie-1.0.2-4.el7.noarch
tfm-rubygem-httpclient-2.8.3-3.el7.noarch
tfm-rubygem-i18n-1.8.2-1.el7.noarch
tfm-rubygem-ipaddress-0.8.0-12.el7.noarch
tfm-rubygem-jwt-2.2.1-2.el7.noarch
tfm-rubygem-kafo-6.2.1-1.el7.noarch
tfm-rubygem-kafo_parsers-1.1.0-3.el7.noarch
tfm-rubygem-kafo_wizards-0.0.2-1.el7.noarch
tfm-rubygem-katello-4.0.0-1.el7.noarch
tfm-rubygem-ldap_fluff-0.5.0-1.el7.noarch
tfm-rubygem-little-plugger-1.1.4-2.el7.noarch
tfm-rubygem-locale-2.0.9-14.el7.noarch
tfm-rubygem-logging-2.3.0-1.el7.noarch
tfm-rubygem-loofah-2.4.0-1.el7.noarch
tfm-rubygem-mail-2.7.1-1.el7.noarch
tfm-rubygem-marcel-0.3.3-1.el7.noarch
tfm-rubygem-memoist-0.16.0-2.el7.noarch
tfm-rubygem-method_source-0.9.2-2.el7.noarch
tfm-rubygem-mime-types-3.2.2-4.el7.noarch
tfm-rubygem-mime-types-data-3.2018.0812-4.el7.noarch
tfm-rubygem-mimemagic-0.3.5-1.el7.noarch
tfm-rubygem-mini_mime-1.0.2-1.el7.noarch
tfm-rubygem-mini_portile2-2.4.0-1.el7.noarch
tfm-rubygem-multi_json-1.14.1-2.el7.noarch
tfm-rubygem-multipart-post-2.0.0-2.el7.noarch
tfm-rubygem-mustermann-1.0.2-4.el7.noarch
tfm-rubygem-net-ldap-0.17.0-1.el7.noarch
tfm-rubygem-net-ping-2.0.1-4.el7.noarch
tfm-rubygem-net-scp-1.2.1-4.el7.noarch
tfm-rubygem-net-ssh-4.2.0-2.el7.noarch
tfm-rubygem-netrc-0.11.0-5.el7.noarch
tfm-rubygem-nio4r-2.5.4-1.el7.x86_64
tfm-rubygem-nokogiri-1.10.9-1.el7.x86_64
tfm-rubygem-oauth-0.5.4-4.el7.noarch
tfm-rubygem-optimist-3.0.0-2.el7.noarch
tfm-rubygem-os-1.0.0-2.el7.noarch
tfm-rubygem-ovirt-engine-sdk-4.4.0-1.el7.x86_64
tfm-rubygem-parse-cron-0.1.4-4.fm2_1.el7.noarch
tfm-rubygem-pg-1.1.4-3.el7.x86_64
tfm-rubygem-polyglot-0.3.5-2.el7.noarch
tfm-rubygem-powerbar-2.0.1-2.el7.noarch
tfm-rubygem-promise.rb-0.7.4-2.el7.noarch
tfm-rubygem-public_suffix-3.0.3-2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.7.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.6.0-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.3-1.el7.noarch
tfm-rubygem-pulp_container_client-2.2.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.8.0-1.el7.noarch
tfm-rubygem-pulp_file_client-1.5.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.10.0-1.el7.noarch
tfm-rubygem-pulpcore_client-3.9.0-1.el7.noarch
tfm-rubygem-puma-5.1.1-1.el7.x86_64
tfm-rubygem-qpid_proton-0.33.0-1.el7.x86_64
tfm-rubygem-rabl-0.14.3-1.el7.noarch
tfm-rubygem-rack-2.2.3-1.el7.noarch
tfm-rubygem-rack-cors-1.0.2-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-9.el7.noarch
tfm-rubygem-rack-protection-2.1.0-1.el7.noarch
tfm-rubygem-rack-test-1.1.0-4.el7.noarch
tfm-rubygem-rails-6.0.3.4-1.el7.noarch
tfm-rubygem-rails-dom-testing-2.0.3-6.el7.noarch
tfm-rubygem-rails-html-sanitizer-1.3.0-1.el7.noarch
tfm-rubygem-rails-i18n-6.0.0-2.el7.noarch
tfm-rubygem-railties-6.0.3.4-1.el7.noarch
tfm-rubygem-rainbow-2.2.1-3.el7.noarch
tfm-rubygem-rb-inotify-0.9.7-5.el7.noarch
tfm-rubygem-rbvmomi-2.2.0-3.el7.noarch
tfm-rubygem-record_tag_helper-1.0.1-3.el7.noarch
tfm-rubygem-redfish_client-0.5.2-1.el7.noarch
tfm-rubygem-redis-4.1.2-2.el7.noarch
tfm-rubygem-representable-3.0.4-2.el7.noarch
tfm-rubygem-responders-3.0.0-3.el7.noarch
tfm-rubygem-rest-client-2.0.2-3.el7.noarch
tfm-rubygem-retriable-3.1.2-2.el7.noarch
tfm-rubygem-rkerberos-0.1.5-19.el7.x86_64
tfm-rubygem-roadie-3.4.0-3.el7.noarch
tfm-rubygem-roadie-rails-2.1.1-2.el7.noarch
tfm-rubygem-robotex-1.0.0-21.el7.noarch
tfm-rubygem-rsec-0.4.3-4.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.1-1.el7.x86_64
tfm-rubygem-ruby2ruby-2.4.2-3.el7.noarch
tfm-rubygem-ruby_parser-3.10.1-3.el7.noarch
tfm-rubygem-rubyipmi-0.10.0-6.el7.noarch
tfm-rubygem-runcible-2.13.1-1.el7.noarch
tfm-rubygem-safemode-1.3.6-1.el7.noarch
tfm-rubygem-scoped_search-4.1.9-1.el7.noarch
tfm-rubygem-sd_notify-0.1.0-1.el7.noarch
tfm-rubygem-secure_headers-6.3.0-2.el7.noarch
tfm-rubygem-sequel-5.7.1-3.el7.noarch
tfm-rubygem-server_sent_events-0.1.2-1.el7.noarch
tfm-rubygem-sexp_processor-4.10.0-6.el7.noarch
tfm-rubygem-sidekiq-5.2.7-3.el7.noarch
tfm-rubygem-signet-0.14.0-1.el7.noarch
tfm-rubygem-sinatra-2.1.0-1.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
tfm-rubygem-sprockets-4.0.2-1.el7.noarch
tfm-rubygem-sprockets-rails-3.2.1-6.el7.noarch
tfm-rubygem-sshkey-1.9.0-4.el7.noarch
tfm-rubygem-statsd-instrument-2.1.4-3.el7.noarch
tfm-rubygem-stomp-1.4.9-1.el7.noarch
tfm-rubygem-thor-1.0.1-2.el7.noarch
tfm-rubygem-thread_safe-0.3.6-5.el7.noarch
tfm-rubygem-tilt-2.0.8-4.el7.noarch
tfm-rubygem-tzinfo-1.2.6-1.el7.noarch
tfm-rubygem-uber-0.1.0-2.el7.noarch
tfm-rubygem-unf-0.1.3-8.el7.noarch
tfm-rubygem-unf_ext-0.0.7.2-3.el7.x86_64
tfm-rubygem-unicode-0.4.4.4-3.el7.x86_64
tfm-rubygem-unicode-display_width-1.0.5-4.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-7.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-5.el7.noarch
tfm-rubygem-websocket-driver-0.7.1-1.el7.x86_64
tfm-rubygem-websocket-extensions-0.1.5-1.el7.noarch
tfm-rubygem-will_paginate-3.1.7-3.el7.noarch
tfm-rubygem-xmlrpc-0.3.0-2.el7.noarch
tfm-rubygem-zeitwerk-2.2.2-1.el7.noarch
tfm-runtime-6.1-4.el7.x86_64

Distribution and version:
CentOS 7.9.2009 up-to-date

Other relevant data:
2021-05-05T16:06:09 [E|bac|dc3608e0] ‘utf-8’ codec can’t decode byte 0xfd in position 0: invalid start byte (Katello::Errors::Pulp3Error)
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/pulp3/abstract_async_task.rb:102:in block in check_for_errors' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/pulp3/abstract_async_task.rb:100:in each’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/pulp3/abstract_async_task.rb:100:in check_for_errors' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/pulp3/abstract_async_task.rb:133:in poll_external_task’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in poll_external_task_with_rescue' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in run' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in block (3 levels) in execute_run' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/middleware/remote_action.rb:16:in block in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/models/katello/concerns/user_extensions.rb:21:in cp_config’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/middleware/remote_action.rb:16:in run' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in wrap’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in with_progress_calculation' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in restore_current_request_id' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in block in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:54:in restore_curent_user' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in run’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in execute’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' dc3608e0 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
dc3608e0 | [ sidekiq ]
dc3608e0 | [ concurrent-ruby ]

1 Like
2

There’s a Pulp 3 issue for this: Issue #8700: CentOS 8 stream repositories fails to synchronize - 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte - Pulp

3

I ran into this today. I think it’s because the modules.yaml is xz’d instead of gzip’d.

1 Like
4

This patch is tiding me over, because I’m not in production.

--- /usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py.old    2021-03-25 13:24:01.000000000 -0400
+++ /usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py.new    2021-05-05 16:13:50.016460535 -0400
@@ -1,5 +1,6 @@
 import asyncio
 import gzip
+import lzma
 import logging
 import os
 import re
@@ -742,7 +743,12 @@ class ModulesMetadataParser:
         """Parse module.yaml, if exists, to create relations between packages."""
         if self.modulemd_result:
             modulemd_index = mmdlib.ModuleIndex.new()
-            open_func = gzip.open if self.modulemd_result.url.endswith(".gz") else open
+            if self.modulemd_result.url.endswith(".gz"):
+                open_func = gzip.open
+            elif self.modulemd_result.url.endswith(".xz"):
+                open_func = lzma.open
+            else:
+                open_func = open
             with open_func(self.modulemd_result.path, "r") as moduleyaml:
                 content = moduleyaml.read()
                 module_content = content if isinstance(content, str) else content.decode()
2 Likes
5

You should submit a patch upstream:

2 Likes
6

Thanks @quartsize
I applied that patch to /usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py and it did the trick!
:+1:

7

PR submitted.

3 Likes
8

This looks like it is also impacting CentOS 8 as well.

CentOS-8 AppStream
CentOS-8 BaseOS
CentOS-8 PowerTools

9

I implemented the fix described above to a 3.18 install, but I was able to get anything to sync afterwards. I have changed back to the original python script version since we don’t have many CentOS 8 systems.

10

I upgraded to Foreman 2.4 and Katello 3.18 earlier this week and I am getting this error with my CentOS 8 repositories. I looked at my synchronizing.py file and it appears I have the patch described above, but I still can’t sync CentOS 8. Luckily my other repos seem to be fine. Any ideas?

11

Just noticed a typo in my last comment.

It still did NOT work for me with those CentOS-8 repos after implementing the updates to the synchronizing.py script.

12

It still works fine for me for CentOS 8 (x4) and CentOS Stream (x4) repos.

def parse(self):

    """Parse module.yaml, if exists, to create relations between packages."""
    if self.modulemd_result:
        modulemd_index = mmdlib.ModuleIndex.new()
        # open_func = gzip.open if self.modulemd_result.url.endswith(".gz") else open

        if self.modulemd_result.url.endswith(".gz"):
            open_func = gzip.open
        elif self.modulemd_result.url.endswith(".xz"):
            open_func = lzma.open
        else:
            open_func = open

        with open_func(self.modulemd_result.path, "r") as moduleyaml:
            content = moduleyaml.read()
            module_content = content if isinstance(content, str) else content.decode()
            modulemd_index.update_from_string(module_content, True)

        self._parse_modulemd_list(modulemd_index)
        self._parse_modulemd_default_names(modulemd_index)
13

have I missed something?

diff /usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py /usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py.org

3d2
< import lzma
746,751c745
<             if self.modulemd_result.url.endswith(".gz"):
<                 open_func = gzip.open
<             elif self.modulemd_result.url.endswith(".xz"):
<                 open_func = lzma.open
<             else:
<                 open_func = open
---
>             open_func = gzip.open if self.modulemd_result.url.endswith(".gz") else open

its still not working here (still the utf-8 error)

14

here the full error:

Jun 10 15:33:51 amon pulpcore-worker-3[918]: pulp [16aeaa33275d4c66b8475a924da5ce87]: rq.worker:ERROR: Traceback (most recent call last):
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    rv = job.perform()
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    self._result = self._execute()
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    return self.func(*self.args, **self.kwargs)
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 255, in synchronize
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    dv = RpmDeclarativeVersion(first_stage=stage, repository=sub_repo)
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    loop.run_until_complete(pipeline)
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    return future.result()
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    await asyncio.gather(*futures)
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    await self.run()
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 439, in run
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    await self.parse_repository_metadata()
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 495, in parse_modules_metadata
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    modules_metadata_parser = ModulesMetadataParser(self.data, modulemd_result)
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 747, in parse
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib64/python3.6/codecs.py", line 321, in decode
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    (result, consumed) = self._buffer_decode(data, self.errors, final)
Jun 10 15:33:51 amon pulpcore-worker-3[918]: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
Jun 10 15:33:51 amon pulpcore-worker-3[918]: Traceback (most recent call last):
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    rv = job.perform()
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    self._result = self._execute()
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    return self.func(*self.args, **self.kwargs)
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 255, in synchronize
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    dv = RpmDeclarativeVersion(first_stage=stage, repository=sub_repo)
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    loop.run_until_complete(pipeline)
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    return future.result()
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    await asyncio.gather(*futures)
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    await self.run()
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 439, in run
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    await self.parse_repository_metadata()
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 495, in parse_modules_metadata
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    modules_metadata_parser = ModulesMetadataParser(self.data, modulemd_result)
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 747, in parse
Jun 10 15:33:51 amon pulpcore-worker-3[918]:  File "/usr/lib64/python3.6/codecs.py", line 321, in decode
Jun 10 15:33:51 amon pulpcore-worker-3[918]:    (result, consumed) = self._buffer_decode(data, self.errors, final)
Jun 10 15:33:51 amon pulpcore-worker-3[918]: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
15

oh… I’ve restarted the whole foreman host and now it seems to work… sorry.

1 Like
16

Hi Christoph
Yep I had to restart too after applying the patch… Sorry for forgetting to mention that detail…

17

Is the recommended fix for this issue in Katello 3.18 to upgrade to 4.0?

18

I’m using Foreman 2.4 with Katello 4.0 and CentOS Repositories are unable to sync as described in Issue #8700: CentOS 8 stream repositories fails to synchronize - 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte - RPM Support - Pulp Although that issue has been closed it doesn’t appear to have been released by Katello yet. I’m not sure when the next raft of bugfixes will be released, there is the workaround described earlier but for production systems I guess we are going to have to wait until there is a new release of Katello which will hopefully include the pulp fix.

19

I am running into the same issue. There is mention of a patch fix here but I can’t seem to find any detail on how to apply this patch… please clarify

'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte

Thanks

20

I am getting this error when trying to sync CentOS 8 Stream repository. I am not sure if this issue related with my config or source repo side. Is there a solution for this issue ?

'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte