Katello 4.3 - Repo Sync Error - Errno1 Operation Not Permitted

Support
1

Problem:
Not able to sync repo, getting this error:

Katello::Errors::Pulp3Error: [Errno 1] Operation not permitted: ‘/var/lib/pulp/media/artifact/b7/9079ad71cc3c5ceb3561fff348a1b67ee37f71f4cddfec09480d4589c191d6’

Expected outcome:
Successful sync with repo

Foreman and Proxy versions:
Foreman 3.10, Katello 4.3
Foreman and Proxy plugin versions:

Installed Packages

candlepin-4.1.8-1.el7.noarch
candlepin-selinux-4.1.8-1.el7.noarch
foreman-3.1.0-1.el7.noarch
foreman-cli-3.1.0-1.el7.noarch
foreman-debug-3.1.0-1.el7.noarch
foreman-dynflow-sidekiq-3.1.0-1.el7.noarch
foreman-installer-3.1.0-1.el7.noarch
foreman-installer-katello-3.1.0-1.el7.noarch
foreman-postgresql-3.1.0-1.el7.noarch
foreman-proxy-3.1.0-1.el7.noarch
foreman-release-3.1.0-1.el7.noarch
foreman-release-scl-7-3.el7.noarch
foreman-selinux-3.1.0-1.el7.noarch
foreman-service-3.1.0-1.el7.noarch
foreman-vmware-3.1.0-1.el7.noarch
katello-4.3.0-0.6.rc4.el7.noarch
katello-ca-consumer-MYHOSTNAME-1.0-1.noarch
katello-certs-tools-2.8.2-1.el7.noarch
katello-client-bootstrap-1.7.7-1.el7.noarch
katello-common-4.3.0-0.6.rc4.el7.noarch
katello-debug-4.3.0-0.6.rc4.el7.noarch
katello-default-ca-1.0-1.noarch
katello-host-tools-3.5.1-2.el7.noarch
katello-host-tools-fact-plugin-3.5.1-2.el7.noarch
katello-repos-4.3.0-0.2.rc4.el7.noarch
katello-selinux-4.0.2-1.el7.noarch
katello-server-ca-1.0-10.noarch
katello-service-3.14.1-1.el7.noarch
pulp-client-1.0-10.noarch
pulpcore-selinux-1.2.6-2.el7.x86_64
python-gofer-qpid-2.12.5-3.el7.noarch
python2-qpid-1.37.0-5.el7.noarch
python2-qpid-qmf-1.39.0-1.el7.x86_64
python36-qpid-proton-0.35.0-1.el7.x86_64
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-client-devel-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.17.0-1.el7.x86_64
qpid-proton-c-0.35.0-1.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
qpid_router_katello_agent-qpid-router-client-1.0-1.noarch
rubygem-foreman_maintain-0.9.2-1.el7.noarch
tfm-pulpcore-python3-aiodns-3.0.0-2.el7.noarch
tfm-pulpcore-python3-aiofiles-0.7.0-2.el7.noarch
tfm-pulpcore-python3-aiohttp-3.7.4-4.el7.x86_64
tfm-pulpcore-python3-aiohttp-socks-0.6.0-2.el7.noarch
tfm-pulpcore-python3-aiohttp-xmlrpc-1.3.2-1.el7.noarch
tfm-pulpcore-python3-aioredis-2.0.0-2.el7.noarch
tfm-pulpcore-python3-ansible-builder-1.0.1-2.el7.noarch
tfm-pulpcore-python3-asgiref-3.4.1-1.el7.noarch
tfm-pulpcore-python3-async-lru-1.0.2-2.el7.noarch
tfm-pulpcore-python3-async-timeout-3.0.1-3.el7.noarch
tfm-pulpcore-python3-asyncio-throttle-1.0.2-2.el7.noarch
tfm-pulpcore-python3-attrs-21.2.0-2.el7.noarch
tfm-pulpcore-python3-backoff-1.11.1-1.el7.noarch
tfm-pulpcore-python3-bandersnatch-4.4.0-4.el7.noarch
tfm-pulpcore-python3-bindep-2.10.1-1.el7.noarch
tfm-pulpcore-python3-bleach-3.3.1-1.el7.noarch
tfm-pulpcore-python3-bleach-allowlist-1.0.3-2.el7.noarch
tfm-pulpcore-python3-certifi-2020.6.20-2.el7.noarch
tfm-pulpcore-python3-cffi-1.15.0-1.el7.x86_64
tfm-pulpcore-python3-chardet-3.0.4-3.el7.noarch
tfm-pulpcore-python3-charset-normalizer-2.0.7-1.el7.noarch
tfm-pulpcore-python3-click-8.0.3-1.el7.noarch
tfm-pulpcore-python3-colorama-0.4.4-2.el7.noarch
tfm-pulpcore-python3-createrepo_c-0.17.6-4.el7.x86_64
tfm-pulpcore-python3-cryptography-3.1.1-1.el7.x86_64
tfm-pulpcore-python3-dateutil-2.8.2-1.el7.noarch
tfm-pulpcore-python3-debian-0.1.42-1.el7.noarch
tfm-pulpcore-python3-defusedxml-0.7.1-2.el7.noarch
tfm-pulpcore-python3-diff-match-patch-20200713-2.el7.noarch
tfm-pulpcore-python3-distro-1.6.0-1.el7.noarch
tfm-pulpcore-python3-django-3.2.9-1.el7.noarch
tfm-pulpcore-python3-django-currentuser-0.5.3-3.el7.noarch
tfm-pulpcore-python3-django-filter-21.1-1.el7.noarch
tfm-pulpcore-python3-django-guardian-2.4.0-3.el7.noarch
tfm-pulpcore-python3-django-guid-3.2.0-2.el7.noarch
tfm-pulpcore-python3-django-import-export-2.6.1-1.el7.noarch
tfm-pulpcore-python3-django-lifecycle-0.9.3-1.el7.noarch
tfm-pulpcore-python3-django-prometheus-2.1.0-2.el7.noarch
tfm-pulpcore-python3-django-readonly-field-1.0.5-3.el7.noarch
tfm-pulpcore-python3-djangorestframework-3.12.4-4.el7.noarch
tfm-pulpcore-python3-djangorestframework-queryfields-1.0.0-4.el7.noarch
tfm-pulpcore-python3-drf-access-policy-1.1.0-1.el7.noarch
tfm-pulpcore-python3-drf-nested-routers-0.93.3-3.el7.noarch
tfm-pulpcore-python3-drf-spectacular-0.20.1-1.el7.noarch
tfm-pulpcore-python3-dynaconf-3.1.7-2.el7.noarch
tfm-pulpcore-python3-ecdsa-0.13.3-3.el7.noarch
tfm-pulpcore-python3-et-xmlfile-1.1.0-1.el7.noarch
tfm-pulpcore-python3-filelock-3.0.12-2.el7.noarch
tfm-pulpcore-python3-flake8-3.9.2-3.el7.noarch
tfm-pulpcore-python3-future-0.18.2-4.el7.noarch
tfm-pulpcore-python3-galaxy-importer-0.4.1-1.el7.noarch
tfm-pulpcore-python3-gnupg-0.4.7-2.el7.noarch
tfm-pulpcore-python3-gunicorn-20.1.0-3.el7.noarch
tfm-pulpcore-python3-idna-3.3-1.el7.noarch
tfm-pulpcore-python3-idna-ssl-1.1.0-4.el7.noarch
tfm-pulpcore-python3-importlib-metadata-1.7.0-2.el7.noarch
tfm-pulpcore-python3-importlib-resources-5.0.0-2.el7.noarch
tfm-pulpcore-python3-inflection-0.5.1-2.el7.noarch
tfm-pulpcore-python3-iniparse-0.4-34.el7.noarch
tfm-pulpcore-python3-jinja2-3.0.2-1.el7.noarch
tfm-pulpcore-python3-jsonschema-3.2.0-7.el7.noarch
tfm-pulpcore-python3-libcomps-0.1.18-1.el7.x86_64
tfm-pulpcore-python3-lxml-4.6.4-1.el7.x86_64
tfm-pulpcore-python3-markdown-3.3.4-4.el7.noarch
tfm-pulpcore-python3-markuppy-1.14-2.el7.noarch
tfm-pulpcore-python3-markupsafe-2.0.1-1.el7.x86_64
tfm-pulpcore-python3-mccabe-0.6.1-2.el7.noarch
tfm-pulpcore-python3-multidict-5.2.0-1.el7.x86_64
tfm-pulpcore-python3-odfpy-1.4.1-5.el7.noarch
tfm-pulpcore-python3-openpyxl-3.0.9-1.el7.noarch
tfm-pulpcore-python3-packaging-21.2-1.el7.noarch
tfm-pulpcore-python3-parsley-1.3-1.el7.noarch
tfm-pulpcore-python3-pbr-5.6.0-1.el7.noarch
tfm-pulpcore-python3-pkginfo-1.7.1-3.el7.noarch
tfm-pulpcore-python3-productmd-1.33-2.el7.noarch
tfm-pulpcore-python3-prometheus-client-0.8.0-2.el7.noarch
tfm-pulpcore-python3-psycopg2-2.9.1-1.el7.x86_64
tfm-pulpcore-python3-pulp-ansible-0.10.1-1.el7.noarch
tfm-pulpcore-python3-pulp-certguard-1.5.1-1.el7.noarch
tfm-pulpcore-python3-pulp-container-2.9.0-1.el7.noarch
tfm-pulpcore-python3-pulp-deb-2.16.0-1.el7.noarch
tfm-pulpcore-python3-pulp-file-1.10.1-1.el7.noarch
tfm-pulpcore-python3-pulp-python-3.5.2-1.el7.noarch
tfm-pulpcore-python3-pulp-rpm-3.16.1-1.el7.noarch
tfm-pulpcore-python3-pulpcore-3.16.1-1.el7.noarch
tfm-pulpcore-python3-pyOpenSSL-19.1.0-2.el7.noarch
tfm-pulpcore-python3-pycairo-1.20.1-2.el7.x86_64
tfm-pulpcore-python3-pycares-4.1.2-1.el7.x86_64
tfm-pulpcore-python3-pycodestyle-2.7.0-4.el7.noarch
tfm-pulpcore-python3-pycparser-2.20-2.el7.noarch
tfm-pulpcore-python3-pycryptodomex-3.11.0-1.el7.x86_64
tfm-pulpcore-python3-pyflakes-2.3.1-4.el7.noarch
tfm-pulpcore-python3-pygobject-3.42.0-1.el7.x86_64
tfm-pulpcore-python3-pygtrie-2.4.2-2.el7.noarch
tfm-pulpcore-python3-pyjwkest-1.4.2-5.el7.noarch
tfm-pulpcore-python3-pyjwt-1.7.1-7.el7.noarch
tfm-pulpcore-python3-pyparsing-2.4.7-2.el7.noarch
tfm-pulpcore-python3-pyrsistent-0.18.0-1.el7.x86_64
tfm-pulpcore-python3-pytz-2021.3-1.el7.noarch
tfm-pulpcore-python3-pyyaml-5.4.1-3.el7.x86_64
tfm-pulpcore-python3-redis-3.5.3-2.el7.noarch
tfm-pulpcore-python3-requests-2.26.0-3.el7.noarch
tfm-pulpcore-python3-requirements-parser-0.2.0-2.el7.noarch
tfm-pulpcore-python3-rhsm-1.19.2-2.el7.x86_64
tfm-pulpcore-python3-semantic-version-2.8.5-2.el7.noarch
tfm-pulpcore-python3-setuptools-1.0-4.el7.x86_64
tfm-pulpcore-python3-six-1.16.0-1.el7.noarch
tfm-pulpcore-python3-socks-1.2.4-2.el7.noarch
tfm-pulpcore-python3-solv-0.7.20-4.el7.x86_64
tfm-pulpcore-python3-sqlparse-0.4.2-2.el7.noarch
tfm-pulpcore-python3-tablib-3.1.0-1.el7.noarch
tfm-pulpcore-python3-typing-extensions-3.10.0.2-1.el7.noarch
tfm-pulpcore-python3-uritemplate-4.1.1-1.el7.noarch
tfm-pulpcore-python3-url-normalize-1.4.3-3.el7.noarch
tfm-pulpcore-python3-urllib3-1.26.7-1.el7.noarch
tfm-pulpcore-python3-urlman-1.4.0-2.el7.noarch
tfm-pulpcore-python3-webencodings-0.5.1-2.el7.noarch
tfm-pulpcore-python3-whitenoise-5.3.0-1.el7.noarch
tfm-pulpcore-python3-xlrd-2.0.1-4.el7.noarch
tfm-pulpcore-python3-xlwt-1.3.0-2.el7.noarch
tfm-pulpcore-python3-yarl-1.7.2-1.el7.x86_64
tfm-pulpcore-python3-zipp-3.4.0-3.el7.noarch
tfm-pulpcore-runtime-1.0-4.el7.x86_64
tfm-rubygem-actioncable-6.0.3.7-1.el7.noarch
tfm-rubygem-actionmailbox-6.0.3.7-1.el7.noarch
tfm-rubygem-actionmailer-6.0.3.7-1.el7.noarch
tfm-rubygem-actionpack-6.0.3.7-1.el7.noarch
tfm-rubygem-actiontext-6.0.3.7-1.el7.noarch
tfm-rubygem-actionview-6.0.3.7-1.el7.noarch
tfm-rubygem-activejob-6.0.3.7-1.el7.noarch
tfm-rubygem-activemodel-6.0.3.7-1.el7.noarch
tfm-rubygem-activerecord-6.0.3.7-1.el7.noarch
tfm-rubygem-activerecord-import-1.1.0-1.el7.noarch
tfm-rubygem-activerecord-session_store-2.0.0-1.el7.noarch
tfm-rubygem-activestorage-6.0.3.7-1.el7.noarch
tfm-rubygem-activesupport-6.0.3.7-1.el7.noarch
tfm-rubygem-addressable-2.8.0-1.el7.noarch
tfm-rubygem-algebrick-0.7.3-8.el7.noarch
tfm-rubygem-amazing_print-1.1.0-2.el7.noarch
tfm-rubygem-ancestry-3.0.7-2.el7.noarch
tfm-rubygem-anemone-0.7.2-23.el7.noarch
tfm-rubygem-angular-rails-templates-1.1.0-2.el7.noarch
tfm-rubygem-ansi-1.5.0-3.el7.noarch
tfm-rubygem-apipie-bindings-0.4.0-2.el7.noarch
tfm-rubygem-apipie-dsl-2.4.0-1.el7.noarch
tfm-rubygem-apipie-params-0.0.5-5.el7.noarch
tfm-rubygem-apipie-rails-0.5.17-4.el7.noarch
tfm-rubygem-audited-4.9.0-4.el7.noarch
tfm-rubygem-awesome_print-1.8.0-6.el7.noarch
tfm-rubygem-azure_mgmt_compute-0.22.0-1.el7.noarch
tfm-rubygem-azure_mgmt_network-0.26.1-1.el7.noarch
tfm-rubygem-azure_mgmt_resources-0.18.2-1.el7.noarch
tfm-rubygem-azure_mgmt_storage-0.23.0-1.el7.noarch
tfm-rubygem-azure_mgmt_subscriptions-0.18.5-1.el7.noarch
tfm-rubygem-bcrypt-3.1.12-4.el7.x86_64
tfm-rubygem-bcrypt_pbkdf-1.1.0-1.el7.x86_64
tfm-rubygem-builder-3.2.4-2.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-6.el7.noarch
tfm-rubygem-clamp-1.1.2-7.el7.noarch
tfm-rubygem-coffee-rails-5.0.0-2.el7.noarch
tfm-rubygem-coffee-script-2.4.1-5.el7.noarch
tfm-rubygem-coffee-script-source-1.12.2-5.el7.noarch
tfm-rubygem-colorize-0.8.1-2.el7.noarch
tfm-rubygem-concurrent-ruby-1.1.6-3.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.6.0-3.fm2_5.el7.noarch
tfm-rubygem-connection_pool-2.2.2-3.el7.noarch
tfm-rubygem-crass-1.0.6-2.el7.noarch
tfm-rubygem-css_parser-1.4.7-5.el7.noarch
tfm-rubygem-daemons-1.2.3-7.el7.noarch
tfm-rubygem-deacon-1.0.0-5.el7.noarch
tfm-rubygem-deep_cloneable-3.0.0-4.el7.noarch
tfm-rubygem-deface-1.5.3-3.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-5.el7.noarch
tfm-rubygem-dynflow-1.6.2-1.fm3_1.el7.noarch
tfm-rubygem-ed25519-1.2.4-1.el7.x86_64
tfm-rubygem-erubi-1.9.0-2.el7.noarch
tfm-rubygem-ethon-0.12.0-2.el7.noarch
tfm-rubygem-excon-0.76.0-2.el7.noarch
tfm-rubygem-execjs-2.7.0-5.el7.noarch
tfm-rubygem-facter-4.0.51-2.el7.x86_64
tfm-rubygem-faraday-0.17.3-2.el7.noarch
tfm-rubygem-faraday-cookie_jar-0.0.6-2.el7.noarch
tfm-rubygem-fast_gettext-1.4.1-5.el7.noarch
tfm-rubygem-ffi-1.12.2-2.el7.x86_64
tfm-rubygem-fog-core-2.1.0-4.el7.noarch
tfm-rubygem-fog-vsphere-3.5.0-2.el7.noarch
tfm-rubygem-foreman-tasks-5.2.0-1.fm3_1.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.6-1.fm2_5.el7.noarch
tfm-rubygem-foreman_azure_rm-2.2.6-1.fm3_1.el7.noarch
tfm-rubygem-foreman_puppet-2.0.0-1.fm3_1.el7.noarch
tfm-rubygem-foreman_remote_execution-5.0.1-1.fm3_1.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.4.8-1.el7.noarch
tfm-rubygem-formatador-0.2.1-13.el7.noarch
tfm-rubygem-friendly_id-5.3.0-2.el7.noarch
tfm-rubygem-fx-0.5.0-2.el7.noarch
tfm-rubygem-get_process_mem-0.2.7-2.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.8.0-3.el7.noarch
tfm-rubygem-gitlab-sidekiq-fetcher-0.6.0-2.el7.noarch
tfm-rubygem-globalid-0.4.2-2.el7.noarch
tfm-rubygem-graphql-1.8.14-3.el7.noarch
tfm-rubygem-graphql-batch-0.3.10-3.el7.noarch
tfm-rubygem-gssapi-1.2.0-8.el7.noarch
tfm-rubygem-hammer_cli-3.1.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-3.1.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-2.el7.noarch
tfm-rubygem-hammer_cli_foreman_puppet-0.0.4-1.fm3_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.16-1.fm3_0.el7.noarch
tfm-rubygem-hammer_cli_katello-1.3.1-1.el7.noarch
tfm-rubygem-hashie-3.6.0-3.el7.noarch
tfm-rubygem-highline-2.0.3-2.el7.noarch
tfm-rubygem-hocon-1.3.1-2.el7.noarch
tfm-rubygem-http-cookie-1.0.2-5.el7.noarch
tfm-rubygem-i18n-1.8.2-2.el7.noarch
tfm-rubygem-jwt-2.2.2-2.el7.noarch
tfm-rubygem-kafo-6.4.0-1.el7.noarch
tfm-rubygem-kafo_parsers-1.2.1-1.el7.noarch
tfm-rubygem-kafo_wizards-0.0.2-2.el7.noarch
tfm-rubygem-katello-4.3.0-0.5.rc4.el7.noarch
tfm-rubygem-ldap_fluff-0.6.0-1.el7.noarch
tfm-rubygem-little-plugger-1.1.4-3.el7.noarch
tfm-rubygem-locale-2.0.9-15.el7.noarch
tfm-rubygem-logging-2.3.0-2.el7.noarch
tfm-rubygem-loofah-2.4.0-2.el7.noarch
tfm-rubygem-mail-2.7.1-2.el7.noarch
tfm-rubygem-marcel-1.0.1-1.el7.noarch
tfm-rubygem-method_source-0.9.2-3.el7.noarch
tfm-rubygem-mime-types-3.3.1-2.el7.noarch
tfm-rubygem-mime-types-data-3.2018.0812-5.el7.noarch
tfm-rubygem-mini_mime-1.0.2-2.el7.noarch
tfm-rubygem-mini_portile2-2.5.1-1.el7.noarch
tfm-rubygem-mqtt-0.5.0-1.el7.noarch
tfm-rubygem-ms_rest-0.7.6-1.el7.noarch
tfm-rubygem-ms_rest_azure-0.12.0-1.el7.noarch
tfm-rubygem-msgpack-1.3.3-2.el7.x86_64
tfm-rubygem-multi_json-1.14.1-3.el7.noarch
tfm-rubygem-multipart-post-2.0.0-3.el7.noarch
tfm-rubygem-mustermann-1.1.1-1.el7.noarch
tfm-rubygem-net-ldap-0.17.0-2.el7.noarch
tfm-rubygem-net-ping-2.0.1-5.el7.noarch
tfm-rubygem-net-scp-1.2.1-5.el7.noarch
tfm-rubygem-net-ssh-4.2.0-3.el7.noarch
tfm-rubygem-net_http_unix-0.2.2-2.el7.noarch
tfm-rubygem-netrc-0.11.0-6.el7.noarch
tfm-rubygem-nio4r-2.5.4-2.el7.x86_64
tfm-rubygem-nokogiri-1.11.3-2.el7.x86_64
tfm-rubygem-oauth-0.5.4-5.el7.noarch
tfm-rubygem-optimist-3.0.0-3.el7.noarch
tfm-rubygem-parallel-1.19.1-2.el7.noarch
tfm-rubygem-parse-cron-0.1.4-5.fm2_5.el7.noarch
tfm-rubygem-pg-1.1.4-4.el7.x86_64
tfm-rubygem-polyglot-0.3.5-3.el7.noarch
tfm-rubygem-powerbar-2.0.1-3.el7.noarch
tfm-rubygem-promise.rb-0.7.4-3.el7.noarch
tfm-rubygem-public_suffix-3.0.3-3.el7.noarch
tfm-rubygem-pulp_ansible_client-0.10.1-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.5.0-1.el7.noarch
tfm-rubygem-pulp_container_client-2.9.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.16.0-1.el7.noarch
tfm-rubygem-pulp_docker_client-4.0.0b6.dev01562331743-1.el7.noarch
tfm-rubygem-pulp_file_client-1.10.0-1.el7.noarch
tfm-rubygem-pulp_ostree_client-2.0.0-0.1.a1.el7.noarch
tfm-rubygem-pulp_python_client-3.5.2-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.16.1-1.el7.noarch
tfm-rubygem-pulpcore_client-3.16.0-1.el7.noarch
tfm-rubygem-puma-5.5.0-1.el7.x86_64
tfm-rubygem-puma-status-1.3-1.el7.noarch
tfm-rubygem-qpid_messaging-1.38.0-1.el7.x86_64
tfm-rubygem-qpid_proton-0.35.0-1.el7.x86_64
tfm-rubygem-rabl-0.14.3-2.el7.noarch
tfm-rubygem-racc-1.5.2-1.el7.x86_64
tfm-rubygem-rack-2.2.3-2.el7.noarch
tfm-rubygem-rack-cors-1.0.2-3.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-10.el7.noarch
tfm-rubygem-rack-protection-2.1.0-2.el7.noarch
tfm-rubygem-rack-test-1.1.0-5.el7.noarch
tfm-rubygem-rails-6.0.3.7-1.el7.noarch
tfm-rubygem-rails-dom-testing-2.0.3-7.el7.noarch
tfm-rubygem-rails-html-sanitizer-1.3.0-2.el7.noarch
tfm-rubygem-rails-i18n-6.0.0-3.el7.noarch
tfm-rubygem-railties-6.0.3.7-1.el7.noarch
tfm-rubygem-rainbow-2.2.2-1.el7.noarch
tfm-rubygem-rake-compiler-1.0.7-4.el7.noarch
tfm-rubygem-rb-inotify-0.9.7-6.el7.noarch
tfm-rubygem-rbvmomi-2.2.0-4.el7.noarch
tfm-rubygem-record_tag_helper-1.0.1-4.el7.noarch
tfm-rubygem-redfish_client-0.5.2-2.el7.noarch
tfm-rubygem-redis-4.1.2-3.el7.noarch
tfm-rubygem-responders-3.0.0-4.el7.noarch
tfm-rubygem-rest-client-2.0.2-4.el7.noarch
tfm-rubygem-rkerberos-0.1.5-20.el7.x86_64
tfm-rubygem-roadie-3.4.0-4.el7.noarch
tfm-rubygem-roadie-rails-2.1.1-3.el7.noarch
tfm-rubygem-robotex-1.0.0-22.el7.noarch
tfm-rubygem-rsec-0.4.3-5.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.1-2.el7.x86_64
tfm-rubygem-ruby2_keywords-0.0.4-1.el7.noarch
tfm-rubygem-ruby2ruby-2.4.2-4.el7.noarch
tfm-rubygem-ruby_parser-3.10.1-4.el7.noarch
tfm-rubygem-rubyipmi-0.11.0-1.el7.noarch
tfm-rubygem-runcible-2.13.1-2.el7.noarch
tfm-rubygem-safemode-1.3.6-2.el7.noarch
tfm-rubygem-scoped_search-4.1.9-2.el7.noarch
tfm-rubygem-sd_notify-0.1.0-2.el7.noarch
tfm-rubygem-secure_headers-6.3.0-3.el7.noarch
tfm-rubygem-sequel-5.42.0-2.el7.noarch
tfm-rubygem-server_sent_events-0.1.2-2.el7.noarch
tfm-rubygem-sexp_processor-4.10.0-7.el7.noarch
tfm-rubygem-sidekiq-5.2.7-4.el7.noarch
tfm-rubygem-sinatra-2.1.0-2.el7.noarch
tfm-rubygem-smart_proxy_dynflow-0.6.1-1.fm3_1.el7.noarch
tfm-rubygem-smart_proxy_dynflow_core-0.4.1-1.fm2_6.el7.noarch
tfm-rubygem-smart_proxy_pulp-3.2.0-1.fm3_1.el7.noarch
tfm-rubygem-smart_proxy_remote_execution_ssh-0.5.0-1.fm3_1.el7.noarch
tfm-rubygem-sprockets-4.0.2-2.el7.noarch
tfm-rubygem-sprockets-rails-3.2.1-7.el7.noarch
tfm-rubygem-sqlite3-1.3.13-7.el7.x86_64
tfm-rubygem-sshkey-1.9.0-5.el7.noarch
tfm-rubygem-statsd-instrument-2.1.4-4.el7.noarch
tfm-rubygem-stomp-1.4.9-2.el7.noarch
tfm-rubygem-thor-1.0.1-3.el7.noarch
tfm-rubygem-thread_safe-0.3.6-6.el7.noarch
tfm-rubygem-tilt-2.0.8-5.el7.noarch
tfm-rubygem-timeliness-0.3.10-2.el7.noarch
tfm-rubygem-typhoeus-1.3.1-2.el7.noarch
tfm-rubygem-tzinfo-1.2.6-2.el7.noarch
tfm-rubygem-unf-0.1.3-9.el7.noarch
tfm-rubygem-unf_ext-0.0.7.2-4.el7.x86_64
tfm-rubygem-unicode-0.4.4.4-4.el7.x86_64
tfm-rubygem-unicode-display_width-1.7.0-2.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-8.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-6.el7.noarch
tfm-rubygem-websocket-driver-0.7.1-2.el7.x86_64
tfm-rubygem-websocket-extensions-0.1.5-2.el7.noarch
tfm-rubygem-will_paginate-3.1.7-4.el7.noarch
tfm-rubygem-xmlrpc-0.3.0-3.el7.noarch
tfm-rubygem-zeitwerk-2.2.2-2.el7.noarch
tfm-runtime-7.0-4.el7.x86_64
MYHOSTNAME-foreman-client-1.0-10.noarch
MYHOSTNAME-foreman-proxy-1.0-16.noarch
MYHOSTNAME-foreman-proxy-client-1.0-10.noarch
MYHOSTNAME-qpid-broker-1.0-11.noarch
MYHOSTNAME-qpid-client-cert-1.0-1.noarch
MYHOSTNAME-qpid-router-client-1.0-1.noarch
MYHOSTNAME-qpid-router-server-1.0-10.noarch

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

Other relevant data:
As this is happening on one repo only (others sync fine), I thought to remove and create it from fresh which didn’t change the symptom. Created a new Content View and added this one repo in, no difference - still get Errno1. Created another new repo and pointed at a different upstream URL but made no difference.

Any help gratefully received, thanks!

file.log (28.5 KB)

2

What is the affected repo and its sync options?

The full traceback of the error (should be included in the failed Foreman task) would also be good.

3

It’s the CentOS 7 Repo I’m trying to sync, those settings are:

Sync Settings

Restrict to architecture
    No restriction
Restrict to
OS version
    No restriction
Upstream URL
    https://mirror.bytemark.co.uk/centos/7/os/x86_64/
Verify SSL
    Yes
Upstream Authorization
Yum Metadata Checksum
    Default
Mirror on Sync
    Yes
HTTP Proxy
    Global Default
Ignore SRPMs
    No
Publish via HTTPS
    Yes
Unprotected
    Yes
Published At
    https://MYHOSTNAME/pulp/content/FNC/Library/custom/CentOS/7_OS_Repo/ 
GPG Key
SSL CA Cert
SSL Client Cert
SSL Client Key
Download Policy
    Immediate

The backtrace is here:

Actions::Pulp3::Repository::Sync

Input:

{"repo_id"=>753,
 "smart_proxy_id"=>1,
 "options"=>{"optimize"=>false},
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"46371912-4d51-4eed-8a2b-6464c5d545e3",
 "current_timezone"=>"UTC",
 "current_organization_id"=>1,
 "current_location_id"=>nil,
 "current_user_id"=>6}

Output:

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/929b44b3-0b56-4254-a4ed-669348ac6c81/",
    "pulp_created"=>"2022-02-08T11:28:39.563+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"46371912-4d51-4eed-8a2b-6464c5d545e3",
    "started_at"=>"2022-02-08T11:28:39.615+00:00",
    "finished_at"=>"2022-02-08T11:29:34.619+00:00",
    "error"=>
     {"traceback"=>
       #<Sequel::SQL::Blob:0xa47421c bytes=139 start="  File \"/o" end="form_task\n"> +
       #<Sequel::SQL::Blob:0xa474230 bytes=35 start="    result" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0xa474244 bytes=137 start="  File \"/o" end="nchronize\n"> +
       #<Sequel::SQL::Blob:0xa474258 bytes=56 start="    repo_v" end="version()\n"> +
       #<Sequel::SQL::Blob:0xa47426c bytes=142 start="  File \"/o" end="in create\n"> +
       #<Sequel::SQL::Blob:0xa474280 bytes=38 start="    loop.r" end="pipeline)\n"> +
       #<Sequel::SQL::Blob:0xa474294 bytes=110 start="  File \"/o" end="_complete\n"> +
       #<Sequel::SQL::Blob:0xa4742a8 bytes=27 start="    return" end=".result()\n"> +
       #<Sequel::SQL::Blob:0xa4742bc bytes=135 start="  File \"/o" end="_pipeline\n"> +
       #<Sequel::SQL::Blob:0xa4742d0 bytes=35 start="    await " end="*futures)\n"> +
       #<Sequel::SQL::Blob:0xa4742e4 bytes=127 start="  File \"/o" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0xa4742f8 bytes=21 start="    await " end="elf.run()\n"> +
       #<Sequel::SQL::Blob:0xa47430c bytes=135 start="  File \"/o" end="9, in run\n"> +
       #<Sequel::SQL::Blob:0xa474320 bytes=62 start="    await " end="_create)(\n"> +
       #<Sequel::SQL::Blob:0xa474334 bytes=114 start="  File \"/o" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0xa474348 bytes=55 start="    ret = " end="out=None)\n"> +
       #<Sequel::SQL::Blob:0xa47435c bytes=94 start="  File \"/o" end=" wait_for\n"> +
       #<Sequel::SQL::Blob:0xa474370 bytes=21 start="    return" end="await fut\n"> +
       #<Sequel::SQL::Blob:0xa474384 bytes=100 start="  File \"/o" end="7, in run\n"> +
       #<Sequel::SQL::Blob:0xa474398 bytes=48 start="    result" end="f.kwargs)\n"> +
       #<Sequel::SQL::Blob:0xa4743ac bytes=120 start="  File \"/o" end="d_handler\n"> +
       #<Sequel::SQL::Blob:0xa4743c0 bytes=33 start="    return" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0xa4743d4 bytes=138 start="  File \"/o" end="or_create\n"> +
       #<Sequel::SQL::Blob:0xa4743e8 bytes=60 start="    return" end="tch_size)\n"> +
       #<Sequel::SQL::Blob:0xa4743fc bytes=131 start="  File \"/o" end="er_method\n"> +
       #<Sequel::SQL::Blob:0xa474410 bytes=63 start="    return" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0xa474424 bytes=127 start="  File \"/o" end="lk_create\n"> +
       #<Sequel::SQL::Blob:0xa474438 bytes=45 start="    return" end="d_insert(\n"> +
       #<Sequel::SQL::Blob:0xa47444c bytes=132 start="  File \"/o" end="ed_insert\n"> +
       #<Sequel::SQL::Blob:0xa474460 bytes=39 start="    insert" end="._insert(\n"> +
       #<Sequel::SQL::Blob:0xa474474 bytes=124 start="  File \"/o" end="n _insert\n"> +
       #<Sequel::SQL::Blob:0xa474488 bytes=73 start="    return" end="g_fields)\n"> +
       #<Sequel::SQL::Blob:0xa47449c bytes=135 start="  File \"/o" end="ecute_sql\n"> +
       #<Sequel::SQL::Blob:0xa4744b0 bytes=38 start="    for sq" end="as_sql():\n"> +
       #<Sequel::SQL::Blob:0xa4744c4 bytes=129 start="  File \"/o" end="in as_sql\n"> +
       #<Sequel::SQL::Blob:0xa4744d8 bytes=58 start="    return" end=".as_sql()\n"> +
       #<Sequel::SQL::Blob:0xa4744ec bytes=130 start="  File \"/o" end="in as_sql\n"> +
       #<Sequel::SQL::Blob:0xa474500 bytes=19 content="    value_rows = [\n"> +
       #<Sequel::SQL::Blob:0xa474514 bytes=134 start="  File \"/o" end="listcomp>\n"> +
       #<Sequel::SQL::Blob:0xa474528 bytes=83 start="    [self." end="n fields]\n"> +
       #<Sequel::SQL::Blob:0xa47453c bytes=134 start="  File \"/o" end="listcomp>\n"> +
       #<Sequel::SQL::Blob:0xa474550 bytes=83 start="    [self." end="n fields]\n"> +
       #<Sequel::SQL::Blob:0xa474564 bytes=136 start="  File \"/o" end="_save_val\n"> +
       #<Sequel::SQL::Blob:0xa474578 bytes=41 start="    return" end="add=True)\n"> +
       #<Sequel::SQL::Blob:0xa47458c bytes=127 start="  File \"/o" end=" pre_save\n"> +
       #<Sequel::SQL::Blob:0xa4745a0 bytes=49 start="    return" end="nce, add)\n"> +
       #<Sequel::SQL::Blob:0xa4745b4 bytes=131 start="  File \"/o" end=" pre_save\n"> +
       #<Sequel::SQL::Blob:0xa4745c8 bytes=48 start="    file.s" end="ve=False)\n"> +
       #<Sequel::SQL::Blob:0xa4745dc bytes=126 start="  File \"/o" end=", in save\n"> +
       #<Sequel::SQL::Blob:0xa4745f0 bytes=83 start="    self.n" end="x_length)\n"> +
       #<Sequel::SQL::Blob:0xa474604 bytes=122 start="  File \"/o" end=", in save\n"> +
       #<Sequel::SQL::Blob:0xa474618 bytes=37 start="    return" end=" content)\n"> +
       #<Sequel::SQL::Blob:0xa47462c bytes=125 start="  File \"/o" end=" in _save\n"> +
       #<Sequel::SQL::Blob:0xa474640 bytes=52 start="    os.chm" end="ons_mode)\n">,
      "description"=>
       "[Errno 1] Operation not permitted: '/var/lib/pulp/media/artifact/b7/9079ad71cc3c5ceb3561fff348a1b67ee37f71f4cddfec09480d4589c191d6'"},
    "worker"=>"/pulp/api/v3/workers/e650fde8-d352-4ebe-bd9f-25b264fa0f70/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>5},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"canceled",
       "done"=>5},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>0},
      {"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"canceled",
       "done"=>2502}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/f5d868a2-607d-4104-9ec3-8b5a83618a50/",
      "shared:/pulp/api/v3/remotes/rpm/rpm/abac2046-1c7d-490d-823b-6a88274f4816/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>22, "failed"=>1}}

Exception:

Katello::Errors::Pulp3Error: [Errno 1] Operation not permitted: '/var/lib/pulp/media/artifact/b7/9079ad71cc3c5ceb3561fff348a1b67ee37f71f4cddfec09480d4589c191d6'

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/pulp3/abstract_async_task.rb:160:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.rc4/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:580:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:580:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:483:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:483:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:483:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:575:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/action.rb:296:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/director.rb:94:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.2/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Action:

Actions::Pulp3::Repository::Sync

Input:

{"repo_id"=>753,
 "smart_proxy_id"=>1,
 "options"=>{"optimize"=>false},
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"46371912-4d51-4eed-8a2b-6464c5d545e3",
 "current_timezone"=>"UTC",
 "current_organization_id"=>1,
 "current_location_id"=>nil,
 "current_user_id"=>6}

Output:

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/929b44b3-0b56-4254-a4ed-669348ac6c81/",
    "pulp_created"=>"2022-02-08T11:28:39.563+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"46371912-4d51-4eed-8a2b-6464c5d545e3",
    "started_at"=>"2022-02-08T11:28:39.615+00:00",
    "finished_at"=>"2022-02-08T11:29:34.619+00:00",
    "error"=>
     {"traceback"=>
       #<Sequel::SQL::Blob:0xa474654 bytes=139 start="  File \"/o" end="form_task\n"> +
       #<Sequel::SQL::Blob:0xa474668 bytes=35 start="    result" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0xa47467c bytes=137 start="  File \"/o" end="nchronize\n"> +
       #<Sequel::SQL::Blob:0xa474690 bytes=56 start="    repo_v" end="version()\n"> +
       #<Sequel::SQL::Blob:0xa4746a4 bytes=142 start="  File \"/o" end="in create\n"> +
       #<Sequel::SQL::Blob:0xa4746b8 bytes=38 start="    loop.r" end="pipeline)\n"> +
       #<Sequel::SQL::Blob:0xa4746cc bytes=110 start="  File \"/o" end="_complete\n"> +
       #<Sequel::SQL::Blob:0xa4746e0 bytes=27 start="    return" end=".result()\n"> +
       #<Sequel::SQL::Blob:0xa4746f4 bytes=135 start="  File \"/o" end="_pipeline\n"> +
       #<Sequel::SQL::Blob:0xa474708 bytes=35 start="    await " end="*futures)\n"> +
       #<Sequel::SQL::Blob:0xa47471c bytes=127 start="  File \"/o" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0xa474730 bytes=21 start="    await " end="elf.run()\n"> +
       #<Sequel::SQL::Blob:0xa474744 bytes=135 start="  File \"/o" end="9, in run\n"> +
       #<Sequel::SQL::Blob:0xa474758 bytes=62 start="    await " end="_create)(\n"> +
       #<Sequel::SQL::Blob:0xa47476c bytes=114 start="  File \"/o" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0xa474780 bytes=55 start="    ret = " end="out=None)\n"> +
       #<Sequel::SQL::Blob:0xa474794 bytes=94 start="  File \"/o" end=" wait_for\n"> +
       #<Sequel::SQL::Blob:0xa4747a8 bytes=21 start="    return" end="await fut\n"> +
       #<Sequel::SQL::Blob:0xa4747bc bytes=100 start="  File \"/o" end="7, in run\n"> +
       #<Sequel::SQL::Blob:0xa4747d0 bytes=48 start="    result" end="f.kwargs)\n"> +
       #<Sequel::SQL::Blob:0xa4747e4 bytes=120 start="  File \"/o" end="d_handler\n"> +
       #<Sequel::SQL::Blob:0xa4747f8 bytes=33 start="    return" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0xa47480c bytes=138 start="  File \"/o" end="or_create\n"> +
       #<Sequel::SQL::Blob:0xa474820 bytes=60 start="    return" end="tch_size)\n"> +
       #<Sequel::SQL::Blob:0xa474834 bytes=131 start="  File \"/o" end="er_method\n"> +
       #<Sequel::SQL::Blob:0xa474848 bytes=63 start="    return" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0xa47485c bytes=127 start="  File \"/o" end="lk_create\n"> +
       #<Sequel::SQL::Blob:0xa474870 bytes=45 start="    return" end="d_insert(\n"> +
       #<Sequel::SQL::Blob:0xa474884 bytes=132 start="  File \"/o" end="ed_insert\n"> +
       #<Sequel::SQL::Blob:0xa474898 bytes=39 start="    insert" end="._insert(\n"> +
       #<Sequel::SQL::Blob:0xa4748ac bytes=124 start="  File \"/o" end="n _insert\n"> +
       #<Sequel::SQL::Blob:0xa4748c0 bytes=73 start="    return" end="g_fields)\n"> +
       #<Sequel::SQL::Blob:0xa4748d4 bytes=135 start="  File \"/o" end="ecute_sql\n"> +
       #<Sequel::SQL::Blob:0xa4748e8 bytes=38 start="    for sq" end="as_sql():\n"> +
       #<Sequel::SQL::Blob:0xa4748fc bytes=129 start="  File \"/o" end="in as_sql\n"> +
       #<Sequel::SQL::Blob:0xa474910 bytes=58 start="    return" end=".as_sql()\n"> +
       #<Sequel::SQL::Blob:0xa474924 bytes=130 start="  File \"/o" end="in as_sql\n"> +
       #<Sequel::SQL::Blob:0xa474938 bytes=19 content="    value_rows = [\n"> +
       #<Sequel::SQL::Blob:0xa47494c bytes=134 start="  File \"/o" end="listcomp>\n"> +
       #<Sequel::SQL::Blob:0xa474960 bytes=83 start="    [self." end="n fields]\n"> +
       #<Sequel::SQL::Blob:0xa474974 bytes=134 start="  File \"/o" end="listcomp>\n"> +
       #<Sequel::SQL::Blob:0xa474988 bytes=83 start="    [self." end="n fields]\n"> +
       #<Sequel::SQL::Blob:0xa47499c bytes=136 start="  File \"/o" end="_save_val\n"> +
       #<Sequel::SQL::Blob:0xa4749b0 bytes=41 start="    return" end="add=True)\n"> +
       #<Sequel::SQL::Blob:0xa4749c4 bytes=127 start="  File \"/o" end=" pre_save\n"> +
       #<Sequel::SQL::Blob:0xa4749d8 bytes=49 start="    return" end="nce, add)\n"> +
       #<Sequel::SQL::Blob:0xa4749ec bytes=131 start="  File \"/o" end=" pre_save\n"> +
       #<Sequel::SQL::Blob:0xa474a00 bytes=48 start="    file.s" end="ve=False)\n"> +
       #<Sequel::SQL::Blob:0xa474a14 bytes=126 start="  File \"/o" end=", in save\n"> +
       #<Sequel::SQL::Blob:0xa474a28 bytes=83 start="    self.n" end="x_length)\n"> +
       #<Sequel::SQL::Blob:0xa474a3c bytes=122 start="  File \"/o" end=", in save\n"> +
       #<Sequel::SQL::Blob:0xa474a50 bytes=37 start="    return" end=" content)\n"> +
       #<Sequel::SQL::Blob:0xa474a64 bytes=125 start="  File \"/o" end=" in _save\n"> +
       #<Sequel::SQL::Blob:0xa474a78 bytes=52 start="    os.chm" end="ons_mode)\n">,
      "description"=>
       "[Errno 1] Operation not permitted: '/var/lib/pulp/media/artifact/b7/9079ad71cc3c5ceb3561fff348a1b67ee37f71f4cddfec09480d4589c191d6'"},
    "worker"=>"/pulp/api/v3/workers/e650fde8-d352-4ebe-bd9f-25b264fa0f70/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>5},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"canceled",
       "done"=>5},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>0},
      {"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"canceled",
       "done"=>2502}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/f5d868a2-607d-4104-9ec3-8b5a83618a50/",
      "shared:/pulp/api/v3/remotes/rpm/rpm/abac2046-1c7d-490d-823b-6a88274f4816/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>22, "failed"=>1}}

Action:

Actions::Pulp3::Repository::SaveVersion

Input:

{"repository_id"=>753,
 "tasks"=>Step(6).output[:pulp_tasks],
 "repository_details"=>nil,
 "force_fetch_version"=>true,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"46371912-4d51-4eed-8a2b-6464c5d545e3",
 "current_timezone"=>"UTC",
 "current_organization_id"=>1,
 "current_location_id"=>nil,
 "current_user_id"=>6}

Output:

{}

Action:

Actions::Pulp3::Repository::CreatePublication

Input:

{"repository_id"=>753,
 "smart_proxy_id"=>1,
 "contents_changed"=>true,
 "skip_publication_creation"=>Step(9).output[:publication_provided],
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"46371912-4d51-4eed-8a2b-6464c5d545e3",
 "current_timezone"=>"UTC",
 "current_organization_id"=>1,
 "current_location_id"=>nil,
 "current_user_id"=>6}

Output:

{}

Action:

Actions::Pulp3::Repository::SavePublication

Input:

{"repository_id"=>753,
 "tasks"=>Step(12).output,
 "contents_changed"=>true,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"46371912-4d51-4eed-8a2b-6464c5d545e3",
 "current_timezone"=>"UTC",
 "current_organization_id"=>1,
 "current_location_id"=>nil,
 "current_user_id"=>6}

Output:

{}

Action:

Actions::Pulp3::Orchestration::Repository::Sync

Input:

{"subaction_output"=>Step(9).output,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"46371912-4d51-4eed-8a2b-6464c5d545e3",
 "current_timezone"=>"UTC",
 "current_organization_id"=>1,
 "current_location_id"=>nil,
 "current_user_id"=>6}

Output:

{}

Action:

Actions::Katello::Repository::Sync

Input:

{"repository"=>
  {"id"=>753,
   "name"=>"CentOS 7 os x86_64",
   "label"=>"CentOS_7_os_x86_64"},
 "product"=>
  {"id"=>68,
   "name"=>"CentOS",
   "label"=>"CentOS",
   "cp_id"=>"675993204103"},
 "provider"=>{"id"=>1, "name"=>"Anonymous"},
 "organization"=>{"id"=>1, "name"=>"ORGANISATION", "label"=>"ORG"},
 "services_checked"=>["pulp3"],
 "id"=>753,
 "sync_result"=>Step(19).output,
 "skip_metadata_check"=>true,
 "validate_contents"=>false,
 "contents_changed"=>nil,
 "current_request_id"=>"46371912-4d51-4eed-8a2b-6464c5d545e3",
 "current_timezone"=>"UTC",
 "current_organization_id"=>1,
 "current_location_id"=>nil,
 "current_user_id"=>6}

Output:

{}

Action:

Actions::Katello::Repository::Sync

Input:

{"repository"=>
  {"id"=>753,
   "name"=>"CentOS 7 os x86_64",
   "label"=>"CentOS_7_os_x86_64"},
 "product"=>
  {"id"=>68,
   "name"=>"CentOS",
   "label"=>"CentOS",
   "cp_id"=>"675993204103"},
 "provider"=>{"id"=>1, "name"=>"Anonymous"},
 "organization"=>{"id"=>1, "name"=>"ORGANISATION", "label"=>"ORG"},
 "services_checked"=>["pulp3"],
 "id"=>753,
 "sync_result"=>Step(19).output,
 "skip_metadata_check"=>true,
 "validate_contents"=>false,
 "contents_changed"=>nil,
 "current_request_id"=>"46371912-4d51-4eed-8a2b-6464c5d545e3",
 "current_timezone"=>"UTC",
 "current_organization_id"=>1,
 "current_location_id"=>nil,
 "current_user_id"=>6}

Output:

{}

Pointing at a different mirror source doesn’t change the problem, it downloads no packages at all.

4

Hmm. I have not seen these <Sequel::SQL::Blob:0xa47421c bytes=139 start= backtraces before. @Justin_Sherrill Any idea what this might be?

5

We have exactly the same problem. Any idea what we can do? Please help!

6

The " Sequel::SQL::Blob" stuff seems to be Ruby-related, those are the only hits I get when I google it. Are you pulling task data out of the database directly?

Anyway… I think that’s unrelated to the pulp task failing. It’s hard to read due to the truncation though.

@JamesV21 or @tomzellner, could you try to fin an un-mutilated version of the traceback? Perhaps looking at the task record via pulp task details --href /pulp/api/v3/tasks/929b44b3-0b56-4254-a4ed-669348ac6c81/? If you have the pulp-cli package installed.

7

I can’t reproduce on the latest version but I haven’t tried the version shipped with 4.3 yet, FWIW.

8

I have just upgraded from 4.2 to 4.3 (again) and see the same error now when syncing the centos 7 os repo.

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/c77c47d7-c526-44b3-8ad0-f0e0320413fe/",
    "pulp_created"=>"2022-03-09T18:03:36.900+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"413226aa-898a-4e38-9a8e-ab48f40e6001",
    "started_at"=>"2022-03-09T18:03:37.296+00:00",
    "finished_at"=>"2022-03-09T18:09:04.085+00:00",
    "error"=>
     {"traceback"=>
       #<Sequel::SQL::Blob:0x36240 bytes=139 start="  File \"/o" end="form_task\n"> +
       #<Sequel::SQL::Blob:0x36254 bytes=35 start="    result" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x36268 bytes=137 start="  File \"/o" end="nchronize\n"> +
       #<Sequel::SQL::Blob:0x3627c bytes=56 start="    repo_v" end="version()\n"> +
       #<Sequel::SQL::Blob:0x36290 bytes=142 start="  File \"/o" end="in create\n"> +
       #<Sequel::SQL::Blob:0x362a4 bytes=38 start="    loop.r" end="pipeline)\n"> +
       #<Sequel::SQL::Blob:0x362b8 bytes=110 start="  File \"/o" end="_complete\n"> +
       #<Sequel::SQL::Blob:0x362cc bytes=27 start="    return" end=".result()\n"> +
       #<Sequel::SQL::Blob:0x362e0 bytes=135 start="  File \"/o" end="_pipeline\n"> +
       #<Sequel::SQL::Blob:0x362f4 bytes=35 start="    await " end="*futures)\n"> +
       #<Sequel::SQL::Blob:0x36308 bytes=127 start="  File \"/o" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0x3631c bytes=21 start="    await " end="elf.run()\n"> +
       #<Sequel::SQL::Blob:0x36330 bytes=135 start="  File \"/o" end="9, in run\n"> +
       #<Sequel::SQL::Blob:0x36344 bytes=62 start="    await " end="_create)(\n"> +
       #<Sequel::SQL::Blob:0x36358 bytes=114 start="  File \"/o" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0x3636c bytes=55 start="    ret = " end="out=None)\n"> +
       #<Sequel::SQL::Blob:0x36380 bytes=94 start="  File \"/o" end=" wait_for\n"> +
       #<Sequel::SQL::Blob:0x36394 bytes=21 start="    return" end="await fut\n"> +
       #<Sequel::SQL::Blob:0x363a8 bytes=100 start="  File \"/o" end="7, in run\n"> +
       #<Sequel::SQL::Blob:0x363bc bytes=48 start="    result" end="f.kwargs)\n"> +
       #<Sequel::SQL::Blob:0x363d0 bytes=120 start="  File \"/o" end="d_handler\n"> +
       #<Sequel::SQL::Blob:0x363e4 bytes=33 start="    return" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x363f8 bytes=138 start="  File \"/o" end="or_create\n"> +
       #<Sequel::SQL::Blob:0x3640c bytes=60 start="    return" end="tch_size)\n"> +
       #<Sequel::SQL::Blob:0x36420 bytes=131 start="  File \"/o" end="er_method\n"> +
       #<Sequel::SQL::Blob:0x36434 bytes=63 start="    return" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x36448 bytes=127 start="  File \"/o" end="lk_create\n"> +
       #<Sequel::SQL::Blob:0x3645c bytes=45 start="    return" end="d_insert(\n"> +
       #<Sequel::SQL::Blob:0x36470 bytes=132 start="  File \"/o" end="ed_insert\n"> +
       #<Sequel::SQL::Blob:0x36484 bytes=39 start="    insert" end="._insert(\n"> +
       #<Sequel::SQL::Blob:0x36498 bytes=124 start="  File \"/o" end="n _insert\n"> +
       #<Sequel::SQL::Blob:0x364ac bytes=73 start="    return" end="g_fields)\n"> +
       #<Sequel::SQL::Blob:0x364c0 bytes=135 start="  File \"/o" end="ecute_sql\n"> +
       #<Sequel::SQL::Blob:0x364d4 bytes=38 start="    for sq" end="as_sql():\n"> +
       #<Sequel::SQL::Blob:0x364e8 bytes=129 start="  File \"/o" end="in as_sql\n"> +
       #<Sequel::SQL::Blob:0x364fc bytes=58 start="    return" end=".as_sql()\n"> +
       #<Sequel::SQL::Blob:0x36510 bytes=130 start="  File \"/o" end="in as_sql\n"> +
       #<Sequel::SQL::Blob:0x36524 bytes=19 content="    value_rows = [\n"> +
       #<Sequel::SQL::Blob:0x36538 bytes=134 start="  File \"/o" end="listcomp>\n"> +
       #<Sequel::SQL::Blob:0x3654c bytes=83 start="    [self." end="n fields]\n"> +
       #<Sequel::SQL::Blob:0x36560 bytes=134 start="  File \"/o" end="listcomp>\n"> +
       #<Sequel::SQL::Blob:0x36574 bytes=83 start="    [self." end="n fields]\n"> +
       #<Sequel::SQL::Blob:0x36588 bytes=136 start="  File \"/o" end="_save_val\n"> +
       #<Sequel::SQL::Blob:0x3659c bytes=41 start="    return" end="add=True)\n"> +
       #<Sequel::SQL::Blob:0x365b0 bytes=127 start="  File \"/o" end=" pre_save\n"> +
       #<Sequel::SQL::Blob:0x365c4 bytes=49 start="    return" end="nce, add)\n"> +
       #<Sequel::SQL::Blob:0x365d8 bytes=131 start="  File \"/o" end=" pre_save\n"> +
       #<Sequel::SQL::Blob:0x365ec bytes=48 start="    file.s" end="ve=False)\n"> +
       #<Sequel::SQL::Blob:0x36600 bytes=126 start="  File \"/o" end=", in save\n"> +
       #<Sequel::SQL::Blob:0x36614 bytes=83 start="    self.n" end="x_length)\n"> +
       #<Sequel::SQL::Blob:0x36628 bytes=122 start="  File \"/o" end=", in save\n"> +
       #<Sequel::SQL::Blob:0x3663c bytes=37 start="    return" end=" content)\n"> +
       #<Sequel::SQL::Blob:0x36650 bytes=125 start="  File \"/o" end=" in _save\n"> +
       #<Sequel::SQL::Blob:0x36664 bytes=52 start="    os.chm" end="ons_mode)\n">,
      "description"=>
       "[Errno 1] Operation not permitted: '/var/lib/pulp/media/artifact/00/9f12969b8e86ea4f15b803c160c721f7f9426c8b3805deccdaf2d9c09bdc46'"},
    "worker"=>"/pulp/api/v3/workers/c1a4b937-d196-4505-a3d3-447a90223152/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"canceled",
       "done"=>5140},
      {"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>5},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"canceled",
       "done"=>5},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>0}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/17e07bde-cdaf-485b-9f27-5316da1c20da/",
      "shared:/pulp/api/v3/remotes/rpm/rpm/eae954f5-3387-43c9-a58f-325ded40354c/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>41, "failed"=>1}}

The file mentioned is the .treeinfo file:

# cat /var/lib/pulp/media/artifact/00/9f12969b8e86ea4f15b803c160c721f7f9426c8b3805deccdaf2d9c09bdc46
[general]
name = CentOS-7
family = CentOS
timestamp = 1603729576.26
variant = 
version = 7
packagedir = 
arch = x86_64

[stage2]
mainimage = LiveOS/squashfs.img

[images-x86_64]
kernel = images/pxeboot/vmlinuz
initrd = images/pxeboot/initrd.img
boot.iso = images/boot.iso

[images-xen]
kernel = images/pxeboot/vmlinuz
initrd = images/pxeboot/initrd.img

Pulp task details retrieved with curl:

# curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key https://foreman.example.com/pulp/api/v3/tasks/c77c47d7-c526-44b3-8ad0-f0e0320413fe/  | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6189  100  6189    0     0  33413      0 --:--:-- --:--:-- --:--:-- 33635
{
    "child_tasks": [],
    "created_resources": [],
    "error": {
        "description": "[Errno 1] Operation not permitted: '/var/lib/pulp/media/artifact/00/9f12969b8e86ea4f15b803c160c721f7f9426c8b3805deccdaf2d9c09bdc46'",
        "traceback": "  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 377, in _perform_task\n    result = func(*args, **kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 544, in synchronize\n    repo_version = dv.create() or repo.latest_version()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 161, in create\n    loop.run_until_complete(pipeline)\n  File \"/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_events.py\", line 616, in run_until_complete\n    return future.result()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 249, in run\n    await sync_to_async(Artifact.objects.bulk_get_or_create)(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/asgiref/sync.py\", line 444, in __call__\n    ret = await asyncio.wait_for(future, timeout=None)\n  File \"/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/tasks.py\", line 455, in wait_for\n    return await fut\n  File \"/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/thread.py\", line 57, in run\n    result = self.fn(*self.args, **self.kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/asgiref/sync.py\", line 486, in thread_handler\n    return func(*args, **kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/content.py\", line 86, in bulk_get_or_create\n    return super().bulk_create(objs, batch_size=batch_size)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/manager.py\", line 85, in manager_method\n    return getattr(self.get_queryset(), name)(*args, **kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 502, in bulk_create\n    returned_columns = self._batched_insert(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 1287, in _batched_insert\n    inserted_rows.extend(self._insert(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 1270, in _insert\n    return query.get_compiler(using=using).execute_sql(returning_fields)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py\", line 1415, in execute_sql\n    for sql, params in self.as_sql():\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django_readonly_field/compiler.py\", line 31, in as_sql\n    return super(ReadonlySQLCompilerMixin, self).as_sql()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py\", line 1358, in as_sql\n    value_rows = [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py\", line 1359, in <listcomp>\n    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py\", line 1359, in <listcomp>\n    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py\", line 1310, in pre_save_val\n    return field.pre_save(obj, add=True)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/fields.py\", line 76, in pre_save\n    return super().pre_save(model_instance, add)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/files.py\", line 302, in pre_save\n    file.save(file.name, file.file, save=False)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/files.py\", line 89, in save\n    self.name = self.storage.save(name, content, max_length=self.field.max_length)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/storage.py\", line 54, in save\n    return self._save(name, content)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/storage.py\", line 94, in _save\n    os.chmod(full_path, self.file_permissions_mode)\n"
    },
    "finished_at": "2022-03-09T18:09:04.085427Z",
    "logging_cid": "413226aa-898a-4e38-9a8e-ab48f40e6001",
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "sync.parsing.packages",
            "done": 5140,
            "message": "Parsed Packages",
            "state": "canceled",
            "suffix": null,
            "total": null
        },
        {
            "code": "sync.downloading.metadata",
            "done": 5,
            "message": "Downloading Metadata Files",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "sync.downloading.artifacts",
            "done": 5,
            "message": "Downloading Artifacts",
            "state": "canceled",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 0,
            "message": "Associating Content",
            "state": "canceled",
            "suffix": null,
            "total": null
        }
    ],
    "pulp_created": "2022-03-09T18:03:36.900012Z",
    "pulp_href": "/pulp/api/v3/tasks/c77c47d7-c526-44b3-8ad0-f0e0320413fe/",
    "reserved_resources_record": [
        "/pulp/api/v3/repositories/rpm/rpm/17e07bde-cdaf-485b-9f27-5316da1c20da/",
        "shared:/pulp/api/v3/remotes/rpm/rpm/eae954f5-3387-43c9-a58f-325ded40354c/"
    ],
    "started_at": "2022-03-09T18:03:37.296968Z",
    "state": "failed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/c1a4b937-d196-4505-a3d3-447a90223152/"
}

Traceback formatted:

  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 377, in _perform_task
    result = func(*args, **kwargs)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 544, in synchronize
    repo_version = dv.create() or repo.latest_version()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
    loop.run_until_complete(pipeline)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
    await asyncio.gather(*futures)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
    await self.run()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 249, in run
    await sync_to_async(Artifact.objects.bulk_get_or_create)(
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/asgiref/sync.py", line 444, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/tasks.py", line 455, in wait_for
    return await fut
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/asgiref/sync.py", line 486, in thread_handler
    return func(*args, **kwargs)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/content.py", line 86, in bulk_get_or_create
    return super().bulk_create(objs, batch_size=batch_size)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 502, in bulk_create
    returned_columns = self._batched_insert(
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1287, in _batched_insert
    inserted_rows.extend(self._insert(
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1415, in execute_sql
    for sql, params in self.as_sql():
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django_readonly_field/compiler.py", line 31, in as_sql
    return super(ReadonlySQLCompilerMixin, self).as_sql()
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1358, in as_sql
    value_rows = [
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1310, in pre_save_val
    return field.pre_save(obj, add=True)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 76, in pre_save
    return super().pre_save(model_instance, add)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/files.py", line 302, in pre_save
    file.save(file.name, file.file, save=False)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/files.py", line 89, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/files/storage.py", line 54, in save
    return self._save(name, content)
  File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/storage.py", line 94, in _save
    os.chmod(full_path, self.file_permissions_mode)

Permissions of the directory with that artifact and some others with similar beginning filename:

# ls -lad /var/lib/pulp/media/artifact/00/{,9f*} 
drwxr-xr-x. 2 pulp   pulp   94208 Mar  8 01:33 /var/lib/pulp/media/artifact/00/
-rw-rw-r--. 1 apache pulp     354 Nov 12  2020 /var/lib/pulp/media/artifact/00/9f12969b8e86ea4f15b803c160c721f7f9426c8b3805deccdaf2d9c09bdc46
-rw-------. 1 pulp   pulp 3095704 Feb  2 07:26 /var/lib/pulp/media/artifact/00/9f1e5fdfe0287c2d456390aacab52be52e77247292e285d5caa965c2330dab
-rw-r--r--. 1 pulp   pulp  129900 Jul 14  2021 /var/lib/pulp/media/artifact/00/9f2e85d869301681b61117ede92ac669c927746ec11e38fdb5beef212f81e3
-rw-rw-r--. 1 apache pulp 1206016 May 12  2020 /var/lib/pulp/media/artifact/00/9f498f5bbadf4103438c37fed783f3ed3f8fdcfd3e7e6de1d6554337da0760
-rw-rw-r--. 1 apache pulp  268204 May 22  2020 /var/lib/pulp/media/artifact/00/9f6dc27ef561a73c03e66fa06bf651205dfd51d51f5c7d34e25ee7411e0d1d

I hope this helps. I am not sure, yet, if I stick with 4.3 this time or I revert back.

9

@gvde This is a migrated system, yes? I’m pretty sure these permissions are supposed to be uniform, and probably the files owned by apache came from Pulp 2… could be a migration bug, which would explain why I can’t reproduce.

10

Yes. Started with Katello 3.15 I think, i.e. with pulp2 and migrated eventually to pulp3.

That could be. The migration steps in the 3.18 docs only set up uniform group and group permissions on everything in the pulp2 content directory.

The artifact causing the problem has owner apache, i.e. if the pulpcore worker running as user pulp tries to change permissions on that file it’ll fail with ‘Operation not permitted’.

Looking at the dates of the files listed it seems everything should be owner/group pulp.pulp and permission 0600?

11

On my fresh install (never 2to3 migrated), the permissions look like this:

# ls -al /var/lib/pulp/media/artifact/00/
total 10372
drwxr-xr-x.   2 pulp pulp    4096 Mar 10 12:04 .
drwxr-xr-x. 257 pulp pulp    8192 Mar 10 12:04 ..
-rw-------.   1 pulp pulp   13040 Mar 10 12:03 655717ec2494a953b1c3f477bc86b080760402b72a926057076144aa8b9c41
-rw-------.   1 pulp pulp 1381996 Mar 10 11:53 9767e4fcb269500c8b2ebe0d2cf7f818b9774858019cbed6c325d7c3e73596
-rw-------.   1 pulp pulp  686508 Mar 10 11:54 b9e63e287f45300d4a4f59b6b88e25918443c932ae3e5845d5761ae193c530
-rw-------.   1 pulp pulp 8515480 Mar 10 12:04 d0de456134668f41bd9ea308a076bc0a6a805180445af8a37209d433f41efe

So yes, I would say you are right.

12

Our system was migrated from pulp2 to pulp3 and the majority of files are owned by apache rather than pulp and have the wrong permissions as well

# ls -la /var/lib/pulp/media/artifact/00
total 1203204
drwxr-xr-x   2 pulp   pulp     32768 Feb 21 10:39 .
drwxr-xr-x 258 pulp   pulp      4096 Jun 22  2021 ..
-rw-rw-r--   1 apache pulp     42540 Mar 27  2020 0149f21cbf7579915c09ea2f659c357395c0da4b9a286c4de09df558098f7b
-rw-------   1 pulp   pulp    125729 Oct 26 08:18 0225aca00d646eb11e18344d3b8d97eb59eaa30fadef186d2bc11524570c1c
-rw-rw-r--   1 apache pulp     71284 Oct 21  2020 047c0995e145e37da62cc328a1566c494d3ef7298450494f23f9f1dfcaa543
-rw-rw-r--   1 apache pulp    343320 Nov 18  2020 063f470db329bdbf0defd2b9622d72db94830b9c0a108b7646f6acb3316563
-rw-rw-r--   1 apache pulp    364064 Mar 27  2020 0787002b6a9fa53a6818a92bbd82d75c0deec343540dc08a66ba630fa1e8e8
-rw-rw-r--   1 apache pulp     34432 Mar 27  2020 085cd8333fafa109179e0d672bb352b10dedd4eef3876fc0835833bafc8de0
-rw-rw-r--   1 apache pulp   5653596 Mar 27  2020 096c7131af7b7f1e7120bfb01651356a15d94f13e38494ccfa07e86af8fef6
-rw-rw-r--   1 apache pulp     13720 Mar 27  2020 0a434f3dc587fcad9a5e566391daedd4f78b38f48b37c59103d0dd51f7610b
-rw-rw-r--   1 apache pulp     46980 Dec 18  2020 0c783ad67a44fbc26df8ad81ecd49cdb80c72bac2e0468445fe1454d75a522
-rw-rw-r--   1 apache pulp     72616 Mar 27  2020 0c82d22234b0c7817153d85ba7a4a66e2b335173ab0fff97178f18a2684dca
-rw-rw-r--   1 apache pulp     76008 Oct 21  2020 0cb20f305398259ef2c1bef9919d6cd7abfae7492f3a75acf25c2ea26d266e
-rw-rw-r--   1 apache pulp     95224 Nov 12  2020 0ea54e38c1c408fcf9548fe7d830eb475e867dc698d6f2f5aa00850c4bd67b
-rw-rw-r--   1 apache pulp     98312 Apr 27  2020 0fd1719737652c9766ebfbb9d8acd63b4c854bb62f4738f3f8ee23eab188f7
-rw-rw-r--   1 apache pulp     57156 Mar 27  2020 0ff36975b410e23a092f6ab0b0b159dab993325d9a7b45565424c6333d326b
-rw-rw-r--   1 apache pulp    952356 Mar 27  2020 119100dc629a7357cf5847e231dc0523e24152e40ae7b9e85660d6e45bb650
-rw-rw-r--   1 apache pulp   2631124 Mar 27  2020 136ed06ddba50e2a66e0241fcfe9085634c5dda843984e2dc9ed87b3928472
-rw-rw-r--   1 apache pulp     56464 Mar 27  2020 164893c28d09401a9fd295082b4ee1e3d4201cede1b65c17fd08fe9322304c
-rw-rw-r--   1 apache pulp   7995824 Oct 21  2020 167451bfc93d622557ee62240a7f90d35e11e21fac2e953f83a9325f3bca28
-rw-r--r--   1 pulp   pulp    182840 Aug  4  2021 16a44310db36215e198957df42b0bdf3fc6098673e2347e4bb789e61b06e6e
-rw-rw-r--   1 apache pulp     65414 Feb 22  2021 1abafa991740446acf0ba79fa50c401b5555a081ce793f72c904f21e8ddddf
-rw-rw-r--   1 apache pulp    740492 Oct 21  2020 1db24815b09f65a0c60cceb4f436cdf71dc9e2742e0af358b378ce4b628780
-rw-rw-r--   1 apache pulp    289120 May  6  2020 21110620b0dce48b05ab0359cc22991cd701f058a1b8099e650c6272081873
-rw-rw-r--   1 apache pulp     23508 Apr 27  2020 2126fe84a7207f9ee184f2acd2618da10c2be4c49ab3b0248bfe2eddb90199
-rw-rw-r--   1 apache pulp     33788 Oct 21  2020 213fd7804aad49be9b2f9e0d949cc3a2c93a32b16ab958c4bf216eb78549c1
-rw-rw-r--   1 apache pulp 115758573 Apr 23  2020 21cb8ca0978f52a8f0ff2d759eb695001a8632cdd8cd4b782de167d66603c3
-rw-rw-r--   1 apache pulp    168484 Mar 27  2020 23356012efcb7eadd3be350868f8113ba41062ec20181d44a37d233e0e0001
-rw-rw-r--   1 apache pulp   2353520 Oct 21  2020 243e95aec094be1112a35536ad5d570d5d59423e2ad1de9cb5beeef0eba075
-rw-rw-r--   1 apache pulp     12756 Mar 27  2020 269ac5b12e96f874e9cee7f5090a80b97bdce1564a7a01819ae9f516559ab9
-rw-rw-r--   1 apache pulp     13730 Feb 27  2021 2a53b5c093bf8f171866dbd27aa667bf2407a400706ebd5365bc9b268d16ab
-rw-rw-r--   1 apache pulp     81540 Apr 29  2020 2b73881adce7c1097c6eacae3126bbc5a8291bcae3287bfd5c6ebd54d3df8a
-rw-rw-r--   1 apache pulp     30012 Mar 27  2020 2c83c87fbc85eea1404fbacb3a4afcea547a55933348b5d5d015dded765a9b
-rw-rw-r--   1 apache pulp     58720 Mar 27  2020 2e52327d7f3d35a74e802763af005e6f3e7c7296601da6c99feaa8234aff16
-rw-r--r--   1 pulp   pulp     99708 Feb 21 10:39 2ea2460b9e864c85bf1ed8a108e4203f562116643f19001914afc7b07ff4d5
13

So a recursive chown on the artifact folder should straighten out the problem with the owner which would allow pulpcore-workers to do chmod on files…

# chown -hR pulp.pulp /var/lib/pulp/media/artifact/

The permissions on files (600, 644, 664) shouldn’t be an issue here…

14

pulp-2to3-migration tries to make hardlinks for artifacts if it can, in order to not double the storage needed when migrating. That’s how you end up with pulp2 perms/users on pulp3 content post-migration, is my guess - your pulp2 and pulp3 storage live on the same volume. We prob need a post-migration step to do the suggested chown; you don’t want to do it until you’re ready to “turn off” pulp2, so as not to impact pulp2 Doing It’s Thing while you’re still preparing Pulp3.

1 Like
15

I am on 4.3 now, so migration was a long time ago. This error, though, is new. It does not happen with 4.2. So I guess it must be related with pulpcore 3.16 or something around that, that it tries to do a chmod on a file…

I have changed owners and the following sync just finished without problems.

16

Hm, good point. I am def surprised to see this being a problem “suddenly” - I would have expected us to hit this A Long Time Ago. “Just upgraded from core/3.15 to core/3.16” is def a big hint, though.

17

pulpcore 3.14 to 3.16 to be exact.

18

@dralley Is this now a pulpcore or a katello problem? How do we get this fixed? Should foreman-installer do a recursive chown on the pulp directory? Or is this something pulpcore should do as part of the migrations? Or how? I guess, a recursive chown can take a long time depending on the number of files in the directories and the speed of the disk, thus it’s probably not an option to do this during each upgrade…

19

We’ve been discussing that, the best place to put it is most probably foreman-maintain content switchover. So, only once, right at the end when Pulp 2 is being shut off.

It could also be done as a separate manual step, but it’s better to avoid making the migration process any more complicated than it is already

20

That’s useful for those you are on 3.18 running pulp2, but doesn’t help those you are running pulp3 and are on 4.0+. This chmod problem for instance only occurs on 4.3/pulpcore 3.16.

At a minimum, katello upgrade instructions needed to be updated, e.g. to include a simple find to see if there is a potential issue:

# find /var/lib/pulp/media/ \! -user pulp -ls

and then instruct the user to do the chown.

But I kind of don’t like these upgrade instructions where users have to fix things, that got broken somehow, when it could be easily scripted…