I think I have found something in the database which may be the cause for this issue. From pulpcore 3.14 to 3.16 it seems some fields in the database are encrypted with a symmetric key in the file /etc/pulp/certs/database_fields.symmetric.key. The migration seems to be broken for some remotes:
pulpcore=# select pulp_id, name, url, username, length(username) from core_remote where username is not null and length(username) = 0;
pulp_id | name | url | username | length
--------------------------------------+---------------------------+----------------------------------------------------------------------------+----------+--------
e49f9e02-43c5-41ac-8ed3-9a491e31c609 | pgdg-common-el7-775402 | https://download.postgresql.org/pub/repos/yum/common/redhat/rhel-7-x86_64/ | | 0
1597297f-6333-4585-bb4e-7eeccf22be28 | samba-411-76723158 | http://mirror.centos.org/centos/7/storage/x86_64/samba-411/ | | 0
89758522-8af3-44fc-8ec0-26615664494c | openstack-rocky-7674227 | http://mirror.centos.org/centos/7/cloud/x86_64/openstack-rocky/ | | 0
27a18f61-21da-449a-965b-c68938e7ba34 | openstack-stein-76759204 | http://mirror.centos.org/centos/7/cloud/x86_64/openstack-stein/ | | 0
84e1651a-7fb0-4618-ae60-5bd7adb6b93f | openstack-queens-76733362 | http://mirror.centos.org/centos/7/cloud/x86_64/openstack-queens/ | | 0
57e722fc-ecf9-4daa-babb-249ca8d87668 | AppStream_x86_64-1715630 | http://mirror.centos.org/centos/8/AppStream/x86_64/os/ | | 0
6f1837c9-d88f-42c2-861d-ff7e931885b9 | epel-1779394 | https://download.fedoraproject.org/pub/epel/8/Everything/x86_64/ | | 0
(7 rows)
I haven’t checked all fields in the database which seem to be encrypted now.
What did notice, though, is that on the old 4.2 server with pulpcore 3.14 all those remotes have an empty username and password set, i.e. an empty string “” and not NULL like many others. However, the list of remotes on the old server with empty string username is longer than the affected ones on my 4.3 server.
So it seems to be an issue with the database migration from 3.14 to 3.16 and encryption of some fields in the database. The migration/encryption seems to fail is some cases and the original content remains in the database which makes decryption failed. This is also true for my affected rhel repository: I can see in the database that the remote contains the RSA private key in client_key, unlike the other rhel repos which contain some base64 encoded string in the client_key fields.
Thus the problem is that for some rows in the database (at least the core_remote table) encryption of the fields client_key, username, password, (proxy_username, proxy_password, too, I guess) fails during migration from 3.14 to 3.16 leaving the original content in the database. This leads to decryption failure in 3.16.
For whatever reason only a few repos are affected, mostly those which have empty strings for username, password instead of NULL like many others. However, even with empty strings for some remotes the encryption works while it doesn’t for others.
I hope it’s clear what I wrote and at least @dralley understands me…