**Version**
After upgrading from Katello 4.2 to 4.3, some users are encounter…ing sync problems with some repositories. A 500 error is triggered when the /repositories/.../sync/ endpoint is used. The error seems to be related to the client_cert / client_key / ca_cert values.
```
Feb 24 11:28:57 foreman pulpcore-api: pulp [e2601bd6-70a5-4a87-ace3-a386280e3e30]: django.request:ERROR: Internal Server Error: /pulp/api/v3/remotes/rpm/rpm/89e5b587-2295-4f1d-a49d-c0e8ed518c67/
Feb 24 11:28:57 foreman pulpcore-api: Traceback (most recent call last):
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 96, in _get_unverified_token_data
Feb 24 11:28:57 foreman pulpcore-api: data = base64.urlsafe_b64decode(token)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/base64.py", line 133, in urlsafe_b64decode
Feb 24 11:28:57 foreman pulpcore-api: return b64decode(s)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/base64.py", line 87, in b64decode
Feb 24 11:28:57 foreman pulpcore-api: return binascii.a2b_base64(s)
Feb 24 11:28:57 foreman pulpcore-api: binascii.Error: Incorrect padding
Feb 24 11:28:57 foreman pulpcore-api: During handling of the above exception, another exception occurred:
Feb 24 11:28:57 foreman pulpcore-api: Traceback (most recent call last):
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
Feb 24 11:28:57 foreman pulpcore-api: response = get_response(request)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
Feb 24 11:28:57 foreman pulpcore-api: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Feb 24 11:28:57 foreman pulpcore-api: return view_func(*args, **kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
Feb 24 11:28:57 foreman pulpcore-api: return self.dispatch(request, *args, **kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
Feb 24 11:28:57 foreman pulpcore-api: response = self.handle_exception(exc)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
Feb 24 11:28:57 foreman pulpcore-api: self.raise_uncaught_exception(exc)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Feb 24 11:28:57 foreman pulpcore-api: raise exc
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
Feb 24 11:28:57 foreman pulpcore-api: response = handler(request, *args, **kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/viewsets/base.py", line 470, in partial_update
Feb 24 11:28:57 foreman pulpcore-api: return self.update(request, *args, **kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/viewsets/base.py", line 452, in update
Feb 24 11:28:57 foreman pulpcore-api: instance = self.get_object()
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/generics.py", line 96, in get_object
Feb 24 11:28:57 foreman pulpcore-api: obj = get_object_or_404(queryset, **filter_kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/generics.py", line 19, in get_object_or_404
Feb 24 11:28:57 foreman pulpcore-api: return _get_object_or_404(queryset, *filter_args, **filter_kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/shortcuts.py", line 76, in get_object_or_404
Feb 24 11:28:57 foreman pulpcore-api: return queryset.get(*args, **kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 431, in get
Feb 24 11:28:57 foreman pulpcore-api: num = len(clone)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 262, in __len__
Feb 24 11:28:57 foreman pulpcore-api: self._fetch_all()
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
Feb 24 11:28:57 foreman pulpcore-api: self._result_cache = list(self._iterable_class(self))
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 68, in __iter__
Feb 24 11:28:57 foreman pulpcore-api: for row in compiler.results_iter(results):
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
Feb 24 11:28:57 foreman pulpcore-api: value = converter(value, expression, connection)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 104, in from_db_value
Feb 24 11:28:57 foreman pulpcore-api: return force_str(self._fernet.decrypt(force_bytes(value)))
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 75, in decrypt
Feb 24 11:28:57 foreman pulpcore-api: timestamp, data = Fernet._get_unverified_token_data(token)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 98, in _get_unverified_token_data
Feb 24 11:28:57 foreman pulpcore-api: raise InvalidToken
Feb 24 11:28:57 foreman pulpcore-api: cryptography.fernet.InvalidToken
Feb 24 11:28:57 foreman pulpcore-api: pulp [e2601bd6-70a5-4a87-ace3-a386280e3e30]: - - [24/Feb/2022:10:28:57 +0000] "PATCH /pulp/api/v3/remotes/rpm/rpm/89e5b587-2295-4f1d-a49d-c0e8ed518c67/ HTTP/1.1" 500 145 "-" "OpenAPI-Generator/3.16.1/ruby"
Feb 24 11:28:58 foreman qdrouterd: SERVER (info) [C464018] Connection from 10.2.1.176:50898 (to :5647) failed: amqp:resource-limit-exceeded local-idle-timeout expired
Feb 24 11:28:58 foreman qdrouterd: ROUTER_CORE (info) [C464018][L1196864] Link detached: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Feb 24 11:28:58 foreman qdrouterd: ROUTER_CORE (info) [C464018] Connection Closed
Feb 24 11:28:58 foreman qdrouterd: ROUTER_CORE (info) [C4][L1196865] Link detached: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Feb 24 11:28:58 foreman pulpcore-api: pulp [e2601bd6-70a5-4a87-ace3-a386280e3e30]: django.request:ERROR: Internal Server Error: /pulp/api/v3/repositories/rpm/rpm/b4a448e8-896c-4fec-8ee2-fcdf4307f2fd/sync/
Feb 24 11:28:58 foreman pulpcore-api: Traceback (most recent call last):
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 96, in _get_unverified_token_data
Feb 24 11:28:58 foreman pulpcore-api: data = base64.urlsafe_b64decode(token)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/base64.py", line 133, in urlsafe_b64decode
Feb 24 11:28:58 foreman pulpcore-api: return b64decode(s)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/base64.py", line 87, in b64decode
Feb 24 11:28:58 foreman pulpcore-api: return binascii.a2b_base64(s)
Feb 24 11:28:58 foreman pulpcore-api: binascii.Error: Incorrect padding
Feb 24 11:28:58 foreman pulpcore-api: During handling of the above exception, another exception occurred:
Feb 24 11:28:58 foreman pulpcore-api: Traceback (most recent call last):
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
Feb 24 11:28:58 foreman pulpcore-api: response = get_response(request)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
Feb 24 11:28:58 foreman pulpcore-api: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Feb 24 11:28:58 foreman pulpcore-api: return view_func(*args, **kwargs)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
Feb 24 11:28:58 foreman pulpcore-api: return self.dispatch(request, *args, **kwargs)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
Feb 24 11:28:58 foreman pulpcore-api: response = self.handle_exception(exc)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
Feb 24 11:28:58 foreman pulpcore-api: self.raise_uncaught_exception(exc)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Feb 24 11:28:58 foreman pulpcore-api: raise exc
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
Feb 24 11:28:58 foreman pulpcore-api: response = handler(request, *args, **kwargs)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/viewsets.py", line 130, in sync
Feb 24 11:28:58 foreman pulpcore-api: serializer.is_valid(raise_exception=True)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/serializers.py", line 220, in is_valid
Feb 24 11:28:58 foreman pulpcore-api: self._validated_data = self.run_validation(self.initial_data)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/serializers.py", line 419, in run_validation
Feb 24 11:28:58 foreman pulpcore-api: value = self.to_internal_value(data)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/serializers.py", line 476, in to_internal_value
Feb 24 11:28:58 foreman pulpcore-api: validated_value = field.run_validation(primitive_value)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/relations.py", line 153, in run_validation
Feb 24 11:28:58 foreman pulpcore-api: return super().run_validation(data)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/fields.py", line 568, in run_validation
Feb 24 11:28:58 foreman pulpcore-api: value = self.to_internal_value(data)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/relations.py", line 373, in to_internal_value
Feb 24 11:28:58 foreman pulpcore-api: return self.get_object(match.view_name, match.args, match.kwargs)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/serializers/base.py", line 322, in get_object
Feb 24 11:28:58 foreman pulpcore-api: return super().get_object(*args, **kwargs).cast()
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/relations.py", line 318, in get_object
Feb 24 11:28:58 foreman pulpcore-api: return queryset.get(**lookup_kwargs)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 431, in get
Feb 24 11:28:58 foreman pulpcore-api: num = len(clone)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 262, in __len__
Feb 24 11:28:58 foreman pulpcore-api: self._fetch_all()
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
Feb 24 11:28:58 foreman pulpcore-api: self._result_cache = list(self._iterable_class(self))
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 68, in __iter__
Feb 24 11:28:58 foreman pulpcore-api: for row in compiler.results_iter(results):
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
Feb 24 11:28:58 foreman pulpcore-api: value = converter(value, expression, connection)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 104, in from_db_value
Feb 24 11:28:58 foreman pulpcore-api: return force_str(self._fernet.decrypt(force_bytes(value)))
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 75, in decrypt
Feb 24 11:28:58 foreman pulpcore-api: timestamp, data = Fernet._get_unverified_token_data(token)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 98, in _get_unverified_token_data
Feb 24 11:28:58 foreman pulpcore-api: raise InvalidToken
Feb 24 11:28:58 foreman pulpcore-api: cryptography.fernet.InvalidToken
Feb 24 11:28:58 foreman pulpcore-api: pulp [e2601bd6-70a5-4a87-ace3-a386280e3e30]: - - [24/Feb/2022:10:28:58 +0000] "POST /pulp/api/v3/repositories/rpm/rpm/b4a448e8-896c-4fec-8ee2-fcdf4307f2fd/sync/ HTTP/1.1" 500 145
```
>
> What did notice, though, is that on the old 4.2 server with pulpcore 3.14 all those remotes have an empty username and password set, i.e. an empty string “” and not NULL like many others. However, the list of remotes on the old server with empty string username is longer than the affected ones on my 4.3 server.
>
> So it seems to be an issue with the database migration from 3.14 to 3.16 and encryption of some fields in the database. The migration/encryption seems to fail is some cases and the original content remains in the database which makes decryption failed. This is also true for my affected rhel repository: I can see in the database that the remote contains the RSA private key in client_key, unlike the other rhel repos which contain some base64 encoded string in the client_key fields.
>
> Thus the problem is that for some rows in the database (at least the core_remote table) encryption of the fields client_key, username, password, (proxy_username, proxy_password, too, I guess) fails during migration from 3.14 to 3.16 leaving the original content in the database. This leads to decryption failure in 3.16.
>
> For whatever reason only a few repos are affected, mostly those which have empty strings for username, password instead of NULL like many others. However, even with empty strings for some remotes the encryption works while it doesn’t for others.
**Describe the bug**
API calls to trigger a sync return a 500 error for some remotes
**To Reproduce**
Unclear
**Expected behavior**
A sync task is correctly created, no 500 error.
**Additional context**
Discourse discussion: https://community.theforeman.org/t/pulprpmclient-apierror-http-500-during-sync-repository/27480/13?u=dralley