I guess this should also be filed as katello bug because anyone with more than 100 repositories/remotes in the database should probably defer upgrading from 4.2 to 4.3 until this has been fixed…
anyone with more than 100 repositories/remotes in the database should probably defer upgrading from 4.2 to 4.3 until this has been fixed…
I’m in general agreement with this
I don’t know if 100 was actually hit in my environment, but I am getting the same issue, 1 foreman server + 1 smart proxy, maybe 60 hosts. Repos are less than 30.
If it’s less then 100 remote repositories I can’t be really the exact same issue. You can easily check the total number in the database as root on your foreman server:
[root@foreman ~]# sudo -u postgres psql pulpcore -c 'select count(*) from core_remote;'
could not change directory to "/root": Permission denied
count
-------
117
(1 row)
If that’s less than 100 (and you didn’t remove lots of repositories since the upgrade to 4.3 of course) I’d say it’s something else. If you have the same cryptography.fernet.InvalidToken although you had less than 100 rows in the table during migration from 4.2 to 4.3 it would require a closer look.
@gvde My count is 35. Indeed, I have removed a few repositories (got rid of CentOS 8) but I would reckon about 15 or so.
Here is what I see when syncing the smart proxy:
Mar 9 20:16:38 proxy pulpcore-api: pulp [fd038a28-78a8-4d5e-8ced-f5acfb108681]: - - [09/Mar/2022:18:16:38 +0000] "GET /pulp/api/v3/remotes/rpm/rpm/?name=1-_CV_Subversion_CentOS_7-Production-1236c46e-6338-4e72-b7fa-598b57a91631 HTTP/1.1" 500 145 "-" "OpenAPI-Generator/3.16.1/ruby"
Mar 9 20:16:38 proxy pulpcore-api: pulp [fd038a28-78a8-4d5e-8ced-f5acfb108681]: django.request:ERROR: Internal Server Error: /pulp/api/v3/remotes/rpm/rpm/
Mar 9 20:16:38 proxy pulpcore-api: Traceback (most recent call last):
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
Mar 9 20:16:38 proxy pulpcore-api: response = get_response(request)
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
Mar 9 20:16:38 proxy pulpcore-api: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Mar 9 20:16:38 proxy pulpcore-api: return view_func(*args, **kwargs)
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
Mar 9 20:16:38 proxy pulpcore-api: return self.dispatch(request, *args, **kwargs)
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
Mar 9 20:16:38 proxy pulpcore-api: response = self.handle_exception(exc)
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/proxy/views.py", line 469, in handle_exception
Mar 9 20:16:38 proxy pulpcore-api: self.raise_uncaught_exception(exc)
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Mar 9 20:16:38 proxy pulpcore-api: raise exc
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
Mar 9 20:16:38 proxy pulpcore-api: response = handler(request, *args, **kwargs)
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/mixins.py", line 40, in list
Mar 9 20:16:38 proxy pulpcore-api: page = self.paginate_queryset(queryset)
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/generics.py", line 171, in paginate_queryset
Mar 9 20:16:38 proxy pulpcore-api: return self.paginator.paginate_queryset(queryset, self.request, view=self)
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/pagination.py", line 395, in paginate_queryset
Mar 9 20:16:38 proxy pulpcore-api: return list(queryset[self.offset:self.offset + self.limit])
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 262, in __len__
Mar 9 20:16:38 proxy pulpcore-api: self._fetch_all()
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
Mar 9 20:16:38 proxy pulpcore-api: self._result_cache = list(self._iterable_class(self))
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 68, in __iter__
Mar 9 20:16:38 proxy pulpcore-api: for row in compiler.results_iter(results):
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
Mar 9 20:16:38 proxy pulpcore-api: value = converter(value, expression, connection)
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 104, in from_db_value
Mar 9 20:16:38 proxy pulpcore-api: return force_str(self._fernet.decrypt(force_bytes(value)))
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 75, in decrypt
Mar 9 20:16:38 proxy pulpcore-api: timestamp, data = Fernet._get_unverified_token_data(token)
Mar 9 20:16:38 proxy pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 101, in _get_unverified_token_data
Mar 9 20:16:38 proxy pulpcore-api: raise InvalidToken
Mar 9 20:16:38 proxy pulpcore-api: cryptography.fernet.InvalidToken
Mar 9 20:16:38 proxy pulpcore-api: pulp [fd038a28-78a8-4d5e-8ced-f5acfb108681]: - - [09/Mar/2022:18:16:38 +0000] "GET /pulp/api/v3/remotes/rpm/rpm/?name=1-CentOS_7-Production-1236c46e-6338-4e72-b7fa-598b57a91631 HTTP/1.1" 500 145 "-" "OpenAPI-Generator/3.16.1/ruby"
O.K. That’s on the proxy. Run the psql command on the proxy and it’ll show a much larger number. It’s the same problem. The proxy loads each publication from the main server, i.e. it has remotes for each content view in each lifecycle environment.
That means for the content proxy, the whole problem is much bigger…
236 … doh … 
Katello updates a remote with the latest remote configuration options during every sync. That’s done in the RefreshRemote action. Are these bad remote options not being updated when Katello runs RefreshRemote? If not then that may be part of the issue as well.
What we’re looking for is the first step on a repo sync dynflow task:
Clicking on it, you should see a pulp task:
I don‘t quite understand: should Katello update the username, password, client key during each sync? I don’t think it does. If it would, the sync should work just fine and double encryption or missing encryption in the pulp database shouldn’t matter, because it would be overwritten immediately… Instead, each sync fails.
We call the Pulp remote update API through the RefreshRemote task that @Justin_Sherrill pointed out above. It should be updating all of those fields every time. We don’t update the fields for remotes immediately when remote configuration is changed in Katello, rather we defer it until sync time.
So, if these fields are still bad, then either the update isn’t working in Pulp, or there’s a Katello bug that’s skipping the update.
Ah okay, the username and password aren’t updated if there aren’t any set for a repository. That is likely related. katello/repository.rb at master · Katello/katello · GitHub
The keys should be updated, however: katello/repository.rb at master · Katello/katello · GitHub
This PR may help if anyone is able to test it: Fixes #34619 - Remotes should have username and password cleared out if set blank by ianballou · Pull Request #10018 · Katello/katello · GitHub
Ignore the test files when patching.
I have checked the repositories I have problems with (mentioned above):
https://download.postgresql.org/pub/repos/yum/common/redhat/rhel-7-x86_64/
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os
http://mirror.centos.org/centos/7/storage/x86_64/samba-411/
http://mirror.centos.org/centos/7/cloud/x86_64/openstack-queens/
http://mirror.centos.org/centos/7/cloud/x86_64/openstack-rocky/
http://mirror.centos.org/centos/7/cloud/x86_64/openstack-stein/
https://download.fedoraproject.org/pub/epel/8/Everything/x86_64/
All those except the redhat repo have the problem that they an empty string username and password in the pulpcore core_remote table, i.e. they did not get encrypted. They fail each time.
I have also checked the redhat repository and it does have a client_key which is also not encrypted. It’s actually the first RefreshRemote that is failing:
So it looks to me as if the update of the remote is already failing…
Ah that’s interesting, what is the error on the Pulp side? I wonder if that’s related to what @dralley is working on.
Yes. I think it’s related. I guess, before you can change the username, etc. in the pulpcore remote, it tries to load it first and thus automatically tries to decrypt the encrypted fields, which fails.
The error message is the exact message as mentioned above, e.g. in the first post…
Looks like the Pulp side of this should be taken care of soon. We’ll make sure the changes get into the Katello repo as soon as possible. I’ll add an entry to the docs to run the repair command for users who migrated to Pulpcore 3.16 without the fix.
1 Like
Don’t forget that this also seems to affect content proxies.
2 Likes
Thanks for the reminder. That’ll be added to the docs as well.