Problem:
we have setup a Katello docker repositories for our project, with its upstream repository URL pointing to private Azure docker registry containing the master files. For credential we are using username and token issued for Azure registry.
We are trying to use sync function of Katello to download the containers from the Azure docker registry where the master files are staged. However, we are getting the following error from katello which indicates that received bearer token from Azure is not valid. I have copied the error logs below.
Note1: we experimented the same thing with dockerhub private repository and we were able to successfully to synchronize. Is there a known OAUTH compatibility issue between Katello and Azure registry?
Note2: we are able to use the same username/token from Azure registry to successfully download the files using docker pull command. So the credentials are good.
Expected outcome:
Upon clicking sync button on Foreman/Katello GUI, we expect katello authenticate to Azure registry using the specified token and username, and download the specified containers .
Foreman and Proxy versions:
Foreman version : 2.2.1 and Katello version: 3.16
Foreman and Proxy plugin versions:
Distribution and version:
Other relevant data:
here is the error log we obtained from Foreman/Katello GUI
Actions::Pulp3::Repository::Sync
Input:
{“repo_id”=>20,
“smart_proxy_id”=>1,
“options”=> {“smart_proxy_id”=>1, “repo_id”=>20, “source_url”=>nil, “options”=>{}},
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“current_request_id”=>nil,
“current_timezone”=>“Asia/Shanghai”,
“current_user_id”=>4,
“current_organization_id”=>3,
“current_location_id”=>2}
Output:
{“pulp_tasks”=>
[{“pulp_href”=>"/pulp/api/v3/tasks/6ab97651-99b4-42b3-818e-509d03733428/",
“pulp_created”=>“2020-11-23T04:51:48.309+00:00”,
“state”=>“failed”,
“name”=>“pulp_container.app.tasks.synchronize.synchronize”,
“started_at”=>“2020-11-23T04:51:48.362+00:00”,
“finished_at”=>“2020-11-23T04:51:56.316+00:00”,
“error”=> {“traceback”=>
" File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 822, in perform_job\n" +
" rv = job.perform()\n" +
" File “/usr/lib/python3.6/site-packages/rq/job.py”, line 605, in perform\n" +
" self._result = self._execute()\n" +
" File “/usr/lib/python3.6/site-packages/rq/job.py”, line 611, in _execute\n" +
" return self.func(*self.args, **self.kwargs)\n" +
" File “/usr/lib/python3.6/site-packages/pulp_container/app/tasks/synchronize.py”, line 44, in synchronize\n" +
" dv.create()\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 149, in create\n" +
" loop.run_until_complete(pipeline)\n" +
" File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete\n" +
" return future.result()\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 209, in create_pipeline\n" +
" await asyncio.gather(*futures)\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call\n" +
" await self.run()\n" +
" File “/usr/lib/python3.6/site-packages/pulp_container/app/tasks/sync_stages.py”, line 59, in run\n" +
" await list_downloader.run(extra_data={‘repo_name’: repo_name})\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 221, in run\n" +
" return await self._run(extra_data=extra_data)\n" +
" File “/usr/lib/python3.6/site-packages/backoff/_async.py”, line 133, in retry\n" +
" ret = await target(*args, **kwargs)\n" +
" File “/usr/lib/python3.6/site-packages/pulp_container/app/downloaders.py”, line 75, in _run\n" +
" await self.update_token(response_auth_header, this_token, repo_name)\n" +
" File “/usr/lib/python3.6/site-packages/pulp_container/app/downloaders.py”, line 133, in update_token\n" +
" self.registry_auth[‘bearer’] = json.loads(token_data)[‘token’]\n",
“description”=>"‘token’"},
“worker”=>"/pulp/api/v3/workers/c67e2a9c-5b96-4949-a0e1-fa4d92fa92c6/",
“progress_reports”=>
[{“message”=>“Downloading tag list”,
“code”=>“downloading.tag_list”,
“state”=>“failed”,
“total”=>1,
“done”=>0}, {“message”=>“Downloading Artifacts”,
“code”=>“downloading.artifacts”,
“state”=>“canceled”,
“done”=>0}, {“message”=>“Associating Content”,
“code”=>“associating.content”,
“state”=>“canceled”,
“done”=>0}],
“created_resources”=>,
“reserved_resources_record”=>
["/pulp/api/v3/repositories/container/container/298528be-628a-4f3a-a8f9-d7b2118599a0/",
“/pulp/api/v3/remotes/container/container/203ef170-b226-4752-a2d6-251fe7b87bba/”]}],
“create_version”=>true,
“poll_attempts”=>{“total”=>10, “failed”=>1}}
Exception:
Katello::Errors::Pulp3Error: ‘token’