Proxy credentials display in clear text

Support
,
1

Problem:
When syncing repositories if the http proxy “goes away” the error messages in the logs display the proxy credentials in clear text.

Expected outcome:
All logged credentials should be obfuscated

Foreman and Proxy versions:
foreman.noarch 2.2.1-1.el7
foreman-cli.noarch 2.2.1-1.el7
foreman-debug.noarch 2.2.1-1.el7
foreman-dynflow-sidekiq.noarch 2.2.1-1.el7
foreman-installer.noarch 1:2.2.1-1.el7
foreman-installer-katello.noarch 1:2.2.1-1.el7
foreman-postgresql.noarch 2.2.1-1.el7
foreman-proxy.noarch 2.2.1-1.el7
foreman-selinux.noarch 2.2.1-1.el7
foreman-service.noarch 2.2.1-1.el7

Foreman and Proxy plugin versions:
katello.noarch 3.17.0-1.el7
katello-certs-tools.noarch 2.7.1-2.el7
katello-client-bootstrap.noarch 1.7.5-1.el7
katello-common.noarch 3.17.0-1.el7
katello-debug.noarch 3.17.0-1.el7
katello-default-ca.noarch 1.0-1
katello-selinux.noarch 3.4.0-1.el7
katello-server-ca.noarch 1.0-1

Distribution and version:
CentOS 7.9

Other relevant data:
using hammer repository sync

Error: 503, message=‘Service Unavailable’, url=URL(‘http://username:Passw0rd@proxy.company.com.au:8080’)

Same appears in the Web UI

2

Valid bug, can you share stacktrace so we pick the correct component which needs to be fixed? Maybe you’ll need to turn on DEBUG mode to see it.

3

Hi @lzap,

My proxy is a little flakey and I can’t even rely on it to fail. Will switch DEBUG mode on and expect huge logs. It’s also in a secure environment so I’ll need to redact some of the content of the stacktrace but it should still be ok for you.

Andy

4

Hi @lzap

It’s not a stacktrace but does this help at all?

image
image1461×550 35.7 KB

5

That will do it, thanks. So it’s probably in the exception.message output, it’s being rendered both via CLI and into the web UI.

Mind looking up the exception stacktrace so I can identify exactly the line and more importantly class? We use about 5 HTTP classes to do HTTP calls :wink: You can find this in produciton.log if you grep this message.

6

Anyway, file a issue. This needs to be investigated.

8

DEBUG enabled, trying to reproduce the error now

9

I am trying to simulate this, but HTTP class in Ruby 2.7 does not reveal its URL or proxy credentials. I’d love to see that backtrace to see where it triggers:

require "net/http"
proxy_addr = "localhost"
proxy_port = 3128
Net::HTTP.new('does-not-exist-at-all.com', nil, proxy_addr, proxy_port).start { |http|
  http.request Net::HTTP::Get.new("/")
}

=> #<Net::HTTPServiceUnavailable 503 Service Unavailable readbody=true>
10

Judging from the stacktrace and format of the error message I’d say the error actually originates in pulp and we just propagate the message from it.

1 Like
11

@aruzicka @lzap

Just wondering which proxy setting this relates to. The one in the general tab displays the username and password in clear text, mostly because it’s a single field of the format:

http://user: password@proxy.server.name:port

Or somesuch.

It doesn’t allow obfuscation of any component of the URL.

Not sure if that helps.

Andy

P.S. great time for my proxy to decide to become stable🤬

12

Can you grep that also in Pulp-related log file (that includes also system journal or /var/log/messages)? This could confirm @aruzicka 's thought.

13

Hi @lzap,

You’re right! it’s all over the place in the messages log from pulpcore workers:

Dec 10 10:03:00 foreman-svr dynflow-sidekiq@worker: 2020-12-09T23:33:00.412Z 8510 TID-2xkvka Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork JID-cc67cbe413dbce63e8a08c92 INFO: done: 0.131 sec
Dec 10 10:03:00 foreman-svr dynflow-sidekiq@orchestrator: 2020-12-09T23:33:00.412Z 8466 TID-2wc8sa Dynflow::Executors::Sidekiq::OrchestratorJobs::WorkerDone JID-1a23e2e868a7847bcf0f4175 INFO: start
Dec 10 10:03:00 foreman-svr dynflow-sidekiq@orchestrator: 2020-12-09T23:33:00.414Z 8466 TID-2wc8sa Dynflow::Executors::Sidekiq::OrchestratorJobs::WorkerDone JID-1a23e2e868a7847bcf0f4175 INFO: done: 0.001 sec
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: pulp: rq.worker:ERROR: Traceback (most recent call last):
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: rv = job.perform()
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: self._result = self._execute()
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: return self.func(*self.args, **self.kwargs)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 266, in synchronize
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: dv.create()
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: loop.run_until_complete(pipeline)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: return future.result()
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: await asyncio.gather(*futures)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: await self.run()
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 152, in run
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: pb.done += task.result() # download_count
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 178, in _handle_content_unit
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: await asyncio.gather(*downloaders_for_content)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: download_result = await downloader.run(extra_data=self.extra_data)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: return await self._run(extra_data=extra_data)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py”, line 87, in _run
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response:
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 1012, in aenter
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: self._resp = await self._coro
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 483, in _request
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: timeout=real_timeout
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 523, in connect
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: proto = await self._create_connection(req, traces, timeout)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 856, in _create_connection
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: req, traces, timeout)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 1083, in _create_proxy_connection
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: headers=resp.headers)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: aiohttp.client_exceptions.ClientHttpProxyError: 503, message=‘Service Unavailable’, url=URL(‘http://proxyuser:proxypass@proxysvr.blah.com:8080’)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: Traceback (most recent call last):
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: rv = job.perform()
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: self._result = self._execute()
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: return self.func(*self.args, **self.kwargs)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 266, in synchronize
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: dv.create()
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: loop.run_until_complete(pipeline)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: return future.result()
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: await asyncio.gather(*futures)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: await self.run()
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 152, in run
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: pb.done += task.result() # download_count
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 178, in _handle_content_unit
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: await asyncio.gather(*downloaders_for_content)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: download_result = await downloader.run(extra_data=self.extra_data)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: return await self._run(extra_data=extra_data)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py”, line 87, in _run
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response:
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 1012, in aenter
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: self._resp = await self._coro
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/client.py”, line 483, in _request
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: timeout=real_timeout
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 523, in connect
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: proto = await self._create_connection(req, traces, timeout)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 856, in _create_connection
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: req, traces, timeout)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: File “/usr/lib64/python3.6/site-packages/aiohttp/connector.py”, line 1083, in _create_proxy_connection
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: headers=resp.headers)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: aiohttp.client_exceptions.ClientHttpProxyError: 503, message=‘Service Unavailable’, url=URL(‘http://proxyuser:proxypass@proxysvr.blah.com:8080’)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: pulp: rq.worker:INFO: Cleaning registries for queue: 29117@foreman-svr.blah.com
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: pulp: rq.worker:INFO: 29117@foreman-svr.blah.com: f44c079e-1f6f-418f-862d-3f2750967b59
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: pulp: rq.worker:INFO: 29117@foreman-svr.blah.com: Job OK (f44c079e-1f6f-418f-862d-3f2750967b59)
Dec 10 10:03:02 foreman-svr pulpcore-worker-2: pulp: rq.worker:INFO: 29117@foreman-svr.blah.com: 0e1b15ba-53bb-4b4b-8281-9c96dc2138bd
Dec 10 10:03:02 foreman-svr pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (37c3b1b7-b17d-47da-8c05-83058de07eb1)
Dec 10 10:03:02 foreman-svr pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: 674105cb-89eb-4b5d-89bb-4e0437377d89
Dec 10 10:03:04 foreman-svr pulpcore-worker-2: pulp: rq.worker:INFO: 29117@foreman-svr.blah.com: Job OK (0e1b15ba-53bb-4b4b-8281-9c96dc2138bd)
Dec 10 10:03:04 foreman-svr pulpcore-worker-2: pulp: rq.worker:INFO: 29117@foreman-svr.blah.com: ba50faec-c595-4965-a20e-12d808b5e2eb
Dec 10 10:03:04 foreman-svr pulpcore-worker-2: pulp: rq.worker:INFO: 29117@foreman-svr.blah.com: Job OK (ba50faec-c595-4965-a20e-12d808b5e2eb)
Dec 10 10:03:04 foreman-svr pulpcore-worker-2: pulp: rq.worker:INFO: 29117@foreman-svr.blah.com: 542d88fe-2dbd-4914-89e0-ab1563ac8553

14

Great, please file a bug at https://pulp.plan.io we just display this. Thanks.

15

FYI pulpcore can’t fix, it’s with aiohttp