Intermittent failure regenerating registry Personal Access Token for Podman login

Support
, , ,
1

Problem: User cannot renew ‘Personal Access Tokens’

Expected outcome: Login and pull Docker images

Foreman and Proxy versions: Foreman 3.17.0 / Katello 4.19.0.1

Foreman and Proxy plugin versions:
Plugins:

  1. Name: foreman-tasks
    Version: 11.0.6
  2. Name: foreman_ansible
    Version: 17.0.2
  3. Name: foreman_puppet
    Version: 9.1.0
  4. Name: foreman_remote_execution
    Version: 16.3.1
  5. Name: foreman_webhooks
    Version: 4.0.2
  6. Name: katello
    Version: 4.19.0.1

Smart Proxies:
Features:

  1. Name: dynflow
    Version: 0.9.4
  2. Name: ansible
    Version: 3.6.1
  3. Name: pulpcore
    Version: 3.4.0
  4. Name: script
    Version: 0.11.7
  5. Name: shellhooks
    Version: 0.9.3
  6. Name: tftp
    Version: 3.17.0
  7. Name: dhcp
    Version: 3.17.0
  8. Name: puppetca
    Version: 3.17.0
  9. Name: puppet
    Version: 3.17.0
    10)Name: logs
    Version: 3.17.0

Distribution and version: AlmaLinux release 9.7 (Moss Jungle Cat)

Podman binary and Podman Ansible collection version: podman version 5.6.0 / containers.podman 1.17.0

Other relevant data:

Hello everyone,

I am experiencing an intermittent issue with authentication against the Foreman/Katello container registry and I would like to know whether this is expected behaviour, a known issue, or a bug.

Use case

I have created a dedicated local Foreman user that is used exclusively by Ansible to authenticate against the container registry.

The Ansible playbook uses the following modules:

containers.podman.podman_login
containers.podman.podman_image

The user authenticates with a standard username and password (not with a Personal Access Token).

The assigned role only contains the following permissions:

Resource Permissions Search
Content Views view_content_views N/A
Image view_images N/A
Personal access token create_personal_access_tokens N/A

The authentication works correctly for some time. However, after a period of time (I have not been able to determine the exact interval), the following error starts occurring during podman login:

Unable to gather info for foreman.example.com:
Error: logging into “foreman.example.com”: invalid username/password

As a consequence, containers.podman.podman_image is also unable to pull images from the registry.

The username and password have not changed.

Initial workaround

Initially, the only workaround I found was:

Delete the Foreman user.

Recreate the user with the same username, password and role.

After recreating the user, authentication immediately starts working again.

Investigation

I inspected the user from the Rails console.

The user is authenticated using the internal authentication source:

u.auth_source
=> AuthSourceInternal

The user also has an automatically created Personal Access Token named registry:

u.personal_access_tokens
#<PersonalAccessToken
  name: "registry"
  expires_at: ...
  revoked: false
>

This registry PAT was not created manually.

Additional information

Ansible tasks are run by a user without root privileges, as follows:

- name: "Create temporary authfile"
  set_fact:
    tmp_authfile: "/tmp/<%= rand_name %>-<%= deployment_name %>_auth.json"


- name: "Login to servidor:8443"
  containers.podman.podman_login:
    username: username
    password: "{{ username_password }}"
    registry: foreman.example.com
    authfile: "{{ tmp_authfile }}"

I mention this in case it is relevant to the observed behaviour.

Any guidance or suggestions would be greatly appreciated.

Thank you in advance.

1 Like
2

Hi @jcaniza ,

Are you perhaps hitting Foreman’s brute force login protection? Do you you see anything about it in the production logs when the login failure occurs? Or, do you see any other errors in the logs around the login failure time?

3

Hi @iballou, thanks for your reply.

If I understand correctly, this shouldn’t be happening, since I don’t have any failed login attempts. I haven’t seen anything relevant in either the production.log or the proxy.log files. If you have any idea where I could check this, please let me know.

I’m trying to reproduce the error.

1 Like
4

Hi again @iballou.

While investigating the issue, I reviewed the Apache SSL access log (/var/log/httpd/foreman-ssl_access_ssl.log) and consistently observed the following authentication flow (the sequence is consistently):

"GET /v2/ HTTP/1.1" 401 58 "-" "containers/5.36.2 (github.com/containers/image)"
"GET /v2/token?account=username&service=foreman.example.com HTTP/1.1" 200 171 "-" "containers/5.36.2 (github.com/containers/image)"
"GET /v2/ HTTP/1.1" 401 58 "-" "containers/5.36.2 (github.com/containers/image)"

Therefore, the /v2/token endpoint successfully authenticates the supplied credentials and returns HTTP 200. However, the subsequent request to /v2/ is still rejected with HTTP 401.

Additionally, the same user can authenticate successfully through the Foreman Web UI using exactly the same username and password, even when podman login fails with invalid username/password.

This suggests that the user credentials themselves are valid and that the issue appears to be specific to the container registry authentication flow.

1 Like
5

While reproducing the issue on another Foreman server, I noticed using foreman-rake console that the affected user had two active Personal Access Tokens named registry.

u.personal_access_tokens.count
=> 2
u.personal_access_tokens
=> 
[#<PersonalAccessToken:0x00007fca2f68fc80
  token: "$2a$08$9e6a55b6b4563e652a23bent...zu6",
  expires_at: Wed, 15 Jul 2026 16:00:09.688549000 UTC +00:00,
  revoked: false,
  created_at: Wed, 15 Jul 2026 15:00:09.719519000 UTC +00:00,
  updated_at: Wed, 15 Jul 2026 15:00:09.719519000 UTC +00:00>,
 #<PersonalAccessToken:0x00007fca2f675088
  id: 6,
  token: "$2a$08$9e6a55b6b4563e652a23begz...MSG",
  name: "registry",
  expires_at: Wed, 15 Jul 2026 16:00:09.693778000 UTC +00:00,
  last_used_at: nil,
  revoked: false,
  user_id: 18,
  created_at: Wed, 15 Jul 2026 15:00:09.726860000 UTC +00:00,
  updated_at: Wed, 15 Jul 2026 15:00:09.726860000 UTC +00:00>]
irb(main):007:0> u.personal_access_tokens.pluck(:id, :name, :expires_at, :last_used_at, :created_at)
=> 
[[5, "registry", Wed, 15 Jul 2026 16:00:09.688549000 UTC +00:00, nil, Wed, 15 Jul 2026 15:00:09.719519000 UTC +00:00],
 [6, "registry", Wed, 15 Jul 2026 16:00:09.693778000 UTC +00:00, nil, Wed, 15 Jul 2026 15:00:09.726860000 UTC +00:00]]

Both tokens:

  • belong to the same user
  • have the same name
  • are not revoked
  • were created only a few milliseconds apart
  • have almost identical expiration timestamps
1 Like
6

Thanks for the details @jcaniza .

You seem to be describing an issue that we have seen very scarcely. I would delete all of the personal access tokens and see if access is recovered. During your next authenticated action with the container registry, a new PAT should be created.

We’ve seen in the past in development environments that a registry request made early in Katello’s boot sequence would result in two PATs getting created and 401s even when logged into the registry.

I don’t think we have a Redmine about this yet, would you be interested in writing one up with information from this thread? Since you mentioned using Ansible, automation could be performing the login with a specific timing that makes the issue more apparent.

7

Thank you for the information @ibaillou.

I don’t think my case matches the scenario you described regarding requests occurring early during the Katello boot sequence.

In my environment, the Foreman/Katello services have been running continuously for days or weeks before the issue appears. There are no service restarts or server reboots immediately before the authentication failures.

In any case, the problem might be related. I’ll create a support ticket with the information provided.

As you mentioned, when all personal access tokens are deleted, a new PAT is successfully created during the next authenticated action with the container registry.