Foreman ansible callback not working

Support
1

Problem: Ansible runs via Foreman finish successfully, but callback/facts gathering doesn’t work, presumably because of Ansible vault.

956:PLAY RECAP *********************************************************************
 957:test.test.com  : ok=205  changed=146  unreachable=0    failed=0    skipped=51   rescued=0    ignored=0[WARNING]: Failure using method (v2_playbook_on_stats) in callback plugin (<ans
 958: ible_collections.theforeman.foreman.plugins.callback.foreman.CallbackModule
 959: object at 0x7efd47dc40d0>): Object of type AnsibleVaultEncryptedUnicode is not
 960: JSON serializable
 961: Exit status: 0

Expected outcome: Ansible-Foreman callback should work, with vault encrypted variables.

Foreman and Proxy versions:

  • Foreman & Proxy 3.11
  • Katello 4.13

Foreman and Proxy plugin versions:

  • ansible-collection-theforeman-foreman-4.0.0-2.el9.noarch
  • rubygem-foreman_ansible-14.0.0-2.fm3_11.el9.noarch
  • rubygem-pulp_ansible_client-0.21.3-1.el9.noarch
  • python3.11-pulp-ansible-0.21.6-1.el9.noarch

Distribution and version:

  • Rocky 9.4

Other relevant data: When I remove the Ansible vault variable, callback/facts gathering works fine. So the ansible-plugin exists and works, the problem is only with the ansible vault variable.

  • Relavent configs in ansible.cfg:
callback_whitelist = foreman
[callback_foreman]
url = https://rocky9-install.test.com
ssl_cert = /etc/foreman-proxy/foreman_ssl_cert.pem
ssl_key = /etc/foreman-proxy/foreman_ssl_key.pem
verify_certs = /etc/foreman-proxy/foreman_ssl_ca.pem
2

It seems that the Foreman callback plugin is attempting to process vault-encrypted data (AnsibleVaultEncryptedUnicode), which isn’t JSON serializable. However, I couldn’t reproduce this issue in my local environment, so I’m unsure of the exact cause.

Could you provide more details on the specific Ansible task you’re running? This might help in identifying what’s triggering the issue.

4

Sorry for the late reply.
We store an encrypted password in a group_var file, which is then used by a task.
I used the following command to encrypt the string:

ansible-vault encrypt_string --vault-password-file .vault_pw_file 'to_be_encrypted' --name 'password_variable'

The Ansible task is basically configuring some Wake-on-lan and sleep statuses via Dell Command Control, which needs the password.
The task look something like this:

command: /opt/dell/dcc/cctk --WakeOnLanOnly=LanOnly --ValSetupPwd={{ password_variable }}

I ran the Ansible task both manually and also using Foreman, and the task ran successfully, so foreman can decrypt the password. It’s only the callback that doesn’t work. I also ran Ansible via Foreman without the password_variable and the callbacks also worked.
Under proxy.log it pretty much shows the same error I posted before, with some extra things like: uuid, playbook, etc. I can also post that if needed.

In ansible.cfg under /etc/ansible, the .vault_pw_file is set as the “vault_passowrd_file”.
Both the .vault_pw_file and ansible.cfg also exist under /usr/share/foreman-proxy/, with a symlink.

5

I also tried adding it to the exclude list, under settings → facts, but that didn’t work either. I assume with that option, Foreman still tries to callback the variable, but it just doesn’t save it.

6

I’ve tried adding a test in add tests for Vault variables by evgeni · Pull Request #1769 · theforeman/foreman-ansible-modules · GitHub and it passes without any changes, weird.

7

This is interesting. This is not the step that records the actual command, but the stats (and facts).
Is the Vault data somehow also part of the facts the host has?

8

I’m not entirely sure if I fully understand the question, but I think the Vault data is for some reason also part of the facts, since when I commented/removed the Vault data, callback and facts-gathering worked as it’s supposed to.

Is there any way I can check and or control this?

9

Any chance you could share the whole playbook? (feel free to mail it to evgeni at redhat dot com, if you do not want to post it publicly)

10

Ha!

    - name: Vault fact
      set_fact:
        crypt: !vault |
              $ANSIBLE_VAULT;1.1;AES256
              62343962363339363461373565656663663734393265636161313466326163666638333735303061
              6134346238366533616262663462396332363535363662660a323339326635373330633230336332
              38363334373037356466383063616532656632303636333839313831626264386132386661303535
              3864663564356332390a633631363962353163316236323038363861623763616265343762366435
              6237

This triggers the error you’re hitting.

It does NOT trigger if the vault data is previously stored in a variable and then you do set_fact: something: "{{ vault_var }}" as that correctly unwraps the vault (and logs the secret in plaintext :thinking: )

11

I have a patch in add tests for Vault variables by evgeni · Pull Request #1769 · theforeman/foreman-ansible-modules · GitHub – if you’re curious you could try it out :slight_smile:

12

Here is a minimalistic version of our Ansible:

Role:

---
- name: echo pw
  command: echo {{ bios_password }} >> /dev/null

Playbook (This is the playbook, we use when we run ansible from the terminal):

---
- name: run role
  roles:
    - 31-cctk
  hosts: workstations

Foreman-playbook (The playbook used when ansible is ran from Foreman):

---
- hosts: all

  tasks:
    - name: Load  group_vars
      include_vars: "/etc/ansible/group_vars/all.yml"
    
    - name: Load host_vars
      include_vars: "/etc/ansible/host_vars/{{ ansible_fqdn }}"
      ignore_errors: yes

    - name: Apply cctk
      include_role:
        name: "31-cctk"
        
    - name: Display ansible info
      debug:

And as mentioned before the encrypted vault data is stored as a variable in a file:

bios_password: !vault |
              $ANSIBLE_VAULT;1.1;AES256
              62343962363339363461373565656663663734393265636161313466326163666638333735303061
              6134346238366533616262663462396332363535363662660a323339326635373330633230336332
              38363334373037356466383063616532656632303636333839313831626264386132386661303535
              3864663564356332390a633631363962353163316236323038363861623763616265343762366435
              6237

We noticed that when Ansible is ran form the terminal, callbacks work, it’s only when Ansible is ran via Foreman itself that the callback doesn’t work.

Besides that, we also noticed that when we run Ansible from foreman we get this deprecation warning, although in ansible.cfg we use the newer standard, ie. “callback_enabled”. We don’t get the deprecation warning when we run Ansible via terminal (By running via terminal, I mean we run ansible on the server that Foreman is also running on).

  1: [DEPRECATION WARNING]: ANSIBLE_CALLBACK_WHITELIST option, normalizing names to 
  2: new standard, use ANSIBLE_CALLBACKS_ENABLED instead. This feature will be 
  3: removed from ansible-core in version 2.15. Deprecation warnings can be disabled
  4: by setting deprecation_warnings=False in ansible.cfg.

If you need more information let me know.

13

I assume bios_password is in one of those files you load via include_vars?

The deprecation warning is “fine” – we right now intentionally set both (old and new) variable names when calling Ansible to support older Ansible releases.

14

Yea, it’s in /etc/ansible/group_vars/all.yml

15

Interesting. Turns out include_vars does set facts. And that’s why it broke for you.

Good news: the linked PR fixes the issue. :slight_smile:

1 Like
16

The patch was merged and collection 4.2.0 released, so whenever you update to it, it will work :slight_smile:

1 Like
17

Thanks! :+1:

18

After updating to the new version, from what I gathered the callbacks work and I can see the facts that foreman/ansible gathered under the facts tab, but for some reason it still shows the error from before.
I now have Foreman version 3.12 and ansible-collection-theforeman-foreman-4.2.0-1.el9.noarch

Failure using method (v2_playbook_on_stats) in callback plugin (<ans
 958: ible_collections.theforeman.foreman.plugins.callback.foreman.CallbackModule
 959: object at 0x7efd47dc40d0>): Object of type AnsibleVaultEncryptedUnicode is not
 960: JSON serializable
 961: Exit status: 0
19

That doesn’t seem right.

You should either see the error, or the values in Foreman. Can’t have both :wink:

Do you maybe have a proxy that is also doing Ansible, and that hasn’t been updated yet?

20

I have Ansible version 2.16.11 installed via pip. If I’m not mistaken it comes with version 3.15.0 .

[root@rocky9-install ansible]# ansible-galaxy collection list | grep foreman
theforeman.foreman                       3.15.0

I tried upgrading it with this command ansible-galaxy collection install theforeman.foreman --upgrade, but now it shows them both being installed.

[root@rocky9-install ansible]# ansible-galaxy collection list | grep foreman
theforeman.foreman                       4.2.0  
theforeman.foreman                       3.15.0

I also couldn’t find a way of removing the older version, as it’s not in the collections folder (neither in /etc/ansible nor anywhere else).
dnf list installed | grep ansible also shows

[root@rocky9-install ansible]# dnf list installed | grep ansible
ansible-collection-theforeman-foreman.noarch               4.2.0-1.el9                      @foreman-plugins

But it still shows the error and also facts. This is both before I tried updating the ansible collection (theforeman.foreman) .
As for proxies, I only have one other proxy which doesn’t run ansible and has the same foreman version (I tested it by disabling the proxy).
When I was still using Foreman version 3.11 it only showed the error, no facts.

This seems to be a really specific issue, and facts seem to be working, so I would understand if you don’t want to invest more time troubleshooting.