I got an error when Ansible plugin for calling back is run... 'CallbackModule' object has no attribute 'ssl_verify'

pjbarbero · February 21, 2018, 10:01am

Hi there,

Please find below the details of the problem I am facing when the callback from Ansible is launched…

Problem:

I got an error when Ansible plugin for calling back is run…

Expected outcome:

No errors and new facts discovered by Ansible should be populated in Foreman.

Foreman and Proxy versions:

1.15.6

Foreman and Proxy plugin versions:

Foreman 1.15.6
Ansible Plugin 2.x

ekohl · February 21, 2018, 12:00pm

Which python version is used on the centospoc03 host? Is this the callback shipped by Ansible itself? If so, which ansible version?

pjbarbero · February 21, 2018, 12:31pm

Hi!!

Thank you for your quick answer… Well these are the versions. And yes the callback is shipped by Ansible once it finishes the execution of the playbook.

Regards.

ekohl · February 21, 2018, 12:59pm

Looking at the source I’d verify requests is installed and at least version 2. Other than that ensure that the foreman URL is correctly configured.

pjbarbero · February 21, 2018, 1:32pm

As mentioned in the documentation,

https://theforeman.org/plugins/foreman_ansible/2.x/index.html

I downloaded the Ansible’s callback module from …

github.com

ansible/ansible/blob/devel/lib/ansible/plugins/callback/foreman.py

# -*- coding: utf-8 -*-
# (c) 2015, 2016 Daniel Lobato <elobatocs@gmail.com>
# (c) 2016 Guido Günther <agx@sigxcpu.org>
# (c) 2017 Ansible Project
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

DOCUMENTATION = '''
    callback: foreman
    type: notification
    short_description: Sends events to Foreman
    description:
      - This callback will report facts and task events to Foreman https://theforeman.org/
      - Before 2.4, if you wanted to use an ini configuration, the file must be placed in the same directory as this plugin and named foreman.ini
      - In 2.4 and above you can just put it in the main Ansible configuration file.
    version_added: "2.2"
    requirements:
      - whitelisting in configuration

This file has been truncated. show original

And EnviromentFile for the “foreman-proxy” (under /etc/sysconfig/foreman-proxy) user contains this variable definition.

FOREMAN_SSL_VERIFY is set to False because I though that If I set it in that way I could by bypass the error… but I getting the same error than when it’s set to True.

Regards.

pjbarbero · February 21, 2018, 4:01pm

Hi there again,

Well after some diving into the Ansible’s callback module trail and error I noticed what is looks like to be a typo…

After changing it i this way now I get another error.

Now the variables I have for the foreman-proxy are:

Now it seems that Foreman server is refusing the callback from the Foreman Smart Proxy… but I am not very sure about it

Regards.

ekohl · February 21, 2018, 4:15pm

Could you try this PR and see if you get better error reporting?

https://github.com/ansible/ansible/pull/36527

pjbarbero · February 22, 2018, 11:31am

Hi again,

After trying the PR you told me I got the same results, before and after changing again the sentence “verify=ssl_verify” =>“verify=_ssl_verify”.

Regards.

dLobatog · February 22, 2018, 12:33pm

Hi again!

Regarding the “connection refused” error, can you share your ‘/var/log/foreman/production.log’?

This error usually happens when you try to run ansible or ansible-playbook with the callback from any computer that is NOT the Foreman host itself, or a foreman-proxy.

Foreman authenticates hosts w/o API credentials only when they’re smart-proxies, or when they are part of the trusted_puppetmaster_hosts (yeah, confusing name, we have to rename it so it makes sense for Ansible).

You have a bit of info about this in the section 3. Configuration in the manual -> Foreman :: Plugin Manuals

tl;dr = Have you added the host you’re trying to run Ansible from to Administer > Settings" trusted puppetmaster hosts" (authentication tab)?

pjbarbero · February 22, 2018, 1:06pm

Hi Daniel,

Well the truth is that Ansible server, from where the Ansible command is being executed, is as well the Foreman Server, the Foreman Smart Proxy and Puppet Master and CA, it is my tiny lab.

I tried with that variable (I mean truested_puppetmaster_hosts) empty as the logic made me think according to the documentation,but after testing and not getting any result It was set with an array with the name of server in all the posible ways (centospoc00, centospoc00.poc,192.168.0.1, localhost and 127.0.0.0) but nothing, same error.

I think the problem it could be, at least in first stage, in that FOREMAN_URL is said to be set as

But the port 3000 is not opened. So hence my question, what should open that port? By the way,is that port correct?

Regards.

dLobatog · February 22, 2018, 1:20pm

@pjbarbero That port has nothing to do with Foreman other than being the default development port. Change your FOREMAN_URL to point to http://centospoc00.poc (or https rather, as I see you have certs)

The callback defaults to localhost:3000 to make development of the callback easier - but in production port 443 or 80 if you only have http are the equivalent.

Apologies for the confusion and if you think this should be changed in the documentation - either tell us here of send us a patch on https://github.com/theforeman/theforeman.org/tree/gh-pages/plugins/foreman_ansible , I’d be glad to merge it.

pjbarbero · February 22, 2018, 3:10pm

Ok don’t worry.

I doesn’t work anyway with port 443 or 80.

Regards.

dLobatog · February 22, 2018, 3:13pm

@pjbarbero Could you share your logs in /var/log/foreman/production.log when you run ansible-playbook? I wonder if the connection refused is at an SELinux level, firewall, or actually Foreman dropping the connection because of some wrong credentials

pjbarbero · February 23, 2018, 1:59pm

Hi again,

I have been checking what you mentioned in your last post, and I saw how SELinux was activated. So that I disabled it and try, but as before absolutely nothing is logged in “/var/log/foreman/production.log”.

By the way the “firewalld” (firewall service in my CentOS OS) was stopped and desactivated.

So that I think the problem points out at the last supposition you made, Foreman must be dropping the connection for some wrong credentials (I think you wanted to mean SSL Certificates verification).

Now I am working on that, but any help from your side is very appreciated.

Any way, there is something I dont understand well about the kind of SSL Certificate verification that is taking place between Foreman Server => and Foreman Smart Proxy, and between Foreman Smart Proxy => Foreman Server . Could you please confirm me this verification is “Certificate based mutual authentication”?

Regards.

dLobatog · February 23, 2018, 3:20pm

If absolutely nothing is logged on /var/log/foreman/production.log when you
try to contact the Foreman host at 80 or 443, it’s most certainly not
Foreman the one returning “connection refused”.

Maybe you can see what’s going on with ‘journalctl -lf’ or even in
/var/log/httpd

pjbarbero · February 26, 2018, 12:54pm

Hi Daniel,

I found which one were my problems. I restored a backup of my VM and I started the installation and configuration of the Ansible plugin again from a scratch.

And here my findings…

First I had an Ansible version higher than 2.2, so I started using the callback module proied by default.

And the second one when configuring it, instead of using the “foreman” user I used the “foreman-proxy” user.

After this, I don’t get any error when invoking any Ansible Playbook or Module.

But I still go on without seen anything in the logs, none requests for upload reports or facts to the Foreman server.

Regards.

dLobatog · February 26, 2018, 1:36pm

Sorry, I don’t quite follow. In the previous messages I saw, you were running ansible-playbook directly through the CLI yourself, as the root user. But on your previous message I see you mention the environment variables were configured for the ‘foreman-proxy’ user, and now you configured the callback variables for the ‘foreman’ user.

And the second one when configuring it, instead of using the “foreman” user I used the “foreman-proxy” user.
…
After this, I don’t get any error when invoking any Ansible Playbook or Module.

How are you invoking these playbooks? Through Foreman/Foreman-proxy itself (via Job templates) or in the CLI?

If it’s on the CLI, ansible-playbook will execute the callback if you’ve set it up the way you said, callback_whitelist = foreman. When the callback runs, it will check the environment variables FOREMAN_HOST, FOREMAN_SSL_CERT, FOREMAN_SSL_KEY. You have to make sure these variables are set for whichever user runs ansible or ansible-playbook.

If you’re running ansible-playbook as root, in bash, you need to make sure the root user has all these environment variables set. FOREMAN_URL, FOREMAN_SSL_CERT, FOREMAN_SSL_KEY. If you’re not sure how to set these variables, there are many options.

To print the variables, you can run echo $FOREMAN_URL, etc… Ensure these variables are set before even running ansible-playbook if you want to get the callback report.
Set them on /etc/environment (log out and log in so that they are available).
Set them on ~/.bashrc , ~/.bash_profile or similar

If you’re running ansible through Foreman Remote Execution, either foreman or foreman-proxy will run the ansible-playbook command in the end. That’s why you need to set up the environment variables for these users. We’re working on making the installer automatically do that as it seems to be a tricky step.

pjbarbero · February 26, 2018, 4:10pm

Ok!! Now it quite clear how is works… Thank you.

I also found a video where is quite clear how to configure it properly.

Regards.