Ansible callback with windows openssh.server produces a 500 Server Error: Internal

Problem:
Attempting to onboard existing windows systems into foreman via Ansible setup. facts are processed but fail to be sent to the Foreman /api/v2/hosts/facts site.
Linux hosts work perfectly, like always.
Ansible runs to the system (win_ping module, playbooks, etc) to the windows host via ansible are also fine. Callback produces the error post run:

[WARNING]: Sending data to Foreman at https://centfor.pate.net failed for servertest.pate.net: 500 Server Error: Internal
Server Error for url: https://centfor.pate.net/api/v2/hosts/facts

Expected outcome:
Setup should report back to Foreman all windows facts and register with system:

Foreman and Proxy versions:
Foreman 3.0.0
|foreman-tasks|5.1.0|
|foreman_ansible|6.4.1|
|foreman_discovery|18.0.0|
|foreman_puppet|1.0.3|
|foreman_remote_execution|4.7.0|

Distribution and version:
CentOS7 for Foreman server.
Windows Server2019 with OpenSSH Server installed:
(Add-WindowsCapability –Online –Name OpenSSH.Server~~~~0.0.1.0)
proxy public key used for authentication

Other relevant data:
Callback addition to ansible.cfg:

callback_whitelist = foreman
callback_plugins = /root/.ansible/collections/ansible_collections/theforeman/foreman/plugins/callback
bin_ansible_callbacks = true

[callback_foreman]
url = ‘https://centfor.pate.net
ssl_cert = /etc/puppetlabs/puppet/ssl/certs/centfor.pate.net.pem
ssl_key = /etc/puppetlabs/puppet/ssl/private_keys/centfor.pate.net.pem
ssl_ca = /etc/puppetlabs/puppet/ssl/certs/ca.pem
verify_certs = /etc/puppetlabs/puppet/ssl/certs/ca.pem
#verify_certs = 1

windows inventory file:
[dev]
servertest. pate. net
servertest2. pate. net

[all:vars]
ansible_shell_type=cmd
ansible_user=Administrator
ansible_ssh_private_key_file="/var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy"
ansible_connection=ssh

Please let me know what else I can provide to help make this effective and contribute!
Logs? etc?

I’ve yet to attempt WinRM. Because SSH is so much more simple to stand up on windows now.

1 Like

We’d need to see the contents of /var/log/foreman/production.log around the time the callback has submitted the data and got the 500 error when uploading facts.

1 Like

did a fresh run and tail -f of the log file.
Hope that helps.

prodlog.log (14.0 KB)

Thanks, yes, this is helpful (at least a bit).

The interesting section from that log is:

2021-10-14T23:25:15 [I|app|aa51d761] Started POST "/api/v2/hosts/facts" for 192.168.1.6 at 2021-10-14 23:25:15 -0700
2021-10-14T23:25:15 [I|app|aa51d761] Processing by Api::V2::HostsController#facts as JSON
2021-10-14T23:25:15 [I|app|aa51d761]   Parameters: {"name"=>"servertest.pate.net", "facts"=>"[FILTERED]", "apiv"=>"v2", "host"=>{"name"=>"servertest.pate.net"}}
2021-10-14T23:25:15 [I|app|aa51d761] Import facts for 'servertest.pate.net' completed. Added: 0, Updated: 16, Deleted 0 facts
2021-10-14T23:25:15 [W|app|aa51d761] Action failed
2021-10-14T23:25:15 [I|app|aa51d761] Backtrace for 'Action failed' error (NoMethodError): undefined method `match' for #<ActiveSupport::HashWithIndifferentAccess:0x000055e580b78930>
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:221:in `block in remove_ignored'
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:220:in `delete_if'
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:220:in `remove_ignored'
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:77:in `interfaces'
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:92:in `suggested_primary_interface'
 aa51d761 | /usr/share/foreman/app/models/host/base.rb:152:in `primary_interface_type'
 aa51d761 | /usr/share/foreman/app/models/host/base.rb:158:in `populate_fields_from_facts'
 aa51d761 | /usr/share/foreman/app/models/host/managed.rb:489:in `populate_fields_from_facts'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:50:in `block (2 levels) in parse_facts'
 aa51d761 | /usr/share/foreman/app/services/foreman/telemetry_helper.rb:27:in `telemetry_duration_histogram'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:49:in `block in parse_facts'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:90:in `block in skipping_orchestration'
 aa51d761 | /usr/share/foreman/app/models/concerns/orchestration.rb:124:in `without_orchestration'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:89:in `skipping_orchestration'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:45:in `parse_facts'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:34:in `import_facts'
 aa51d761 | /usr/share/foreman/app/controllers/api/v2/hosts_controller.rb:310:in `facts'

Which indicates at this section in the code:

Now why it is getting a HashWithIndifferentAccess there, is a good question…

Can you capture those facts and attach them too?

We haven’t tried this with facts from Windows, that’s why. Who knows what Ansible sends there…

So to make things a little more “ODD”…
I manually created the host in Foreman. (set hostname, mac address, etc…) very basic.

ran a playbook to basic install of some apps via choco.
same error 500 message.

But…
Facts were uploaded to the Manually created host.

@lzap are you asking me for the facts pre failure when setup is ran?

Yeah, our code assumes a fact that is a string and your facts do appear to have a hash there. Something is very different on Windows, I assume.

Hah, this was actually reported in the past:

https://projects.theforeman.org/issues/23936

Bwhaahaha!

How difficult will it be for allowing the name “Ethernet” and “Ethernet #x” to setup?

Facts info:
setup-facts.log (14.0 KB)

1 Like

The problem is not the name of the interface, but the fact (lol) that on Linux, Ansible reports the interfaces as ['lo', 'eth0'] (= a list of strings), while on Windows it’s [{'connection_name':'Ethernet', …}] (= a list of hashes), and our code doesn’t cope with the later :frowning:

I do not have time for full implementation of NIC parsing for Windows, however, I see you are blocked by this therefore I propose that Ansible parser returns no interfaces:

That should unblock you, please test and report back.

Thanks @lzap I’ll attempt the pull a little later (busy with at my workplace)
Expect a report back in a few days.

I have about 600+ windows systems (and about 2000 linux) that need imported in my production environment. so this would be a huge lifesaver.

FYI, OpenSSH server on Windows + Ansible is exceptionally streamlined and useful. Way easier to setup than WinRM instances.
Happy to provide windows side erb of ssh setup script I have when completed.

~B

I regret poor git-fu.
Quite confused on how to pull the changes form this.
Using stable 3.0 branch installed via yum.