Ansible callback with windows openssh.server produces a 500 Server Error: Internal

Problem:
Attempting to onboard existing windows systems into foreman via Ansible setup. facts are processed but fail to be sent to the Foreman /api/v2/hosts/facts site.
Linux hosts work perfectly, like always.
Ansible runs to the system (win_ping module, playbooks, etc) to the windows host via ansible are also fine. Callback produces the error post run:

[WARNING]: Sending data to Foreman at https://centfor.pate.net failed for servertest.pate.net: 500 Server Error: Internal
Server Error for url: https://centfor.pate.net/api/v2/hosts/facts

Expected outcome:
Setup should report back to Foreman all windows facts and register with system:

Foreman and Proxy versions:
Foreman 3.0.0
|foreman-tasks|5.1.0|
|foreman_ansible|6.4.1|
|foreman_discovery|18.0.0|
|foreman_puppet|1.0.3|
|foreman_remote_execution|4.7.0|

Distribution and version:
CentOS7 for Foreman server.
Windows Server2019 with OpenSSH Server installed:
(Add-WindowsCapability –Online –Name OpenSSH.Server~~~~0.0.1.0)
proxy public key used for authentication

Other relevant data:
Callback addition to ansible.cfg:

callback_whitelist = foreman
callback_plugins = /root/.ansible/collections/ansible_collections/theforeman/foreman/plugins/callback
bin_ansible_callbacks = true

[callback_foreman]
url = ‘https://centfor.pate.net
ssl_cert = /etc/puppetlabs/puppet/ssl/certs/centfor.pate.net.pem
ssl_key = /etc/puppetlabs/puppet/ssl/private_keys/centfor.pate.net.pem
ssl_ca = /etc/puppetlabs/puppet/ssl/certs/ca.pem
verify_certs = /etc/puppetlabs/puppet/ssl/certs/ca.pem
#verify_certs = 1

windows inventory file:
[dev]
servertest. pate. net
servertest2. pate. net

[all:vars]
ansible_shell_type=cmd
ansible_user=Administrator
ansible_ssh_private_key_file="/var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy"
ansible_connection=ssh

Please let me know what else I can provide to help make this effective and contribute!
Logs? etc?

I’ve yet to attempt WinRM. Because SSH is so much more simple to stand up on windows now.

1 Like

We’d need to see the contents of /var/log/foreman/production.log around the time the callback has submitted the data and got the 500 error when uploading facts.

1 Like

did a fresh run and tail -f of the log file.
Hope that helps.

prodlog.log (14.0 KB)

Thanks, yes, this is helpful (at least a bit).

The interesting section from that log is:

2021-10-14T23:25:15 [I|app|aa51d761] Started POST "/api/v2/hosts/facts" for 192.168.1.6 at 2021-10-14 23:25:15 -0700
2021-10-14T23:25:15 [I|app|aa51d761] Processing by Api::V2::HostsController#facts as JSON
2021-10-14T23:25:15 [I|app|aa51d761]   Parameters: {"name"=>"servertest.pate.net", "facts"=>"[FILTERED]", "apiv"=>"v2", "host"=>{"name"=>"servertest.pate.net"}}
2021-10-14T23:25:15 [I|app|aa51d761] Import facts for 'servertest.pate.net' completed. Added: 0, Updated: 16, Deleted 0 facts
2021-10-14T23:25:15 [W|app|aa51d761] Action failed
2021-10-14T23:25:15 [I|app|aa51d761] Backtrace for 'Action failed' error (NoMethodError): undefined method `match' for #<ActiveSupport::HashWithIndifferentAccess:0x000055e580b78930>
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:221:in `block in remove_ignored'
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:220:in `delete_if'
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:220:in `remove_ignored'
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:77:in `interfaces'
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:92:in `suggested_primary_interface'
 aa51d761 | /usr/share/foreman/app/models/host/base.rb:152:in `primary_interface_type'
 aa51d761 | /usr/share/foreman/app/models/host/base.rb:158:in `populate_fields_from_facts'
 aa51d761 | /usr/share/foreman/app/models/host/managed.rb:489:in `populate_fields_from_facts'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:50:in `block (2 levels) in parse_facts'
 aa51d761 | /usr/share/foreman/app/services/foreman/telemetry_helper.rb:27:in `telemetry_duration_histogram'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:49:in `block in parse_facts'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:90:in `block in skipping_orchestration'
 aa51d761 | /usr/share/foreman/app/models/concerns/orchestration.rb:124:in `without_orchestration'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:89:in `skipping_orchestration'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:45:in `parse_facts'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:34:in `import_facts'
 aa51d761 | /usr/share/foreman/app/controllers/api/v2/hosts_controller.rb:310:in `facts'

Which indicates at this section in the code:

Now why it is getting a HashWithIndifferentAccess there, is a good question…

Can you capture those facts and attach them too?

We haven’t tried this with facts from Windows, that’s why. Who knows what Ansible sends there…

So to make things a little more “ODD”…
I manually created the host in Foreman. (set hostname, mac address, etc…) very basic.

ran a playbook to basic install of some apps via choco.
same error 500 message.

But…
Facts were uploaded to the Manually created host.

@lzap are you asking me for the facts pre failure when setup is ran?

Yeah, our code assumes a fact that is a string and your facts do appear to have a hash there. Something is very different on Windows, I assume.

Hah, this was actually reported in the past:

https://projects.theforeman.org/issues/23936

Bwhaahaha!

How difficult will it be for allowing the name “Ethernet” and “Ethernet #x” to setup?

Facts info:
setup-facts.log (14.0 KB)

1 Like

The problem is not the name of the interface, but the fact (lol) that on Linux, Ansible reports the interfaces as ['lo', 'eth0'] (= a list of strings), while on Windows it’s [{'connection_name':'Ethernet', …}] (= a list of hashes), and our code doesn’t cope with the later :frowning:

I do not have time for full implementation of NIC parsing for Windows, however, I see you are blocked by this therefore I propose that Ansible parser returns no interfaces:

That should unblock you, please test and report back.

Thanks @lzap I’ll attempt the pull a little later (busy with at my workplace)
Expect a report back in a few days.

I have about 600+ windows systems (and about 2000 linux) that need imported in my production environment. so this would be a huge lifesaver.

FYI, OpenSSH server on Windows + Ansible is exceptionally streamlined and useful. Way easier to setup than WinRM instances.
Happy to provide windows side erb of ssh setup script I have when completed.

~B

1 Like

So how was the testing?

Not so great @lzap

My virt enviroment for test cases took a bad poweroutage and I’m reconstructing it all. (better than production though! hah!)

Hey, I’m not really the best PR’s from Git, what it the best method in pulling said changes for verification? Or should I be standing a new foreman-devl instance up for changes?

It is as easy as:

cd /usr/share/foreman
wget https://github.com/theforeman/foreman/pull/8848.patch
patch -p1 < 8848.patch

Or you can just locate the app/services/ansible_fact_parser.rb and add the line which you can see in the patch diff view. It was a one line change, all the rest are just tests. Restart foreman afterwards.

1 Like

Appreciate it. will test later today and report.

@lzap

Patch didn’t like to apply, so I added the 1 line:

return [] if facts[:ansible_os_family] == 'Windows'

Still receive the 500 error. system doesnt populate in foreman. (unless pre-populated)

What kind of error do you get? Please pastebin it here again.

No ansible -m setup errors outside the 500 error when ran…

here’s the prod log.

prodlog2.log (23.4 KB)

foreman-rake console does show machine, but it never is inputted into the UI if not pre-populated prior to running setup.

irb(main):002:0> Host.find_by_name(“patetest2.guest.pate.net”)
=> #<Host::Managed id: 16, name: “patetest2.guest.pate.net”, last_compile: nil, last_report: [FILTERED], updated_at: “2021-11-03 17:16:00”, created_at: “2021-11-03 17:16:00”, root_pass: nil,…