Ansible callback with windows openssh.server produces a 500 Server Error: Internal

platypusoffury · October 14, 2021, 5:57am

Problem:
Attempting to onboard existing windows systems into foreman via Ansible setup. facts are processed but fail to be sent to the Foreman /api/v2/hosts/facts site.
Linux hosts work perfectly, like always.
Ansible runs to the system (win_ping module, playbooks, etc) to the windows host via ansible are also fine. Callback produces the error post run:

[WARNING]: Sending data to Foreman at https://centfor.pate.net failed for servertest.pate.net: 500 Server Error: Internal
Server Error for url: https://centfor.pate.net/api/v2/hosts/facts

Expected outcome:
Setup should report back to Foreman all windows facts and register with system:

Foreman and Proxy versions:
Foreman 3.0.0
|foreman-tasks|5.1.0|
|foreman_ansible|6.4.1|
|foreman_discovery|18.0.0|
|foreman_puppet|1.0.3|
|foreman_remote_execution|4.7.0|

Distribution and version:
CentOS7 for Foreman server.
Windows Server2019 with OpenSSH Server installed:
(Add-WindowsCapability –Online –Name OpenSSH.Server~~~~0.0.1.0)
proxy public key used for authentication

Other relevant data:
Callback addition to ansible.cfg:

callback_whitelist = foreman
callback_plugins = /root/.ansible/collections/ansible_collections/theforeman/foreman/plugins/callback
bin_ansible_callbacks = true

[callback_foreman]
url = ‘https://centfor.pate.net’
ssl_cert = /etc/puppetlabs/puppet/ssl/certs/centfor.pate.net.pem
ssl_key = /etc/puppetlabs/puppet/ssl/private_keys/centfor.pate.net.pem
ssl_ca = /etc/puppetlabs/puppet/ssl/certs/ca.pem
verify_certs = /etc/puppetlabs/puppet/ssl/certs/ca.pem
#verify_certs = 1

windows inventory file:
[dev]
servertest. pate. net
servertest2. pate. net

[all:vars]
ansible_shell_type=cmd
ansible_user=Administrator
ansible_ssh_private_key_file="/var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy"
ansible_connection=ssh

Please let me know what else I can provide to help make this effective and contribute!
Logs? etc?

I’ve yet to attempt WinRM. Because SSH is so much more simple to stand up on windows now.

evgeni · October 15, 2021, 5:39am

We’d need to see the contents of /var/log/foreman/production.log around the time the callback has submitted the data and got the 500 error when uploading facts.

platypusoffury · October 15, 2021, 6:31am

did a fresh run and tail -f of the log file.
Hope that helps.

prodlog.log (14.0 KB)

evgeni · October 15, 2021, 6:48am

Thanks, yes, this is helpful (at least a bit).

The interesting section from that log is:

2021-10-14T23:25:15 [I|app|aa51d761] Started POST "/api/v2/hosts/facts" for 192.168.1.6 at 2021-10-14 23:25:15 -0700
2021-10-14T23:25:15 [I|app|aa51d761] Processing by Api::V2::HostsController#facts as JSON
2021-10-14T23:25:15 [I|app|aa51d761]   Parameters: {"name"=>"servertest.pate.net", "facts"=>"[FILTERED]", "apiv"=>"v2", "host"=>{"name"=>"servertest.pate.net"}}
2021-10-14T23:25:15 [I|app|aa51d761] Import facts for 'servertest.pate.net' completed. Added: 0, Updated: 16, Deleted 0 facts
2021-10-14T23:25:15 [W|app|aa51d761] Action failed
2021-10-14T23:25:15 [I|app|aa51d761] Backtrace for 'Action failed' error (NoMethodError): undefined method `match' for #<ActiveSupport::HashWithIndifferentAccess:0x000055e580b78930>
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:221:in `block in remove_ignored'
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:220:in `delete_if'
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:220:in `remove_ignored'
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:77:in `interfaces'
 aa51d761 | /usr/share/foreman/app/services/fact_parser.rb:92:in `suggested_primary_interface'
 aa51d761 | /usr/share/foreman/app/models/host/base.rb:152:in `primary_interface_type'
 aa51d761 | /usr/share/foreman/app/models/host/base.rb:158:in `populate_fields_from_facts'
 aa51d761 | /usr/share/foreman/app/models/host/managed.rb:489:in `populate_fields_from_facts'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:50:in `block (2 levels) in parse_facts'
 aa51d761 | /usr/share/foreman/app/services/foreman/telemetry_helper.rb:27:in `telemetry_duration_histogram'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:49:in `block in parse_facts'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:90:in `block in skipping_orchestration'
 aa51d761 | /usr/share/foreman/app/models/concerns/orchestration.rb:124:in `without_orchestration'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:89:in `skipping_orchestration'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:45:in `parse_facts'
 aa51d761 | /usr/share/foreman/app/services/host_fact_importer.rb:34:in `import_facts'
 aa51d761 | /usr/share/foreman/app/controllers/api/v2/hosts_controller.rb:310:in `facts'

Which indicates at this section in the code:

github.com

theforeman/foreman/blob/3.0.0/app/services/fact_parser.rb#L219-L226

    
      
          def remove_ignored(interfaces)
            interfaces.clone.delete_if do |identifier|
              if (remove = identifier.match(ignored_interfaces))
                logger.debug { "skipping interface with identifier '#{identifier}' since it was matched by 'ignored_interface_identifiers' setting " }
              end
              remove
            end
          end

Now why it is getting a HashWithIndifferentAccess there, is a good question…

lzap · October 15, 2021, 6:53am

Can you capture those facts and attach them too?

lzap · October 15, 2021, 6:56am

We haven’t tried this with facts from Windows, that’s why. Who knows what Ansible sends there…

platypusoffury · October 15, 2021, 6:59am

So to make things a little more “ODD”…
I manually created the host in Foreman. (set hostname, mac address, etc…) very basic.

ran a playbook to basic install of some apps via choco.
same error 500 message.

But…
Facts were uploaded to the Manually created host.

@lzap are you asking me for the facts pre failure when setup is ran?

lzap · October 15, 2021, 7:00am

Yeah, our code assumes a fact that is a string and your facts do appear to have a hash there. Something is very different on Windows, I assume.

evgeni · October 15, 2021, 7:07am

Hah, this was actually reported in the past:

https://projects.theforeman.org/issues/23936

platypusoffury · October 15, 2021, 7:31am

Bwhaahaha!

How difficult will it be for allowing the name “Ethernet” and “Ethernet #x” to setup?

Facts info:
setup-facts.log (14.0 KB)

evgeni · October 15, 2021, 7:46am

The problem is not the name of the interface, but the fact (lol) that on Linux, Ansible reports the interfaces as ['lo', 'eth0'] (= a list of strings), while on Windows it’s [{'connection_name':'Ethernet', …}] (= a list of hashes), and our code doesn’t cope with the later

lzap · October 15, 2021, 8:59am

I do not have time for full implementation of NIC parsing for Windows, however, I see you are blocked by this therefore I propose that Ansible parser returns no interfaces:

That should unblock you, please test and report back.

platypusoffury · October 15, 2021, 6:24pm

Thanks @lzap I’ll attempt the pull a little later (busy with at my workplace)
Expect a report back in a few days.

I have about 600+ windows systems (and about 2000 linux) that need imported in my production environment. so this would be a huge lifesaver.

FYI, OpenSSH server on Windows + Ansible is exceptionally streamlined and useful. Way easier to setup than WinRM instances.
Happy to provide windows side erb of ssh setup script I have when completed.

~B

lzap · October 25, 2021, 7:31am

So how was the testing?

platypusoffury · October 25, 2021, 1:51pm

Not so great @lzap…

My virt enviroment for test cases took a bad poweroutage and I’m reconstructing it all. (better than production though! hah!)

Hey, I’m not really the best PR’s from Git, what it the best method in pulling said changes for verification? Or should I be standing a new foreman-devl instance up for changes?

lzap · October 25, 2021, 1:56pm

It is as easy as:

cd /usr/share/foreman
wget https://github.com/theforeman/foreman/pull/8848.patch
patch -p1 < 8848.patch

Or you can just locate the app/services/ansible_fact_parser.rb and add the line which you can see in the patch diff view. It was a one line change, all the rest are just tests. Restart foreman afterwards.

platypusoffury · October 25, 2021, 1:59pm

Appreciate it. will test later today and report.

platypusoffury · October 27, 2021, 1:30am

@lzap

Patch didn’t like to apply, so I added the 1 line:

return [] if facts[:ansible_os_family] == 'Windows'

Still receive the 500 error. system doesnt populate in foreman. (unless pre-populated)

lzap · November 3, 2021, 7:57am

What kind of error do you get? Please pastebin it here again.

platypusoffury · November 3, 2021, 5:25pm

No ansible -m setup errors outside the 500 error when ran…

here’s the prod log.

prodlog2.log (23.4 KB)

foreman-rake console does show machine, but it never is inputted into the UI if not pre-populated prior to running setup.

irb(main):002:0> Host.find_by_name(“patetest2.guest.pate.net”)
=> #<Host::Managed id: 16, name: “patetest2.guest.pate.net”, last_compile: nil, last_report: [FILTERED], updated_at: “2021-11-03 17:16:00”, created_at: “2021-11-03 17:16:00”, root_pass: nil,…