Foreman 2.1.2 + salt 3001: undefined method `merge’ for nil:NilClass

fbarco · September 9, 2020, 1:13pm

Hello,

I’ve seen this problem already reported but i didn’t found any solution.
I can accept a salt key in foreman smart proxy, but the salt_minion host is not created when salt_master execute foreman-node

It 's OK if i run foreman-node my-salt-minion-host-name manually or if i run salt-call --grains from salt-minion host.

Foreman and Proxy versions:

foreman-installer.noarch                                1:2.1.2-1.el8                                    @foreman
foreman.noarch                                          2.1.2-1.el8                                      @foreman
foreman-proxy.noarch                                    2.1.2-1.el8                                      @foreman

Foreman and Proxy plugin versions:

rubygem-foreman_salt.noarch                             13.2.3-1.fm2_1.el8                               @foreman-plugins 
rubygem-smart_proxy_salt.noarch                         3.1.2-3.fm2_1.el8                                @foreman-plugins 
rubygem-smart_proxy_salt_core.noarch                    0.0.3-1.el8                                      @foreman-plugins

Distribution and version:

CentOS Linux release 8.1.1911 (Core)

Other relevant data:

2020-09-09 12:29:55,999 [salt.loaded.int.module.cmdmod:854 ][ERROR   ][8429] Command '['/usr/bin/foreman-node', 'sphynx.ac-test.fr']' failed with return code: 1
2020-09-09 12:29:56,000 [salt.loaded.int.module.cmdmod:856 ][ERROR   ][8429] stdout: Could not get grains: undefined method `merge' for nil:NilClass
Couldn't retrieve ENC data: Could not send facts to Foreman: exit
2020-09-09 12:29:56,000 [salt.loaded.int.module.cmdmod:860 ][ERROR   ][8429] retcode: 1
2020-09-09 12:29:56,000 [salt.loaded.int.module.cmdmod:1218][ERROR   ][8429] Command '/usr/bin/foreman-node sphynx.ac-test.fr' failed with return code: 1
2020-09-09 12:29:56,001 [salt.loaded.int.module.cmdmod:1223][ERROR   ][8429] output: Could not get grains: undefined method `merge' for nil:NilClass
Couldn't retrieve ENC data: Could not send facts to Foreman: exit
2020-09-09 12:29:56,001 [salt.loaded.int.pillar.puppet:31  ][CRITICAL][8429] YAML data from /usr/bin/foreman-node failed to parse

fbarco · September 14, 2020, 1:01pm

It seems grains cache is created after the script foreman-node tries to get it.
Is there a timeout or something else i can configure ?

Bernhard_Suttner · September 15, 2020, 11:28am

Looks like, no grains can be gathered from salt.
Can you try to use the following command to get the cached grains for the minion specified with

salt-run -l quit --output=json cache.grains <minion>

fbarco · September 15, 2020, 11:51am

I’ve already tried this and i have the same problem :
salt-key -a <minion> -y && salt-run -l quiet --output=json cache.grains <minion>
If i wait some seconds after that, cache is built (/var/cache/salt/master/<minion>/data.p file).
I’ve written a dirty bash script to see if this is a cache problem and replace foreman-node by it in ext_pillar config and it works :

#!/bin/bash
/usr/bin/salt "$1" -l quiet --out no_out_quiet saltutils.sync_grains
/usr/bin/foreman-node $1

Bernhard_Suttner · September 15, 2020, 11:55am

In this case, I would suggest to try

github.com/theforeman/smart_proxy_salt

add option to retrieve cache from files instead of using salt-run

theforeman:master ← bochi:foreman-node-filecache

opened 06:26AM - 18 Jun 20 UTC

bochi

+30 -5

This adds an option "filecache". If set to true, foreman-node will no longer u…se salt-run to retrieve the cache but directly deserialize the data from the files in /var/cache/salt/master/minions We are running foreman-salt in a relatively large deployment, with 1500 minions on the largest proxy. When scaling up the saltmaster to more than 20 threads foreman-node will fail to retrieve the grains and return the ENC data and blocks salt processes. Comparision: When getting grains from cache using salt-run: `# time scl enable rh-ruby25 tfm -- foreman-node <host>` `[output stripped...]` `real 0m7.033s` `user 0m6.073s` `sys 0m0.430s` When using filecache: `# time scl enable rh-ruby25 tfm ./foreman-node2 <host>` `[output stripped...]` `real 0m0.780s` `user 0m0.236s` `sys 0m0.056s` This has been tested and is working fine on multiple nodes. We'd love to get the changes upstream rather than working with a self-patched version in our environment :) Thanks

fbarco · September 15, 2020, 1:38pm

Thanks, but how can i install rubygem-msgpack ?

Bernhard_Suttner · September 15, 2020, 1:44pm

Try this: https://yum.theforeman.org/plugins/nightly/el7/x86_64/tfm-rubygem-msgpack-1.3.3-1.el7.x86_64.rpm if possible

fbarco · September 15, 2020, 2:17pm

I’ll try with https://yum.theforeman.org/releases/nightly/el8/x86_64/foreman-release.rpm because there is dependency problem.
I’ve installed foreman with https://yum.theforeman.org/releases/2.1/el8/x86_64/foreman-release.rpm

fbarco · September 16, 2020, 12:33pm

It’s the same problem :

2020-09-16 12:28:20,774 [salt.loaded.int.module.cmdmod:854 ][ERROR   ][29748] Command '['/usr/bin/foreman-node', 'sphynx.ac-test.fr']' failed with return code: 1
2020-09-16 12:28:20,774 [salt.loaded.int.module.cmdmod:856 ][ERROR   ][29748] stdout: Could not get grains: No such file or directory @ rb_sysopen - /var/cache/salt/master/minions/sphynx.ac-test.fr/data.p
Couldn't retrieve ENC data: Could not send facts to Foreman: exit
2020-09-16 12:28:20,775 [salt.loaded.int.module.cmdmod:860 ][ERROR   ][29748] retcode: 1

After a few seconds, the data.p file is created, but it’s too late.
For both get_grains_from_saltrun and get_grains_from_filecache, it should wait a little if cache is not ready immediatly, don’t you think ?