States will not be applied/appended (foreman/saltstack)

BadMuff · February 3, 2022, 9:56am

Problem:

Running Highstate through Foreman - no states will be appended to top.sls

Expected outcome:

highstate with the added/imported states for the vm

Foreman and Proxy versions:

2.5

Foreman and Proxy plugin versions:

2.5

Distribution and version:

Debian 11 (buster)

Other relevant data:

I am able to import salt states via foreman - if i am running highstate via salt-call no states will be appended to the standard top.sls

top.sls (compound should not affect anything actually)

main:
  'G@role:SALT-MASTER':
    - match: compound
    - salt.api-user          #lumiserve-salt git

  'G@role:API':
    - match: compound
    - common.node_environment #lumiserve-salt git

master.conf

auto_accept: True
order_masters: True
default_include: master.d/*.conf
interface: 192.168.0.50
ipv6: False
publish_port: 4505
user: root
enable_ssh_minions: True
ret_port: 4506
log_level: debug
show_timeout: True
cli_summary: False
use_yamlloader_old: False

master_tops:
  ext_nodes: /usr/bin/foreman-node

ext_pillar:
  - puppet: /usr/bin/foreman-node
  - git:
      - git@xxx.xxx/saltstack-pillar-dev.git

autosign_file: /etc/salt/autosign.conf

publisher_acl:
  foreman-proxy:
    - .*

external_auth:
  pam:
    saltuser:
      - .*
      - '@runner'
      - '@wheel'
      - '@jobs'
  rest:
    saltuser:
      - .*
      - '@runner'
      - '@wheel'
      - '@jobs'

rest_cherrypy:
  port: 8080
  host: 0.0.0.0
  debug: True
  disable_ssl: false
  ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/foreman01.uuxoi.local.pem
  ssl_crt: /etc/puppetlabs/puppet/ssl/certs/foreman01.uuxoi.local.pem
  ssl_ca: /etc/puppetlabs/puppet/ssl/certs/ca.pem

ext_pillar_first: True
pillarenv_from_saltenv: False
pillar_raise_on_missing: True
git_pillar_provider: pygit2
git_pillar_pubkey: /root/.ssh/id_rsa.pub
git_pillar_privkey: /root/.ssh/id_rsa

gitfs_user: anyusr
gitfs_global_lock: False
gitfs_privkey: /root/.ssh/id_rsa
gitfs_pubkey: /root/.ssh/id_rsa.pub
gitfs_provider: pygit2
gitfs_update_interval: 60
gitfs_ssl_verify: False

foreman.verifyssl: False
foreman.url: https://foreman01.uuxoi.local/foreman_api
foreman.user: admin # default is admin
foreman.password: blafasel # default is changeme

state_top: top.sls
state_top_saltenv: main

gitfs_remotes:
  - git@xxxx.xxx:renz/saltstack-gitfs-dev.git

gitfs_saltenv:
  - main:
    - mountpoint: salt://
    - ref: main

gitfs_ref_types:
  - branch

gitfs_refspecs:
  - '+refs/heads/*:refs/remotes/origin/*'
  - '+refs/tags/*:refs/tags/*'
  - '+refs/pull/*/head:refs/remotes/origin/pr/*'
  - '+refs/pull/*/merge:refs/remotes/origin/merge/*'

fileserver_backend:
  - gitfs

gitfs_env_whitelist:
    - main

salt_env_order:
    - main

top_file_merging_strategy: merge

event_return_whitelist:
  - salt/key

state_output: changes

#default_include: master.d/*.conf

output of foreman-node saltserver


---
classes:
- states.basic.remove_unattended_upgrades
- states.dfn-cert
- states.motd
- states.needrestart
parameters:
  foreman:
    hostname: foreman01
    fqdn: foreman01.anyfqdn.local
    hostgroup: default
    foreman_subnets: []
    foreman_interfaces:
    - ip: 192.168.0.50
      ip6: 2003:ed:e71a:4300:20c:29ff:fe52:d9fb
      mac: 00:0c:29:52:d9:fb
      name: foreman01.anyfqdn.local
      attrs: {}
      virtual: false
      link: true
      identifier: ens33
      managed: true
      primary: true
      provision: true
      subnet: 
      subnet6: 
      tag: 
      attached_to: 
      type: Interface
    location: Default Location
    location_title: Default Location
    organization: Default Organization
    organization_title: Default Organization
    domainname: uuxoi.local
    owner_name: anyuser
    owner_email: anyuser@0ea1.net
    ssh_authorized_keys:
    - ecdsa-sha2-nistp521 AAAAE2Vj.........
      anyuser@foreman01.anyfqdn.local
    foreman_users:
      anyuser:
        firstname: anyuser
        lastname: anyuser
        mail: anyuser@anymail.de
        description: ''
        fullname: anyuser
        name: anyuser
        ssh_authorized_keys:
        - type: ecdsa-sha2-nistp521
          key: AAAAE2...................
          comment: anyuser@foreman01.anyfqdn.local
    root_pw: 
    foreman_config_groups: []
    puppetmaster: foreman01.anyfqdn.local
    puppet_ca: foreman01.anyfqdn.local
    foreman_env: production
    host_packages: ''
    host_registration_insights: false
    host_registration_remote_execution: true
    remote_execution_ssh_keys:
    - ssh-rsa AAAAB3N..................
      foreman-proxy@foreman01
    remote_execution_ssh_user: root
    remote_execution_effective_user_method: sudo
    remote_execution_connect_by_ip: false
    salt_master: foreman01.anyfqdn.local
    saltenv: main
environment: main

highstate output:

[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: foreman01.uuxoi.local
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Override  __grains__: <module 'salt.loaded.int.log_handlers.sentry_mod' from '/usr/local/lib/python3.8/site-packages/salt/log/handlers/sentry_mod.py'>
[DEBUG   ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from '/usr/local/lib/python3.8/site-packages/salt/grains/zfs.py'>
[DEBUG   ] /etc/resolv.conf: The domain and search keywords are mutually exclusive.
[DEBUG   ] Unable to resolve address fe80::20c:29ff:fe52:d9fb: [Errno 1] Unknown host
[DEBUG   ] Elapsed time getting FQDNs: 0.18523001670837402 seconds
[DEBUG   ] LazyLoaded zfs.is_supported
[DEBUG   ] Connecting to master. Attempt 1 of 1
[DEBUG   ] Master URI: tcp://192.168.0.50:4506
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'foreman01.uuxoi.local', 'tcp://192.168.0.50:4506')
[DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (7980)
[DEBUG   ] Setting zmq_reconnect_ivl to '7980ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.0.50:4506
[DEBUG   ] Trying to connect to: tcp://192.168.0.50:4506
[DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] salt.crypt._get_key_with_evict: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] Connecting the Minion to the Master publish port, using the URI: tcp://192.168.0.50:4505
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Determining pillar cache
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'foreman01.uuxoi.local', 'tcp://192.168.0.50:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.0.50:4506
[DEBUG   ] Trying to connect to: tcp://192.168.0.50:4506
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded state.highstate
[DEBUG   ] LazyLoaded direct_call.execute
[DEBUG   ] Override  __grains__: <module 'salt.loaded.int.module.grains' from '/usr/local/lib/python3.8/site-packages/salt/modules/grains.py'>
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] LazyLoaded config.get
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'foreman01.uuxoi.local', 'tcp://192.168.0.50:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.0.50:4506
[DEBUG   ] Trying to connect to: tcp://192.168.0.50:4506
[DEBUG   ] Gathering pillar data for state run
[DEBUG   ] Finished gathering pillar data for state run
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] In saltenv 'main', looking at rel_path 'top.sls' to resolve 'salt://top.sls'
[DEBUG   ] In saltenv 'main', ** considering ** path '/var/cache/salt/minion/files/main/top.sls' to resolve 'salt://top.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/main/top.sls
[DEBUG   ] Jinja search path: ['/var/cache/salt/minion/files/main']
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'foreman01.uuxoi.local', 'tcp://192.168.0.50:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.0.50:4506
[DEBUG   ] Trying to connect to: tcp://192.168.0.50:4506
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/main/top.sls' using 'jinja' renderer: 0.01361703872680664
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/main/top.sls:
main:
  'G@role:SALT-MASTER':
    - match: compound
    - salt.api-user          #lumiserve-salt git

  'G@role:API':
    - match: compound
    - common.node_environment #lumiserve-salt git

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('main', OrderedDict([('G@role:SALT-MASTER', [OrderedDict([('match', 'compound')]), 'salt.api-user']), ('G@role:API', [OrderedDict([('match', 'compound')]), 'common.node_environment'])]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/main/top.sls' using 'yaml' renderer: 0.0006673336029052734
[DEBUG   ] LazyLoaded confirm_top.confirm_top
[DEBUG   ] LazyLoaded compound_match.match
[DEBUG   ] compound_match: foreman01.uuxoi.local ? G@role:SALT-MASTER
[DEBUG   ] LazyLoaded grain_match.match
[DEBUG   ] grains target: role:SALT-MASTER
[DEBUG   ] Attempting to match 'SALT-MASTER' in 'role' using delimiter ':'
[DEBUG   ] compound_match foreman01.uuxoi.local ? "G@role:SALT-MASTER" => "False"
[DEBUG   ] LazyLoaded compound_match.match
[DEBUG   ] compound_match: foreman01.uuxoi.local ? G@role:API
[DEBUG   ] LazyLoaded grain_match.match
[DEBUG   ] grains target: role:API
[DEBUG   ] Attempting to match 'API' in 'role' using delimiter ':'
[DEBUG   ] compound_match foreman01.uuxoi.local ? "G@role:API" => "False"
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] LazyLoaded state.check_result
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'foreman01.uuxoi.local', 'tcp://192.168.0.50:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.0.50:4506
[DEBUG   ] Trying to connect to: tcp://192.168.0.50:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] LazyLoaded highstate.output
local:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or master_tops data matches found. Please see master log for details.
     Changes:   

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms

/usr/sbin/upload-salt-reports foreman01.uuxoi.local

Success 20220203094427698609: b'{"task_id":"68bfb468-33bd-47cc-bc19-ea9bd460c646"}'

All seems to be fine - but states will not be added/appended (classes) to existing top.sls

mcorr · February 4, 2022, 6:04pm

Anyone in @atix have any ideas what might be happening?

Bernhard_Suttner · February 7, 2022, 10:59am

Which job template do you use? ssh / or salt provider?

Is maybe the following specification necessary:

file_roots:
  base:
    - /srv/salt

If I understand you correctly, foreman-node works as expected and does list the configured states?

brimioulle · February 8, 2022, 8:26am

Hi, this is a bug in saltstack. The file /usr/lib/python3.6/site-packages/salt/tops/ext_nodes.py needs to be adjusted in order to assign salt states from the foreman gui. See the following pull request:

github.com/saltstack/salt

Fixes #60872 - ext_nodes does not pass minion hostname

saltstack:master ← ATIX-AG:fix_ext_nodes_subprocess

opened 12:58PM - 13 Sep 21 UTC

bastian-src

+9 -5

### What issues does this PR fix or reference? Fixes: [#60872](https://github.c…om/saltstack/salt/issues/60872). When using `shell=True`, the command is supposed to be passed as a String. Otherwise, the parameters are **not** evaluated and therefore not passed to the command. More information can be found [here](https://docs.python.org/3/library/subprocess.html#popen-constructor): > The shell argument (which defaults to False) specifies whether to use the shell as the program to execute. If shell is True, it is recommended to pass args as a string rather than as a sequence. ### Previous Behavior Pass an array to `subprocess.run` ### New Behavior Pass a string to `subprocess.run` ### Commits signed with GPG? No

BadMuff · February 8, 2022, 11:42am

Hi Brimioule,

thanks for the hint. I’ve done some installations on different vms, sometimes with ubuntu sometimes with debian but always the same dejavue. Will checkout your hint - thanks a lot!!

My old Installations runs fine, but there is a reporting bug. Sometime the reports for a vm are broken and the poor fix is - rm /etc/salt/master/last_uploaded. did try some other methods too, need to investigate.

Best regards, Alex

BadMuff · February 8, 2022, 12:20pm

Thanks! Fix is working well, you saved me a lot of time.

THANKS!