Upload-salt-reports hanging or killed

Jeff_Sparrow · January 19, 2024, 1:26pm

We’re trying to get salt reports uploaded to foreman and hitting a hang/kill issue in the script. I added some print commands to each of the functions. Looks like it gets to the run block where it appears to want to read /etc/salt/master config:

def run(*args, **kwargs):
    print("run")
    __opts__ = salt.config.master_config(
            os.environ.get('SALT_MASTER_CONFIG', '/etc/salt/master'))
    #print(__opts__)
    runner = salt.runner.Runner(__opts__)
    #print(runner)
    with io.open(os.devnull, 'w') as f:
        print("io open block")
        print(f)
        stdout_bak, sys.stdout = sys.stdout, f
        print("stdout block")
        try:
            ret = runner.cmd(*args, **kwargs)
            #print(ret)
        finally:
            sys.stdout = stdout_bak
    return ret['data'] if 'data' in ret else ret

Here is what that looks like when I run it:

[root@10-222-76-237 salt]# /usr/sbin/upload-salt-reports
get_lock
upload(jobs)
salt_config
jobs_to_upload
run
io open block
f is <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>
Killed

The Killed is the output of the script, not a print command.

Any idea what is going on here? I know that most of our salt config lives in /etc/salt/master.d/<confs> is this the issue?

Jeff_Sparrow · January 19, 2024, 7:12pm

Interestingly, when I run the script my CPU (16 core) and RAM (24GB) maxes out. There is literally nothing going on with this server, something seems very wrong.

Bernhard_Suttner · January 22, 2024, 8:12am

Are you using a user to run the script, who is not allowed to open os.devnull? E.g. selinux is preventing it? Can you check the logs? /var/log/audit/audit.log ?

Additionally, the “new” way to upload reports is this: Configuring Hosts Using Salt

Would it be possible to switch to this method?

Jeff_Sparrow · January 22, 2024, 12:16pm

Thank you very much for the reply.

The user running the script is root:root so should have access to everything. I see nothing in /var/log/audit/audit.log when running the script.

I am currently using the new way to upload reports. This is reproducible on any and all of our saltmasters. Run the script, gets stuck at the same point, locks up the entire virtual machine and it has to be rebooted from vsphere.

I have some suspicion that its having issues reading our config. Here is the __opts__ var it gets from the run block:

{'interface': '0.0.0.0', 'publish_port': 4505, 'zmq_backlog': 1000, 'pub_hwm': 1000, 'auth_mode': 1, 'user': 'salt', 'worker_threads': 6, 'sock_dir': '/var/run/salt/master', 'sock_pool_size': 1, 'ret_port': 4506, 'timeout': 15, 'keep_jobs': 24, 'archive_jobs': False, 'root_dir': '/', 'pki_dir': '/etc/salt/pki/master', 'key_cache': '', 'cachedir': '/var/cache/salt/master', 'file_roots': {'__env__': ['/srv/salt']}, 'master_roots': {'base': ['/srv/salt-master']}, 'pillar_roots': {'base': ['/srv/pillar', '/srv/spm/pillar']}, 'on_demand_ext_pillar': ['libvirt', 'virtkey'], 'decrypt_pillar': [], 'decrypt_pillar_delimiter': ':', 'decrypt_pillar_default': 'gpg', 'decrypt_pillar_renderers': ['gpg'], 'gpg_decrypt_must_succeed': False, 'thoriumenv': None, 'thorium_top': 'salt://<redacted>-thorium/top.sls', 'thorium_interval': 0.5, 'thorium_roots': {'base': ['/srv/thorium']}, 'top_file_merging_strategy': 'merge', 'env_order': [], 'saltenv': None, 'lock_saltenv': False, 'pillarenv': None, 'default_top': 'base', 'file_client': 'local', 'local': True, 'roots_update_interval': 60, 'azurefs_update_interval': 60, 'gitfs_update_interval': 60, 'git_pillar_update_interval': 60, 'hgfs_update_interval': 60, 'minionfs_update_interval': 60, 's3fs_update_interval': 60, 'svnfs_update_interval': 60, 'git_pillar_base': 'master', 'git_pillar_branch': 'master', 'git_pillar_env': '', 'git_pillar_fallback': '', 'git_pillar_root': '', 'git_pillar_ssl_verify': True, 'git_pillar_global_lock': True, 'git_pillar_user': '', 'git_pillar_password': '', 'git_pillar_insecure_auth': False, 'git_pillar_privkey': '/etc/salt/pki/master/salt_id_rsa', 'git_pillar_pubkey': '/etc/salt/pki/master/salt_id_rsa.pub', 'git_pillar_passphrase': '', 'git_pillar_refspecs': ['+refs/heads/*:refs/remotes/origin/*', '+refs/tags/*:refs/tags/*'], 'git_pillar_includes': True, 'git_pillar_verify_config': True, 'gitfs_remotes': [{'git@code.<redacted>.dev:salt/<redacted>-baseline.git': [{'name': 'baseline'}, {'mountpoint': 'salt://<redacted>-baseline'}, {'saltenv': [{'base': [{'ref': 'master'}]}, {'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/arch_gitfstesting.git': [{'name': 'precedence'}, {'saltenv': [{'base': [{'ref': 'master'}]}, {'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-orchestration.git': [{'name': 'old-orchestration'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-orchestration.git': [{'name': 'orchestration'}, {'mountpoint': 'salt://<redacted>-orchestration'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/awd.git': [{'mountpoint': 'salt://awd'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-database.git': [{'name': 'old-database'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-database.git': [{'name': 'database'}, {'mountpoint': 'salt://<redacted>-database'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-middleware.git': [{'name': 'old-middleware'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-middleware.git': [{'name': 'middleware'}, {'mountpoint': 'salt://<redacted>-middleware'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-monitoring.git': [{'mountpoint': 'salt://<redacted>-monitoring'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-windows.git': [{'mountpoint': 'salt://<redacted>-windows'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-unix.git': [{'mountpoint': 'salt://<redacted>-unix'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-vmware.git': [{'mountpoint': 'salt://<redacted>-vmware'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-hsappsupport.git': [{'mountpoint': 'salt://<redacted>-hsappsupport'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-algo.git': [{'mountpoint': 'salt://<redacted>-algo'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-alps.git': [{'mountpoint': 'salt://<redacted>-alps'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-ars.git': [{'mountpoint': 'salt://<redacted>-ars'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-confluent.git': [{'mountpoint': 'salt://<redacted>-confluent'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-directoryservices.git': [{'mountpoint': 'salt://<redacted>-directoryservices'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-onesecurity.git': [{'mountpoint': 'salt://<redacted>-onesecurity'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-tnr.git': [{'mountpoint': 'salt://<redacted>-tnr'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-pmg.git': [{'mountpoint': 'salt://<redacted>-pmg'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-ccs.git': [{'mountpoint': 'salt://<redacted>-ccs'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-thorium.git': [{'name': 'thorium'}, {'mountpoint': 'salt://<redacted>-thorium'}, {'saltenv': [{'base': [{'ref': 'master'}]}, {'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-innovest.git': [{'name': 'innovest'}, {'mountpoint': 'salt://<redacted>-innovest'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-advent.git': [{'name': 'advent'}, {'mountpoint': 'salt://<redacted>-advent'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-common.git': [{'name': 'common'}, {'mountpoint': 'salt://<redacted>-common'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-observability.git': [{'name': 'observability'}, {'mountpoint': 'salt://<redacted>-observability'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-citrix.git': [{'name': 'citrix'}, {'mountpoint': 'salt://<redacted>-citrix'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-advs-devops-saltrepo.git': [{'name': 'advs-devops-saltrepo'}, {'mountpoint': 'salt://<redacted>-advs-devops-saltrepo'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-launch': [{'name': '<redacted>-launch'}, {'mountpoint': 'salt://<redacted>-launch'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-business_services.git': [{'name': 'business_services'}, {'mountpoint': 'salt://<redacted>-business_services'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-evare.git': [{'name': 'evare'}, {'mountpoint': 'salt://<redacted>-evare'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-vdi-services.git': [{'name': 'vdi-services'}, {'mountpoint': 'salt://<redacted>-vdi-services'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-blackdiamond.git': [{'name': 'blackdiamond'}, {'mountpoint': 'salt://<redacted>-blackdiamond'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-<redacted>es.git': [{'name': '<redacted>es'}, {'mountpoint': 'salt://<redacted>-<redacted>es'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-gidsemea.git': [{'name': 'gidsemea'}, {'mountpoint': 'salt://<redacted>-gidsemea'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-skyline.git': [{'name': 'skyline'}, {'mountpoint': 'salt://<redacted>-skyline'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}, {'git@code.<redacted>.dev:salt/<redacted>-aloha-db-ops.git': [{'name': 'aloha-db-ops'}, {'mountpoint': 'salt://<redacted>-aloha-db-ops'}, {'saltenv': [{'prod': [{'ref': 'master'}]}, {'uat': [{'ref': 'uat'}]}, {'devtest': [{'ref': 'devtest'}]}, {'sandbox': [{'ref': 'sandbox'}]}]}]}], 'gitfs_mountpoint': '', 'gitfs_root': '', 'gitfs_base': 'master', 'gitfs_fallback': 'can_be_anything', 'gitfs_user': '', 'gitfs_password': '', 'gitfs_insecure_auth': False, 'gitfs_privkey': '/etc/salt/pki/master/salt_id_rsa', 'gitfs_pubkey': '/etc/salt/pki/master/salt_id_rsa.pub', 'gitfs_passphrase': '', 'gitfs_saltenv_whitelist': [], 'gitfs_saltenv_blacklist': [], 'gitfs_global_lock': False, 'gitfs_ssl_verify': True, 'gitfs_saltenv': [], 'gitfs_ref_types': ['branch', 'tag', 'sha'], 'gitfs_refspecs': ['+refs/heads/*:refs/remotes/origin/*', '+refs/tags/*:refs/tags/*'], 'gitfs_disable_saltenv_mapping': False, 'hgfs_remotes': [], 'hgfs_mountpoint': '', 'hgfs_root': '', 'hgfs_base': 'default', 'hgfs_branch_method': 'branches', 'hgfs_saltenv_whitelist': [], 'hgfs_saltenv_blacklist': [], 'show_timeout': True, 'show_jid': False, 'unique_jid': False, 'svnfs_remotes': [], 'svnfs_mountpoint': '', 'svnfs_root': '', 'svnfs_trunk': 'trunk', 'svnfs_branches': 'branches', 'svnfs_tags': 'tags', 'svnfs_saltenv_whitelist': [], 'svnfs_saltenv_blacklist': [], 'max_event_size': 1048576, 'master_stats': False, 'master_stats_event_iter': 60, 'minionfs_env': 'base', 'minionfs_mountpoint': '', 'minionfs_whitelist': [], 'minionfs_blacklist': [], 'ext_pillar': [{'puppet': '/usr/bin/foreman-node'}, {'git': ['master git@code.<redacted>.dev:salt/<redacted>-pillar.git']}], 'pillar_version': 2, 'pillar_opts': False, 'pillar_safe_render_error': True, 'pillar_source_merging_strategy': 'smart', 'pillar_merge_lists': False, 'pillar_includes_override_sls': False, 'pillar_cache': False, 'pillar_cache_ttl': 3600, 'pillar_cache_backend': 'disk', 'gpg_cache': False, 'gpg_cache_ttl': 86400, 'gpg_cache_backend': 'disk', 'ping_on_rotate': False, 'peer': {}, 'preserve_minion_cache': False, 'syndic_master': 'masterofmasters', 'syndic_failover': 'random', 'syndic_forward_all_events': False, 'syndic_log_file': '/var/log/salt/syndic', 'syndic_pidfile': '/var/run/salt-syndic.pid', 'outputter_dirs': [], 'runner_dirs': ['/usr/local/lib/python3.6/site-packages/sseape/runners'], 'utils_dirs': ['/var/cache/salt/master/extmods/utils'], 'client_acl_verify': True, 'publisher_acl': {'foreman-proxy': ['state.template_str']}, 'publisher_acl_blacklist': {}, 'sudo_acl': False, 'external_auth': {'pam': {'sre%': ['.*', '@wheel', '@runner', '@jobs'], 'svc_iscloudapi': ['file.get_sum'], 'sn_patch': [{'@runner': ['state.orch']}], 'cdp_salt': [{'@wheel': ['key.delete']}], 'saltuser': ['.*', '@runner'], 'root%': ['.*', '@runner'], 'salt': ['.*']}}, 'token_expire': 1576800000, 'token_expire_user_override': False, 'permissive_acl': False, 'keep_acl_in_token': False, 'eauth_acl_module': '', 'eauth_tokens': 'localfs', 'extension_modules': '/var/cache/salt/master/extmods', 'module_dirs': ['/usr/local/lib/python3.6/site-packages/sseape/modules'], 'file_recv': False, 'file_recv_max_size': 100, 'file_buffer_size': 1048576, 'file_ignore_regex': [], 'file_ignore_glob': [], 'fileserver_backend': ['roots', 'gitfs'], 'fileserver_followsymlinks': True, 'fileserver_ignoresymlinks': False, 'fileserver_verify_config': True, 'max_open_files': 100000, 'hash_type': 'sha256', 'optimization_order': [0, 1, 2], 'conf_file': '/etc/salt/master', 'open_mode': False, 'auto_accept': False, 'renderer': 'jinja|yaml', 'renderer_whitelist': [], 'renderer_blacklist': [], 'failhard': False, 'state_top': 'top.sls', 'state_top_saltenv': None, 'master_tops': {'ext_nodes': '/usr/bin/foreman-node'}, 'master_tops_first': False, 'order_masters': False, 'job_cache': True, 'ext_job_cache': '', 'master_job_cache': 'sseapi', 'job_cache_store_endtime': False, 'minion_data_cache': True, 'enforce_mine_cache': False, 'ipc_mode': 'ipc', 'ipc_write_buffer': 0, 'req_server_niceness': None, 'pub_server_niceness': None, 'fileserver_update_niceness': None, 'mworker_niceness': None, 'mworker_queue_niceness': None, 'maintenance_niceness': None, 'event_return_niceness': None, 'event_publisher_niceness': None, 'reactor_niceness': None, 'ipv6': None, 'tcp_master_pub_port': 4512, 'tcp_master_pull_port': 4513, 'tcp_master_publish_pull': 4514, 'tcp_master_workers': 4515, 'log_file': '/var/log/salt/master', 'log_level': 'warning', 'log_level_logfile': False, 'log_datefmt': '%H:%M:%S', 'log_datefmt_logfile': '%Y-%m-%d %H:%M:%S', 'log_fmt_console': '[%(levelname)-8s] %(message)s', 'log_fmt_logfile': '%(asctime)s,%(msecs)03d [%(name)-17s:%(lineno)-4d][%(levelname)-8s][%(process)d] %(message)s', 'log_fmt_jid': '[JID: %(jid)s]', 'log_granular_levels': {'salt': 'warning', 'sseape': 'info', 'sseapiclient': 'warning', 'salt.loaded.int.engines': 'info'}, 'log_rotate_max_bytes': 0, 'log_rotate_backup_count': 0, 'pidfile': '/var/run/salt-master.pid', 'publish_session': 86400, 'range_server': 'range:80', 'reactor': [{'salt/beacon/*/inotify//etc/filebeat/filebeat.yml': ['salt://<redacted>-thorium/reactor/filebeat_inotify_reactor.sls']}, {'salt/auth': ['/usr/share/foreman-proxy/salt/reactors/foreman_minion_auth.sls']}, {'salt/job/*/ret/*': ['/usr/share/foreman-proxy/salt/reactors/foreman_report_upload.sls']}, {'salt/beacon/*/service/NwADASitSvc': ['salt://<redacted>-thorium/reactor/netwrix_service_reactor.sls']}, {'salt/beacon/*/service/NwArchiveSvc': ['salt://<redacted>-thorium/reactor/netwrix_service_reactor.sls']}, {'salt/beacon/*/service/NwCfgServerSvc': ['salt://<redacted>-thorium/reactor/netwrix_service_reactor.sls']}, {'salt/beacon/*/service/NwCoreSvc': ['salt://<redacted>-thorium/reactor/netwrix_service_reactor.sls']}, {'salt/beacon/*/service/NwDataCollectionCoreSvc': ['salt://<redacted>-thorium/reactor/netwrix_service_reactor.sls']}, {'salt/beacon/*/service/NwNLASvc': ['salt://<redacted>-thorium/reactor/netwrix_service_reactor.sls']}, {'salt/beacon/*/service/NwSyslogCollectionSvc': ['salt://<redacted>-thorium/reactor/netwrix_service_reactor.sls']}, {'salt/beacon/*/service/NwManagementSvc': ['salt://<redacted>-thorium/reactor/netwrix_service_reactor.sls']}, {'salt/beacon/*/service/NwWatchdogSvc': ['salt://<redacted>-thorium/reactor/netwrix_service_reactor.sls']}, {'salt/beacon/*/service/winlogbeat': ['salt://<redacted>-thorium/reactor/service.sls']}, {'salt/beacon/*/service/filebeat': ['salt://<redacted>-thorium/reactor/service.sls']}, {'salt/beacon/*/service/Spooler': ['salt://<redacted>-thorium/reactor/service.sls']}], 'reactor_refresh_interval': 60, 'reactor_worker_threads': 10, 'reactor_worker_hwm': 10000, 'engines': [{'sseapi': {}}, {'eventqueue': {}}, {'rpcqueue': {}}, {'keyauth': {}}, {'jobcompletion': {}}, {'stalekey': {'interval': 3600, 'expire': 14400}}, {'thorium': {}}], 'event_return': ['sseapi'], 'event_return_queue': 1000, 'event_return_whitelist': [], 'event_return_blacklist': ['salt/auth'], 'event_match_type': 'startswith', 'runner_returns': True, 'serial': 'msgpack', 'test': False, 'state_verbose': True, 'state_output': 'changes', 'state_output_diff': False, 'state_output_profile': True, 'state_auto_order': True, 'state_events': False, 'state_aggregate': False, 'search': '', 'loop_interval': 60, 'nodegroups': {'win_ykt': ['WIN-6E7EO9576UV', 'WIN-CM35BL43B7M', 'WIN-354LIHUINQN', 'WIN-QELLIQ4PDN5', 'WIN-NQUTAINUBJ9.<redacted>direct.com'], 'win_kc': ['WIN-IPC21QJ4I0A'], 'win_stl': None, 'dc_fix': ['10-234-23-43.<redacted>-corp.global', '10-234-23-51.<redacted>-corp.global', '10-234-23-70-cloudad.<redacted>ad.global', '10-234-26-29.cloudad.<redacted>ad.global', '10-234-26-30.cloudad.<redacted>ad.global', '10-234-26-31.cloudad.<redacted>ad.global', '10-234-26-52.spla.<redacted>ad.global', '10-234-26-55.<redacted>-corp.cloud', '10-234-26-56.cloudad.<redacted>ad.global', '10-234-26-57.<redacted>-corp.global', '10-234-27-17.<redacted>-corp.cloud', '10-234-27-24.<redacted>-corp.cloud', '10-234-27-43.<redacted>-corp.cloud', '10-234-27-47.cloudad.<redacted>ad.global', '10-234-27-48.cloudad.<redacted>ad.global', '10-234-27-52.spla.<redacted>ad.global', '10-234-28-15.<redacted>-corp.global', '10-234-28-24.<redacted>-corp.global', '10-234-28-27.cloudad.<redacted>ad.global', '10-234-28-41.<redacted>-corp.cloud', '10-234-28-43.<redacted>-corp.cloud', '10-234-28-47.cloudad.<redacted>ad.global', '10-234-28-52.spla.<redacted>ad.global', '10-234-28-56.cloudad.<redacted>ad.global', '10-234-28-58.<redacted>-corp.cloud', '10-234-32-16.<redacted>-corp.cloud', '10-234-32-18.<redacted>-corp.cloud', '10-234-32-43.<redacted>-corp.cloud', '10-234-32-47.cloudad.<redacted>ad.global', '10-234-32-49.<redacted>-corp.cloud', '10-234-32-50.<redacted>-corp.cloud', '10-234-32-52.<redacted>-corp.cloud', '10-234-52-139.<redacted>-corp.cloud', '10-234-52-142.<redacted>-corp.cloud', '10-234-64-41.<redacted>-corp.cloud', '10-234-66-45.<redacted>-corp.cloud', '10-234-67-47.<redacted>-corp.cloud', '10-234-71-43.<redacted>-corp.cloud', '10-234-71-45.<redacted>-corp.cloud']}, 'ssh_list_nodegroups': {}, 'ssh_use_home_key': False, 'cython_enable': False, 'enable_gpu_grains': False, 'key_logfile': '/var/log/salt/key', 'verify_env': True, 'permissive_pki_access': True, 'key_pass': None, 'signing_key_pass': None, 'default_include': 'master.d/*.conf', 'winrepo_dir': '/srv/salt/win/repo', 'winrepo_dir_ng': '/srv/salt/win/repo-ng', 'winrepo_cachefile': 'winrepo.p', 'winrepo_remotes': ['https://github.com/saltstack/salt-winrepo.git'], 'winrepo_remotes_ng': ['https://github.com/saltstack/salt-winrepo-ng.git'], 'winrepo_branch': 'master', 'winrepo_fallback': '', 'winrepo_ssl_verify': True, 'winrepo_user': '', 'winrepo_password': '', 'winrepo_insecure_auth': False, 'winrepo_privkey': '', 'winrepo_pubkey': '', 'winrepo_passphrase': '', 'winrepo_refspecs': ['+refs/heads/*:refs/remotes/origin/*', '+refs/tags/*:refs/tags/*'], 'syndic_wait': 5, 'jinja_env': {'trim_blocks': True, 'lstrip_blocks': True}, 'jinja_sls_env': {}, 'jinja_lstrip_blocks': False, 'jinja_trim_blocks': False, 'tcp_keepalive': True, 'tcp_keepalive_idle': 300, 'tcp_keepalive_cnt': -1, 'tcp_keepalive_intvl': -1, 'sign_pub_messages': True, 'keysize': 2048, 'transport': 'zeromq', 'gather_job_timeout': 10, 'syndic_event_forward_timeout': 0.5, 'syndic_jid_forward_cache_hwm': 100, 'regen_thin': False, 'ssh_passwd': '', 'ssh_priv_passwd': '', 'ssh_port': '22', 'ssh_sudo': False, 'ssh_sudo_user': '', 'ssh_timeout': 60, 'ssh_user': 'root', 'ssh_scan_ports': '22', 'ssh_scan_timeout': 0.01, 'ssh_identities_only': False, 'ssh_log_file': '/var/log/salt/ssh', 'ssh_config_file': '/root/.ssh/config', 'cluster_mode': False, 'sqlite_queue_dir': '/var/cache/salt/master/queues', 'queue_dirs': [], 'cli_summary': False, 'max_minions': 0, 'master_sign_key_name': 'master_sign', 'master_sign_pubkey': True, 'master_pubkey_signature': 'master_pubkey_signature', 'master_use_pubkey_signature': False, 'zmq_filtering': False, 'zmq_monitor': False, 'con_cache': False, 'rotate_aes_key': True, 'cache_sreqs': True, 'dummy_pub': False, 'http_connect_timeout': 20.0, 'http_request_timeout': 3600.0, 'http_max_body': 107374182400, 'cache': 'localfs', 'memcache_expire_seconds': 0, 'memcache_max_items': 1024, 'memcache_full_cleanup': False, 'memcache_debug': False, 'thin_extra_mods': '', 'min_extra_mods': '', 'ssl': None, 'extmod_whitelist': {}, 'extmod_blacklist': {}, 'clean_dynamic_modules': True, 'django_auth_path': '', 'django_auth_settings': '', 'allow_minion_key_revoke': True, 'salt_cp_chunk_size': 98304, 'require_minion_sign_messages': False, 'drop_messages_signature_fail': False, 'discovery': False, 'schedule': {}, 'auth_events': True, 'minion_data_cache_events': True, 'enable_ssh_minions': False, 'netapi_allow_raw_shell': False, 'fips_mode': False, 'detect_remote_minions': False, 'remote_minions_port': 22, '__role': 'master', 'beacons_dirs': ['/usr/local/lib/python3.6/site-packages/sseape/beacons'], 'engines_dirs': ['/usr/local/lib/python3.6/site-packages/sseape/engines'], 'fileserver_dirs': ['/usr/local/lib/python3.6/site-packages/sseape/fileserver'], 'pillar_dirs': ['/usr/local/lib/python3.6/site-packages/sseape/pillar'], 'returner_dirs': ['/usr/local/lib/python3.6/site-packages/sseape/returners'], 'roster_dirs': ['/usr/local/lib/python3.6/site-packages/sseape/roster'], 'proxy_dirs': ['/usr/local/lib/python3.6/site-packages/sseape/proxy'], 'metaproxy_dirs': ['/usr/local/lib/python3.6/site-packages/sseape/metaproxy'], 'states_dirs': ['/usr/local/lib/python3.6/site-packages/sseape/states'], 'autosign_grains_dir': '/etc/salt/autosign_grains', 'autosign_file': '/etc/salt/autosign.conf', 'rest_cherrypy': {'port': 9191, 'ssl_key': '/etc/pki/tls/certs/localhost.key', 'ssl_crt': '/etc/pki/tls/certs/localhost.crt'}, 'git_pillar_provider': 'pygit2', 'gitfs_provider': 'pygit2', 'mattermost': {'hook': 'ugsbgr5dwbd9jph59sjq4o6yzy', 'api_url': 'https://chat.<redacted>tech.com/hooks', 'username': 'saltstack-test', 'channel': '<redacted>-Patching'}, 'id': 'sparrow-sb-master', 'sseapi_pubkey_path': '/etc/salt/pki/master/sseapi_key.pub', 'presence_events': True, 'sseapi_update_interval': 60, 'sseapi_server': 'https://sse-test-01.<redacted>-corp.cloud', 'sseapi_ssl_validate_cert': True, 'sseapi_event_queue': {'name': 'sseapi-events', 'strategy': 'always', 'push_interval': 10, 'batch_limit': 2000, 'age_limit': 129600, 'size_limit': 35000000, 'vacuum_interval': 86400, 'vacuum_limit': 350000, 'forward': []}, 'sseapi_rpc_queue': {'name': 'sseapi-rpc', 'strategy': 'always', 'push_interval': 10, 'batch_limit': 500, 'age_limit': 3600, 'size_limit': 360000, 'vacuum_interval': 86400, 'vacuum_limit': 100000}, 'sseapi_local_cache': {'load': 3600, 'tgt': 300}, 'event_return_queue_max_seconds': 10, '__cli': 'upload-salt-reports', 'token_dir': '/var/cache/salt/master/tokens', 'syndic_dir': '/var/cache/salt/master/syndics'}

Bernhard_Suttner · January 22, 2024, 1:57pm

why do you want to use the old way with the upload_salt_reports script?

Jeff_Sparrow · January 22, 2024, 2:22pm

Oh! I see. I missed where it says "Alternatively … you can upload manually … " I didnt realize that was referring to an old method.

Ok, let me poke more into the reactor runner service. Thanks.

Jeff_Sparrow · January 22, 2024, 2:25pm

Im assuming my reactor system is working, or attempting to work. Is this the expected logging of when its attempting to upload a report and/or grains?

2024-01-22 14:24:09,941 [salt.loaded.int.module.cmdmod:907 ][ERROR   ][611439] stdout: Couldn't retrieve ENC data: Could not send facts to Foreman: Neither PUB key nor PRIV key: nested asn1 error
2024-01-22 14:24:09,941 [salt.loaded.int.module.cmdmod:911 ][ERROR   ][611439] retcode: 1
2024-01-22 14:24:09,941 [salt.loaded.int.module.cmdmod:1328][ERROR   ][611439] Command '/usr/bin/foreman-node' failed with return code: 1
2024-01-22 14:24:09,942 [salt.loaded.int.module.cmdmod:1333][ERROR   ][611439] output: Couldn't retrieve ENC data: Could not send facts to Foreman: Neither PUB key nor PRIV key: nested asn1 error
2024-01-22 14:24:09,942 [salt.loaded.int.pillar.puppet:27  ][CRITICAL][611439] YAML data from /usr/bin/foreman-node failed to parse

The above is what I see in the /var/log/salt/master logs after I do a simple test.ping on a minion.

Jeff_Sparrow · January 22, 2024, 3:11pm

Im not totally understanding the keys here:

2024-01-22 15:08:57,704 [salt.loaded.int.module.cmdmod:905 ][ERROR   ][629296] Command '/usr/bin/foreman-node' failed with return code: 1
2024-01-22 15:08:57,705 [salt.loaded.int.module.cmdmod:907 ][ERROR   ][629296] stdout: Couldn't retrieve ENC data: Could not send facts to Foreman: Permission denied @ rb_sysopen - /etc/puppetlabs/puppet/ssl/private_keys/10-222-76-237.<redacted>-corp.cloud.pem
2024-01-22 15:08:57,705 [salt.loaded.int.module.cmdmod:911 ][ERROR   ][629296] retcode: 1
2024-01-22 15:08:57,705 [salt.loaded.int.module.cmdmod:1328][ERROR   ][629296] Command '/usr/bin/foreman-node' failed with return code: 1
2024-01-22 15:08:57,705 [salt.loaded.int.module.cmdmod:1333][ERROR   ][629296] output: Couldn't retrieve ENC data: Could not send facts to Foreman: Permission denied @ rb_sysopen - /etc/puppetlabs/puppet/ssl/private_keys/10-222-76-237.<redacted>-corp.cloud.pem
2024-01-22 15:08:57,706 [salt.loaded.int.pillar.puppet:27  ][CRITICAL][629296] YAML data from /usr/bin/foreman-node failed to parse

This key is owned by puppet:puppet which is required for foreman-proxy to start. However, the salt master is assuming its own by salt:salt - How can both be true?

Jeff_Sparrow · January 22, 2024, 7:30pm

ok. Well after working on this all day, here is where I am. I cannot get past SSL issues whatsoever. Ive gotten a hostname mismatch error, so I regenerated them all with the hostname, and all alt-names. But I still hit the same issue:

2024-01-22 19:21:19,724 [salt.loaded.int.module.cmdmod:907 ][ERROR   ][715839] stdout: Couldn't retrieve ENC data: Could not send facts to Foreman: SSL_connect returned=1 errno=0 state=error: certificate verify failed (Hostname mismatch)
2024-01-22 19:21:19,725 [salt.loaded.int.module.cmdmod:911 ][ERROR   ][715839] retcode: 1
2024-01-22 19:21:19,725 [salt.loaded.int.module.cmdmod:1328][ERROR   ][715839] Command '/usr/bin/foreman-node' failed with return code: 1
2024-01-22 19:21:19,725 [salt.loaded.int.module.cmdmod:1333][ERROR   ][715839] output: Couldn't retrieve ENC data: Could not send facts to Foreman: SSL_connect returned=1 errno=0 state=error: certificate verify failed (Hostname mismatch)
2024-01-22 19:21:19,725 [salt.loaded.int.pillar.puppet:27  ][CRITICAL][715839] YAML data from /usr/bin/foreman-node failed to parse

[root@10-222-76-237 ssl]# hostname
10-222-76-237.<redacted>-corp.cloud

[root@10-222-76-237 ssl]# puppetserver ca list --all
Signed Certificates:
    10-222-76-237.<redacted>-corp.cloud       (SHA256)  70:D2:FC:87:59:E5:9D:B8:58:40:A8:75:7E:53:CD:D7:90:40:85:A6:07:81:89:57:34:2E:CF:C6:64:27:98:A5        alt names: ["DNS:10-222-76-237.<redacted>-corp.cloud", "DNS:sparrow-sb-master", "DNS:sparrow-sb-master.<redacted>-corp.cloud"]        authorization extensions: [pp_cli_auth: true]

Im really at a loss here. Im not sure if uploading reports is even going to work once I get past this.

Any help would be super duper greatly appreciated.

Jeff_Sparrow · January 22, 2024, 9:16pm

ok! I may have gotten somewhere. I had to add localhost and 127.0.0.1 to the alt-names, which is a bit loony. However, I now am hitting a 404 issue with uploading:

2024-01-22T21:04:21  [D] accept: 10.222.76.237:46508
2024-01-22T21:04:21  [D] Rack::Handler::WEBrick is invoked.
2024-01-22T21:04:21 b298f45a [I] Started POST /api/hosts/facts
2024-01-22T21:04:21 b298f45a [I] Finished POST /api/hosts/facts with 404 (0.48 ms)
2024-01-22T21:04:21  [D] close: 10.222.76.237:46508
2024-01-22T21:04:21  [D] accept: 10.222.76.237:46520
2024-01-22T21:04:21  [D] Rack::Handler::WEBrick is invoked.
2024-01-22T21:04:21 cec33784 [I] Started GET /salt/node/10-222-10-5.<redacted>-corp.cloud format=yml
2024-01-22T21:04:21 cec33784 [D] verifying remote client 10.222.76.237 against trusted_hosts ["10-222-76-237.<redacted>-corp.cloud"]
2024-01-22T21:04:21 cec33784 [I] Finished GET /salt/node/10-222-10-5.<redacted>-corp.cloud with 404 (0.55 ms)
2024-01-22T21:04:21  [D] close: 10.222.76.237:46520

Im running the script ad-hoc, seeing the grains and all so thats good. But its failing to find foreman-proxy, which I dont really understand why:

I added a puts of the url its attempting to GET:
https://10-222-76-237.<redacted>-corp.cloud:8443/api/hosts/facts
and the service is certainly running and listening:

[root@10-222-76-237 ssl]# lsof -i :8443
COMMAND      PID          USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
smart-pro 725757 foreman-proxy    7u  IPv4 10496547      0t0  TCP *:pcsync-https (LISTEN)
smart-pro 725757 foreman-proxy    8u  IPv6 10496548      0t0  TCP *:pcsync-https (LISTEN)

and I can manually curl features:

curl --cert /etc/puppetlabs/puppet/ssl/certs/10-222-76-237.<redacted>-corp.cloud.pem --key /etc/puppetlabs/puppet/ssl/private_keys/10-222-76-237.<redacted>-corp.cloud.pem --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem https://localhost:8443/features
["dynflow","logs","puppet","puppetca","salt"][root@10-222-76-237 ssl]

What is possibly missing?

Jeff_Sparrow · January 23, 2024, 2:12pm

I modified the /etc/salt/foreman.yaml and changed the host/port/username/password to match foreman, not foreman-proxy.