Node.rb failing to find node after upgrade to 1.24.3

Problem:
After upgrading foreman to version 1.24.3 from version 1.23.2, all puppet runs fail with the error:

Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Server Error: Failed to find myHost via exec: Execution of ‘/etc/puppetlabs/puppet/node.rb myHost’ returned 1:
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node myHost: Failed to find myHost via exec: Execution of ‘/etc/puppetlabs/puppet/node.rb myHost’ returned 1:
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

We deploy foreman using puppet and have no issues until the version is changed to 1.24.3.

Expected outcome:
Node.rb gets the specified node definition.

Other relevant data:
In the foreman production log:

2020-06-09T03:48:17 [I|app|03dea261] Started GET “/node/foreman.hcilabs.hpecorp.net?format=yml” for 16.100.202.19 at 2020-06-09 03:48:17 -0400
2020-06-09T03:48:17 [I|app|03dea261] Processing by HostsController#externalNodes as YML
2020-06-09T03:48:17 [I|app|03dea261] Parameters: {“name”=>“myForemanHost”}
2020-06-09T03:48:17 [W|app|03dea261] Failed to generate external nodes for myForemanHost
2020-06-09T03:48:17 [I|app|03dea261] Rendering text template
2020-06-09T03:48:17 [I|app|03dea261] Rendered text template (0.0ms)
2020-06-09T03:48:17 [I|app|03dea261] Completed 412 Precondition Failed in 127ms (Views: 0.4ms | ActiveRecord: 13.8ms)

412 Precondition Failed also seems to get thrown every time node.rb is run.

can you try changing the logging level to debug (Foreman :: Manual) and running the request again so we have a full stack trace please? Also, can you share the facter output from a node that is failing?

Here is the facter output:

{
  "aio_agent_version": "5.5.20",
  "augeas": {
    "version": "1.12.0"
  },
  "disks": {
    "sda": {
      "model": "Virtual disk",
      "size": "64.00 GiB",
      "size_bytes": 68719476736,
      "vendor": "VMware"
    },
    "sdb": {
      "model": "Virtual disk",
      "size": "128.00 GiB",
      "size_bytes": 137438953472,
      "vendor": "VMware"
    },
    "sdc": {
      "model": "Virtual disk",
      "size": "128.00 GiB",
      "size_bytes": 137438953472,
      "vendor": "VMware"
    },
    "sr0": {
      "model": "VMware IDE CDR00",
      "size": "1.00 GiB",
      "size_bytes": 1073741312,
      "vendor": "NECVMWar"
    }
  },
  "dmi": {
    "bios": {
      "release_date": "06/19/2019",
      "vendor": "VMware, Inc.",
      "version": "VMW71.00V.13989454.B64.1906190538"
    },
    "board": {
      "manufacturer": "Intel Corporation",
      "product": "440BX Desktop Reference Platform",
      "serial_number": "None"
    },
    "chassis": {
      "asset_tag": "No Asset Tag",
      "type": "Other"
    },
    "manufacturer": "VMware, Inc.",
    "product": {
      "name": "VMware7,1",
      "serial_number": "VMware-42 37 ed a6 af e8 6e 65-e1 4e f4 1a b5 53 bf 0b",
      "uuid": "A6ED3742-E8AF-656E-E14E-F41AB553BF0B"
    }
  },
  "facterversion": "3.11.13",
  "filesystems": "ext2,ext3,ext4,vfat,xfs",
  "fips_enabled": false,
  "hypervisors": {
    "vmware": {}
  },
  "identity": {
    "gid": 0,
    "group": "root",
    "privileged": true,
    "uid": 0,
    "user": "root"
  },
  "is_virtual": true,
  "kernel": "Linux",
  "kernelmajversion": "3.10",
  "kernelrelease": "3.10.0-1127.10.1.el7.x86_64",
  "kernelversion": "3.10.0",
  "load_averages": {
    "15m": 0.41,
    "1m": 0.27,
    "5m": 0.54
  },
  "memory": {
    "swap": {
      "available": "3.87 GiB",
      "available_bytes": 4160745472,
      "capacity": "0%",
      "total": "3.87 GiB",
      "total_bytes": 4160745472,
      "used": "0 bytes",
      "used_bytes": 0
    },
    "system": {
      "available": "6.07 GiB",
      "available_bytes": 6519066624,
      "capacity": "20.51%",
      "total": "7.64 GiB",
      "total_bytes": 8200601600,
      "used": "1.57 GiB",
      "used_bytes": 1681534976
    }
  },
  "mountpoints": {
    "/": {
      "available": "37.36 GiB",
      "available_bytes": 40111083520,
      "capacity": "5.60%",
      "device": "/dev/mapper/centos-root",
      "filesystem": "xfs",
      "options": [
        "rw",
        "seclabel",
        "relatime",
        "attr2",
        "inode64",
        "noquota"
      ],
      "size": "39.57 GiB",
      "size_bytes": 42492706816,
      "used": "2.22 GiB",
      "used_bytes": 2381623296
    },
    "/boot": {
      "available": "872.52 MiB",
      "available_bytes": 914903040,
      "capacity": "13.95%",
      "device": "/dev/sda2",
      "filesystem": "xfs",
      "options": [
        "rw",
        "seclabel",
        "relatime",
        "attr2",
        "inode64",
        "noquota"
      ],
      "size": "1014.00 MiB",
      "size_bytes": 1063256064,
      "used": "141.48 MiB",
      "used_bytes": 148353024
    },
    "/boot/efi": {
      "available": "188.77 MiB",
      "available_bytes": 197939200,
      "capacity": "5.51%",
      "device": "/dev/sda1",
      "filesystem": "vfat",
      "options": [
        "rw",
        "relatime",
        "fmask=0077",
        "dmask=0077",
        "codepage=437",
        "iocharset=ascii",
        "shortname=winnt",
        "errors=remount-ro"
      ],
      "size": "199.79 MiB",
      "size_bytes": 209489920,
      "used": "11.02 MiB",
      "used_bytes": 11550720
    },
    "/datastore01": {
      "available": "125.81 GiB",
      "available_bytes": 135083810816,
      "capacity": "0.05%",
      "device": "/dev/sdb1",
      "filesystem": "ext4",
      "options": [
        "rw",
        "seclabel",
        "relatime",
        "data=ordered"
      ],
      "size": "125.87 GiB",
      "size_bytes": 135146749952,
      "used": "60.02 MiB",
      "used_bytes": 62939136
    },
    "/datastore02": {
      "available": "125.33 GiB",
      "available_bytes": 134570110976,
      "capacity": "0.43%",
      "device": "/dev/sdc1",
      "filesystem": "ext4",
      "options": [
        "rw",
        "seclabel",
        "relatime",
        "data=ordered"
      ],
      "size": "125.87 GiB",
      "size_bytes": 135146749952,
      "used": "549.93 MiB",
      "used_bytes": 576638976
    },
    "/dev": {
      "available": "3.81 GiB",
      "available_bytes": 4088217600,
      "capacity": "0%",
      "device": "devtmpfs",
      "filesystem": "devtmpfs",
      "options": [
        "rw",
        "seclabel",
        "nosuid",
        "size=3992400k",
        "nr_inodes=998100",
        "mode=755"
      ],
      "size": "3.81 GiB",
      "size_bytes": 4088217600,
      "used": "0 bytes",
      "used_bytes": 0
    },
    "/dev/hugepages": {
      "available": "0 bytes",
      "available_bytes": 0,
      "capacity": "100%",
      "device": "hugetlbfs",
      "filesystem": "hugetlbfs",
      "options": [
        "rw",
        "seclabel",
        "relatime"
      ],
      "size": "0 bytes",
      "size_bytes": 0,
      "used": "0 bytes",
      "used_bytes": 0
    },
    "/dev/mqueue": {
      "available": "0 bytes",
      "available_bytes": 0,
      "capacity": "100%",
      "device": "mqueue",
      "filesystem": "mqueue",
      "options": [
        "rw",
        "seclabel",
        "relatime"
      ],
      "size": "0 bytes",
      "size_bytes": 0,
      "used": "0 bytes",
      "used_bytes": 0
    },
    "/dev/pts": {
      "available": "0 bytes",
      "available_bytes": 0,
      "capacity": "100%",
      "device": "devpts",
      "filesystem": "devpts",
      "options": [
        "rw",
        "seclabel",
        "nosuid",
        "noexec",
        "relatime",
        "gid=5",
        "mode=620",
        "ptmxmode=000"
      ],
      "size": "0 bytes",
      "size_bytes": 0,
      "used": "0 bytes",
      "used_bytes": 0
    },
    "/dev/shm": {
      "available": "3.82 GiB",
      "available_bytes": 4100300800,
      "capacity": "0%",
      "device": "tmpfs",
      "filesystem": "tmpfs",
      "options": [
        "rw",
        "seclabel",
        "nosuid",
        "nodev"
      ],
      "size": "3.82 GiB",
      "size_bytes": 4100300800,
      "used": "0 bytes",
      "used_bytes": 0
    },
    "/home": {
      "available": "19.29 GiB",
      "available_bytes": 20708864000,
      "capacity": "0.16%",
      "device": "/dev/mapper/centos-home",
      "filesystem": "xfs",
      "options": [
        "rw",
        "seclabel",
        "relatime",
        "attr2",
        "inode64",
        "noquota"
      ],
      "size": "19.32 GiB",
      "size_bytes": 20742930432,
      "used": "32.49 MiB",
      "used_bytes": 34066432
    },
    "/run": {
      "available": "3.80 GiB",
      "available_bytes": 4082327552,
      "capacity": "0.44%",
      "device": "tmpfs",
      "filesystem": "tmpfs",
      "options": [
        "rw",
        "seclabel",
        "nosuid",
        "nodev",
        "mode=755"
      ],
      "size": "3.82 GiB",
      "size_bytes": 4100300800,
      "used": "17.14 MiB",
      "used_bytes": 17973248
    },
    "/run/user/6026": {
      "available": "782.07 MiB",
      "available_bytes": 820060160,
      "capacity": "0%",
      "device": "tmpfs",
      "filesystem": "tmpfs",
      "options": [
        "rw",
        "seclabel",
        "nosuid",
        "nodev",
        "relatime",
        "size=800840k",
        "mode=700",
        "uid=6026",
        "gid=6026"
      ],
      "size": "782.07 MiB",
      "size_bytes": 820060160,
      "used": "0 bytes",
      "used_bytes": 0
    },
    "/run/user/996": {
      "available": "782.07 MiB",
      "available_bytes": 820060160,
      "capacity": "0%",
      "device": "tmpfs",
      "filesystem": "tmpfs",
      "options": [
        "rw",
        "seclabel",
        "nosuid",
        "nodev",
        "relatime",
        "size=800840k",
        "mode=700",
        "uid=996",
        "gid=993"
      ],
      "size": "782.07 MiB",
      "size_bytes": 820060160,
      "used": "0 bytes",
      "used_bytes": 0
    },
    "/sys/fs/cgroup": {
      "available": "3.82 GiB",
      "available_bytes": 4100300800,
      "capacity": "0%",
      "device": "tmpfs",
      "filesystem": "tmpfs",
      "options": [
        "ro",
        "seclabel",
        "nosuid",
        "nodev",
        "noexec",
        "mode=755"
      ],
      "size": "3.82 GiB",
      "size_bytes": 4100300800,
      "used": "0 bytes",
      "used_bytes": 0
    },
    "/var/lib/nfs/rpc_pipefs": {
      "available": "0 bytes",
      "available_bytes": 0,
      "capacity": "100%",
      "device": "sunrpc",
      "filesystem": "rpc_pipefs",
      "options": [
        "rw",
        "relatime"
      ],
      "size": "0 bytes",
      "size_bytes": 0,
      "used": "0 bytes",
      "used_bytes": 0
    }
  },
  "networking": {
    "domain": "hcilabs.hpecorp.net",
    "fqdn": "foreman.hcilabs.hpecorp.net",
    "hostname": "foreman",
    "interfaces": {
      "ens160": {
        "bindings": [
          {
            "address": "16.100.202.11",
            "netmask": "255.255.254.0",
            "network": "16.100.202.0"
          }
        ],
        "bindings6": [
          {
            "address": "fe80::250:56ff:feb7:940c",
            "netmask": "ffff:ffff:ffff:ffff::",
            "network": "fe80::"
          }
        ],
        "ip": "16.100.202.11",
        "ip6": "fe80::250:56ff:feb7:940c",
        "mac": "00:50:56:b7:94:0c",
        "mtu": 1500,
        "netmask": "255.255.254.0",
        "netmask6": "ffff:ffff:ffff:ffff::",
        "network": "16.100.202.0",
        "network6": "fe80::",
        "scope6": "link"
      },
      "lo": {
        "bindings": [
          {
            "address": "127.0.0.1",
            "netmask": "255.0.0.0",
            "network": "127.0.0.0"
          }
        ],
        "bindings6": [
          {
            "address": "::1",
            "netmask": "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff",
            "network": "::1"
          }
        ],
        "ip": "127.0.0.1",
        "ip6": "::1",
        "mtu": 65536,
        "netmask": "255.0.0.0",
        "netmask6": "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff",
        "network": "127.0.0.0",
        "network6": "::1",
        "scope6": "host"
      }
    },
    "ip": "16.100.202.11",
    "ip6": "fe80::250:56ff:feb7:940c",
    "mac": "00:50:56:b7:94:0c",
    "mtu": 1500,
    "netmask": "255.255.254.0",
    "netmask6": "ffff:ffff:ffff:ffff::",
    "network": "16.100.202.0",
    "network6": "fe80::",
    "primary": "ens160",
    "scope6": "link"
  },
  "os": {
    "architecture": "x86_64",
    "family": "RedHat",
    "hardware": "x86_64",
    "name": "CentOS",
    "release": {
      "full": "7.8.2003",
      "major": "7",
      "minor": "8"
    },
    "selinux": {
      "config_mode": "permissive",
      "config_policy": "targeted",
      "current_mode": "permissive",
      "enabled": true,
      "enforced": false,
      "policy_version": "31"
    }
  },
  "partitions": {
    "/dev/mapper/centos-home": {
      "filesystem": "xfs",
      "mount": "/home",
      "size": "19.33 GiB",
      "size_bytes": 20753416192,
      "uuid": "9d04344a-dd60-44b2-b07a-1c3200fb03c7"
    },
    "/dev/mapper/centos-root": {
      "filesystem": "xfs",
      "mount": "/",
      "size": "39.59 GiB",
      "size_bytes": 42513465344,
      "uuid": "53317a05-9b79-469c-b455-c58108a5e2de"
    },
    "/dev/mapper/centos-swap": {
      "filesystem": "swap",
      "size": "3.88 GiB",
      "size_bytes": 4160749568,
      "uuid": "fde44ecf-c7fb-4e24-a322-839b7efa1736"
    },
    "/dev/sda1": {
      "filesystem": "vfat",
      "mount": "/boot/efi",
      "partlabel": "EFI System Partition",
      "partuuid": "885675a7-4045-42f6-8536-51541eee5892",
      "size": "200.00 MiB",
      "size_bytes": 209715200,
      "uuid": "F851-F31A"
    },
    "/dev/sda2": {
      "filesystem": "xfs",
      "mount": "/boot",
      "partuuid": "8924a11d-9cce-4e2d-a037-7f87faff476a",
      "size": "1.00 GiB",
      "size_bytes": 1073741824,
      "uuid": "5fe3cbe8-ccd9-4c32-b885-fa928b1a5aef"
    },
    "/dev/sda3": {
      "filesystem": "LVM2_member",
      "partuuid": "2719c5db-30ac-452c-a7e6-b87a4c170732",
      "size": "62.80 GiB",
      "size_bytes": 67433922560,
      "uuid": "zWCTFY-rFWV-gbiQ-wmou-0CnD-JLkS-3rRe90"
    },
    "/dev/sdb1": {
      "filesystem": "ext4",
      "label": "None",
      "mount": "/datastore01",
      "partuuid": "fc61ec64-4b73-43ec-b10b-1ad3d139796a",
      "size": "128.00 GiB",
      "size_bytes": 137437888000,
      "uuid": "f1af64e6-a248-4b28-9fde-6f89b81d03da"
    },
    "/dev/sdc1": {
      "filesystem": "ext4",
      "label": "None",
      "mount": "/datastore02",
      "partuuid": "61c6cf73-bb0e-4d77-b8f2-0a155e4d2714",
      "size": "128.00 GiB",
      "size_bytes": 137437888000,
      "uuid": "3ab1dfbc-b04d-475e-b63c-4fad7140e6b4"
    }
  },
  "path": "/sbin:/bin:/usr/sbin:/usr/bin:/opt/puppetlabs/bin",
  "processors": {
    "count": 4,
    "isa": "x86_64",
    "models": [
      "Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz",
      "Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz",
      "Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz",
      "Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz"
    ],
    "physicalcount": 4
  },
  "ruby": {
    "platform": "x86_64-linux",
    "sitedir": "/opt/puppetlabs/puppet/lib/ruby/site_ruby/2.4.0",
    "version": "2.4.10"
  },
  "ssh": {
    "ecdsa": {
      "fingerprints": {
        "sha1": "SSHFP 3 1 edce836f752a528d085f5a0be85aef500fc851d5",
        "sha256": "SSHFP 3 2 6a4740375902a34e28095b91f0e1542deb2e46e28d806739e056c22deebf93c8"
      },
      "key": "REDACTED",
      "type": "ecdsa-sha2-nistp256"
    },
    "ed25519": {
      "fingerprints": {
        "sha1": "SSHFP 4 1 3a9ad433c63147bdd4b8d9069f66a8239e796be5",
        "sha256": "SSHFP 4 2 83209db0d026916627649f18901852089ba0c624cdb32bafbcbb69732388ab17"
      },
      "key": "AAAAC3NzaC1lZDI1NTE5AAAAIN7rPTei0c7FX6QUlJ37k8jQ1H2AQ7qo7BXYT9h1MQzY",
      "type": "ssh-ed25519"
    },
    "rsa": {
      "fingerprints": {
        "sha1": "SSHFP 1 1 613441ff1d8361a7ca6672c95a917639bee253fc",
        "sha256": "SSHFP 1 2 e8b9927c9d22f7be4ae55e64f5b0d29795be95527ea463ee140661be35a44760"
      },
      "key": "REDACTED",
      "type": "ssh-rsa"
    }
  },
  "system_uptime": {
    "days": 1,
    "hours": 42,
    "seconds": 151927,
    "uptime": "1 day"
  },
  "timezone": "EDT",
  "virtual": "vmware"
}

As for the production log, nothing seemed to change in the foreman production.log when I changed the log level. Is there some other log that I should look at?

Did you restart foreman after changing the log level? a restart is required for the new setting to take affect.

I believe this is the full stack trace from the attempt to run the puppet agent on the foreman master:

2020-06-18T11:55:17 [I|app|11e7e7c3] Started GET "/node/puppetmaster03.hcilabs.hpecorp.net?format=yml" for 16.100.202.19 at 2020-06-18 11:55:17 -0400
2020-06-18T11:55:17 [I|app|11e7e7c3] Processing by HostsController#externalNodes as YML
2020-06-18T11:55:17 [I|app|11e7e7c3]   Parameters: {"name"=>"puppetmaster03.hcilabs.hpecorp.net"}
2020-06-18T11:55:17 [D|app|11e7e7c3] Examining client certificate to extract dn and sans
2020-06-18T11:55:17 [D|app|11e7e7c3] Client sent certificate with subject 'puppetmaster03.hcilabs.hpecorp.net' and subject alt names '["puppetmaster03.hcilabs.hpecorp.net"]'
2020-06-18T11:55:17 [D|app|11e7e7c3] Verifying request from ["puppetmaster03.hcilabs.hpecorp.net"] against ["puppetmaster03.hcilabs.hpecorp.net"]
2020-06-18T11:55:17 [W|app|11e7e7c3] Failed to generate external nodes for puppetmaster03.hcilabs.hpecorp.net
2020-06-18T11:55:17 [D|app|11e7e7c3] Backtrace for 'Failed to generate external nodes for puppetmaster03.hcilabs.hpecorp.net' error (NoMethodError): undefined method `enabled_taxonomies' for #<Class:0x000000000b628f30>
Did you mean?  enable_auditing
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/dynamic_matchers.rb:22:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.8.2/app/models/concerns/foreman_remote_execution/host_extensions.rb:74:in `remote_execution_proxies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.8.2/app/models/concerns/foreman_remote_execution/host_extensions.rb:88:in `remote_execution_ssh_keys'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.8.2/app/models/concerns/foreman_remote_execution/host_extensions.rb:50:in `host_params_hash'
/usr/share/foreman/app/models/concerns/host_params.rb:54:in `host_params'
/usr/share/foreman/app/models/concerns/host_params.rb:11:in `params'
/usr/share/foreman/app/models/host_info_providers/host_params_info.rb:4:in `host_info'
/usr/share/foreman/app/models/concerns/host_info_extensions.rb:29:in `block in info'
/usr/share/foreman/app/models/concerns/host_info_extensions.rb:27:in `each'
/usr/share/foreman/app/models/concerns/host_info_extensions.rb:27:in `info'
/usr/share/foreman/app/controllers/hosts_controller.rb:199:in `block in externalNodes'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/mime_responds.rb:197:in `respond_to'
/usr/share/foreman/app/controllers/hosts_controller.rb:195:in `externalNodes'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/rack/agent_hooks.rb:30:in `traced_call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/rack/browser_monitoring.rb:32:in `traced_call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/conditional_get.rb:25:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/newrelic_rpm-6.2.0.354/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
/usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-06-18T11:55:17 [I|app|11e7e7c3]   Rendering text template
2020-06-18T11:55:17 [I|app|11e7e7c3]   Rendered text template (0.0ms)
2020-06-18T11:55:17 [I|app|11e7e7c3] Completed 412 Precondition Failed in 165ms (Views: 0.5ms | ActiveRecord: 17.5ms)

This is odd. The lowest version in 1.24 repos is 1.8.4. Guess the upgrade didn’t go through fully?

What you’re describing sounds a lot like this issue, which was fixed in foreman_remote_execution-2.0.0 and newer.

hmm strange. I’ll look into if that version was pinned somewhere. I’ve only recently been working on these system so its possible.