BMC Errors After Upgrade

Problem: After upgrading from 2.1.0 to 2.2.3, encountering BMC errors: “undefined method `bmc’ for nil:NilClass”. Editing and attempting to save the interface results in 500 Internal Server Error.

Expected outcome: BMC should be working.

Foreman and Proxy versions: 2.2.3

Foreman and Proxy plugin versions: Foreman Discovery 16.2.0

Distribution and version: CentOS 7.9

Other relevant data:
[root@ ~]# foreman-rake errors:fetch_log request_id=ed9474b0
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Warning: Logging layout is not multiline_request_pattern.
This output of this command can be incomplete.
2021-04-13T17:58:48 [I|app|ed9474b0] Started GET “/hosts/c78087-1qjnd43/bmc” for 127.0.0.1 at 2021-04-13 17:58:48 -0400
2021-04-13T17:58:48 [I|app|ed9474b0] Processing by HostsController#bmc as HTML
2021-04-13T17:58:48 [I|app|ed9474b0] Parameters: {“id”=>“c78087-1qjnd43”}
2021-04-13T17:58:48 [W|app|ed9474b0] undefined method bmc' for nil:NilClass 2021-04-13T17:58:48 [I|app|ed9474b0] Backtrace for 'undefined method bmc’ for nil:NilClass’ error (NoMethodError): undefined method `bmc’ for nil:NilClass
2021-04-13T17:58:48 [I|app|ed9474b0] Rendering common/500.html.erb
2021-04-13T17:58:48 [I|app|ed9474b0] Rendered common/500.html.erb (Duration: 1.7ms | Allocations: 746)
2021-04-13T17:58:48 [I|app|ed9474b0] Completed 500 Internal Server Error in 19ms (Views: 2.6ms | ActiveRecord: 2.6ms | Allocations: 7276)

Foreman version: 2.2.3
Plugins:
 - foreman_discovery 16.2.0

    [root@ ~]# foreman-rake errors:fetch_log request_id=7b2e8214
    API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
    Warning: Logging layout is not multiline_request_pattern.
    This output of this command can be incomplete.
    2021-04-13T17:46:34 [I|app|7b2e8214] Started PATCH "/hosts/346" for 127.0.0.1 at 2021-04-13 17:46:34 -0400
    2021-04-13T17:46:34 [I|app|7b2e8214] Processing by HostsController#update as */*
    2021-04-13T17:46:34 [I|app|7b2e8214]   Parameters: {"utf8"=>"✓", "authenticity_token"=>"pgDDJbpGBpkgDMYrvwX6tb82wILx1QTaAkTfXMWqL0/DkDeaR6TLozOhDySoIC6GIkmm1KRrcv110YiA7EdFRQ==", "host"=>{"name"=>"", "hostgroup_id"=>"", "environment_id"=>"1", "puppet_proxy_id"=>"1", "puppet_ca_proxy_id"=>"1", "puppetclass_ids"=>[""], "managed"=>"true", "progress_report_id"=>"[FILTERED]", "interfaces_attributes"=>{"0"=>{"_destroy"=>"0", "mac"=>"e4:43:4b:de:46:74", "identifier"=>"eno1", "name"=>"c78087-1qjnd43", "domain_id"=>"1", "subnet_id"=>"1", "ip"=>"10.133.200.248", "ip6"=>"", "managed"=>"1", "primary"=>"0", "provision"=>"1", "tag"=>"", "attached_to"=>"", "id"=>"1722"}, "1"=>{"_destroy"=>"0", "mac"=>"e4:43:4b:de:46:75", "identifier"=>"eno2", "name"=>"", "domain_id"=>"", "subnet_id"=>"", "ip"=>"", "ip6"=>"", "managed"=>"1", "primary"=>"0", "provision"=>"0", "tag"=>"", "attached_to"=>"", "id"=>"1723"}, "2"=>{"_destroy"=>"0", "mac"=>"e4:43:4b:de:46:76", "identifier"=>"eno3", "name"=>"", "domain_id"=>"", "subnet_id"=>"", "ip"=>"", "ip6"=>"", "managed"=>"0", "primary"=>"0", "provision"=>"0", "tag"=>"", "attached_to"=>"", "id"=>"1724"}, "3"=>{"_destroy"=>"0", "mac"=>"e4:43:4b:de:46:77", "identifier"=>"eno4", "name"=>"", "domain_id"=>"", "subnet_id"=>"", "ip"=>"", "ip6"=>"", "managed"=>"0", "primary"=>"0", "provision"=>"0", "tag"=>"", "attached_to"=>"", "id"=>"1725"}, "4"=>{"_destroy"=>"1", "mac"=>"f4:02:70:eb:b4:7a", "identifier"=>"ipmi", "name"=>"", "domain_id"=>"", "subnet_id"=>"", "ip"=>"10.133.198.252", "ip6"=>"", "managed"=>"1", "primary"=>"0", "provision"=>"0", "username"=>"root", "provider"=>"IPMI", "id"=>"1726"}, "1618350365097"=>{"type"=>"Nic::BMC", "mac"=>"", "identifier"=>"ipmi", "name"=>"", "domain_id"=>"", "subnet_id"=>"", "ip"=>"10.133.198.252", "ip6"=>"", "managed"=>"1", "primary"=>"0", "provision"=>"0", "username"=>"root", "password"=>"[FILTERED]", "provider"=>"IPMI"}}, "architecture_id"=>"3", "operatingsystem_id"=>"2", "medium_id"=>"8", "ptable_id"=>"127", "pxe_loader"=>"PXELinux BIOS", "disk"=>"", "host_parameters_attributes"=>{"0"=>{"name"=>"serial-number", "parameter_type"=>"string", "value"=>"[FILTERED]", "hidden_value"=>"[FILTERED]", "_destroy"=>"false", "id"=>"203"}, "1"=>{"name"=>"tag", "parameter_type"=>"string", "value"=>"[FILTERED]", "hidden_value"=>"[FILTERED]", "_destroy"=>"false", "id"=>"16"}}, "is_owned_by"=>"4-Users", "enabled"=>"1", "model_id"=>"2", "comment"=>"", "overwrite"=>"false"}, "fakepassword"=>"[FILTERED]", "id"=>"346"}
    2021-04-13T17:46:34 [W|app|7b2e8214] Host::Base#name= delegated to primary_interface.name=, but primary_interface is nil: #<Host::Managed id: 346, name: "c78087-1qjnd43", last_compile: nil, last_report: nil, updated_at: "2020-08-28 22:32:58", created_at: "2020-07-24 23:30:45", root_pass: [FILTERED], architecture_id: 3, operatingsystem_id: 2, environment_id: 1, ptable_id: 127, medium_id: 8, build: false, comment: "", disk: "", installed_at: "2020-07-27 21:05:09", model_id: 2, hostgroup_id: nil, owner_id: 4, owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: nil, puppet_proxy_id: 1, certname: nil, image_id: nil, organization_id: 1, location_id: 2, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: nil, provision_method: nil, grub_pass: "$5$NHqtwoMfsqTlBGyY$dopggiuw2RqhMhfkZ00jyMQBiC/mgW...", global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: "PXELinux BIOS", initiated_at: "2020-07-27 20:54:43", build_errors: nil, discovery_rule_id: nil>
    2021-04-13T17:46:34 [I|app|7b2e8214] Backtrace for 'Host::Base#name= delegated to primary_interface.name=, but primary_interface is nil: #<Host::Managed id: 346, name: "c78087-1qjnd43", last_compile: nil, last_report: nil, updated_at: "2020-08-28 22:32:58", created_at: "2020-07-24 23:30:45", root_pass: [FILTERED], architecture_id: 3, operatingsystem_id: 2, environment_id: 1, ptable_id: 127, medium_id: 8, build: false, comment: "", disk: "", installed_at: "2020-07-27 21:05:09", model_id: 2, hostgroup_id: nil, owner_id: 4, owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: nil, puppet_proxy_id: 1, certname: nil, image_id: nil, organization_id: 1, location_id: 2, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: nil, provision_method: nil, grub_pass: "$5$NHqtwoMfsqTlBGyY$dopggiuw2RqhMhfkZ00jyMQBiC/mgW...", global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: "PXELinux BIOS", initiated_at: "2020-07-27 20:54:43", build_errors: nil, discovery_rule_id: nil>' error (Module::DelegationError): Host::Base#name= delegated to primary_interface.name=, but primary_interface is nil: #<Host::Managed id: 346, name: "c78087-1qjnd43", last_compile: nil, last_report: nil, updated_at: "2020-08-28 22:32:58", created_at: "2020-07-24 23:30:45", root_pass: [FILTERED], architecture_id: 3, operatingsystem_id: 2, environment_id: 1, ptable_id: 127, medium_id: 8, build: false, comment: "", disk: "", installed_at: "2020-07-27 21:05:09", model_id: 2, hostgroup_id: nil, owner_id: 4, owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: nil, puppet_proxy_id: 1, certname: nil, image_id: nil, organization_id: 1, location_id: 2, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: nil, provision_method: nil, grub_pass: "$5$NHqtwoMfsqTlBGyY$dopggiuw2RqhMhfkZ00jyMQBiC/mgW...", global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: "PXELinux BIOS", initiated_at: "2020-07-27 20:54:43", build_errors: nil, discovery_rule_id: nil>
    2021-04-13T17:46:34 [I|app|7b2e8214]   Rendering common/500.html.erb
    2021-04-13T17:46:34 [I|app|7b2e8214]   Rendered common/500.html.erb (Duration: 0.8ms | Allocations: 377)
    2021-04-13T17:46:34 [I|app|7b2e8214] Completed 500 Internal Server Error in 26ms (Views: 1.3ms | ActiveRecord: 2.9ms | Allocations: 13541)

    Foreman version: 2.2.3
    Plugins:
     - foreman_discovery 16.2.0

In 2.2 we have changed how BMC proxy is found. Previously it was any proxy with BMC features, now it is BMC proxy you have associated with your subnet.

https://projects.theforeman.org/issues/28357

Solution: Associate BMC proxies with your subnets.

Now, be completely honest - have you seen our Upgrade notes when you were upgrading your instance? I think it is burried in our document below ton of information (list of fixed bugs) and we need to do something with this:

https://theforeman.org/manuals/2.2/index.html#Upgradewarnings

If you read carefully, there is the action you need to take described. @mcorr @tbrisker @ekohl

Anyway, can you share backtrace by enabling debug mode? If there is a nil exception we should probably fix this and provide an useful message for others who would skip this. Thanks.

I saw this but dismissed it because the linked bug tracker had a more specific error message:

[Foreman::BMCFeatureException]: There is no proxy with BMC feature set up. Associate a BMC feature with a subnet.

Even when I went to the subnet and Smart Proxy pages, there was no error / message / indicator that this was not configured.

I’ll look into it again.

One further question:

The BMC subnet is not managed by an external DHCP server that is used to do Dell zero-touch provisioning of BIOS/RAID/Firmware automatically, without Foreman being involved. As such, we do not have the BMC subnet configured in Foreman or attached to any Smart Proxies by design.

I assume we just have to add the subnet to Foreman? I do not want Foreman managing those IP addresses. BMC addresses are populated through the auto discovery process instead.

Thanks.

Meant to say that the BMC subnet is not managed by Foreman, but by an external DHCP server.

There is no DHCP requirements for BMC integration to work. As long as the IP address of a BMC interface in Foreman is correct, you should be fine.

Thanks.

I am a bit confused on if there is a specific way to associate the proxy with the BMC subnet. In Foreman, the “BMC Proxy” is already configured for all of my subnets - including adding a new subnet to Foreman for the BMC addresses (in case that was needed).

The Smart Proxy and Foreman server are the same system - there are no dedicated Smart Proxies doing BMC.

Pictures and backtraces attached.

[root@c78087-fore001 ~]# foreman-rake errors:fetch_log request_id=55a6fb45

2021-04-15T17:20:50 [I|app|55a6fb45] Started PATCH "/hosts/346" for 127.0.0.1 at 2021-04-15 17:20:50 -0400
2021-04-15T17:20:50 [I|app|55a6fb45] Processing by HostsController#update as */*
2021-04-15T17:20:50 [I|app|55a6fb45]   Parameters: {"utf8"=>"✓", "authenticity_token"=>"mpM3YsBzz/cW6x5ksslzrE9OqsodNrcUGm0Ct2lGk75hLZMaBMS9px+AnaHBI9isNFzYldPwiAtVqYtnSY2jyQ==", "host"=>{"name"=>"", "hostgroup_id"=>"", "environment_id"=>"1", "puppet_proxy_id"=>"1", "puppet_ca_proxy_id"=>"1", "puppetclass_ids"=>[""], "managed"=>"true", "progress_report_id"=>"[FILTERED]", "interfaces_attributes"=>{"0"=>{"_destroy"=>"0", "mac"=>"e4:43:4b:de:46:74", "identifier"=>"eno1", "name"=>"c78087-1qjnd43", "domain_id"=>"1", "subnet_id"=>"1", "ip"=>"10.133.200.248", "ip6"=>"", "managed"=>"1", "primary"=>"0", "provision"=>"1", "tag"=>"", "attached_to"=>"", "id"=>"1722"}, "1"=>{"_destroy"=>"0", "mac"=>"e4:43:4b:de:46:75", "identifier"=>"eno2", "name"=>"", "domain_id"=>"", "subnet_id"=>"", "ip"=>"", "ip6"=>"", "managed"=>"1", "primary"=>"0", "provision"=>"0", "tag"=>"", "attached_to"=>"", "id"=>"1723"}, "2"=>{"_destroy"=>"0", "mac"=>"e4:43:4b:de:46:76", "identifier"=>"eno3", "name"=>"", "domain_id"=>"", "subnet_id"=>"", "ip"=>"", "ip6"=>"", "managed"=>"0", "primary"=>"0", "provision"=>"0", "tag"=>"", "attached_to"=>"", "id"=>"1724"}, "3"=>{"_destroy"=>"0", "mac"=>"e4:43:4b:de:46:77", "identifier"=>"eno4", "name"=>"", "domain_id"=>"", "subnet_id"=>"", "ip"=>"", "ip6"=>"", "managed"=>"0", "primary"=>"0", "provision"=>"0", "tag"=>"", "attached_to"=>"", "id"=>"1725"}, "4"=>{"_destroy"=>"0", "mac"=>"f4:02:70:eb:b4:7a", "identifier"=>"ipmi", "name"=>"", "domain_id"=>"", "subnet_id"=>"", "ip"=>"10.133.198.252", "ip6"=>"", "managed"=>"1", "primary"=>"0", "provision"=>"0", "username"=>"root", "provider"=>"IPMI", "id"=>"1726"}}, "architecture_id"=>"3", "operatingsystem_id"=>"2", "medium_id"=>"8", "ptable_id"=>"127", "pxe_loader"=>"PXELinux BIOS", "disk"=>"", "host_parameters_attributes"=>{"0"=>{"name"=>"serial-number", "parameter_type"=>"string", "value"=>"[FILTERED]", "hidden_value"=>"[FILTERED]", "_destroy"=>"false", "id"=>"203"}, "1"=>{"name"=>"tag", "parameter_type"=>"string", "value"=>"[FILTERED]", "hidden_value"=>"[FILTERED]", "_destroy"=>"false", "id"=>"16"}}, "is_owned_by"=>"4-Users", "enabled"=>"1", "model_id"=>"2", "comment"=>"", "overwrite"=>"false"}, "fakepassword"=>"[FILTERED]", "id"=>"346"}
2021-04-15T17:20:50 [W|app|55a6fb45] Host::Base#name= delegated to primary_interface.name=, but primary_interface is nil: #<Host::Managed id: 346, name: "c78087-1qjnd43", last_compile: nil, last_report: nil, updated_at: "2020-08-28 22:32:58", created_at: "2020-07-24 23:30:45", root_pass: [FILTERED], architecture_id: 3, operatingsystem_id: 2, environment_id: 1, ptable_id: 127, medium_id: 8, build: false, comment: "", disk: "", installed_at: "2020-07-27 21:05:09", model_id: 2, hostgroup_id: nil, owner_id: 4, owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: nil, puppet_proxy_id: 1, certname: nil, image_id: nil, organization_id: 1, location_id: 2, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: nil, provision_method: nil, grub_pass: "$5$NHqtwoMfsqTlBGyY$dopggiuw2RqhMhfkZ00jyMQBiC/mgW...", global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: "PXELinux BIOS", initiated_at: "2020-07-27 20:54:43", build_errors: nil, discovery_rule_id: nil>
2021-04-15T17:20:50 [I|app|55a6fb45] Backtrace for 'Host::Base#name= delegated to primary_interface.name=, but primary_interface is nil: #<Host::Managed id: 346, name: "c78087-1qjnd43", last_compile: nil, last_report: nil, updated_at: "2020-08-28 22:32:58", created_at: "2020-07-24 23:30:45", root_pass: [FILTERED], architecture_id: 3, operatingsystem_id: 2, environment_id: 1, ptable_id: 127, medium_id: 8, build: false, comment: "", disk: "", installed_at: "2020-07-27 21:05:09", model_id: 2, hostgroup_id: nil, owner_id: 4, owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: nil, puppet_proxy_id: 1, certname: nil, image_id: nil, organization_id: 1, location_id: 2, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: nil, provision_method: nil, grub_pass: "$5$NHqtwoMfsqTlBGyY$dopggiuw2RqhMhfkZ00jyMQBiC/mgW...", global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: "PXELinux BIOS", initiated_at: "2020-07-27 20:54:43", build_errors: nil, discovery_rule_id: nil>' error (Module::DelegationError): Host::Base#name= delegated to primary_interface.name=, but primary_interface is nil: #<Host::Managed id: 346, name: "c78087-1qjnd43", last_compile: nil, last_report: nil, updated_at: "2020-08-28 22:32:58", created_at: "2020-07-24 23:30:45", root_pass: [FILTERED], architecture_id: 3, operatingsystem_id: 2, environment_id: 1, ptable_id: 127, medium_id: 8, build: false, comment: "", disk: "", installed_at: "2020-07-27 21:05:09", model_id: 2, hostgroup_id: nil, owner_id: 4, owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: nil, puppet_proxy_id: 1, certname: nil, image_id: nil, organization_id: 1, location_id: 2, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: nil, provision_method: nil, grub_pass: "$5$NHqtwoMfsqTlBGyY$dopggiuw2RqhMhfkZ00jyMQBiC/mgW...", global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: "PXELinux BIOS", initiated_at: "2020-07-27 20:54:43", build_errors: nil, discovery_rule_id: nil>
 55a6fb45 | /usr/share/foreman/app/models/host/base.rb:104:in `rescue in name='
 55a6fb45 | /usr/share/foreman/app/models/host/base.rb:104:in `name='
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-6.0.3.1/lib/active_model/attribute_assignment.rb:50:in `public_send'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-6.0.3.1/lib/active_model/attribute_assignment.rb:50:in `_assign_attribute'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-6.0.3.1/lib/active_model/attribute_assignment.rb:43:in `block in _assign_attributes'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-6.0.3.1/lib/active_model/attribute_assignment.rb:42:in `each'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-6.0.3.1/lib/active_model/attribute_assignment.rb:42:in `_assign_attributes'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/attribute_assignment.rb:21:in `_assign_attributes'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-6.0.3.1/lib/active_model/attribute_assignment.rb:35:in `assign_attributes'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/persistence.rb:620:in `block in update'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:212:in `transaction'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/persistence.rb:619:in `update'
 55a6fb45 | /usr/share/foreman/app/controllers/hosts_controller.rb:116:in `block in update'
 55a6fb45 | /usr/share/foreman/app/models/taxonomy.rb:76:in `block in no_taxonomy_scope'
 55a6fb45 | /usr/share/foreman/app/models/taxonomy.rb:83:in `block (2 levels) in as_taxonomy'
 55a6fb45 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:195:in `as_location'
 55a6fb45 | /usr/share/foreman/app/models/taxonomy.rb:82:in `block in as_taxonomy'
 55a6fb45 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:154:in `as_org'
 55a6fb45 | /usr/share/foreman/app/models/taxonomy.rb:81:in `as_taxonomy'
 55a6fb45 | /usr/share/foreman/app/models/taxonomy.rb:75:in `no_taxonomy_scope'
 55a6fb45 | /usr/share/foreman/app/controllers/hosts_controller.rb:113:in `update'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:195:in `process_action'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
 55a6fb45 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 55a6fb45 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 55a6fb45 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:139:in `run_callbacks'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in `block in instrument'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in `instrument'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:136:in `process'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.1/lib/action_view/rendering.rb:39:in `process'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:190:in `dispatch'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:254:in `dispatch'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `each'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `serve'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:834:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.2.9/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
 55a6fb45 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 55a6fb45 | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:40:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 55a6fb45 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/cookies.rb:648:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:101:in `run_callbacks'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:37:in `call_app'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:28:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 55a6fb45 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `public_send'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `method_missing'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/configuration.rb:228:in `call'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:682:in `handle_request'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:472:in `process_client'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:328:in `block in run'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/thread_pool.rb:134:in `block in spawn_thread'
 55a6fb45 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-04-15T17:20:50 [I|app|55a6fb45]   Rendering common/500.html.erb
2021-04-15T17:20:50 [I|app|55a6fb45]   Rendered common/500.html.erb (Duration: 0.7ms | Allocations: 375)
2021-04-15T17:20:50 [I|app|55a6fb45] Completed 500 Internal Server Error in 24ms (Views: 1.2ms | ActiveRecord: 3.2ms | Allocations: 12886)

Foreman version: 2.2.3
Plugins:
 - foreman_discovery 16.2.0

[root@c78087-fore001 ~]# foreman-rake errors:fetch_log request_id=d99b30ed
2021-04-15T17:19:38 [I|app|d99b30ed] Started GET "/hosts/c78087-1qjnd43/bmc" for 127.0.0.1 at 2021-04-15 17:19:38 -0400
2021-04-15T17:19:38 [I|app|d99b30ed] Processing by HostsController#bmc as HTML
2021-04-15T17:19:38 [I|app|d99b30ed]   Parameters: {"id"=>"c78087-1qjnd43"}
2021-04-15T17:19:38 [W|app|d99b30ed] undefined method `bmc' for nil:NilClass
2021-04-15T17:19:38 [I|app|d99b30ed] Backtrace for 'undefined method `bmc' for nil:NilClass' error (NoMethodError): undefined method `bmc' for nil:NilClass
 d99b30ed | /usr/share/foreman/app/controllers/hosts_controller.rb:272:in `rescue in bmc'
 d99b30ed | /usr/share/foreman/app/controllers/hosts_controller.rb:266:in `bmc'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:195:in `process_action'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
 d99b30ed | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 d99b30ed | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 d99b30ed | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:139:in `run_callbacks'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in `block in instrument'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in `instrument'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:136:in `process'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.1/lib/action_view/rendering.rb:39:in `process'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:190:in `dispatch'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:254:in `dispatch'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `each'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `serve'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:834:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.2.9/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
 d99b30ed | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 d99b30ed | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 d99b30ed | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/cookies.rb:648:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:101:in `run_callbacks'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:37:in `call_app'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:28:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 d99b30ed | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `public_send'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `method_missing'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/configuration.rb:228:in `call'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:682:in `handle_request'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:472:in `process_client'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:328:in `block in run'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/thread_pool.rb:134:in `block in spawn_thread'
 d99b30ed | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-04-15T17:19:38 [I|app|d99b30ed]   Rendering common/500.html.erb
2021-04-15T17:19:38 [I|app|d99b30ed]   Rendered common/500.html.erb (Duration: 0.8ms | Allocations: 379)
2021-04-15T17:19:38 [I|app|d99b30ed] Completed 500 Internal Server Error in 39ms (Views: 1.6ms | ActiveRecord: 17.0ms | Allocations: 18207)

Foreman version: 2.2.3
Plugins:
 - foreman_discovery 16.2.0

This error is different from the one you originally reported.

This one was thrown because the host does not appear to have exactly one NIC marked as primary one. Can you check this? We expect hosts to have one provision and one primary interface.

This is the exact same error I originally posted:

*“undefined method bmc’ for nil:NilClass”*

All the hosts are doing this, and the hosts are configured with primary and provision interfaces because these are constantly being provisioned/re-provisioned. Everything works on 2.1.0. Upgrading to 2.2+ breaks BMC and the ability to edit network interfaces due to the BMC error.

The issues tracker says we would have seen the "Failed to fetch power status: ERF88-9474 [Foreman::BMCFeatureException]" error if a proxy wasn’t configured, but that error is not presenting itself in any part of the upgrade or troubleshooting.

I need to see the whole stacktrace, enable debug to see it, paste it here please.

What version are you on? Even if I identify and fix an issue, 2.2 no longer receives updates.

LZ

Will have to wait a few days before I can re-run the upgrade again with the client.