Cannot boot anymore Discovery image

Marco,

I had also one server that resulted constantly in the GRUB-prompt
Only the first 1GB network interface for UEFI PXE was configured as boot option.
By configuring UEFI HTTP Boot as extra boot option for the same interface, but as first option and as second one the PXE I got rid of the GRUB-prompt and got the expected menu to select the desired Foreman Discovery Image option.

This was the case for my Fujitsu RX-2530-M5 System, no idea how it is for HP.
Beside this, we also have HP Apollo 4200 Gen 9 and HP DL-360 Gen9 & 10 servers.
But were installed in the past with cobbler. The idea is to build them from scratch in the future with Foreman.

I also tried to use the grubx64.efi from Fedora Rawhide, but that did not solved our problem as well.

tcpdump from our Foreman-server rsults in connection loss as of packet 66678 for the initrd0.img file.

So, I will try the workaround by adding/configuring the host manually in Foreman and building the provisioning file. Hope this might be a temporary solution fo rme as well.

Anyway, thnx for your feedback.

I hope they will find and provide soon a solution for this issue.

@lzap that was the root cause.

On the smart-proxy, I downgraded grub2-efi-x64 to v0.81 (once updated, it become v0.86)

yum remove grub2-common grub2-tools grub2-tools-extra grub2-efi  grub2-common grub2-efi-x64-modules  grub2-tools-minimal grub2-pc-modules
yum install grub2-common-2.02-0.81.el7.centos.noarch grub2-tools-minimal-2.02-0.81.el7.centos.x86_64 grub2-efi-x64-2.02-0.81.el7.centos

and then replaced grubx64.efi in tftpboot with the old version

mv /var/lib/tftpboot/grub2/grubx64.efi /var/lib/tftpboot/grub2/grubx64.efi.bak
cp /boot/efi/EFI/centos/grubx64.efi /var/lib/tftpboot/grub2/grubx64.efi
chmod 644 /var/lib/tftpboot/grub2/grubx64.efi

Now the discovery image works again using the standard Grub2 UEFI boot (that uses tftp). Yuppy :smiley:

And now I am back to the good old problem I had before upgrading foreman and centos7…

Screenshot from 2020-08-21 15-59-31

with this message in foreman’s production.log

2020-08-21T15:47:28 [W|app|1fbe3132] Host discovery failed, facts: {xxx}
2020-08-21T15:47:28 [I|app|1fbe3132] Completed 422 Unprocessable Entity in 512ms (Views: 0.2ms | ActiveRecord: 250.8ms | Allocations: 147840)

:sweat_smile:

1 Like

If you temporarily downgrade grub2-efi on your smart-proxy, the discovery image will restart working fine.

Give it a try

Hi Marco,

Thanks for this feedback. This is wonderful news and I will apply these actions as first job, next Monday.

I also had the “FAILURE” ststus during the first discovery in version 1.24.x of Foreman.
As a solution I applied “Resend” and it then went well.
Give it a try, hope it will work for you as well.

Thanks for the suggestion, but unfortunately the Resend does not fix the validation issue.

Let me know how it goes on Monday. Good luck

Actually, the full log with debug enabled from production.log says

Host discovery failed, facts: {xxx ...., "discovery_proxy_type"=>"foreman", "_type"=>:foreman_discovery}' error (ActiveRecord::RecordInvalid): Validation failed: Name has already been taken
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/validations.rb:80:in `raise_validation_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/validations.rb:53:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:318:in `block in save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:318:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/suppressor.rb:48:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-16.1.0/app/models/host/discovered.rb:133:in `populate_discovery_fields_from_facts'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-16.1.0/app/models/host/discovered.rb:124:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/base.rb:168:in `block in parse_facts'
/usr/share/foreman/app/services/foreman/telemetry_helper.rb:27:in `telemetry_duration_histogram'
/usr/share/foreman/app/models/host/base.rb:167:in `parse_facts'
/usr/share/foreman/app/models/host/base.rb:153:in `import_facts'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-16.1.0/app/models/host/discovered.rb:105:in `import_facts'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-16.1.0/app/models/host/discovered.rb:95:in `import_host'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-16.1.0/app/controllers/api/v2/discovered_hosts_controller.rb:104:in `block in facts'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-16.1.0/app/controllers/api/v2/discovered_hosts_controller.rb:103:in `facts'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:195:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:139:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in `block in instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:136:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.1/lib/action_view/rendering.rb:39:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:190:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:254:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:49:in `block in serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:834:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.2.2/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/etag.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/conditional_get.rb:40:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/session/abstract/id.rb:266:in `context'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/session/abstract/id.rb:260:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/cookies.rb:648:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:101:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:37:in `call_app'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/method_override.rb:24:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/sendfile.rb:110:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:74:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/configuration.rb:228:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:682:in `handle_request'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:472:in `process_client'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:328:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/thread_pool.rb:134:in `block in spawn_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-08-21T17:49:28 [D|app|6bcc20ad] Body: {"message":"Validation failed: Name has already been taken"}
2020-08-21T17:49:28 [I|app|6bcc20ad] Completed 422 Unprocessable Entity in 876ms (Views: 0.2ms | ActiveRecord: 342.5ms | Allocations: 211517)

Hi Marco,

I can confirm that the grub2 package version 0.81 works as expected.
Now the Foreman Discovery Image is well transferred to the new host.

As I said, during the first discovery, it results in a failure, however, with a different error.

image

But after a resend, it went well.

image

Glad to hear the downgrade works well for you too!

Unfortunately the “Resend” does not work for me, still something to solve to go back to normal operations

Marco,

See also following remarks regarding this fix:

Which location the host is initially discovered in? And in which ones is the subnet?

@lzap
I have configured one organization with only one location and one domain.
There are 2 subnets, no VLANs, no network restrictions.
In the first subnet there is the foreman master with one smart proxy.
In the second subnet just the smart proxy (this was needed for the dhcp of the discovered hosts).
Host is discovered in 2nd subnet, facts are sent directly to foreman master in 1st subnet.

Are subnets in those one org/loc? And what do you have in Administer - Setting - Discovery - Initial location and organization?

@lzap
Both subnets are configured with the right organization/location.
I checked the settings, they seems configured properly:

Puppet -> Default location -> MyLocation
Puppet -> Default organization -> MyOrganization
Puppet -> Location fact -> foreman_location
Puppet -> Organization fact -> foreman_organization
Discovered -> Discovery location -> MyLocation
Discovered -> Discovery organization -> MyOrganization

Can you enable debug and pastebin whole production.log transaction from the first and then the second fact upload? There must be some more detailed error thrown.

@lzap many thanks for your help!

Here is the requested log file

Is it possible that there already is a host with mac48df374de058 hostname probably in a different organization or even managed? Some leftover, when transaction errors out Foreman can have an invalid record in the db. Delete it.

Hi @lzap

Indeed… I played directly with the foreman DB, and I found an invalid record under the hosts table. By invalid, I mean that this record does not appear in the web interface.

To properly delete this host (id=511), I used these commands

foremandb=> DELETE FROM fact_values WHERE host_id=511;                                                                                                                                       
DELETE 421
foremandb=> DELETE FROM nics WHERE host_id=511;                                                                                                                                              
DELETE 7
foremandb=> DELETE FROM hosts WHERE id = 511;                                                                                                                                                
DELETE 1

I was so happy… until after rebooting the server and starting the discovery… the “Name has already been taken” error appeared again, and new records for this hosts/nics/facts have been added in the database tables.

How can I fix it?

@lzap one more thing.

I think I have found a way to reproduce it… I have a 2nd server to provision. The discovery was working well for this server, facts were sent, and the host appeared under “Discovered hosts”.

Yesterday afternoon I tried to provision it, and received an error. Did not take a screenshot, but the related log from production.log is here

The thing is, now the discovery fails with the “Name has already been taken” also for this host.

My guess is that a failed provisioning from a discovered host generates something invalid inside the DB.

Yes, we have seen that. Let’s investigate that, would you mind opening a new thread? This is a different story.

We do not use SQL transactions because of orchestration framework, therefore when something bad happens during provisioning there is no rollback :frowning:

Absolutely no problem, you are right :wink:

I opened it here

1 Like