Cannot boot anymore Discovery image

[marcoskv]

Problem: Cannot boot anymore Discovery image.

Had a working installation of foreman (1.24.x, foreman discovery image 3.5.1, do not remember the version of foreman discovery plugin). With that version I discovered and provisioned 30/40 bare metal servers.
Upgraded to v2.0 (+ installed foreman datacenter plugin) a couple of months ago.
Tried to discover new bare metal servers in the last week.
Initially, discovery image was booting, but facts were not sent to foreman. Had this error: Validation failed: Name has already been taken
While trying to troubleshoot that error, I decided to delete /var/lib/tftpboot and let foreman recreate the directory on both foreman master AND foreman proxy servers (did this be re-running the foreman-installer command with the same parameters used during the installation/upgrade).
Since that moment, I was no more able to boot the discovery image. I see the Grub2 boot screen, select Foreman Discovery Image EFI, screen become black, and after a few seconds, this error appears: error: timeout reading ‘boot/fdi-image/initrd0.img’. Press any key to continue. Boot process continues but then it fails.
On the smart proxy tftp server, I see this in the logs

Aug 10 12:13:28 s157p dhcpd: DHCPDISCOVER from 48:df:37:4d:f0:6c via enp70s0f0
Aug 10 12:13:29 s157p dhcpd: none: host unknown.
Aug 10 12:13:29 s157p dhcpd: DHCPOFFER on 192.168.240.254 to 48:df:37:4d:f0:6c via enp70s0f0
Aug 10 12:13:31 s157p dhcpd: DHCPREQUEST for 192.168.240.254 (192.168.240.157) from 48:df:37:4d:f0:6c via enp70s0f0
Aug 10 12:13:31 s157p dhcpd: DHCPACK on 192.168.240.254 to 48:df:37:4d:f0:6c via enp70s0f0
Aug 10 12:13:31 s157p in.tftpd[2499]: RRQ from 192.168.240.254 filename grub2/shim.efi
Aug 10 12:13:31 s157p in.tftpd[2499]: Error code 8: User aborted the transfer
Aug 10 12:13:31 s157p in.tftpd[2500]: RRQ from 192.168.240.254 filename grub2/shim.efi
Aug 10 12:13:31 s157p in.tftpd[2500]: Client 192.168.240.254 finished grub2/shim.efi
Aug 10 12:13:31 s157p in.tftpd[2501]: RRQ from 192.168.240.254 filename grub2/grubx64.efi
Aug 10 12:13:31 s157p in.tftpd[2501]: Client 192.168.240.254 finished grub2/grubx64.efi
Aug 10 12:13:32 s157p in.tftpd[2502]: RRQ from 192.168.240.254 filename grub2/grub.cfg-01-48-df-37-4d-f0-6c
Aug 10 12:13:32 s157p in.tftpd[2502]: Client 192.168.240.254 File not found grub2/grub.cfg-01-48-df-37-4d-f0-6c
Aug 10 12:13:32 s157p in.tftpd[2503]: RRQ from 192.168.240.254 filename grub2/grub.cfg-8BBFF0FE
Aug 10 12:13:32 s157p in.tftpd[2503]: Client 192.168.240.254 File not found grub2/grub.cfg-8BBFF0FE
Aug 10 12:13:32 s157p in.tftpd[2504]: RRQ from 192.168.240.254 filename grub2/grub.cfg-8BBFF0F
Aug 10 12:13:32 s157p in.tftpd[2504]: Client 192.168.240.254 File not found grub2/grub.cfg-8BBFF0F
Aug 10 12:13:32 s157p in.tftpd[2505]: RRQ from 192.168.240.254 filename grub2/grub.cfg-8BBFF0
Aug 10 12:13:32 s157p in.tftpd[2505]: Client 192.168.240.254 File not found grub2/grub.cfg-8BBFF0
Aug 10 12:13:32 s157p in.tftpd[2506]: RRQ from 192.168.240.254 filename grub2/grub.cfg-8BBFF
Aug 10 12:13:32 s157p in.tftpd[2506]: Client 192.168.240.254 File not found grub2/grub.cfg-8BBFF
Aug 10 12:13:32 s157p in.tftpd[2507]: RRQ from 192.168.240.254 filename grub2/grub.cfg-8BBF
Aug 10 12:13:32 s157p in.tftpd[2507]: Client 192.168.240.254 File not found grub2/grub.cfg-8BBF
Aug 10 12:13:32 s157p in.tftpd[2508]: RRQ from 192.168.240.254 filename grub2/grub.cfg-8BB
Aug 10 12:13:32 s157p in.tftpd[2508]: Client 192.168.240.254 File not found grub2/grub.cfg-8BB
Aug 10 12:13:32 s157p in.tftpd[2509]: RRQ from 192.168.240.254 filename grub2/grub.cfg-8B
Aug 10 12:13:32 s157p in.tftpd[2509]: Client 192.168.240.254 File not found grub2/grub.cfg-8B
Aug 10 12:13:32 s157p in.tftpd[2510]: RRQ from 192.168.240.254 filename grub2/grub.cfg-8
Aug 10 12:13:32 s157p in.tftpd[2510]: Client 192.168.240.254 File not found grub2/grub.cfg-8
Aug 10 12:13:32 s157p in.tftpd[2511]: RRQ from 192.168.240.254 filename grub2/grub.cfg
Aug 10 12:13:32 s157p in.tftpd[2511]: Client 192.168.240.254 finished grub2/grub.cfg
Aug 10 12:13:32 s157p in.tftpd[2512]: RRQ from 192.168.240.254 filename /EFI/centos/x86_64-efi/command.lst
Aug 10 12:13:32 s157p in.tftpd[2512]: Client 192.168.240.254 File not found /EFI/centos/x86_64-efi/command.lst
Aug 10 12:13:32 s157p in.tftpd[2513]: RRQ from 192.168.240.254 filename /EFI/centos/x86_64-efi/fs.lst
Aug 10 12:13:32 s157p in.tftpd[2513]: Client 192.168.240.254 File not found /EFI/centos/x86_64-efi/fs.lst
Aug 10 12:13:32 s157p in.tftpd[2514]: RRQ from 192.168.240.254 filename /EFI/centos/x86_64-efi/crypto.lst
Aug 10 12:13:32 s157p in.tftpd[2514]: Client 192.168.240.254 File not found /EFI/centos/x86_64-efi/crypto.lst
Aug 10 12:13:32 s157p in.tftpd[2515]: RRQ from 192.168.240.254 filename /EFI/centos/x86_64-efi/terminal.lst
Aug 10 12:13:32 s157p in.tftpd[2515]: Client 192.168.240.254 File not found /EFI/centos/x86_64-efi/terminal.lst
Aug 10 12:13:32 s157p in.tftpd[2516]: RRQ from 192.168.240.254 filename grub2/grub.cfg
Aug 10 12:13:32 s157p in.tftpd[2516]: Client 192.168.240.254 finished grub2/grub.cfg
Aug 10 12:13:32 s157p in.tftpd[2517]: RRQ from 192.168.240.254 filename /httpboot/grub2/grub.cfg-48:df:37:4d:f0:6c
Aug 10 12:13:32 s157p in.tftpd[2517]: Client 192.168.240.254 File not found /httpboot/grub2/grub.cfg-48:df:37:4d:f0:6c
Aug 10 12:13:32 s157p in.tftpd[2518]: RRQ from 192.168.240.254 filename /grub2/grub.cfg-48:df:37:4d:f0:6c
Aug 10 12:13:32 s157p in.tftpd[2518]: Client 192.168.240.254 File not found /grub2/grub.cfg-48:df:37:4d:f0:6c
Aug 10 12:13:32 s157p in.tftpd[2519]: RRQ from 192.168.240.254 filename grub2/grub.cfg-48:df:37:4d:f0:6c
Aug 10 12:13:32 s157p in.tftpd[2519]: Client 192.168.240.254 File not found grub2/grub.cfg-48:df:37:4d:f0:6c
Aug 10 12:13:32 s157p in.tftpd[2520]: RRQ from 192.168.240.254 filename grub.cfg-48:df:37:4d:f0:6c
Aug 10 12:13:32 s157p in.tftpd[2520]: Client 192.168.240.254 File not found grub.cfg-48:df:37:4d:f0:6c
Aug 10 12:13:39 s157p in.tftpd[2521]: RRQ from 192.168.240.254 filename boot/fdi-image/vmlinuz0
Aug 10 12:13:39 s157p in.tftpd[2521]: Client 192.168.240.254 finished boot/fdi-image/vmlinuz0
Aug 10 12:13:39 s157p in.tftpd[2522]: RRQ from 192.168.240.254 filename boot/fdi-image/initrd0.img
Aug 10 12:14:16 s157p in.tftpd[2522]: Client 192.168.240.254 finished boot/fdi-image/initrd0.img

The timeout error appears when the message Client 192.168.240.254 finished boot/fdi-image/initrd0.img is written in the server log.

In addition, I have tried the following things

• tried to upgrade foreman/smart-proxy to v2.1.1. Nothing changed
• I made sure TFTP works, by booting a live cd on the server to discover, and after it was loaded, tried to retrieve the image from the smart proxy using TFTP. The download was working.
• I tried to Network boot using a different port of the same NIC. Nothing changed
• I tried to discover another new server. Same timeout issue.

What could be the issue? Any suggestion? Many thanks in advance

Expected outcome: Discovery image boot

Foreman and Proxy versions:
Foreman master 2.1.1
Foreman proxy (in a different subnet): 2.1.1

Foreman and Proxy plugin versions:
Foreman discovery plugin 16.1.0
Foreman discovery image 3.5.7
Foreman datacenter 2.0.1

Distribution and version:

CentOS 7.8

[lzap]

Hello,

@marcoskv

timeout error

this usually indicates there is a firewall misconfiguration. TFTP is a stateless protocol, it cannot go through NAT, it’s UDP. But in your case you can download small files but the initramdisk of discovery which is 300 MB fails - this indicates network problem. UDP packets gets lost.

@marcoskv

Any suggestion

Ditch TFTP for once and forever and start using HTTP UEFI boot if you can. We have added support in 2.1. If you still need BIOS, then what you can do iPXE chainbooting.

https://docs.theforeman.org/master/Provisioning_Guide/index-foreman.html#Configuring_Networking-Configuring_gPXE_to_Reduce_Provisioning_Times

What kind of hardware is this?

[marcoskv]

Many thanks for the prompt reply @lzap

Will give it a try with HTTP UEFI then. Will report back once I manage to get it working.

We have an heterogeneous set of servers, including HP ProLiant Gen8, 9, 10 , plus some IBM x3550

[langesmalle]

Hi Marco,

I’m experiencing exactly the same issue after upgrading foreman 1.24.x to 2.0 on Fujitsu hardware.
So far I was only able to install 1 server out of the 4 I tried, and have another 8 to go.

I am wondering if you found the reason and what you applied as solution?

Thnx for your feedback.

[lzap]

Weird, we haven’t changed anything in this regard. Can you compare TFTP server versions before/after the upgrade? But I doubt there were any changes in this ancient package.

[marcoskv]

Hi @langesmalle

I did not manage to find the root reason of why the discovery image stopped working.

I tried, as suggested from @lzap , to use UEFI http boot, but without success.

We have many HP DL380 Gen9 e Gen10. I tried to use the “Boot from URL” feature, by using an URL like http://IP_SMART_PROXY/httpboot/grub2/grubx64.efi.

This returns a grub> terminal with

Minimal BASH-like line editing is supported. For the first word, TAB lists possible command completions. Anywhere else TAB lists possible device or file completions.

httpboot was enabled on both master and smart-proxy. I had to redirect with a firewall rule port 8000 to 80.

As a reference, I followed (and found interesting) the following guides/tutorial:

https://lzap.fedorapeople.org/foreman-doc-ng/doc-Provisioning_Guide/index-foreman.html

UEFI HTTP Boot over tagged VLAN with discovery

This is a tutorial how to set up UEFI HTTP Boot over tagged VLAN provisioning with Foreman 1.23 and few more patches. Hopefully all of these can be merged for Foreman 1.24, they are all small changes. Requirements: A host for foreman, a host for dedicated smart-proxy with connection to foreman and a managed host you want to provision. I’ve created a simple setup in libvirt where smart-proxy host has two NICs: Foreman VM foreman.nat.lan (NIC: 192.168.199.0) Smart Proxy VM gwvlan.nat.lan (NIC1…

My workaround (for now ) was to use the “Create Host” option in foreman, manually type all the MAC addresses of the server, configure OS/partitioning and then directly run the provisioning. But this is a bit annoying, since the discovery saves a lot of time / manual work.

@lzap what do you suggest to try to fix httpboot?

[lzap]

For the record, Red Hat QA have identified a regression in (likely) a grub2 update in RHEL7. We are currently investigating the problem. Suggestion is to either downgrade or upgrade to Fedora Rawhide grub, until we find out what patch caused this.

https://bugzilla.redhat.com/show_bug.cgi?id=1870035

Edit: I have extracted all the builds from RHEL7 so you can only download/replace the grubx64.efi file:

http://people.redhat.com/~lzapleta/grub/

The working version for EFI/PXE is:

http://people.redhat.com/~lzapleta/grub/grub2-efi-x64-2.02-0.81.el7.x86_64/grubx64.efi

Edit 2: RH bootloader team identified the regression, it was caused by one of the CVE fixes in grub2. Here is a WIP patch that includes all security patches as well as correction of the TFTP sizing problem:

http://people.redhat.com/~lzapleta/grub/grub2-efi-x64-cdboot-2.02-0.87.el7.hotfix/grubx64.efi

Keep in mind SecureBoot will not work as this file is not signed.

[langesmalle]

Marco,

I had also one server that resulted constantly in the GRUB-prompt
Only the first 1GB network interface for UEFI PXE was configured as boot option.
By configuring UEFI HTTP Boot as extra boot option for the same interface, but as first option and as second one the PXE I got rid of the GRUB-prompt and got the expected menu to select the desired Foreman Discovery Image option.

This was the case for my Fujitsu RX-2530-M5 System, no idea how it is for HP.
Beside this, we also have HP Apollo 4200 Gen 9 and HP DL-360 Gen9 & 10 servers.
But were installed in the past with cobbler. The idea is to build them from scratch in the future with Foreman.

I also tried to use the grubx64.efi from Fedora Rawhide, but that did not solved our problem as well.

tcpdump from our Foreman-server rsults in connection loss as of packet 66678 for the initrd0.img file.

So, I will try the workaround by adding/configuring the host manually in Foreman and building the provisioning file. Hope this might be a temporary solution fo rme as well.

Anyway, thnx for your feedback.

I hope they will find and provide soon a solution for this issue.

[marcoskv]

@lzap that was the root cause.

On the smart-proxy, I downgraded grub2-efi-x64 to v0.81 (once updated, it become v0.86)

yum remove grub2-common grub2-tools grub2-tools-extra grub2-efi  grub2-common grub2-efi-x64-modules  grub2-tools-minimal grub2-pc-modules
yum install grub2-common-2.02-0.81.el7.centos.noarch grub2-tools-minimal-2.02-0.81.el7.centos.x86_64 grub2-efi-x64-2.02-0.81.el7.centos

and then replaced grubx64.efi in tftpboot with the old version

mv /var/lib/tftpboot/grub2/grubx64.efi /var/lib/tftpboot/grub2/grubx64.efi.bak
cp /boot/efi/EFI/centos/grubx64.efi /var/lib/tftpboot/grub2/grubx64.efi
chmod 644 /var/lib/tftpboot/grub2/grubx64.efi

Now the discovery image works again using the standard Grub2 UEFI boot (that uses tftp). Yuppy

And now I am back to the good old problem I had before upgrading foreman and centos7…

with this message in foreman’s production.log

2020-08-21T15:47:28 [W|app|1fbe3132] Host discovery failed, facts: {xxx}
2020-08-21T15:47:28 [I|app|1fbe3132] Completed 422 Unprocessable Entity in 512ms (Views: 0.2ms | ActiveRecord: 250.8ms | Allocations: 147840)

[marcoskv]

If you temporarily downgrade grub2-efi on your smart-proxy, the discovery image will restart working fine.

Give it a try

[langesmalle]

Hi Marco,

Thanks for this feedback. This is wonderful news and I will apply these actions as first job, next Monday.

I also had the “FAILURE” ststus during the first discovery in version 1.24.x of Foreman.
As a solution I applied “Resend” and it then went well.
Give it a try, hope it will work for you as well.

[marcoskv]

Thanks for the suggestion, but unfortunately the Resend does not fix the validation issue.

Let me know how it goes on Monday. Good luck

[marcoskv]

Actually, the full log with debug enabled from production.log says

Host discovery failed, facts: {xxx ...., "discovery_proxy_type"=>"foreman", "_type"=>:foreman_discovery}' error (ActiveRecord::RecordInvalid): Validation failed: Name has already been taken
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/validations.rb:80:in `raise_validation_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/validations.rb:53:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:318:in `block in save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:318:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/suppressor.rb:48:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-16.1.0/app/models/host/discovered.rb:133:in `populate_discovery_fields_from_facts'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-16.1.0/app/models/host/discovered.rb:124:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/base.rb:168:in `block in parse_facts'
/usr/share/foreman/app/services/foreman/telemetry_helper.rb:27:in `telemetry_duration_histogram'
/usr/share/foreman/app/models/host/base.rb:167:in `parse_facts'
/usr/share/foreman/app/models/host/base.rb:153:in `import_facts'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-16.1.0/app/models/host/discovered.rb:105:in `import_facts'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-16.1.0/app/models/host/discovered.rb:95:in `import_host'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-16.1.0/app/controllers/api/v2/discovered_hosts_controller.rb:104:in `block in facts'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-16.1.0/app/controllers/api/v2/discovered_hosts_controller.rb:103:in `facts'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:195:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:139:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in `block in instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:136:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.1/lib/action_view/rendering.rb:39:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:190:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:254:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:49:in `block in serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `serve'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:834:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.2.2/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/etag.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/conditional_get.rb:40:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/session/abstract/id.rb:266:in `context'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/session/abstract/id.rb:260:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/cookies.rb:648:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:101:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:37:in `call_app'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/method_override.rb:24:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/sendfile.rb:110:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:74:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/configuration.rb:228:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:682:in `handle_request'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:472:in `process_client'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:328:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/thread_pool.rb:134:in `block in spawn_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-08-21T17:49:28 [D|app|6bcc20ad] Body: {"message":"Validation failed: Name has already been taken"}
2020-08-21T17:49:28 [I|app|6bcc20ad] Completed 422 Unprocessable Entity in 876ms (Views: 0.2ms | ActiveRecord: 342.5ms | Allocations: 211517)

[langesmalle]

Hi Marco,

I can confirm that the grub2 package version 0.81 works as expected.
Now the Foreman Discovery Image is well transferred to the new host.

As I said, during the first discovery, it results in a failure, however, with a different error.

But after a resend, it went well.

[marcoskv]

Glad to hear the downgrade works well for you too!

Unfortunately the “Resend” does not work for me, still something to solve to go back to normal operations

[langesmalle]

Marco,

See also following remarks regarding this fix:

Foreman efi discovery fails on Fujitsu RX2530

Hi Lukas, Some extra feedback. As said above, the Foreman Discovery Image efi works when using grubx64.efi from the grub2-efi v0.81. However, once I have finished the host-config en submit it for final server installation, it ends up with a grub-prompt. In /var/log/messages I see following for the configured host: [image] And then it ends with the grub-prompt. But, once I use the Rawhide version of grbx64.efi in /var/lib/tftboot/grub2, it will go further with loading the grub-cgf file for…

[lzap]

Which location the host is initially discovered in? And in which ones is the subnet?

[marcoskv]

@lzap
I have configured one organization with only one location and one domain.
There are 2 subnets, no VLANs, no network restrictions.
In the first subnet there is the foreman master with one smart proxy.
In the second subnet just the smart proxy (this was needed for the dhcp of the discovered hosts).
Host is discovered in 2nd subnet, facts are sent directly to foreman master in 1st subnet.

[lzap]

Are subnets in those one org/loc? And what do you have in Administer - Setting - Discovery - Initial location and organization?

[marcoskv]

@lzap
Both subnets are configured with the right organization/location.
I checked the settings, they seems configured properly:

Puppet -> Default location -> MyLocation
Puppet -> Default organization -> MyOrganization
Puppet -> Location fact -> foreman_location
Puppet -> Organization fact -> foreman_organization
Discovered -> Discovery location -> MyLocation
Discovered -> Discovery organization -> MyOrganization