iPXE "No more network devices"

Support
1

Problem:
When attempting boot into the discovery environment using iPXE, I get a couple error messages:

  • I get a “Network unreachable” error message immediately once iPXE is loaded and the net0 interface is brought up.
  • I then get a “No more network devices” message after iPXE appears to successfully access http://10.254.52.26/unattended/iPXE?bootstrap=1 and pulls a “81 byte” script.

Here is a screenshot of this occurrence: https://imgur.com/sSE57Ci

Expected outcome:
I would expect the process to continue booting into the discovery environment.

Foreman and Proxy versions: 2.0.3

Foreman and Proxy plugin versions: N/A

Distribution and version: CentOS 7

Other relevant data:

Here is an excerpt from the production.log while this occurred:

2020-11-12T21:28:29 [I|app|a7484b4d] Started GET "/unattended/iPXE?bootstrap=1" for 10.59.246.5 at 2020-11-12 21:28:29 +0000
2020-11-12T21:28:29 [I|app|a7484b4d] Processing by UnattendedController#host_template as TEXT
2020-11-12T21:28:29 [I|app|a7484b4d]   Parameters: {"bootstrap"=>"1", "kind"=>"iPXE"}
2020-11-12T21:28:29 [I|app|a7484b4d]   Rendering text template
2020-11-12T21:28:29 [I|app|a7484b4d]   Rendered text template (0.0ms)
2020-11-12T21:28:29 [I|app|a7484b4d] Completed 200 OK in 51ms (Views: 0.4ms | ActiveRecord: 12.9ms)

If I boot an OS on this hardware, set my IP settings to match what DHCP gave me during PXE, and attempt to pull /unattended/iPXE?bootstrap=1 or even http://10.254.52.26:8000/httpboot/boot/fdi-image/vmlinuz0, it is successful, which makes me think it is not a firewall or connectivity related issue at play here (unless there is some other connectivity going on that I’m unaware of).

Here is my dhcpd.conf (my DHCP server is not managed by Foreman):

default-lease-time 600;
max-lease-time 7200;
option architecture code 93 = unsigned integer 16;

if exists user-class and option user-class = "iPXE" {
  filename "http://10.254.52.26/unattended/iPXE?bootstrap=1";
} else {
  filename "undionly.0";
}

subnet 10.59.246.0 netmask 255.255.255.224 {
  range 10.59.246.3 10.59.246.20;
  option routers 10.59.246.1;
  option domain-name-servers 10.92.128.40, 10.92.128.41;
  option domain-name "mydomain.net";
  next-server 10.254.52.26; 
}

I’m new to the whole iPXE thing, so apologies if I am missing something obvious!

2

Folks, it is official. I’m a dum-dum :slight_smile:

This host already existed in Foreman and I was expecting it to PXE boot into the discovery environment, which wouldn’t make much sense… I just forgot it was already a known host.

After deleting the host and trying again, all went just swimmingly. There are still a few kinks/customizations for me to work out, but I should hopefully have it from here… If not, no doubt I’ll be back!

Be well!

1 Like
3

I wanted to say 81 bytes is not expected and your iPXE driver probably lost connection but… okay.

4

Hey lzap! Hope you’ve been well.

It being 81 bytes is odd, isn’t it? I thought so too…

That only happens when a host is already “known” to Foreman. However, isn’t the “script” which is downloaded (http://10.254.52.26/unattended/iPXE) generated from the “iPXE intermediate script” template?

If I pull up that template up in Foreman and “preview” it for this particular host, I get:

#!ipxe
# Intermediate iPXE script to report MAC address to Foreman

:net0
isset ${net0/mac} || goto no_nic
dhcp net0 || goto net1
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet0%2Fmac%7D || goto net1

:net1
isset ${net1/mac} || goto no_nic
dhcp net1 || goto net2
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet1%2Fmac%7D || goto net2

:net2
isset ${net2/mac} || goto no_nic
dhcp net2 || goto net3
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet2%2Fmac%7D || goto net3

:net3
isset ${net3/mac} || goto no_nic
dhcp net3 || goto net4
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet3%2Fmac%7D || goto net4

:net4
isset ${net4/mac} || goto no_nic
dhcp net4 || goto net5
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet4%2Fmac%7D || goto net5

:net5
isset ${net5/mac} || goto no_nic
dhcp net5 || goto net6
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet5%2Fmac%7D || goto net6

:net6
isset ${net6/mac} || goto no_nic
dhcp net6 || goto net7
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet6%2Fmac%7D || goto net7

:net7
isset ${net7/mac} || goto no_nic
dhcp net7 || goto net8
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet7%2Fmac%7D || goto net8

:net8
isset ${net8/mac} || goto no_nic
dhcp net8 || goto net9
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet8%2Fmac%7D || goto net9

:net9
isset ${net9/mac} || goto no_nic
dhcp net9 || goto net10
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet9%2Fmac%7D || goto net10

:net10
isset ${net10/mac} || goto no_nic
dhcp net10 || goto net11
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet10%2Fmac%7D || goto net11

:net11
isset ${net11/mac} || goto no_nic
dhcp net11 || goto net12
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet11%2Fmac%7D || goto net12

:net12
isset ${net12/mac} || goto no_nic
dhcp net12 || goto net13
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet12%2Fmac%7D || goto net13

:net13
isset ${net13/mac} || goto no_nic
dhcp net13 || goto net14
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet13%2Fmac%7D || goto net14

:net14
isset ${net14/mac} || goto no_nic
dhcp net14 || goto net15
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet14%2Fmac%7D || goto net15

:net15
isset ${net15/mac} || goto no_nic
dhcp net15 || goto net16
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet15%2Fmac%7D || goto net16

:net16
isset ${net16/mac} || goto no_nic
dhcp net16 || goto net17
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet16%2Fmac%7D || goto net17

:net17
isset ${net17/mac} || goto no_nic
dhcp net17 || goto net18
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet17%2Fmac%7D || goto net18

:net18
isset ${net18/mac} || goto no_nic
dhcp net18 || goto net19
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet18%2Fmac%7D || goto net19

:net19
isset ${net19/mac} || goto no_nic
dhcp net19 || goto net20
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet19%2Fmac%7D || goto net20

:net20
isset ${net20/mac} || goto no_nic
dhcp net20 || goto net21
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet20%2Fmac%7D || goto net21

:net21
isset ${net21/mac} || goto no_nic
dhcp net21 || goto net22
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet21%2Fmac%7D || goto net22

:net22
isset ${net22/mac} || goto no_nic
dhcp net22 || goto net23
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet22%2Fmac%7D || goto net23

:net23
isset ${net23/mac} || goto no_nic
dhcp net23 || goto net24
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet23%2Fmac%7D || goto net24

:net24
isset ${net24/mac} || goto no_nic
dhcp net24 || goto net25
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet24%2Fmac%7D || goto net25

:net25
isset ${net25/mac} || goto no_nic
dhcp net25 || goto net26
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet25%2Fmac%7D || goto net26

:net26
isset ${net26/mac} || goto no_nic
dhcp net26 || goto net27
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet26%2Fmac%7D || goto net27

:net27
isset ${net27/mac} || goto no_nic
dhcp net27 || goto net28
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet27%2Fmac%7D || goto net28

:net28
isset ${net28/mac} || goto no_nic
dhcp net28 || goto net29
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet28%2Fmac%7D || goto net29

:net29
isset ${net29/mac} || goto no_nic
dhcp net29 || goto net30
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet29%2Fmac%7D || goto net30

:net30
isset ${net30/mac} || goto no_nic
dhcp net30 || goto net31
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet30%2Fmac%7D || goto net31

:net31
isset ${net31/mac} || goto no_nic
dhcp net31 || goto net32
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet31%2Fmac%7D || goto net32

:net32
isset ${net32/mac} || goto no_nic
dhcp net32 || goto net33
chain http://foremanserver:8000/unattended/iPXE?mac=%24%7Bnet32%2Fmac%7D || goto net33

:net33
goto no_nic

exit 0

:no_nic
echo Failed to chainload from any network interface
sleep 30
exit 1

There is no way that is 80 bytes (its actually about 5000 bytes).

However if I curl that script from the host once its booted up the OS and pass the bootstrap=1 HTTP parameter (like DHCP instructs iPXE to do), I do get a 80 byte script:

$ curl http://10.254.52.26/unattended/iPXE?bootstrap=1
#!ipxe

# Skips booting from network and continues booting from next device
exit

Which is awesome, its exactly what I need to happen. And if I take a step further and directly curl the URL which is in the previewed iPXE template (by passing the MAC address as a HTTP paramater in the URL):

$ curl http://10.254.52.26:8000/unattended/iPXE?mac=40:f2:e9:d6:5c:1a
#!ipxe

# Skips booting from network and continues booting from next device
exit

I have no idea what is going on under the hood when you pass that bootstrap=1 parameter, but it seems to go straight to passing on the iPXE script of booting the hard disk. Its all black magic to me, which is exciting!

I continue to be impressed with how all of this operates and how it mostly “just works”. I thought setting up iPXE was going to be a real pain in the butt, but all it took was downloading the iPXE boot file, throwing it in my TFTP root and updating my dhcpd.conf to hand it over to PXE clients. The whole discovery process over iPXE worked flawlessly after that. Very nice!

5

I have found one interesting thing however which I could use some guidance on… I know this isn’t 100% the exact issue this thread was originally made for, so if anyone thinks I should create a new thread, I will gladly do so.

I’ve been discovering and building bare-metal on a couple hosts for the past couple days using iPXE. However, one of those hosts suddenly continues to always boot into the discovery environment, even though it is a known host. What is odd is that I can delete and build the other host just fine, without ever running into this issue. I’m also pretty darn sure I had the “problem host” operating just fine yesterday…

So lets walk through some of this… My host with the issue has been discovered and comes up in my “host” list. It has an OS on it, so if I SSH in, I can start to try and emulate the HTTP communication iPXE would handle.

  1. The first thing iPXE should do after loading is use the “filename” option and chain load http://10.254.52.26/unattended/iPXE?bootstrap=1 
# curl http://10.254.52.26/unattended/iPXE?bootstrap=1
#!ipxe
# Intermediate iPXE script to report MAC address to Foreman

:net0
isset ${net0/mac} || goto no_nic
dhcp net0 || goto net1
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet0%2Fmac%7D || goto net1

:net1
isset ${net1/mac} || goto no_nic
dhcp net1 || goto net2
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet1%2Fmac%7D || goto net2

:net2
isset ${net2/mac} || goto no_nic
dhcp net2 || goto net3
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet2%2Fmac%7D || goto net3

:net3
isset ${net3/mac} || goto no_nic
dhcp net3 || goto net4
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet3%2Fmac%7D || goto net4

:net4
isset ${net4/mac} || goto no_nic
dhcp net4 || goto net5
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet4%2Fmac%7D || goto net5

:net5
isset ${net5/mac} || goto no_nic
dhcp net5 || goto net6
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet5%2Fmac%7D || goto net6

:net6
isset ${net6/mac} || goto no_nic
dhcp net6 || goto net7
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet6%2Fmac%7D || goto net7

:net7
isset ${net7/mac} || goto no_nic
dhcp net7 || goto net8
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet7%2Fmac%7D || goto net8

:net8
isset ${net8/mac} || goto no_nic
dhcp net8 || goto net9
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet8%2Fmac%7D || goto net9

:net9
isset ${net9/mac} || goto no_nic
dhcp net9 || goto net10
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet9%2Fmac%7D || goto net10

:net10
isset ${net10/mac} || goto no_nic
dhcp net10 || goto net11
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet10%2Fmac%7D || goto net11

:net11
isset ${net11/mac} || goto no_nic
dhcp net11 || goto net12
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet11%2Fmac%7D || goto net12

:net12
isset ${net12/mac} || goto no_nic
dhcp net12 || goto net13
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet12%2Fmac%7D || goto net13

:net13
isset ${net13/mac} || goto no_nic
dhcp net13 || goto net14
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet13%2Fmac%7D || goto net14

:net14
isset ${net14/mac} || goto no_nic
dhcp net14 || goto net15
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet14%2Fmac%7D || goto net15

:net15
isset ${net15/mac} || goto no_nic
dhcp net15 || goto net16
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet15%2Fmac%7D || goto net16

:net16
isset ${net16/mac} || goto no_nic
dhcp net16 || goto net17
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet16%2Fmac%7D || goto net17

:net17
isset ${net17/mac} || goto no_nic
dhcp net17 || goto net18
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet17%2Fmac%7D || goto net18

:net18
isset ${net18/mac} || goto no_nic
dhcp net18 || goto net19
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet18%2Fmac%7D || goto net19

:net19
isset ${net19/mac} || goto no_nic
dhcp net19 || goto net20
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet19%2Fmac%7D || goto net20

:net20
isset ${net20/mac} || goto no_nic
dhcp net20 || goto net21
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet20%2Fmac%7D || goto net21

:net21
isset ${net21/mac} || goto no_nic
dhcp net21 || goto net22
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet21%2Fmac%7D || goto net22

:net22
isset ${net22/mac} || goto no_nic
dhcp net22 || goto net23
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet22%2Fmac%7D || goto net23

:net23
isset ${net23/mac} || goto no_nic
dhcp net23 || goto net24
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet23%2Fmac%7D || goto net24

:net24
isset ${net24/mac} || goto no_nic
dhcp net24 || goto net25
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet24%2Fmac%7D || goto net25

:net25
isset ${net25/mac} || goto no_nic
dhcp net25 || goto net26
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet25%2Fmac%7D || goto net26

:net26
isset ${net26/mac} || goto no_nic
dhcp net26 || goto net27
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet26%2Fmac%7D || goto net27

:net27
isset ${net27/mac} || goto no_nic
dhcp net27 || goto net28
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet27%2Fmac%7D || goto net28

:net28
isset ${net28/mac} || goto no_nic
dhcp net28 || goto net29
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet28%2Fmac%7D || goto net29

:net29
isset ${net29/mac} || goto no_nic
dhcp net29 || goto net30
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet29%2Fmac%7D || goto net30

:net30
isset ${net30/mac} || goto no_nic
dhcp net30 || goto net31
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet30%2Fmac%7D || goto net31

:net31
isset ${net31/mac} || goto no_nic
dhcp net31 || goto net32
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet31%2Fmac%7D || goto net32

:net32
isset ${net32/mac} || goto no_nic
dhcp net32 || goto net33
chain http://foremanserver/unattended/iPXE?mac=%24%7Bnet32%2Fmac%7D || goto net33

:net33
goto no_nic

exit 0

:no_nic
echo Failed to chainload from any network interface
sleep 30
exit 1
  1. As you can see, based on the iPXE script, the host should continue on to chain loading http://chluxforeman01p.na.odcorp.net/unattended/iPXE?mac=%24%7Bnet0%2Fmac%7D. If I’m not mistaken, this essentially gets translated to http://chluxforeman01p.na.odcorp.net/unattended/iPXE?mac=40:f2:e9:d6:4f:4a, no?
# curl http://foremanserver/unattended/iPXE?mac=40:f2:e9:d6:4f:4a
#!gpxe

kernel http://foremanserver/pulp/repos/ORG/DEV/Kiosk/custom/CentOS_8_Linux_for_x86_64/CentOS8-BaseOS-x86_64//images/pxeboot/vmlinuz initrd=initrd.img ks=http://foremanserver:8000/unattended/provision?token=5ed56c59-997c-44d0-b709-53b89f510f20&static=yes inst.stage2=http://foremanserver/pulp/repos/ORGDEV/Kiosk/custom/CentOS_8_Linux_for_x86_64/CentOS8-BaseOS-x86_64/  ksdevice=40:f2:e9:d6:4f:4a network kssendmac ks.sendmac inst.ks.sendmac ip=${netX/ip}::${netX/gateway}:${netX/netmask}:ods00298r02.odretail.net:enp6s0:none nameserver=${dns} inst.dd=http://foremanserver/pulp/repos/ORG/Library/custom/odr/Extras/Packages/k/kmod-megaraid_sas-07.710.50.00-1.el8_2.elrepo.x86_64.rpm
initrd http://foremanserver/pulp/repos/ORG/DEV/Kiosk/custom/CentOS_8_Linux_for_x86_64/CentOS8-BaseOS-x86_64//images/pxeboot/initrd.img
imgstat
sleep 2
boot

Following that logic, my iPXE booting client should be chain loading into the CentOS installation media.

Here is what happens in the production.log while the host go through the process:

2020-11-18T17:07:34 [I|app|2ae097cb] Started GET "/unattended/iPXE?bootstrap=1" for 127.0.0.1 at 2020-11-18 17:07:34 +0000
2020-11-18T17:07:34 [I|app|2ae097cb] Processing by UnattendedController#host_template as TEXT
2020-11-18T17:07:34 [I|app|2ae097cb]   Parameters: {"bootstrap"=>"1", "kind"=>"iPXE"}
2020-11-18T17:07:34 [I|app|2ae097cb]   Rendering text template
2020-11-18T17:07:34 [I|app|2ae097cb]   Rendered text template (Duration: 0.0ms | Allocations: 3)
2020-11-18T17:07:34 [I|app|2ae097cb] Completed 200 OK in 42ms (Views: 0.5ms | ActiveRecord: 4.4ms | Allocations: 28609)
2020-11-18T17:07:50 [I|app|cfe25896] Started GET "/unattended/iPXE?mac=${net0/mac}" for 127.0.0.1 at 2020-11-18 17:07:50 +0000
2020-11-18T17:07:50 [I|app|cfe25896] Processing by UnattendedController#host_template as TEXT
2020-11-18T17:07:50 [I|app|cfe25896]   Parameters: {"mac"=>"${net0/mac}", "kind"=>"iPXE"}
2020-11-18T17:07:50 [I|app|cfe25896]   Rendering text template
2020-11-18T17:07:50 [I|app|cfe25896]   Rendered text template (Duration: 0.0ms | Allocations: 3)
2020-11-18T17:07:50 [I|app|cfe25896] Completed 200 OK in 19ms (Views: 0.4ms | ActiveRecord: 3.4ms | Allocations: 11685)

I’m not even sure how to approach troubleshooting this. The fact that the other host has no issues really makes me scratch my head…

6

Well, normally I’d say that “black magic” is our de-facto standard, but in this case I have a full chapter which explains this workflow hopefully to the desired detail. Reach out to me if you find anything unclear:

The secret sauce is the bootstrap flag, when set, Foreman returns a template called iPXE intermediate script which can be set in Administer - Settings - Provisioning. It’s a kind of a “global” template that performs additional request now with the MAC address which is later used to find the matching host entry.

iPXE in BIOS mode is indeed very nice, if drivers are available for the hardware. I highly recommend iPXE for VM provisioning (it is guaranteed to work in most virtual environments). When using libvirt/oVirt/RHV VMs actually directly use iPXE. It’s fast and reliable.

1 Like
7

Thank you so much again for sharing the knowledge Lukas!

Not sure if you saw my previous post in this thread (before your last reply) about a particular host of mine that repeats the discovery process over and over, even after being discovered and comes up in my Foreman host list.

Looking at the documentation you provided, the issue is happening at about this point:

The previous workflow repeats, but Foreman recognizes the host’s remote IP address and instead of the intermediate template, the host receives a regular iPXE template.

This particular statement rang a bell in my head… After going through the discovery process, rebooting and loading iPXE again, if it doesn’t get the same IP leased to it from DHCP that it was discovered with, will it be able to “match” its self with its object in Foreman?

  • I have one host that never has issues with this. However, it has a DHCP reservation based on its MAC address, so it always gets the same IP.
  • The other host continues to have this problem, however it gets its lease is assigned by DHCP out of a pool. Unfortunately due to the nature of the kind of hosts I want to build, I can’t use MAC address reservations. However, as test, I setup a MAC address reservation and the entire build process works fine like that.

This is of course an unmanaged DHCP server.

For some reason I thought I read somewhere that this process would use the MAC address rather than the IP? Is there some way to allow that kind of configuration?

8

You have discovered a new bug, an overlook when I was refactoring the template:

I will fix it today.

1 Like
9

Here you are, hotfix your instance and get back to us: https://github.com/theforeman/foreman/pull/8143

1 Like
10

Ah, I thought that looked a little bit odd, but just shrugged it off as something I simply didn’t understand.

I implemented the hot fix and I think I’m now golden. I built my host, going through the discovery process, CentOS installation and hard disk booting without any issues. I’ll continue to test and let you know if I run into any other issues.

Once again, thanks so much!