Unattended setup PXE not working as intended

Support
#1

Thought this was better suited for the mailing list , but hanging out on
IRC as well (AbyssOne).

I am having some issues getting foreman to play using PXE.

The environment uses puppet 3.0 , and I use nightly foreman and
foreman-proxy all on Debian 6.0 foreman runs virtualised puppet on a
physical box only difference is the kernel version due to hardware support
on the Dell servers.

I have a Puppet server in net a, a foreman server in net b (behind a
firewall NAT's etc. work)
I have in net B (this is a lab net virtualised) TFTP server running on my
foreman server and a DHCP which I manually manage the leases on for testing
purposes. This works as the systems both physical and virtual try to net
boot.

But this is where the issues start , I am not well versed enough in PXE to
see what I am missing and something is missing. For test purposes I am
using the templates for provisioning as supplied with foreman. I have
associated the pressed templates with my debian 6 OS so so far that should
be good , that would be the pressed pxe , provisioning and finish
templates. I have not associated these with environments as it doesn't fit
my end needs as things like domains etc are host based as I am wanting to
use puppet/foreman to manage 3 data centers where IP , domain names etc all
vary but the packages etc should be the same. In essence all hosts are the
same except for IP and domain names.

Anyways back on track, when I click the "Build PXE Default" I get a file
generated but this default file looks like the "PXE Default File" and this
file doesnt work with the error "could not find kernel image: menu". So
after some debugging and manual file installation I got it to work , but I
thought this part should be automated. SO nwo I think I am missing
something I am just not sure where.

By the way I have associated the "preseed default" and so on with a
hostgroup and then my default changes but it will still generate the same
error if I do not manually edit the file.

First line read : " DEFAULT menu" this doesn't work if I change this to
reflect the LABEL of the lines added via the hostgroup it will boot and
fetch a boot image.

I think my problem lays somewhere in the initial setup of the templates vs
hostgroups vs hosts vs OS, but I am not sure and I am not seeing it at this
point. I get down stream errors with the preseed templates but I think this
is related to the initial setup.

Thanks for any help.

Oh the log files , I have turned on debugging , and the foreman-proxy log
file is surprisingly empty occasional :

I, [2013-01-03T12:24:02.764746 #11489] INFO – : TFTP:
/srv/tftp/pxelinux.cfg/default entry created successfully
D, [2013-01-03T12:24:02.785457 #11489] DEBUG – : Starting task (total: 0):
wget --timeout=10 --tries=3 --no-check-certificate -nv -c
http://ftp.de.debian.org/debian/dists/squeeze/main/installer-amd64/current/images/netboot/debian-installer/amd64/linux
-O "/srv/tftp/boot/Debian-6.0-x86_64-linux"
D, [2013-01-03T12:24:02.794239 #11489] DEBUG – : Starting task (total: 0):
wget --timeout=10 --tries=3 --no-check-certificate -nv -c
http://ftp.de.debian.org/debian/dists/squeeze/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz
-O "/srv/tftp/boot/Debian-6.0-x86_64-initrd.gz"

by the way it should fetch several different boot images as I have up to 9
different CPU types and OS version combinations.Unless I understand this
fetch wrong , but it only does this if I add the pressed PXE default to a
hostgroup not in any other way.

#2

> Anyways back on track, when I click the "Build PXE Default" I get a file
> generated but this default file looks like the "PXE Default File" and this
> file doesnt work with the error "could not find kernel image: menu". So
> after some debugging and manual file installation I got it to work , but I
> thought this part should be automated. SO nwo I think I am missing something
> I am just not sure where.

That's menu.c32 from syslinux that's missing in your tftp folder.

Do (adjust paths according to your OS):
cp /usr/share/syslinux/menu.c32 /var/lib/tftpboot

But that menu is not needed for the server to be provisioned. It's
just the default - what the host falls back to when there is no host
specific pxelinux.cfg/mac-add-ress … Simply put you normally only
need to generate the PXELinux default once.

> I think my problem lays somewhere in the initial setup of the templates vs
> hostgroups vs hosts vs OS, but I am not sure and I am not seeing it at this
> point. I get down stream errors with the preseed templates but I think this
> is related to the initial setup.

The tftpboot/pxelinux.cfg/0a-ma-ca-dd-re-ss file doesn't get written
until you actually click "Build" on a host.

So, revisit your host and templates, make sure (for Debian) that you
have 3 templates assigned. They are:
PXELinux Template
finish Template
provision Template.

Go view a host in Foreman - click the Templates tab on the left hand
side of the host overview to verify all your templates look ok.
Especially the PXELinux Template in this case, as PXE is the first
step that needs to work.

> Oh the log files , I have turned on debugging , and the foreman-proxy log
> file is surprisingly empty occasional :
>
> I, [2013-01-03T12:24:02.764746 #11489] INFO – : TFTP:
> /srv/tftp/pxelinux.cfg/default entry created successfully
> D, [2013-01-03T12:24:02.785457 #11489] DEBUG – : Starting task (total: 0):
> wget --timeout=10 --tries=3 --no-check-certificate -nv -c
> http://ftp.de.debian.org/debian/dists/squeeze/main/installer-amd64/current/images/netboot/debian-installer/amd64/linux
> -O "/srv/tftp/boot/Debian-6.0-x86_64-linux"
> D, [2013-01-03T12:24:02.794239 #11489] DEBUG – : Starting task (total: 0):
> wget --timeout=10 --tries=3 --no-check-certificate -nv -c
> http://ftp.de.debian.org/debian/dists/squeeze/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz
> -O "/srv/tftp/boot/Debian-6.0-x86_64-initrd.gz"
>
> by the way it should fetch several different boot images as I have up to 9
> different CPU types and OS version combinations.Unless I understand this
> fetch wrong , but it only does this if I add the pressed PXE default to a
> hostgroup not in any other way.

Per above, it only does this when you actually build a host so unless
you clicked build already on all 9 types of hosts you will not get all
images downloaded.

··· On Thu, Jan 3, 2013 at 12:52 PM, Jelle B. wrote:

–
Mikael

#3

Ok so found it , after some back and forth on IRC , thanks again for that,
it turns out it is a configuration piece that is missing but not very
obvious and also not very documented.

Ok what it was, in the subnet configuration in foreman you have to enter ,
via some pull down menus, the tftp proxy , dns and dhcp (where applicable)
without this foreman will silently not try to make any PXE configs for you.
The logical reason being that it doesn't know where to put it , and if you
have ,like me, multiple subnets configured it is even more reasonable.
However it is silently not doing anything and that was kinda not so good :wink:
Anyways I know it will change :slight_smile:

Also while on the subject be aware that de fault unattended scripts install
puppet 2.6.x files and as most will be setting up on puppet 3 by now this
causes some weird errors on debian , just upgrade the puppet client and all
will be good :slight_smile:

··· On Thursday, January 3, 2013 12:52:45 PM UTC+1, Jelle B. wrote: > > Thought this was better suited for the mailing list , but hanging out on > IRC as well (AbyssOne). > > I am having some issues getting foreman to play using PXE. > > The environment uses puppet 3.0 , and I use nightly foreman and > foreman-proxy all on Debian 6.0 foreman runs virtualised puppet on a > physical box only difference is the kernel version due to hardware support > on the Dell servers. > > I have a Puppet server in net a, a foreman server in net b (behind a > firewall NAT's etc. work) > I have in net B (this is a lab net virtualised) TFTP server running on my > foreman server and a DHCP which I manually manage the leases on for testing > purposes. This works as the systems both physical and virtual try to net > boot. > > But this is where the issues start , I am not well versed enough in PXE to > see what I am missing and something is missing. For test purposes I am > using the templates for provisioning as supplied with foreman. I have > associated the pressed templates with my debian 6 OS so so far that should > be good , that would be the pressed pxe , provisioning and finish > templates. I have not associated these with environments as it doesn't fit > my end needs as things like domains etc are host based as I am wanting to > use puppet/foreman to manage 3 data centers where IP , domain names etc all > vary but the packages etc should be the same. In essence all hosts are the > same except for IP and domain names. > > Anyways back on track, when I click the "Build PXE Default" I get a file > generated but this default file looks like the "PXE Default File" and this > file doesnt work with the error "could not find kernel image: menu". So > after some debugging and manual file installation I got it to work , but I > thought this part should be automated. SO nwo I think I am missing > something I am just not sure where. > > By the way I have associated the "preseed default" and so on with a > hostgroup and then my default changes but it will still generate the same > error if I do not manually edit the file. > > First line read : " DEFAULT menu" this doesn't work if I change this to > reflect the LABEL of the lines added via the hostgroup it will boot and > fetch a boot image. > > I think my problem lays somewhere in the initial setup of the templates vs > hostgroups vs hosts vs OS, but I am not sure and I am not seeing it at this > point. I get down stream errors with the preseed templates but I think this > is related to the initial setup. > > Thanks for any help. > > Oh the log files , I have turned on debugging , and the foreman-proxy log > file is surprisingly empty occasional : > > I, [2013-01-03T12:24:02.764746 #11489] INFO -- : TFTP: > /srv/tftp/pxelinux.cfg/default entry created successfully > D, [2013-01-03T12:24:02.785457 #11489] DEBUG -- : Starting task (total: > 0): wget --timeout=10 --tries=3 --no-check-certificate -nv -c > http://ftp.de.debian.org/debian/dists/squeeze/main/installer-amd64/current/images/netboot/debian-installer/amd64/linux-O "/srv/tftp/boot/Debian-6.0-x86_64-linux" > D, [2013-01-03T12:24:02.794239 #11489] DEBUG -- : Starting task (total: > 0): wget --timeout=10 --tries=3 --no-check-certificate -nv -c > http://ftp.de.debian.org/debian/dists/squeeze/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz-O "/srv/tftp/boot/Debian-6.0-x86_64-initrd.gz" > > by the way it should fetch several different boot images as I have up to 9 > different CPU types and OS version combinations.Unless I understand this > fetch wrong , but it only does this if I add the pressed PXE default to a > hostgroup not in any other way. > > > > >
#4

As I am coping with the same subject, I'll try to revive this thread
instead of opening the next one. The last two weeks, I tried different
setups and always failed when setting up provisioning.
As I didn't manage to get Foreman 1.9 with bootdisk-plugin and full image
provisioning up and running, I (for now) settled for setting up a
DHCP/TFTP/Puppetmaster instance in every hosting subnet.

My current setup looks like this:

  • Foreman instance with external DB and CA in our 10.3.0.0/19 management net
  • Puppetmaster with TFTP/DHCP/Discovery in 10.5.0.0./19 customer management
    subnet
  • Host (to be provisioned) with primary interface in public net and
    provisioning interface in customer management subnet
  • System OS: Debian 8.2, all software lates stable versions
  • Provisioned OS: Debian 8 with preseed default PT, Debian Mirror and
    Preseed default/finish/PXELinux (unchanged, for now)
  • Static IP and unmananged interfaces on the host to be provisioned, no IPAM

ENC, DHCP, default PXE template upload, TFTP/PXE in itself and Discovery
work as expected.
The smart proxy is registered in foreman, the provisioning interface of the
new host is bound to the management subnet and the TFTP assigned in the
subnet record.
The foreman debug log shows no obvious errors, but also no entry regarding
the smart proxy. Besides the SQL-queries, all logentries deal with local
template rendering.
The smart proxy debug.log only shows the feature GET request by foreman
when building, but no further communication afterwards.

I double checked permissions on the TFTP directory structure and correct
entries in the settings file, according to the guides in setting up
provisioning.
I also tried it without discovery in a second snapshot. Additionaly, I set
up a host in our own management subnet and tried with the TFTP running on
the foreman server, also to no avail.
Also, no pxelinux.cfg entries are generated.

Maybe someone has an idea, what I am missing, I just can't get
foreman/proxy to deploy the PXE config for the new hosts.

··· -- Marcel
#5

Thanks Jelie. Glad to hear you sorted it out. Any chance you can file
redmine tickets, for the issues that you think we should fix?
http://theforeman.org/projects/foreman/issues

··· On Fri, Jan 4, 2013 at 3:06 AM, Jelle B. wrote:

Ok what it was, in the subnet configuration in foreman you have to enter ,
via some pull down menus, the tftp proxy , dns and dhcp (where applicable)
without this foreman will silently not try to make any PXE configs for you.
The logical reason being that it doesn’t know where to put it , and if you
have ,like me, multiple subnets configured it is even more reasonable.
However it is silently not doing anything and that was kinda not so good :wink:
Anyways I know it will change :slight_smile:

Also while on the subject be aware that de fault unattended scripts
install puppet 2.6.x files and as most will be setting up on puppet 3 by
now this causes some weird errors on debian , just upgrade the puppet
client and all will be good :slight_smile:

#6

> Maybe someone has an idea, what I am missing, I just can't get
> foreman/proxy to deploy the PXE config for the new hosts.

Please run foreman-tail on both server and proxy and do the
provisioning, then pastebin both outputs so we can see the flow there.

··· -- Later, Lukas #lzap Zapletal
#7

Thanks for your reply. Here is what I got from the logs:

Proxy:
http://pastebin.com/yNU5ZuuD

Server:
http://pastebin.com/TWaFSLqy

··· --

BTW, I had to edit the foreman-tail script to get it running:

[…]
if [ -z “$1” ]; then
tail $opts $(eval echo $(cat $dir//)) 2>/dev/null
else
[…]

it seems to look for debian by /etc/debian_release while building, which is
(since Debian 7?) /etc/debian_version

–
Marcel

#8

> Proxy:
> http://pastebin.com/yNU5ZuuD
>
> Server:
> http://pastebin.com/TWaFSLqy

There is no provisioning communication indeed. Have you turned your
server on? :wink: Is it physically connected to the network? You should see
DHCP request on the DHCP server, then proceeding with TFTP download of
the PXELinux configuration.

How have you installed the DHCP service? With our installer?

··· -- Later, Lukas #lzap Zapletal
#9

>
> There is no provisioning communication indeed. Have you turned your
> server on? :wink: Is it physically connected to the network?

Well, I just went checking again, I asked myself that exact same question a
lot during the last days :))

> You should see DHCP request on the DHCP server, then proceeding with TFTP
> download of
> the PXELinux configuration.
>

The DHCP and TFTP services were installed manually before integration from
the debian repositories and worked as standalone solution without Foreman.
But that one gave me some idea and hopefully, I was able to get it to work
now.

I think, I made some grave conceptual mistakes from the beginning:

  • Initially, I set up the DHCP service only for PXE on a very small range
    at the end of the subnet (10.5.31.200-10.5.31.250)
  • I then set the provisioning interface of the new host to "unmanaged",
    "provisioning", fixed ip (10.5.0.81) and static boot in subnet
  • I further assumed, as in the common provisioning tutorials, there should
    be a pxelinux.cfg based on the given mac address, which didn't work out as
    expected, the host never got a custom PXE bootconfig

As I set the interface to managed and gave the DHCP server control over the
whole subnet (or at least the complete range, that I want to deploy to) it
worked.
To be honest, I'm still not completely sure, if it was the "managed" flag
or the extended DHCP range or both that finally led to success.

··· -- Marcel
#10

Mosty likely it's this - if the Provisioning interface is not Managed,
then Foreman won't try and write the "01-mac" file to pxelinux.cfg.

Glad you got it working!

Greg

··· On 6 October 2015 at 16:20, Marcel Vapore wrote: > To be honest, I'm still not completely sure, if it was the "managed" flag > or the extended DHCP range or both that finally led to success.
#11

> Mosty likely it's this - if the Provisioning interface is not Managed,
> then Foreman won't try and write the "01-mac" file to pxelinux.cfg.

Shouldn't we insist on both?

··· -- Later, Lukas #lzap Zapletal
#12

>
> Shouldn't we insist on both?
>

I think we should :slight_smile:

Personally, I'd recommend, at least every beginner at foreman and
unattended installation, setting up a fully managed deployment subnet.
That would have saved lots of time, coffee and headache. Suddenly, the
whole unattended installation works like a charm, from discovery to ssh
login and first puppet runs.

··· -- Marcel
#13

Not sure what two things you're referring to. I will admit my English
was less than perfect there, it should probably read "Foreman won't
try to write", which may be the confusion?

Greg

··· On 7 October 2015 at 08:58, Lukas Zapletal wrote: >> Mosty likely it's this - if the Provisioning interface is not Managed, >> then Foreman won't try and write the "01-mac" file to pxelinux.cfg. > > Shouldn't we insist on both?
#14

Unmanaged interfaces have value, we can't insist that every interface
is managed.

You do raise a good point about beginner setups. I've got plans for a
Beginner's Guide to Foreman (as a Youtube seres) which people can play
along with. I'll be starting to record these later this month.
Hopefully that will resolve a lot of these issues.

Greg

··· On 7 October 2015 at 12:05, Marcel Vapore wrote: > I think we should :) > > Personally, I'd recommend, at least every beginner at foreman and unattended > installation, setting up a fully managed deployment subnet. > That would have saved lots of time, coffee and headache. Suddenly, the whole > unattended installation works like a charm, from discovery to ssh login and > first puppet runs.
#15

>
> You do raise a good point about beginner setups. I've got plans for a
> Beginner's Guide to Foreman (as a Youtube seres) which people can play
> along with. I'll be starting to record these later this month.
> Hopefully that will resolve a lot of these issues.
>
> That sounds nice, available information to start with is a little bit
cluttered and sometimes even contradictory.
If I can be of any help regarding topics, that troubled me most at the
beginning, feel free to contact me :slight_smile:

··· -- Marcel
#16

Thanks! I currently have plans for at least 4 episodes (initial setup
/ installer, provisioning, puppet, and virtualization) but I'll
definitely be looking for more content as we go forwards. Anything you
feel isn't covered (or not covered enough) will be welcome!

··· On 8 October 2015 at 15:21, Marcel Vapore wrote: >> You do raise a good point about beginner setups. I've got plans for a >> Beginner's Guide to Foreman (as a Youtube seres) which people can play >> along with. I'll be starting to record these later this month. >> Hopefully that will resolve a lot of these issues. >> > That sounds nice, available information to start with is a little bit > cluttered and sometimes even contradictory. > If I can be of any help regarding topics, that troubled me most at the > beginning, feel free to contact me :)