Smart-proxy template setup problem

I've been trying to setup a smart-proxy template server in a "remote" pop
(not in same location as foreman prime server) as suggested by a kind
gentleman in this thread
https://groups.google.com/forum/#!topic/foreman-users/2KuJ6l2KiOo

Unfortunately I've not been able to get it to work. Here is a brief
description.

pop1

  • ESXi host1
    -forman prime (also alpha-puppet master)

pop2

  • ESXi host2
    -foreman-proxy (dhcp,tftp, templates) (puppet agent of alpha-puppet
    master) interface with ip 10.0.10.10
    -testbuild host (simple host that will when built be an agent of foreman
    proxy listed directly above. provisioning ip 10.0.10.19

I've configured all of this and the initial build of testbuild get's to the
point it needs templates (gets past the tftpboot) but fails with the
following error from http logs:
==> httpd/access_log <==
10.0.10.19 - - [05/Oct/2016:15:19:05 +0000] "GET
/unattended/provision?token=290e1cce-21d6-43a2-97c9-47a2d114d9b2 HTTP/1.1"
404 218 "-" "curl/7.29.0"

templates.yml config is:
:enabled: true
:template_url: http:/10.0.10.10:8000

I did add this to settings.yml in /etc/foreman-proxy/

:http_port: 8000

I restarted the foreman-proxy and httpd on the proxy site. I refreshed the
smart-proxy listing of the foreman-proxy on the foreman prime server and it
then listed "Templates" (as well as dhcp/tftp).

I tested that ports responded and they did. To be thorough I disabled
iptables/firewalld for the duration of working on this piece.

Is there some some setting or some feature I'm missing here? Should the
template_url be the foreman-prime server or the proxy? I've tried both
(also configured the http_port 8000 on the foreman-prime for that test) but
neither worked for what it's worth.

Is there a log that would give me more information? I'm currently at
"debug" on the foreman prime and the foreman-proxy. I've tweaked the
configuration in various ways trying to figure out the cause but
unfortunately nothing has resolved the problem.

> Is there some some setting or some feature I'm missing here? Should
> the template_url be the foreman-prime server or the proxy? I've tried
> both (also configured the http_port 8000 on the foreman-prime for
> that test) but neither worked for what it's worth.

It needs to be either https://foreman:443 or https://proxy:8443 (assumi
ng you haven't changed default ports)

Also one thing to check is "Token Duration" setting in the Administer
section - make sure you give your server enough minutes to finish
provisioning, otherwise the "built" call will return 404 as the token
expires.

··· -- Later, Lukas "lzap" Zapletal

Thank you for the tips Lukas. I tried a few options you mentioned (443 or
8443) but neither seemed to work.

Currently I have "/etc/foreman-proxy/settings.d/templates.yml" set to.
:enabled: true
:template_url: http://10.0.10.10:8000

And the unattended build fails with (error from 10.0.10.10 apache log):
==> httpd/access_log <==
10.0.10.19 - - [12/Oct/2016:14:27:48 +0000] "GET
/unattended/provision?token=9c9a1410-dada-4c82-b824-72d183d3ecde HTTP/1.1"
404 218 "-" "curl/7.29.0"

However, if I telnet to 10.0.10.10 8000 and issue GET of the unattended URL
(from 10.0.10.10):

telnet 10.0.10.10 8000
Trying 10.0.10.10…
Connected to 10.0.10.10.
Escape character is '^]'.
GET /unattended/provision?token=9c9a1410-dada-4c82-b824-72d183d3ecde

install
url --url http://10.0.10.10/media/mirror.centos.org/7/os/x86_64
lang en_US.UTF-8
selinux --enforcing
keyboard us
etc…

For reference during that the log showed:
==> foreman-proxy/proxy.log <==
D, [2016-10-12T14:20:23.017328 #30738] DEBUG – : accept: 10.0.10.10:60134
D, [2016-10-12T14:20:26.849604 #30738] DEBUG – : Rack::Handler::WEBrick is
invoked.
D, [2016-10-12T14:20:26.850923 #30738] DEBUG – : Retrieving a template
from
http://10.0.10.10:8000//unattended/provision?token=9c9a1410-dada-4c82-b824-72d183d3ecde&url=http%3A%2F%2F10.0.10.10%3A8000
D, [2016-10-12T14:20:26.850994 #30738] DEBUG – : HTTP headers:
{"X-Forwarded-For"=>"10.0.10.10, 10.0.10.10"}
D, [2016-10-12T14:20:30.488181 #30738] DEBUG – : Template: request for
provision using {"token"=>"9c9a1410-dada-4c82-b824-72d183d3ecde",
:url=>"http://10.0.10.10:8000"} at george-dev.iss.gin.ntt.net
I, [2016-10-12T14:20:30.488661 #30738] INFO – : 10.0.10.10 - -
[12/Oct/2016:14:20:30 +0000] "GET
/unattended/provision?token=9c9a1410-dada-4c82-b824-72d183d3ecde HTTP/1.1"
200 5171 3.6386

D, [2016-10-12T14:20:30.488921 #30738] DEBUG – : close: 10.0.10.10:60134

Why would the build host not be able to get the template but I can manually
run the URL and it work?

PS: I will keep an eye on the token expiration. The Foreman page for the
host highlights that in red if it's expired.

··· On Tuesday, October 11, 2016 at 2:47:43 AM UTC-5, Lukas Zapletal wrote: > > > > Is there some some setting or some feature I'm missing here? Should > > the template_url be the foreman-prime server or the proxy? I've tried > > both (also configured the http_port 8000 on the foreman-prime for > > that test) but neither worked for what it's worth. > > It needs to be either https://foreman:443 or https://proxy:8443 (assumi > ng you haven't changed default ports) >

> Why would the build host not be able to get the template but I can manually
> run the URL and it work?
>
> PS: I will keep an eye on the token expiration. The Foreman page for the
> host highlights that in red if it's expired.

Check if TFTP is proxy associated with the subnet, otherwise templating
does not work for that interface and it falls back to non-proxied one.

··· -- Later, Lukas #lzap Zapletal

I believe it does have that set right.


That is the subnet 10.0.10.0/24. "overseer" is the smart-proxy for
tftp/dhcp/template . The boot images (pxe/*.img file) DO get loaded from
tftp if that helps any in debugging this. The kickstart is where it fails.

Here is the last few lines from the console of the host trying to build.

··· On Wednesday, October 12, 2016 at 10:27:25 AM UTC-5, Lukas Zapletal wrote: > > > Why would the build host not be able to get the template but I can > manually > > run the URL and it work? > > > > PS: I will keep an eye on the token expiration. The Foreman page for the > > host highlights that in red if it's expired. > > Check if TFTP is proxy associated with the subnet, otherwise templating > does not work for that interface and it falls back to non-proxied one. > > -- > Later, > Lukas #lzap Zapletal >

>
> >
> > It's interesting that you can get it via telnet - that implies the proxy
> > side is OK. If you tail the Foreman logs, do you see anything there when
> > the host requests kickstart? You may need to enable debug logging, just
> to
> > be on the safe side, but it should log the connection and give some
> info.
> > If it doesn't, then the proxy isn't forwarding the request, and we know
> to
> > focus on that.
> >
> > Greg
> >
>

Unfortunately I get nothing in the foreman-proxy logs (I do have debug
enabled). The httpd log is the only log for the command I've found so far.
The only time I see anything there (foreman-proxy logs) is when I manually
issue the command.

Here is the log from when I canceled the build from the foreman web
interface. Then issued the "build" command issued from start till the httpd
log entry.

D, [2016-10-12T16:18:11.619257 #31062] DEBUG – : accept: [FILTERED
IP]:37181
D, [2016-10-12T16:18:11.698010 #31062] DEBUG – : Rack::Handler::WEBrick is
invoked.
I, [2016-10-12T16:18:11.699030 #31062] INFO – : [FILTERED IP] - -
[12/Oct/2016:16:18:11 +0000] "GET /features HTTP/1.1" 200 36 0.0005

D, [2016-10-12T16:18:11.817480 #31062] DEBUG – : close: [FILTERED IP]:37181
D, [2016-10-12T16:18:19.913639 #31062] DEBUG – : accept: [FILTERED
IP]:37185
D, [2016-10-12T16:18:19.995816 #31062] DEBUG – : Rack::Handler::WEBrick is
invoked.
D, [2016-10-12T16:18:19.996772 #31062] DEBUG – : verifying remote client
[FILTERED IP] against trusted_hosts ["george-dev.[FILTERED HOSTNAME]"]
D, [2016-10-12T16:18:19.997227 #31062] DEBUG – : TFTP: entry for
00:50:56:84:f6:60 created successfully
I, [2016-10-12T16:18:19.997520 #31062] INFO – : [FILTERED IP] - -
[12/Oct/2016:16:18:19 +0000] "POST /tftp/syslinux/00:50:56:84:f6:60
HTTP/1.1" 200 - 0.0010

D, [2016-10-12T16:18:20.037692 #31062] DEBUG – : close: [FILTERED IP]:37185
D, [2016-10-12T16:18:23.182764 #31062] DEBUG – : accept: [FILTERED
IP]:37187
D, [2016-10-12T16:18:23.265975 #31062] DEBUG – : Rack::Handler::WEBrick is
invoked.
D, [2016-10-12T16:18:23.266754 #31062] DEBUG – : verifying remote client
[FILTERED IP] against trusted_hosts ["george-dev.[FILTERED HOSTNAME]"]
D, [2016-10-12T16:18:23.267331 #31062] DEBUG – : Starting task:
/usr/bin/wget --timeout=10 --tries=3 --no-check-certificate -nv -c
"http://10.0.10.10/media/mirror.centos.org/7/os/x86_64/images/pxeboot/vmlinuz&quot;
-O "/var/lib/tftpboot/boot/CentOS-7-x86_64-vmlinuz"
I, [2016-10-12T16:18:23.270902 #31062] INFO – : [FILTERED IP] - -
[12/Oct/2016:16:18:23 +0000] "POST /tftp/fetch_boot_file HTTP/1.1" 200 -
0.0043

==> httpd/access_log <==
10.0.10.10 - - [12/Oct/2016:16:18:23 +0000] "GET
/media/mirror.centos.org/7/os/x86_64/images/pxeboot/vmlinuz HTTP/1.1" 416
314 "-" "Wget/1.14 (linux-gnu)"

==> foreman-proxy/proxy.log <==
D, [2016-10-12T16:18:23.312618 #31062] DEBUG – : close: [FILTERED IP]:37187
D, [2016-10-12T16:18:26.439044 #31062] DEBUG – : accept: [FILTERED
IP]:37189
D, [2016-10-12T16:18:26.516534 #31062] DEBUG – : Rack::Handler::WEBrick is
invoked.
D, [2016-10-12T16:18:26.517556 #31062] DEBUG – : verifying remote client
[FILTERED IP] against trusted_hosts ["george-dev.[FILTERED HOSTNAME]"]
D, [2016-10-12T16:18:26.518106 #31062] DEBUG – : Starting task:
/usr/bin/wget --timeout=10 --tries=3 --no-check-certificate -nv -c
"http://10.0.10.10/media/mirror.centos.org/7/os/x86_64/images/pxeboot/initrd.img&quot;
-O "/var/lib/tftpboot/boot/CentOS-7-x86_64-initrd.img"
I, [2016-10-12T16:18:26.520773 #31062] INFO – : [FILTERED IP] - -
[12/Oct/2016:16:18:26 +0000] "POST /tftp/fetch_boot_file HTTP/1.1" 200 -
0.0035

==> httpd/access_log <==
10.0.10.10 - - [12/Oct/2016:16:18:26 +0000] "GET
/media/mirror.centos.org/7/os/x86_64/images/pxeboot/initrd.img HTTP/1.1"
416 314 "-" "Wget/1.14 (linux-gnu)"

==> foreman-proxy/proxy.log <==
D, [2016-10-12T16:18:26.559492 #31062] DEBUG – : close: [FILTERED IP]:37189

==> httpd/access_log <==
10.0.10.21 - - [12/Oct/2016:16:18:47 +0000] "GET
/unattended/provision?token=a7eed16e-ebdc-4676-833c-b0ee1a690627 HTTP/1.1"
404 218 "-" "curl/7.29.0"

So I just noticed that the IP listed there for the host "10.0.10.21"
doesn't match the ip address I configured for the provisining interface.


It should be 10.0.10.18?

Going to poke around.

··· > On Wednesday, October 12, 2016 at 11:00:17 AM UTC-5, Greg Sutcliffe wrote:

Okay, so it turns out that I had the wrong vSwitch selected… however.

When looking at the compute profile for that device in THAT cluster, it's
listing interface names from ANOTHER cluster/resource, not just the
interfaces in the cluster I selected for that profile.


I renamed the interface on that esxi device from "internal" to
"internal-mlpsca01" and then realized it was trying to use an interface
from another vhost in an entirely different cluster.

Would it be possible to make it so that when you set these variables:


It will only list the vswitch names from those in the drop downs?

Altering the network selection did allow me to get the host build process
rolling! Thanks for the help very much!

It's interesting that you can get it via telnet - that implies the proxy
side is OK. If you tail the Foreman logs, do you see anything there when
the host requests kickstart? You may need to enable debug logging, just to
be on the safe side, but it should log the connection and give some info.
If it doesn't, then the proxy isn't forwarding the request, and we know to
focus on that.

Greg