Foreman/Windows Provisioning/NATd Corp Environment

Hi Guys,

We are trying to provision windows hosts in a corporate environment with
many geographically dispersed networks. To date we deploy many flavours of
linux and hypervisors using smart proxies in each location. Using tokens,
we are able to provision in many subnets with ease. This has been working
for over a year now.

We updated our lab foreman instance and smart proxies to 1.9.2 earlier this
week and are very happy with the new version so thanks goes to the devs!

Environment:
Centos 6.7
Foreman 1…9.8

Soo fast forward to today we are tackling Windows provisioning again and
are getting a little stuck. We need to PXE boot an ISO basically rather
than utilise templates as we deploy to many differing HW/hypervisor
platforms. The issue we are currently having is we cannot find a way of
passing anything via the PXE environment in order to curl the unattended
provision url with a token (similar to the append KS directive). We have
windows provisioning working in environments where access directly to
Foreman is available (as it relies on source IP) but NATd/Proxied
environments aren't playing ball. All examples of provisioning windows I've
seen so far appear to be template based for this reason.

Floundering as we are, the best we can come up with is trying to modify the
request headers, we've proxied the http call for the provision script via
nginx adding X-Forwarded-For headers and confirmed they were coming through
via TCPDUMP

…,/GET /unattended/provision HTTP/1.0
X-Real-IP: 10.12.2.237
X-Forwarded-For: 10.12.2.237 <- This is the IP
of the server we wish to provision
Host: 10.12.2.110 <- This is
the foreman proxy in front of our foreman server (in the lab they are on
the same subnet and we could have gone directly but we are trying to
provision via the proxy to replicate our other envs)

unfortunately we see this reply

…HTTP/1.1 405 Method Not Allowed
Server: Apache/2.2.15 (CentOS)
X-Frame-Options: SAMEORIGIN
Content-Security-Policy: default-src 'self'; connect-src 'self' ws: wss:;
font-src 'self'; frame-src 'self'; img-src 'self' *.gravatar.com data:;
media-src 'self'; object-src 'self'; script-src 'unsafe-eval'
'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self';
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-UA-Compatible: IE=Edge,chrome=1
Cache-Control: no-cache
X-Request-Id: 218dc66c8ee21c2d16ee885ddc5f05ef
X-Runtime: 0.020545
X-Rack-Cache: miss
X-Powered-By: Phusion Passenger 4.0.18
Set-Cookie: request_method=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Status: 405 Method Not Allowed
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=utf-8

when we looked in foreman production log we could see the same kind of
output (note the for 10.12.2.110 is the foreman proxy NOT the machi)

> Started GET "/unattended/provision" for 10.12.2.110 at 2015-09-25
11:04:50 +1000
2015-09-25 11:04:50 [app] [I] Processing by UnattendedController#provision
as HTML
2015-09-25 11:04:50 [app] [I] Found frmnprxy01.ourdomain
2015-09-25 11:04:50 [app] [I] Filter chain halted as :allowed_to_install?
rendered or redirected
2015-09-25 11:04:50 [app] [I] Completed 405 Method Not Allowed in 14ms
(ActiveRecord: 1.6ms)
2015-09-25 11:05:11 [app] [I] Connecting to database specified by
database.yml
2015-09-25 11:05:14 [app] [I] Finished registering 1 hooks for
Host::Managed#destroy
2015-09-25 11:05:22 [sql] [W] Creating scope :completer_scope. Overwriting
existing method Organization.completer_scope.
2015-09-25 11:05:22 [sql] [W] Creating scope :completer_scope. Overwriting
existing method Location.completer_scope.
2015-09-25 11:05:38 [foreman-tasks/dynflow] [I] start terminating client
dispatcher…
2015-09-25 11:05:38 [foreman-tasks/dynflow] [I] stop listening for new
events…
2015-09-25 11:05:38 [foreman-tasks/dynflow] [I] start terminating clock…
2015-09-25 11:06:24 [app] [I]

So it looks like the X-Forwarded-For header aren't being honoured. Now this
may be because it is a security risk perhaps, either way there must be some
means of proxying this.

When we looked in
/usr/share/foreman/app/controllers/unattended_controller.rb it certainly
looked like this should have worked ;(

This method updates the IP held by Foreman from the incoming request.

Useful on unmanaged DHCP systems, with token-based installs where

Foreman

doesn't know the IP in advance (and has been given a fake one just to

make

the form save)

def update_ip
ip = ip_from_request_env
logger.debug "Built notice from #{ip}, current host ip is #{@host.ip},
updating" if @host.ip != ip

# @host has been changed even if the save fails, so we have to change 

it back
old_ip = @host.ip
@host.ip = old_ip unless @host.update_attributes({'ip' => ip})
end

def ip_from_request_env
ip = request.env['REMOTE_ADDR']

# check if someone is asking on behalf of another system (load balance 

etc)
if request.env['HTTP_X_FORWARDED_FOR'].present? and (ip =~
Regexp.new(Setting[:remote_addr]))
ip = request.env['HTTP_X_FORWARDED_FOR']
end

ip

end

If anyone has any ideas I would be super keen to hear them. I'm sure we
have made a stupid assumption somewhere or overlooked something so I'm all
ears at this stage.

Lastly, a feature request, it would nice if there were a new role added to
the smart-proxies which performed this action by default, using the api and
SSL certs. (We looked into writing a small service on the proxy to do this
but couldnt find an endpoint suitable for retrieving a provision script.)
Also another nice feature to add would be the same role being responsible
for sync'ing /var/li/tftp from foreman to all the proxies rather than
having to do this manually.

Thanks
Matt

··· Date: Fri, 25 Sep 2015 01:04:50 GMT

Matthew I remember having the same problem a few weeks ago, but can't be
sure until were I got.
I remember the ip =~ Regexp.new(Setting[:remote_addr]) being part of the
problem.
In the this variable is located in Settings -> Provisioning -> remote_addr.
Just checked in my Foreman instance in our lab and found the regexp there
for the vlans we use for provisioning:

(192.168.\d.\d{1,3}|10.0.7.\d{1,3})

Hope this helps.
Danilo

··· On Thursday, 24 September 2015 22:39:39 UTC-3, Matthew Wilmott wrote: > > Hi Guys, > > We are trying to provision windows hosts in a corporate environment with > many geographically dispersed networks. To date we deploy many flavours of > linux and hypervisors using smart proxies in each location. Using tokens, > we are able to provision in many subnets with ease. This has been working > for over a year now. > > We updated our lab foreman instance and smart proxies to 1.9.2 earlier > this week and are very happy with the new version so thanks goes to the > devs! > > Environment: > Centos 6.7 > Foreman 1..9.8 > > Soo fast forward to today we are tackling Windows provisioning again and > are getting a little stuck. We need to PXE boot an ISO basically rather > than utilise templates as we deploy to many differing HW/hypervisor > platforms. The issue we are currently having is we cannot find a way of > passing anything via the PXE environment in order to curl the unattended > provision url with a token (similar to the append KS directive). We have > windows provisioning working in environments where access directly to > Foreman is available (as it relies on source IP) but NATd/Proxied > environments aren't playing ball. All examples of provisioning windows I've > seen so far appear to be template based for this reason. > > Floundering as we are, the best we can come up with is trying to modify > the request headers, we've proxied the http call for the provision script > via nginx adding X-Forwarded-For headers and confirmed they were coming > through via TCPDUMP > > .....,/GET /unattended/provision HTTP/1.0 > X-Real-IP: 10.12.2.237 > X-Forwarded-For: 10.12.2.237 <- This is the IP > of the server we wish to provision > Host: 10.12.2.110 <- This is > the foreman proxy in front of our foreman server (in the lab they are on > the same subnet and we could have gone directly but we are trying to > provision via the proxy to replicate our other envs) > > unfortunately we see this reply > > ...HTTP/1.1 405 Method Not Allowed > Date: Fri, 25 Sep 2015 01:04:50 GMT > Server: Apache/2.2.15 (CentOS) > X-Frame-Options: SAMEORIGIN > Content-Security-Policy: default-src 'self'; connect-src 'self' ws: wss:; > font-src 'self'; frame-src 'self'; img-src 'self' *.gravatar.com data:; > media-src 'self'; object-src 'self'; script-src 'unsafe-eval' > 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'; > X-XSS-Protection: 1; mode=block > X-Content-Type-Options: nosniff > X-Download-Options: noopen > X-UA-Compatible: IE=Edge,chrome=1 > Cache-Control: no-cache > X-Request-Id: 218dc66c8ee21c2d16ee885ddc5f05ef > X-Runtime: 0.020545 > X-Rack-Cache: miss > X-Powered-By: Phusion Passenger 4.0.18 > Set-Cookie: request_method=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT > Status: 405 Method Not Allowed > Vary: Accept-Encoding > Connection: close > Content-Type: text/html; charset=utf-8 > > when we looked in foreman production log we could see the same kind of > output (note the for 10.12.2.110 is the foreman proxy NOT the machi) > > > Started GET "/unattended/provision" for 10.12.2.110 at 2015-09-25 > 11:04:50 +1000 > 2015-09-25 11:04:50 [app] [I] Processing by UnattendedController#provision > as HTML > 2015-09-25 11:04:50 [app] [I] Found frmnprxy01.ourdomain > 2015-09-25 11:04:50 [app] [I] Filter chain halted as :allowed_to_install? > rendered or redirected > 2015-09-25 11:04:50 [app] [I] Completed 405 Method Not Allowed in 14ms > (ActiveRecord: 1.6ms) > 2015-09-25 11:05:11 [app] [I] Connecting to database specified by > database.yml > 2015-09-25 11:05:14 [app] [I] Finished registering 1 hooks for > Host::Managed#destroy > 2015-09-25 11:05:22 [sql] [W] Creating scope :completer_scope. Overwriting > existing method Organization.completer_scope. > 2015-09-25 11:05:22 [sql] [W] Creating scope :completer_scope. Overwriting > existing method Location.completer_scope. > 2015-09-25 11:05:38 [foreman-tasks/dynflow] [I] start terminating client > dispatcher... > 2015-09-25 11:05:38 [foreman-tasks/dynflow] [I] stop listening for new > events... > 2015-09-25 11:05:38 [foreman-tasks/dynflow] [I] start terminating clock... > 2015-09-25 11:06:24 [app] [I] > > So it looks like the X-Forwarded-For header aren't being honoured. Now > this may be because it is a security risk perhaps, either way there must be > some means of proxying this. > > When we looked in > /usr/share/foreman/app/controllers/unattended_controller.rb it certainly > looked like this should have worked ;( > > # This method updates the IP held by Foreman from the incoming request. > # Useful on unmanaged DHCP systems, with token-based installs where > Foreman > # doesn't know the IP in advance (and has been given a fake one just to > make > # the form save) > def update_ip > ip = ip_from_request_env > logger.debug "Built notice from #{ip}, current host ip is #{@host.ip}, > updating" if @host.ip != ip > > # @host has been changed even if the save fails, so we have to change > it back > old_ip = @host.ip > @host.ip = old_ip unless @host.update_attributes({'ip' => ip}) > end > > def ip_from_request_env > ip = request.env['REMOTE_ADDR'] > > # check if someone is asking on behalf of another system (load balance > etc) > if request.env['HTTP_X_FORWARDED_FOR'].present? and (ip =~ > Regexp.new(Setting[:remote_addr])) > ip = request.env['HTTP_X_FORWARDED_FOR'] > end > > ip > end > > If anyone has any ideas I would be super keen to hear them. I'm sure we > have made a stupid assumption somewhere or overlooked something so I'm all > ears at this stage. > > Lastly, a feature request, it would nice if there were a new role added to > the smart-proxies which performed this action by default, using the api and > SSL certs. (We looked into writing a small service on the proxy to do this > but couldnt find an endpoint suitable for retrieving a provision script.) > Also another nice feature to add would be the same role being responsible > for sync'ing /var/li/tftp from foreman to all the proxies rather than > having to do this manually. > > Thanks > Matt > >

OMG thank you thank you thank you, you can't imagine how happy you've made
us lol

That worked perfectly! I found the same line and was looking through the
config files themselves, never thought to look in foreman UI itself, doh!

Matt

··· On Friday, September 25, 2015 at 11:22:39 PM UTC+10, Danilo Sousa wrote: > > Matthew I remember having the same problem a few weeks ago, but can't be > sure until were I got. > I remember the `ip =~ Regexp.new(Setting[:remote_addr])` being part of the > problem. > In the this variable is located in Settings -> Provisioning -> remote_addr. > Just checked in my Foreman instance in our lab and found the regexp there > for the vlans we use for provisioning: > > `(192.168.\d.\d{1,3}|10.0.7.\d{1,3})` > > Hope this helps. > Danilo > > On Thursday, 24 September 2015 22:39:39 UTC-3, Matthew Wilmott wrote: >> >> Hi Guys, >> >> We are trying to provision windows hosts in a corporate environment with >> many geographically dispersed networks. To date we deploy many flavours of >> linux and hypervisors using smart proxies in each location. Using tokens, >> we are able to provision in many subnets with ease. This has been working >> for over a year now. >> >> We updated our lab foreman instance and smart proxies to 1.9.2 earlier >> this week and are very happy with the new version so thanks goes to the >> devs! >> >> Environment: >> Centos 6.7 >> Foreman 1..9.8 >> >> Soo fast forward to today we are tackling Windows provisioning again and >> are getting a little stuck. We need to PXE boot an ISO basically rather >> than utilise templates as we deploy to many differing HW/hypervisor >> platforms. The issue we are currently having is we cannot find a way of >> passing anything via the PXE environment in order to curl the unattended >> provision url with a token (similar to the append KS directive). We have >> windows provisioning working in environments where access directly to >> Foreman is available (as it relies on source IP) but NATd/Proxied >> environments aren't playing ball. All examples of provisioning windows I've >> seen so far appear to be template based for this reason. >> >> Floundering as we are, the best we can come up with is trying to modify >> the request headers, we've proxied the http call for the provision script >> via nginx adding X-Forwarded-For headers and confirmed they were coming >> through via TCPDUMP >> >> .....,/GET /unattended/provision HTTP/1.0 >> X-Real-IP: 10.12.2.237 >> X-Forwarded-For: 10.12.2.237 <- This is the IP >> of the server we wish to provision >> Host: 10.12.2.110 <- This is >> the foreman proxy in front of our foreman server (in the lab they are on >> the same subnet and we could have gone directly but we are trying to >> provision via the proxy to replicate our other envs) >> >> unfortunately we see this reply >> >> ...HTTP/1.1 405 Method Not Allowed >> Date: Fri, 25 Sep 2015 01:04:50 GMT >> Server: Apache/2.2.15 (CentOS) >> X-Frame-Options: SAMEORIGIN >> Content-Security-Policy: default-src 'self'; connect-src 'self' ws: wss:; >> font-src 'self'; frame-src 'self'; img-src 'self' *.gravatar.com data:; >> media-src 'self'; object-src 'self'; script-src 'unsafe-eval' >> 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'; >> X-XSS-Protection: 1; mode=block >> X-Content-Type-Options: nosniff >> X-Download-Options: noopen >> X-UA-Compatible: IE=Edge,chrome=1 >> Cache-Control: no-cache >> X-Request-Id: 218dc66c8ee21c2d16ee885ddc5f05ef >> X-Runtime: 0.020545 >> X-Rack-Cache: miss >> X-Powered-By: Phusion Passenger 4.0.18 >> Set-Cookie: request_method=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT >> Status: 405 Method Not Allowed >> Vary: Accept-Encoding >> Connection: close >> Content-Type: text/html; charset=utf-8 >> >> when we looked in foreman production log we could see the same kind of >> output (note the for 10.12.2.110 is the foreman proxy NOT the machi) >> >> > Started GET "/unattended/provision" for 10.12.2.110 at 2015-09-25 >> 11:04:50 +1000 >> 2015-09-25 11:04:50 [app] [I] Processing by >> UnattendedController#provision as HTML >> 2015-09-25 11:04:50 [app] [I] Found frmnprxy01.ourdomain >> 2015-09-25 11:04:50 [app] [I] Filter chain halted as :allowed_to_install? >> rendered or redirected >> 2015-09-25 11:04:50 [app] [I] Completed 405 Method Not Allowed in 14ms >> (ActiveRecord: 1.6ms) >> 2015-09-25 11:05:11 [app] [I] Connecting to database specified by >> database.yml >> 2015-09-25 11:05:14 [app] [I] Finished registering 1 hooks for >> Host::Managed#destroy >> 2015-09-25 11:05:22 [sql] [W] Creating scope :completer_scope. >> Overwriting existing method Organization.completer_scope. >> 2015-09-25 11:05:22 [sql] [W] Creating scope :completer_scope. >> Overwriting existing method Location.completer_scope. >> 2015-09-25 11:05:38 [foreman-tasks/dynflow] [I] start terminating client >> dispatcher... >> 2015-09-25 11:05:38 [foreman-tasks/dynflow] [I] stop listening for new >> events... >> 2015-09-25 11:05:38 [foreman-tasks/dynflow] [I] start terminating clock... >> 2015-09-25 11:06:24 [app] [I] >> >> So it looks like the X-Forwarded-For header aren't being honoured. Now >> this may be because it is a security risk perhaps, either way there must be >> some means of proxying this. >> >> When we looked in >> /usr/share/foreman/app/controllers/unattended_controller.rb it certainly >> looked like this should have worked ;( >> >> # This method updates the IP held by Foreman from the incoming request. >> # Useful on unmanaged DHCP systems, with token-based installs where >> Foreman >> # doesn't know the IP in advance (and has been given a fake one just to >> make >> # the form save) >> def update_ip >> ip = ip_from_request_env >> logger.debug "Built notice from #{ip}, current host ip is >> #{@host.ip}, updating" if @host.ip != ip >> >> # @host has been changed even if the save fails, so we have to change >> it back >> old_ip = @host.ip >> @host.ip = old_ip unless @host.update_attributes({'ip' => ip}) >> end >> >> def ip_from_request_env >> ip = request.env['REMOTE_ADDR'] >> >> # check if someone is asking on behalf of another system (load >> balance etc) >> if request.env['HTTP_X_FORWARDED_FOR'].present? and (ip =~ >> Regexp.new(Setting[:remote_addr])) >> ip = request.env['HTTP_X_FORWARDED_FOR'] >> end >> >> ip >> end >> >> If anyone has any ideas I would be super keen to hear them. I'm sure we >> have made a stupid assumption somewhere or overlooked something so I'm all >> ears at this stage. >> >> Lastly, a feature request, it would nice if there were a new role added >> to the smart-proxies which performed this action by default, using the api >> and SSL certs. (We looked into writing a small service on the proxy to do >> this but couldnt find an endpoint suitable for retrieving a provision >> script.) Also another nice feature to add would be the same role being >> responsible for sync'ing /var/li/tftp from foreman to all the proxies >> rather than having to do this manually. >> >> Thanks >> Matt >> >>