Discovery: always go via proxy

UXabre · October 23, 2018, 7:55pm

problem
I love the MaaS functionality, being able to discover “physical” servers and reporting it to foreman. I’ve found this works perfect if the foreman API is freely accessible. However, in many cases, servers live in different subnets and a smart-proxy is used. Even more, when these are used it’s not uncommon that this subnet has no access to the foreman API directly and thus we have to go via a proxy.
Luckily, the discovery image allows for picking up an SRV record which points to the proxy; if this exists, the discovery data is first directed to the proxy, which in turn directs this to foreman.
The “problem” is that I currently create this SRV record manually; which implies that, each time I reinstall foreman I lose this SRV record.

proposal
Would it not be nice to have an option to enable this from the installer, in case DNS and Discovery are active, the SRV record is just created. Could, IMO even just be created in case just DNS is created.

I’m more than willing to create a patch for always creating an SRV record, in case DNS is activated from the installer, if people would see value in this.

I’m currently posting this in support (instead of RFC), as there might be people whom also came across this “issue” and perhaps found a workaround for this? Which I always try to figure out first before pushing changes to this great project!

lzap · October 25, 2018, 11:31am

Hello, it is a good idea. I think an option in our puppet/installer to create the SRV record would be appropriate. I think it can be the default behavior if discovery proxy plugin is turned on, otherwise false. There is one little problem, the current implementation in the image is poor and it is worth updating it a bit. It assumes that the endpoint is type “foreman” and you cannot set it via SRV entry (only via PXE kernel command line) which is lame:

github.com

theforeman/foreman-discovery-image/blob/2f991f46b18aee21767c6b04ca5d5f8741e8446d/root/usr/lib64/ruby/vendor_ruby/discovery.rb#L127-L148


def discover_by_dns_srv
default = Resolv::DNS::Config.default_config_hash
conf = {
  :nameserver => cmdline("fdi.dns_nameserver", default[:nameserver]),
  :search => cmdline("fdi.dns_search", default[:search]),
  :ndots => cmdline("fdi.dns_ndots", default[:ndots]).to_i,
}
resolver = Resolv::DNS.new(conf)
type = Resolv::DNS::Resource::IN::SRV
result = resolver.getresources("_x-foreman._tcp", type).first
hostname = result.target.to_s
if [443, 8443, 9090].include?(result.port)
  scheme = 'https'
else
  scheme = 'http'
end
uri = "#{scheme}://#{hostname}:#{result.port}"
log_msg "Discovered by SRV: #{uri}"
URI.parse(uri)
rescue

This file has been truncated. show original

Another issue is whether to use HTTP or HTTPS protocol, today we assume this from port number. This is also lame, this was written when there was just foreman endpoint via http.

I suggest if you can create a small PR in FDI as well and modify the code so the discover_by_dns_srv discovers the following SRV records in this order:

_x-foreman-proxy-https._tcp
_x-foreman-proxy-http._tcp
_x-foreman-https._tcp
_x-foreman-http._tcp

I’d ditch backward compatibility in this case, it’s an easy fix however if you prefer you can remain it (there is some magic code trying to “guess” http/https but I’d love to remove it honestly).

I will happily test the PR and merge it for discovery image for 1.20 if you are fast enough Then do the installer part - I think the installer should only set the “foreman-proxy” variants, setting this up to foreman is probably too difficult (if you can do it - that should be used in case discovery proxy plugin is disabled).

UXabre · October 29, 2018, 1:08am

Hey @lzap
,
Great & super! Thank you for the code pointers!

Currently working on some other PRs first (but almost done with those, just want to write some extra tests before pushing). After that, I will make the suggested PRs for the FDI project & the Installer.

lzap · November 29, 2018, 11:05am

What is the status please? I would love to get the SRV in the FDI fixed.

UXabre · November 29, 2018, 2:46pm

Hey @lzap,
Thank you for following up. I’ve started looking into the installation scripts for the smart-proxy; however, I was trying to learn if there was a post-install script somewhere. Reasoning for this script would be that, some DNS plugins already support SRV records so would be nice to just call the smart-proxy API (or something very similar) and just create these. The alternative would be to adjust the template for bind9 DNS servers, but this feels like a very limited use-case?

Now, from what i can tell, there’s a lot of puppet in the installation so I’m very hopeful there’s a way… but I’m no “puppeteer” myself which is not ideal because I’m reading up on understanding what the best option could be and if it’s possible at all by adjusting the smart-proxy puppet installation script.

Will dig into this later today (probably I’ll have to request some support from the smart-proxy installation team to better understand if I’m on the right track and what they see as best option in this scenario)

(sorry for the late update on this, didn’t realize a month has gone by already, this huawei VRP patch got me a bit off-track so it seems)

lzap · December 3, 2018, 8:52am

@ekohl to confim, but I think this is the file you need to modify: https://github.com/theforeman/puppet-dns/blob/master/templates/zone.header.erb

UXabre · December 5, 2018, 3:25pm

Hi @lzap,

I think that file is the correct one to adapt if we want to support just Bind9 and this only if we control the installation of this instance.
However, I hoped in finding/learning a better way by calling the smart-proxy API after the installation has completed, and, via the API creating SRV records. AFAIK this is supported; I think it would be a nice path since this would allow us to add this SRV record to remotely managed DNS servers (that have support for SRV record creation via API).

I’ve set-up my install-dev environment to try and do just that But, perhaps @ekohl could share his vision on such an implementation in the installer or what the best way forward would be for this?

(I’m really sorry this is taking longer then I hoped for, the installer is quite a beast for me as I have a lot of missing affinity with puppet and how this all fits together )

ekohl · December 5, 2018, 3:37pm

I’d prefer to avoid doing this via the installer. Instead, I’d suggest to do this at runtime. It’s a more complicated setup, but it’ll be much more flexible. For example, when something changes. Another benefit is that we support it for all our support DNS servers and also for existing domains.

You’re correct that the proxy can already create SRV records though AFAIK there’s no code in Foreman using this.

Not sure where it should hook into the process though. Perhaps there needs to be something to configure a domain for discovery.

UXabre · December 6, 2018, 6:00pm

Hey @ekohl,
I actually can agree with your statement and seems to make sense if you think about it. Just pitching a workflow here; but could we not simply piggyback on the following:

As a subnet is configured, one also can set a discovery proxy as well as the domains for which this applies
Perhaps we could add the SRV records for each domain configured? (so basically, calling the API for SRV-record creation on each managed domain) or this foolsman talk?

Not sure how this will behave when multiple domains are configured in multiple subnets which have disccovery on different proxies? Will it return multiple SRV records when queried and do we simply need to figure out which of these is the correct one? Or will a SRV query for foreman return the responses in a round-robin fashion? The latter is easily figured out nevertheless…

Pros, cons? Better workflows?
Love to hear your insights on this

ekohl · December 6, 2018, 6:50pm

My main issue is that I don’t fully see how you determine the domain on a discovered host so first I’d need to understand that. It looks like it uses the DNS search domains. AFAIK Foreman doesn’t know about these so there’s no nice way to give some guidance here.

That means you could set a discovery proxy for a domain and assigning that would set the record. Unsetting the proxy would remove it.

I don’t see a useful way to integrate subnets here with the current database models. The proxy could expose the search domains and store that so when you import subnets, Foreman knows. It looks like the proxy already reports the domain_name option when available.

lzap · December 10, 2018, 7:31am

Yeah, it’s complicated to implement it in a generic way. I wonder if it makes sense to write only a blogpost about this on how to set this via curl command using proxy REST API naming all the constraints and limitations of this.

Also maybe it’s easier to implement adding discovery URL via Build PXE Defaults button, we slightly discussed this with @rabajaj the other day. But there is also a problem - when multiple subnets are associated with a single (Discovery) smart-proxy then we don’t know which Discovery URL should we render. At the first phase we should probably raise an error, later on we can add support for multiple URLs or better rendering multiple default templates per subnet IP.

rabajaj · December 11, 2018, 9:48am

@lzap to my understanding, there are two senario’s:

There are two ranges created, and we want to discover the host in one range
while provisioning the host in other. Here the DHCP leases stores the mac address of the host while discovery and assigns an IP address to the host and then at the time of provisioning it reassigns the same IP that it assigned while discovery.
When multiple subnets are associated with a single (Discovery) smart-proxy then we don’t know which Discovery URL should we render.

ekohl · December 11, 2018, 1:04pm

I don’t understand why this is an issue. At the moment we have just 1 name for a proxy and that’s the one Foreman knows. There is no multi homing. In the future it could be added, but you can start with the one we know.

UXabre · December 12, 2018, 7:38am

Hi everyone, I think it makes sense, as @lzap suggested, to do the SRV-record creation on “Build PXE Default”-button press. As this is mandatory anyway.
I’ll investige tonight and see if all is (logically) possible, provided the “limitation” of not having multi-homing (as @ekohl suggested). Will let you guys know what the status is; would really love to see this thing fly !!