Duplicate "Hosts" After Customized Build Process

Hello,

So sorry for the book. This is a question for Foreman 2.1.2 with Katello 3.16.0.

I’d like to start with a little background, as I think what I’m trying to do with Foreman may be a little “unusual”. Giving the background may help this question make at least a little more sense.

I’m currenting doing a proof-of-concept for using Foreman to provision around 5000 “kiosk” devices across various branch offices. These kiosks have host names which represent its “branch number”, which is a 5-digit number and also a “kiosk” number, which is a two digit number.

So a typical kiosk would have a host name like “00712K01” for kiosk “1” at branch “00712”. This offers several benefits, but it mostly tells support and system administrators where a particular kiosk is. It also decides what IP address the kiosk gets.

I need the ability for end-users to assign these names to kiosks when they are either rebuilt/replaced or new to a branch. I’ve implemented a way of performing this, by adding a snippet in my kickstart which pauses in the %pre section, brings up a clean tty and using the Python curses library, prompts for a simple “kiosk number” (the “branch” number is pulled from a host group parameter).

Once the number is entered and the full hostname is assembled, the script dynamically creates a “network.ks” file which is included into the kickstart build. This file is basically generated like so:

network --bootproto=static --device=<%= @host.mac %> --gateway=<%= @host.subnet.gateway %> --ip=${ipAddress} --nameserver=<%= @host.subnet.dns_primary %> --netmask=<%= @host.subnet.mask %> --noipv6 --activate" > /tmp/network.ks

This surprisingly works! The kiosk will boot its OS with the expected hostname and IP address. I have no idea if it is efficient, if its breaking things I don’t know about, or if its even the best way to go about this.

Any who, so now you see what my challenge is and how I’ve been overcoming it. Now let me explain the additional challenge this creates for me.

  1. I’ve created the products, content views and activation keys for the kiosks.
  2. I assign the activation key to a particular branch’s host group.
  3. I boot up a kiosk. It PXE boots into the Foreman discovery environment.
  4. The kiosk is automatically discovered using a rule, assigned to its appropriate host group and is given the default name of mac242ffa021773 per the discovery process. It shows up in the “Hosts” section of Foreman as mac242ffa021773.example.com.
  5. The kiosk boots into the installation media with its kickstart. I input my kiosk number. Using the network directive, it sets the hostname to “00712K01” for example.
  6. Now comes time for the kiosk to subscribe to Katello. Based on the production.log, it subscribes as 00712k01.example.com, which I suppose I would expect.
2020-10-08T15:53:03 [I|app|50704986] Started POST "/rhsm/consumers?owner=my_company&activation_keys=centos_8_kiosk_key_dev" for 127.0.0.1 at 2020-10-0
8 15:53:03 -0400
2020-10-08T15:53:03 [I|app|50704986] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON
2020-10-08T15:53:03 [I|app|50704986]   Parameters: {"type"=>"system", "name"=>"00712k01.example.com", "facts"=>"[FILTERED]", "contentTags"=>[], "role"=>"", "addOns"
=>[], "usage"=>"", "serviceLevel"=>"", "owner"=>"my_company", "activation_keys"=>"centos_8_kiosk_key_dev"}
2020-10-08T15:53:04 [I|aud|50704986] Nic::Managed (163) create event on mac
2020-10-08T15:53:04 [I|aud|50704986] Nic::Managed (163) create event on ip
2020-10-08T15:53:04 [I|aud|50704986] Nic::Managed (163) create event on type Nic::Managed
2020-10-08T15:53:04 [I|aud|50704986] Nic::Managed (163) create event on name 00712k01.example.com

What I find odd however is on the screen of the kiosk, I see it running something like the below:

subscription-manager register --name="mac242ffa021773.example.com" --org='myOrg' --activationkey='centos_8_kiosk_key_dev'

As you can see, it is using the original default name which the discovery plugin gave it.

Once all is done, I end up with two “hosts” in Foreman. One is mac242ffa021773.example.com and the other is 00712K01.example.com.

Is there anyway to work around the duplicate entries or is my best bet to delete the original host? These systems call back into Ansible Tower for a provisioning call back, it wouldn’t be too painful to have it simply delete the original object via the API.

Again, if you read all of this, I apologize for the book. It is a weird scenario which I know may not make the most sense. I would love to hear what the community thinks of how I could tackle this new challenge.

Thanks!

1 Like

Hosts in foreman must not start with numbers, this breaks some things with some backend services. That’s why discovered hosts have the “mac” prefix.

From what I can tell, this is the correct behavior. Your %pre solution updates kickstart with the correct hostname, however this does not (and cannot) change ERB variables like @host.name. The %post section gets rendered on server before a human enters the new hostname.

What you need to do I think is to remove the RHSM snippet from your kickstart and create new “dynamic” %post snippet that also uses the new hostname entered by human. Something like:

echo "%pre \n subscription-manager --blah blah $NEW_HOSTNAME \n %end" > /tmp/network.ks
2 Likes

Would you mind sharing the whole script/template with us? I have a customer who would be interested in this, we could ship this with Foreman as a sippet that could be enabled via ask_for_hostname host param.

2 Likes

Hey Lzap, of course! With as much as you’ve helped me with my Foreman journey, I would be absolutely honored to assist you.

Please note, my script/code is really, really messy and I was planning on returning to it eventually to make it cleaner. It also has some things in it that are probably unique to my environment. I tried to take as much of that out to make it more “generic”.

%pre --log=/tmp/ks-pre-stage.log --interpreter=/bin/bash

# Create python script that handles the prompting workflow
cat > /tmp/get_hostname.py << 'EOL'
import os, uuid, re, curses

# If "prompt" is set to "True", prompt end-user for system name using the Python curses library
prompt = True
if prompt:
    screen = curses.initscr()
    num_rows, num_cols = screen.getmaxyx()
    middle_row = int(num_rows / 2)
    middle_column = int(num_cols / 2)
    curses.echo()
    curses.start_color()
    curses.init_pair(1, curses.COLOR_WHITE, curses.COLOR_RED)
    curses.init_pair(2, curses.COLOR_WHITE, curses.COLOR_BLUE)
    verify = None
    while verify != "y":
        # Build main window with a title and sub-title
        titleWindow = curses.newwin(4, 40, (middle_row - 14), (middle_column - 20))
        titleWindow.border('|', '|', '-', '-', '+', '+', '+', '+')
        titleWindow.bkgd(' ', curses.color_pair(1) | curses.A_BOLD)
        titleWindow.addstr(1, 11, "Title", curses.A_UNDERLINE)
        titleWindow.addstr(2, 13, "Sub-Title")
        titleWindow.refresh()

        # Build hostname prompt window
        hostnameWindow = curses.newwin(3, 40, (middle_row - 10), (middle_column - 20))
        hostnameWindow.border('|', '|', '-', '-', '+', '+', '+', '+')
        hostnameWindow.bkgd(' ', curses.color_pair(1) | curses.A_BOLD)

        # Prompt for end-user to enter hostname
        hostnameWindow.addstr(1, 3, "Enter hostname: ")
        posHostnameBytes = hostnameWindow.getstr(1, 19, 11)
        posHostname = posHostnameBytes.decode('utf-8')
        <% end -%>
        hostnameWindow.refresh()

        # Build window for verifying input
        verifyWindow = curses.newwin(3, 40, (middle_row - 8), (middle_column - 20))
        verifyWindow.border('|', '|', '-', '-', '+', '+', '+', '+')
        verifyWindow.bkgd(' ', curses.color_pair(1) | curses.A_BOLD)
        verifyWindow.addstr(1, 3, "Correct? (y/n): ")

        # Gather input and store into "verify" variable, break loop if contains "y"
        verifyBytes = verifyWindow.getstr(1, 19, 11)
        verify = verifyBytes.decode('utf-8')
        verifyWindow.refresh()

        # Keep screen clean
        screen.clear()
        screen.refresh()

    # Exit curses menu
    curses.endwin()

# Write the system name to "/tmp/hostname" to be collected later when the interface is configured
f = open("/tmp/hostname", "w")
f.write(userHostname)
f.close()
EOL

# Switch the TTY and execute the Python script
exec < /dev/tty6 > /dev/tty6 2> /dev/tty6
chvt 6
/usr/libexec/platform-python /tmp/get_hostname.py
chvt 1
exec < /dev/tty1 > /dev/tty1 2> /dev/tty1

# Pull the set system name from "/tmp/hostname"
userHostname=`cat /tmp/hostname`

# Create the "/tmp/network.ks" file with the `network` command with the system's unique IP configuration
echo "network --hostname=${userHostname}" >> /tmp/network.ks
%end

And then in my main kickstart file, I just include /tmp/network.ks in the command section:

%include /tmp/network.ks

One thing to note is that I put a restriction on the length of the hostname, as it will always be a specific length in my environment. This can be modified in the hostnameWindow.getstr(1, 19, 11) statement.

Also, although the prompt variable is hardcoded to True, I want to eventually make it a parameter. This way if I’m doing a rebuild, I can set it to False and it will not prompt, but instead use <%= @host.name %>.

Any who, that is it. I’m sure there is a lot of room for improvement. I hope at the very least it can be used as a starting-point for your customer.


So in regards to your suggestion to my challenge, first of all thanks. I’ve been using the redhat_register provisioning template that ships with Foreman to handle the registration part. So as a test, I edited with my a hardcoded name to see how it would register:

    #subscription-manager register --name="<%= @host.name %>" --org='myOrg' --activationkey='myActivationKey'
    subscription-manager register --name="ods00298k02.mydomain.net" --org='myOrg' --activationkey='myActivationKey'

I know this isn’t exactly what you had suggested, but its the same kind of idea right? I am just curious to see what would happen if I explicitly told it what name to use before implementing a more permanent process.

After doing the above, I watched it build and it did indeed subscribe as “ods00298k02.mydomain.net”, which is showing up in my “Hosts” list. However, I still have a “mac40f2e9d64f4a.odretail.net” as well.

I suppose this may be what you’d expect however, right? One object is a “Content Host” and the other is a regular “Host”, even though the represent the same system I’d think? I typically have these systems do an Ansible Tower provisioning callback on first boot which renames the host to its new, user-set hostname, however that fails now because it already exists with that name as well (as a Content Host)!

I’m thinking a way to work around this is not to try and subscribe in the build. Instead, have my Ansible Tower job first rename the host from its default discover name and then subscribe using the activation key. Unless you know of a better way to approach this (or I’ve misunderstood your original suggestion), I know its not your typical Foreman build process.

Thanks again for all the help.

1 Like

I tried to experiment with something and got interesting results.

  1. I built a system with no activation key assigned to its host group. This way is doesn’t even attempt to “register” for content per the redhat_register snippet. It came up just fine.
  2. Then I renamed the host in Foreman’s “Host” section to its real name of “ODS00298K02.mydomain.net” from “mac40f2e9d64f4a.odretail.net”.
  3. Now I register it using subscription-manager register --name="ods00298k02.example.net" --org="myOrg" --activationkey="myActivationKey" --force
  4. It then creates a separate Content Host of ods00298k02 (no domain name). Is this to be expected?

My pleasure, this is my job.

Unfortunately, yes. When a host is created either manually or via discovery, its name is stored in Foreman db and everything else is orchestrated (DHCP, DNS, Realm). If you want to change hosts name you need to do this before it’s created. It is possible to rename host, Foreman then triggers full orchestration, e.g. delete old DHCP record, create new one, delete DNS name, create new one, delete Puppet cert with the CN, create new one and so on and so on. I guess your ad-hoc interactive workflow is great for asking things like Keyboard Layout or Timezone, but for such important thing like hostname this is too late - the train already departed the station.

I think you should focus your attention more to the discovery phase. There are two approaches you can take:

  1. Use discovery TUI and let operators to provide a custom fact. You can pre-fill the last screen via fdi.factname1=hostname so they only need to enter one field. Granted the screen is a bit clunky, you need to pass multiple screens (NIC, Discover via DHCP, Foreman URL), but most of these can be actually pre-filled using fdi. kernel options so most of the work is just ENTER. There is one snag tho, TUI only shows up when discovery is not PXE-booted and there is no option to force it via an option. It tests for BOOTIF kernel command line argument, so what you can actually do is remove IPAPPEND 2 or BOOTIF in PXELinux or PXEGrub2 templates respectively to make it appear.
  1. The second option is write a custom fact that will automatically find out the branch office code from the environment. You can perform a DNS query for example, or reach out to HTTP IP address and via REMOTE_IP it would return the code. Then you can extend the image via custom fact, FDI has capability to read custom facts and scripts from ZIP files over HTTP or TFTP during boot. To append a number, there was no macro available in Foreman until yesterday when I wrote one: https://github.com/theforeman/foreman/pull/8072 - with all of that you can create autodiscovery rule which can append those two together (branch number + sequence). The macro currently does not support “0001” prefixing bug I will add that for you in a moment.

Thanks, there were two typos so I fixed them and created a gist for it. Nice one.

Ah, this is so interesting… Although it can be frustrating at times, I love this kind of stuff. I would really like to explore both your recommendations a bit.

Here is my thoughts on the two suggestions. Hopefully I understood them properly, as I’m still a bit green with this stuff.

  1. Use the discovery TUI: This would be a fairly straight-forward and pragmatic solution. The only challenge is what you pointed out, navigating the menus. Though with proper documentation, I think the end-users can adapt. My only questions is, lets say that fdi.factname1 fact gets set with new hostname, how does it then become the hostname of the system? Would I simply set the “Hostname” option in the discovery rule to @host.facts['fdi.factname1']?

  2. Write a custom fact that will automatically find out the branch office code: This is really, really neat. I didn’t even know of this functionality until now and reading the “extending the image” is helping me understand how it works. Here is the challenge I’m seeing though. The “branch number” I could probably have a discovery “extension” script figure out systematically from the environment, however the “kiosk number” which identifies the individual kiosk is the toughie. I think this is really something that someone has to “assign”, if that makes any sense. So here is my question, could I make a discovery extension that simply prompts for the end-user to enter the hostname (similar to how my kickstart snippet works)? It will then take that input and store it into a new, custom fact? This is basically mimicking what the first suggestion does, but just more straight-forward for the end-user.

Those are my thoughts/questions, hopefully I’m somewhere in the ballpark of what you are suggesting!


Only if you are interested, here is a little background on what brought me here to the first place. Its mostly me just venting/chit-chatting, so no pressure for you to read it!

We’re a retailer in the US and these kiosks are basically point-of-sale devices. We currently have SLES deployed to our branch hosts and using their “SUSE Linux Enterprise for Point of Sale” product. We’re at a point of evaluating CentOS to save a huge chunk on OS licensing costs (chain store math is brutal), therefor I’m looking to Foreman to try and replace the “SLEPOS” product, which currently handles identity management via a LDAP database and provisioning with images deployed via a PXE boot image/environment, kind of similar to the Foreman discovery solution.

To oversimply it, the PXE boot image prompts for hostname and image version… It then creates a fresh LDAP entry accordingly with the system details, then using TFTP, copies over the system image from the branch server, piping straight into the dd program for writing to the disk. It also uses the MAC address to cross-reference the LDAP database to see if it already exists, and won’t prompt for hostname on re-image.

I personally would like to get away from monolithic disk images due to their lack of flexibility, which is what attracted me to Foreman in the first place. Also, Katello will easily replace our “SUSE Manager” server (SUSE supported Spacewalk clone) for content management.

So that is basically what brought me to Foreman. I’m very excited to start putting it in production. Thanks to its flexibility, I think it will make a fantastic solution, I just need to work out a few small kinks!

Yes, sorry I thought it’s obvious :slight_smile:

Oh that’s too bad, I thought this is just a counter. If you were implementing SAP you could simply tell them “it’s cheaper for you to adapt your process than me re-implementing the process”, but this is Foreman! :wink:

You can do anything, this is opensource. Problem is, discovery TUI was designed later and currently there is a background service (started by systemd) which performs discovery after boot automatically:

There should probably be condition to start it and only if BOOTIF= is present on the kernel command line, this was never added tho.

Anyway. You need to make sure your custom script starts up before this service, you can perhaps turn it off completely as well as the TUI (which is another service started via discovery-menu.service). Run your own one instead and after you gather all facts, perform the discovery manually (in a loop perhaps). You can either drop custom facts for facter and then start the service or if you write it in Ruby then you can call Facter directly.

With all that said, I am planning to kill the TUI completely (it’s quite complex Ruby code with newt native library) and unify the background service creating single point of registration - a shell script with whiptail-based menu. Here is a POC:

This would unify both PXE and PXE-less environments, simplify maintenance, we could even drop Ruby someday. But at this point I would be happy removing the complex TUI with something lightweight, easier to test or debug or even modify. One of the options could be something like “custom script” where user would provide his/her own set of menus.

Asking for a hostname could be one of the build-in menu items tho, this can be pretty useful for many users. We would default to macAABBCCDDEEFF but users could easily replace this with their own one.

By the way, Anaconda in CentOS 7.5+ does support image-based provisioning. You have a golden image (as a tarball or loop-mountable image) and using liveimg kickstart option you instruct Anaconda to paritition disks and then copy files over. Best of both worlds, this is quite “hidden” feature it was added for RHV product (oVirt hypervisors are installed using this technique).

If you like this, I can probably prioritize the POC and we could work together on the improved FDI. This could be a first step towards more lightweight and better discovery:

  1. Upgrade to Facter4 (this is an ongoing effort to support IPv6)
  2. Get rid of TUI
  3. Unify PXE and PXE-less menu experience
  4. Replace foreman-proxy with a small python API
  5. Make discovered nodes available via Remote Execution
  6. Replace Facter with uFacter
  7. Remove Ruby from the image completely
  8. Upgrade to CentOS 8
2 Likes

I love that idea and I would be happy to help in anyway possible on the future state of the discovery process. I probably have some fairly large “knowledge gaps” in regards to this, but I am doing my best to fill those in as much as I can. Just let me know how I can assist and I will make it so.

Today I spent some time getting familiar with the discovery image, the various services built into it and processes. I’ve learned a good few things, but feel like I’ve only scratched the surface thus far.

Now I’ve just started experimenting with extensions. I’m at the point where I’ve built one that runs a quick test script, which works. I can stop the discovery-menu and discovery-register services to “pause” the discovery process.

I am learning the how factor works and how custom facts are created. Obviously the extension process makes this simple, by allowing facts to be defined in /facts/ of the extension zip. Depending on how things are ordered, I suppose I could dynamically generate a /facts/hostname.rb file that will create the fact or write the extension script in Ruby and manipulate factor directly using the library, which I think you were alluding to in your previous post. I do see there is a custom_fact array/list that gets passed to the upload function in the discovery Ruby library.

Like I said, still warming up to how this all works, but I’m enjoying the process :slight_smile:

As always, thanks for all the help thus far.

Yeah good start.

Or just have a script that reads /tmp/myhostname.

Note the fact, named discovery_hostname for instance, will not be picked by discovery plugin automatically. You need to change Hostname facts Adminster setting to “discovery_hostname,discovery_bootif” and Hostname prefix to “” (empty string).

Thanks Lukáš!

So I’ve had some success today. With the idea of taking baby-steps, I attempted to simply recreate what I’ve been doing in my kickstart and instead prompt, in the same manner with the same Python code, inside of the discovery image.

I put together an extension zip and booted the image with it. It executes the below script automatically:

#!/bin/bash

##################################
# Pause the discovery process
##################################
systemctl stop discovery-register
systemctl stop discovery-menu
systemctl disable discovery-register
systemctl disable discovery-menu

##################################
# Create discovery-prompts.service
##################################
cat > /etc/systemd/system/discovery-prompts.service << 'EOL'
[Unit]
Description=Discovery Prompts
Wants=basic.target
After=basic.target network-online.target nss-lookup.target
ConditionPathExists=/dev/tty1
[Service]
Type=idle
ExecStart=/usr/bin/discovery-prompts
ExecStartPre=/usr/sbin/sysctl -w kernel.printk=0
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=5
KillMode=process
TimeoutStopSec=5
StandardInput=tty
StandardError=tty
StandardOutput=tty
TTYPath=/dev/tty1
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
EOL

##################################
# Create discovery-prompts script
##################################
cat > /usr/bin/discovery-prompts << 'EOL'
#!/usr/bin/env python
import os, uuid, re, curses
screen = curses.initscr()
num_rows, num_cols = screen.getmaxyx()
middle_row = int(num_rows / 2)
middle_column = int(num_cols / 2)
curses.echo()
curses.start_color()
curses.init_pair(1, curses.COLOR_WHITE, curses.COLOR_RED)
curses.init_pair(2, curses.COLOR_WHITE, curses.COLOR_BLUE)
verify = None
while verify != "y":
    # Build main window
    titleWindow = curses.newwin(4, 40, (middle_row - 10), (middle_column - 20))
    titleWindow.border('|', '|', '-', '-', '+', '+', '+', '+')
    titleWindow.bkgd(' ', curses.color_pair(1) | curses.A_BOLD)
    titleWindow.addstr(1, 11, "Title 1", curses.A_UNDERLINE)
    titleWindow.addstr(2, 13, "Title 2")
    titleWindow.refresh()
    # Build hostname prompt window
    hostnameWindow = curses.newwin(3, 40, (middle_row - 6), (middle_column - 20))
    hostnameWindow.border('|', '|', '-', '-', '+', '+', '+', '+')
    hostnameWindow.bkgd(' ', curses.color_pair(1) | curses.A_BOLD)
    # Ask for hostname, limiting to 11 characters
    hostnameWindow.addstr(1, 3, "Enter hostname: ")
    hostnameBytes = hostnameWindow.getstr(1, 19, 11)
    hostname = hostnameBytes.decode('utf-8')
    hostnameWindow.refresh()
    # Build window for verifying input
    verifyWindow = curses.newwin(3, 40, (middle_row - 4), (middle_column - 20))
    verifyWindow.border('|', '|', '-', '-', '+', '+', '+', '+')
    verifyWindow.bkgd(' ', curses.color_pair(1) | curses.A_BOLD)
    verifyWindow.addstr(1, 3, "Correct? (y/n): ")
    # Gather input and store into "verify" variable, break loop if contains "y"
    verifyBytes = verifyWindow.getstr(1, 19, 11)
    verify = verifyBytes.decode('utf-8')
    verifyWindow.refresh()
    # Keep screen clean
    screen.clear()
    screen.refresh()
curses.endwin()
# Write the system name to "/tmp/hostname" to be collected later when the interface is configured
f = open("/tmp/hostname", "w")
f.write(hostname)
f.close()
EOL

##################################
# Execute discovery-prompts service
##################################
chmod +x /usr/bin/discovery-prompts
systemctl daemon-reload
systemctl start discovery-prompts

while [[ ! -e /tmp/hostname ]]; do
  sleep 2
done

##################################
# Prepare fact(s)
##################################
discoveryHostname=$(cat /tmp/hostname)

cat > /opt/extension/facts/hostname.rb << EOF
Facter.add("discovery_hostname") do
  setcode do
    "$discoveryHostname"
  end
end
EOF

##################################
# Resume discovery process
##################################
systemctl stop discovery-prompts
systemctl start discovery-register
systemctl start discovery-menu

I know it is messy and there is probably a billion better ways of doing this, but again, baby steps!

I was having some trouble getting my prompts to take control of the TTY, so I stole your discovery-menu.service file and suited it to my needs as a discovery-prompts.service.

I went with just “dynamically” generating a /opt/extension/facts/hostname.rb file for the discovery-register service to pick up. Again, not sure if how “clean” this is, but it works easy enough.

I’m not too concerned with the host’s name as a discovered system, as my discovery rule automatically assigns the hostname just fine using <%= @host.facts['discovery_hostname'] %>.

I got to say, this is awesome and makes things so much easier for me in so many ways:

  1. I can now remove so many extra lines and conditional nonsense from my kickstarts. Simpler is better.
  2. I don’t have to worry about renaming the host in Foreman ad hoc with my Ansible Tower job.
  3. My system subscribes to Katello correctly, with no duplicates or mess.
  4. Now even with this quick solution, I can go to my team and management and show how not only is Foreman going to be able to replace what SLEPOS does for us, but it will be a much more flexible and expandible framework.
  5. The discovery image’s method of extension is awesome. Being able to do stuff like this without completely rebuilding the image makes things so much easier.

I will continue to explore the discovery process and look try to understand your future roadmap as well. If you need testing or see anywhere you can use a hand, please let me know.

2 Likes

Good work!

I still think discovery node is currently too complex, many services, TUI, background service, various conditions to met in order to have one or the other working. I would like to spend some cycles into unifying this as I stated above. One day there will be a simple shell-based TUI and Remote Execution integration so things can be done also from Foreman.

1 Like