Ubuntu 20.04/22.04 LTS not auto/unattended installing with user-data

Problem:
Attempting to auto install (unattended) Ubuntu 20.04/22.04 LTS hosts with foreman (3.3.0.20) packaged with Red Hat Satellite 6.12.2. I have everything working with iPXE at this point to load the iso, however it does not seem to be using my user data template. Please see my templates below:

Preseed_default_iPXE_2004 - AUTOINSTALL - template

#!gpxe
<%#
kind: iPXE
name: Preseed_default_iPXE_2004 - AUTOINSTALL
model: ProvisioningTemplate
oses:
- Debian
- Ubuntu
%>
<% if @host.operatingsystem.name == 'Debian' -%>
  <%- keyboard_params = "auto=true domain=#{@host.domain}" -%>
<% else -%>
  <%- keyboard_params = 'console-setup/ask_detect=false console-setup/layout=USA console-setup/variant=USA keyboard-configuration/layoutcode=us' -%>
<% end -%>

<% subnet = @host.subnet -%>
<% if subnet.nil? || subnet.dhcp_boot_mode? -%>
  <%- provision_url_suffix = '' -%>
  <%- netcfg_args = '' -%>
<% else -%>
  <%- provision_url_suffix = (@host.token.nil? ? '?' : '&') + 'static=yes' -%>
  <%- netcfg_args = 'netcfg/disable_dhcp=true netcfg/get_ipaddress=${netX/ip} netcfg/get_netmask=${netX/netmask} netcfg/get_gateway=${netX/gateway} netcfg/get_nameservers=${dns} netcfg/confirm_static=true' -%>
<% end -%>

set gfxpayload=keep
kernel /pub/installation_media/ubuntu/20.04-x86_64/casper/vmlinuz
initrd /pub/installation_media/ubuntu/20.04-x86_64/casper/initrd
imgargs vmlinuz ramdisk_size=10800 initrd=initrd boot=casper maybe-ubiquity verbose url=http://<mirror_URL>/provisioning/ubuntu/20.04/ubuntu-20.04.6-live-server-amd64.iso ip=dhcp cloud-config-url=/dev/null autoinstall ds=nocloud-net;s=http://<%= foreman_request_addr %>/userdata/

boot

Preseed Autoinstall cloud-init user data - template

<%#
kind: user_data
name: Preseed Autoinstall cloud-init user data
model: ProvisioningTemplate
oses:
- ubuntu
test_on:
- ubuntu_autoinst4dhcp
-%>
<%-
username_to_create = host_param('username_to_create', 'root')
realname_to_create = host_param('realname_to_create') || username_to_create
password_to_create = host_param('password_to_create') || @host.root_pass
host_interface = @host.provision_interface
host_mac = @host.mac
-%>
#cloud-config
autoinstall:
  version: 1
  apt:
    geoip: false
    preserve_sources_list: false
    primary:
      - arches: [amd64, i386]
        uri: http://archive.ubuntu.com/ubuntu
      - arches: [default]
        uri: http://ports.ubuntu.com/ubuntu-ports
        
  users:
    - { name: user1, gecos: Administrator, primary_group: sudo, passwd:<hashed_Pass>, shell: /bin/bash, homedir: /home/user1 }
    - { name: user2, gecos: Provisioning Admin, sudo: ALL=(root) /bin/apt, /bin/usermod, /bin/adjoin, /bin/adleave, /bin/df, /sbin/fdisk -l, passwd:<hashed_Pass>, shell: /bin/bash, homedir: /home/user2 }
    
<%= indent(4) { snippet_if_exists(template_name + " custom apt") } -%>
  identity:
    hostname: <%= @host.name %>
    realname: <%= realname_to_create %>
    username: <%= username_to_create %>
    password: "<%= password_to_create %>"
  keyboard: {layout: us, toggle: null, variant: ''}
  locale: en_US.UTF-8
  
  packages:
    - aide
    - ubuntu-desktop
    - cifs-utils
    - at
    - kexec-tools
    - perl
    - -gnome-boxes
    - -tuned

  
<%= snippet 'preseed_netplan_setup' -%>
  ssh:
    allow-pw: true
    authorized-keys: []
    install-server: true
  updates: security
  
  network:
    version: 1
    config:
      - type: physical
        name: <% host_interface %>
        mac_address: <% host_mac %>
        subnets:
          - type: dhcp
  
<%= indent(2) { @host.diskLayout } -%>
<%= indent(2) { snippet_if_exists(template_name + " custom root") } -%>
  user-data:
    runcmd:
      - wget -Y off <%= @static ? "'#{foreman_url('finish', static: 'true')}'" : foreman_url('finish') %> -O /tmp/finish.sh
      - chmod +x /tmp/finish.sh
      - /tmp/finish.sh

Expected outcome:

Unattended provisioning of Ubuntu 20.04 and 22.04 LTS Desktops using iPXE from foreman.

Foreman and Proxy versions:

  • foreman-3.3.0.20-1.el8sat.noarch
  • foreman-proxy-3.3.0-1.el8sat.noarch

Foreman and Proxy plugin versions:

  • foreman-tasks: 6.0.3
  • foreman_ansible: 7.1.4.1
  • foreman_azure_rm: 2.2.6
  • foreman_bootdisk: 19.0.7
  • foreman_discovery: 21.0.5
  • foreman_hooks: 0.3.17
  • foreman_leapp: 0.1.10
  • foreman_openscap: 5.2.2
  • foreman_remote_execution: 7.2.2
  • foreman_rh_cloud: 6.0.44
  • foreman_templates: 9.3.0
  • foreman_theme_satellite: 10.0.0.4
  • foreman_virt_who_configure: 0.5.9
  • foreman_webhooks: 3.0.5
  • katello: 4.5.0.24

Distribution and version:

Foreman: Red Hat Satellite version 6.12.2

  • foreman-3.3.0.20-1.el8sat.noarch
  • foreman-proxy-3.3.0-1.el8sat.noarch

Hosts to deploy:

  • Red Hat Enterprise Linux 8 = Working
  • Ubuntu 20.04 LTS = kicks off the ISO for manual installation, not accepting user-data
  • Ubuntu 22.04 LTS = kicks off the ISO for manual installation, not accepting user-data

Other relevant data:

Due to sensitivity, I am required to redact any business sensitive content. I am happy to provide any needed data that may help resolve this. Due to the mass scale of system provisioning, with limited Linux experience in my Desktop department I am required to have an automated solution in place. Due to the switch from preseed to subiquity/cloud-init I have to implement a new process with a short time line.

I would like assistance to determine if I am missing a step as this seem to be undocumented, aside from creating the installation media, which I did following steps found here:
2.5. Creating an Operating System for Ubuntu

I have my ISO file hosted on an internal mirror server, this seems to work. The issue is that as soon as the ISO kicks off, it brings me to a menu asking to setup the keyboard, languge, etc. This should not be the case, as I am using the foreman’s user-data template, with some modifications, and autoinstall is suppose to complete regardless of input.

I thank you for any assistance in advance, as this is a high priority for my organization.

Thanks,
Joe

Hi,

if you are directed to the language selection during the Autoinstall you have usually issues with the user-data template.
One thing could be a problem with the format - so no valid yaml syntax. This could be caused for example by specifying the wrong partition table (one meant for preseed) or a custom partitiontable with wrong indentation or something similar. Please check the rendered template for a host if you find something and if the “late-command” is properly indented.
Another thing is the userdata-template as it is provided by foreman. This is usually using port 80 on the main foreman and on proxies port 8000.
Depending on the instance you could find hints in the corresponding logfiles (either /var/log/httpd/foreman_access.log or /var/log/foreman-proxy/proxy.log)

@jtruestedt My user-data template is attached above and from my limited knowledge of autoinstall it looks accurate. This is based on the foreman provided template.

@lzap you seem to be heavily involved and based on the tutorial you referenced quite a bit using the api for user-data. We have been working on this for several months without success and this is impacting our organization. Are you able to assist or point me to someone who can work with me on this issue?

Thanks,
Joe

Hi.

I would bisect the problem by first verifying that the rendered user-data template is actually working. Depending on your capabilities this could be done by using a local (e.g. your workstation) QEMU VM.

  • Mount Ubuntu ISO, copy kernel+initrd to workdir
  • Setup minimal file structure on a web server (e.g. local nginx container or your Satellite pub)
    • Ubuntu ISO
    • Rendered user-data
  • Start kernel+initrd in workdir directly via QEMU with suitable options and specify append option to point to the web server
  • See if its fetching the ISO and the user-data

If this does not work, your template is broken. If its working, you have another problem. I can provide corresponding QEMU parameters if you like.

Hey, I ran into this before, essentially when you get prompted for a manual install it means there’s an issue with cloud-init (in my case it was bad formatting for my user-data file 9 times out of 10).

You can look-up how to escape from the installer prompt and poke around the shell for error logs or complete the installation manually and poke around at cloud-init logs for clues.

Shameless self promotion, I wrote a blog that goes very deep into cloud-init operations and it might help fill in any missing gaps or tip you towards a new direction:

Hope it helps!

1 Like

As @jtruestedt and @jimangel mentioned before, the manual installation you are prompted to is usually caused by an issue with your Autoinstall/cloud-init template.

As a first step, I would check whether your host actually downloads the Autoinstall template. Therefore, you can check your foreman log at /var/log/foreman/production and search for an incoming request to userdata/user-data.

If you see such request, have another look at your rendered Autoinstall template as mentioned before. @jtruestedt and @Jan mentioned that your rendered template could be invalid due to a wrong partition table for example. The template you provided is not the rendered version, this means we cannot see if referenced snippets introduce an error. So please check it to be a valid yaml or post it here such that we can have a look at it (masquerading all sensitive information of course).

Hello @bastian-src, @jtruestedt, and @jimangel thank you all for the responses.

I have tested my rendered user-data file locally and it works without an issue, please see my redacted user-data file and iPXE file:

user-data

#cloud-config
autoinstall:
  version: 1
  apt:
    disable_components: []
    geoip: true
    preserve_sources_list: false
    primary:
    - arches:
      - default
      uri: http://localmirror.example.com
  drivers:
    install: false
  identity:
    hostname: localhostName
    username: root
    password: hashed_pass
  keyboard: {layout: us, toggle: null, variant: ''}
  locale: en_US.UTF-8
  storage:
    config:
    - { ptable: gpt, path: /dev/sda, preserve: false, name: '', grub_device: false, type: disk, id: disk-sda }
    - { device: disk-sda, size: 1127219200, flag: boot, number: 1, preserve: false, grub_device: true, type: partition, id: partition-sda1 }
    - { device: disk-sda, size: 2147483648, wipe: superblock, flag: linux, number: 2, preserve: false, grub_device: false, type: partition, id: partition-sda2 }
    - { fstype: xfs, volume: partition-sda2, preserve: false, type: format, id: format-9}
    - { device: disk-sda, size: 125572218880, wipe: superblock, flag: linux, number: 3, preserve: false, grub_device: false, type: partition, id: partition-sda3}
    - { volume: partition-sda3, key: 'temp', preserve: false, type: dm_crypt, id: dm_crypt-0 }
    - name: vg0
      devices:
      - dm_crypt-0
      preserve: false
      type: lvm_volgroup
      id: lvm_volgroup-0
    - { name: lv-swap, volgroup: lvm_volgroup-0, size: 1073741824B, wipe: superblock, preserve: false, type: lvm_partition, id: lvm_partition-0 }
    - { fstype: swap, volume: lvm_partition-0, preserve: false, type: format, id: format-1 }
    - { path: '', device: format-1, type: mount, id: mount-1 }
    - { name: lv-var, volgroup: lvm_volgroup-0, size: 19327352832B, wipe: superblock, preserve: false, type: lvm_partition, id: lvm_partition-1 }
    - { fstype: xfs, volume: lvm_partition-1, preserve: false, type: format, id: format-2 }
    - { name: lv-tmp, volgroup: lvm_volgroup-0, size: 12884901888B, wipe: superblock, preserve: false, type: lvm_partition, id: lvm_partition-2 }
    - { fstype: xfs, volume: lvm_partition-2, preserve: false, type: format, id: format-3 }
    - { name: lv-vartmp, volgroup: lvm_volgroup-0, size: 12884901888B, wipe: superblock, preserve: false, type: lvm_partition, id: lvm_partition-3 }
    - { fstype: xfs, volume: lvm_partition-3, preserve: false, type: format, id: format-4 }
    - { name: lv-varlog, volgroup: lvm_volgroup-0, size: 25769803776B, wipe: superblock, preserve: false, type: lvm_partition, id: lvm_partition-4 }
    - { fstype: xfs, volume: lvm_partition-4, preserve: false, type: format, id: format-5 }
    - { name: lv-audit, volgroup: lvm_volgroup-0, size: 25769803776B, wipe: superblock, preserve: false, type: lvm_partition, id: lvm_partition-5 }
    - { fstype: xfs, volume: lvm_partition-5, preserve: false, type: format, id: format-6 }
    - { name: lv-home, volgroup: lvm_volgroup-0, size: 6442450944B, wipe: superblock, preserve: false, type: lvm_partition, id: lvm_partition-6 }
    - { fstype: xfs, volume: lvm_partition-6, preserve: false, type: format, id: format-7 }
    - { name: lv-root, volgroup: lvm_volgroup-0, size: 21399339008B, wipe: superblock, preserve: false, type: lvm_partition, id: lvm_partition-7 }
    - { fstype: xfs, volume: lvm_partition-7, preserve: false, type: format, id: format-8 }
    - { path: /, device: format-8, type: mount, id: mount-8 }
    - { path: /boot, device: format-9, type: mount, id: mount-9 }
    - { fstype: fat32, volume: partition-sda1, preserve: false, type: format, id: format-partition-sda1 }
    - { fstype: fat32, volume: partition-sda1, preserve: false, type: format, id: format-partition-sda1 }
    - { path: /boot/efi, device: format-partition-sda1, type: mount, id: mount-0 }
    - { path: /var, device: format-2, type: mount, id: mount-2 }
    - { path: /tmp, device: format-3, type: mount, id: mount-3 }
    - { path: /var/tmp, device: format-4, type: mount, id: mount-4 }
    - { path: /var/log, device: format-5, type: mount, id: mount-5 }
    - { path: /var/log/audit, device: format-6, type: mount, id: mount-6 }
    - { path: /home, device: format-7, type: mount, id: mount-7 }
  updates: security
  packages:
    - update
    - ubuntu-desktop
  shutdown: poweroff
  user-data:
    disable_root: false
    users:
      - { name: admin, gecos: Administrator, lock_passwd: false, primary_group: sudo, passwd: hashed_pass, shell: /bin/bash, homedir: /home/admin }
      - { name: build, gecos: Provisioning Admin, lock_passwd: false, sudo: ALL=(root) /bin/apt, /bin/usermod, /bin/adjoin, /bin/adleave, /bin/df, /sbin/fdisk -l, passwd: hashed_pass, shell: /bin/bash, homedir: /home/build }

iPXE:

#!gpxe



set gfxpayload=keep
kernel /pub/installation_media/ubuntu/22.04-x86_64/casper/vmlinuz
initrd /pub/installation_media/ubuntu/22.04-x86_64/casper/initrd
imgargs vmlinuz ramdisk_size=10800 initrd=initrd boot=casper maybe-ubiquity verbose url=http://localmirror.example.com/provisioning/ubuntu/22.04/22.04-x86_64.iso ip=dhcp cloud-config-url=/dev/null autoinstall ds=nocloud-net;s=http://forman.example.com/pub/userdata/

boot

Do I need to point my iPXE template to a certain location to pickup the templates I have in satellite (foreman)?

Thanks,
Joe

Can you see that the request for the userdata-template is successful?
If it is your foreman and not a proxy, that should be in /var/log/httpd/foreman_access.log
If it is a smart proxy the port should be 8000 in the iPXE-Template and then the log is /var/log/foreman-proxy/proxy.log

The address where the host is supposed to download the template is defined as s=http://forman.example.com/pub/userdata/ in your iPXE template. Are you aware that this is not the user-data interface of the foreman?
In case you want to use the actual API (which I recommend you to do), you must put here s=http://forman.example.com/userdata/.

In case you did that on purpose and you did put your user-data template manually to http://forman.example.com/pub/userdata/user-data, you can check the log as @jtruestedt mentioned.

@bastian-src and @jtruestedt I checked the logs and I am getting a loop of the following from my satellite (foreman) server:

<IP_Addresss> - - [28/Apr/2023:08:15:26 -0400] “GET /userdata/meta-data HTTP/1.1” 404 44 “-” “Cloud-Init/22.4.2-0ubuntu0~22.04.1”

Not really sure what the issue is, looks like a 404? Should something on the firewall be configured to allow this connection to the server? The rendered template is identical to the one that works when pointing to /userdata/user-data for testing purposes that worked.

Thanks,
Joe

That is known, but not actually an issue. Besides userdata/user-data, Ubuntu hosts are also looking for userdata/meta-data but it is not necessary for a successful deployment of the host - so, foreman does not provide such API endpoint.

Do you also see a successful request of your host to GET userdata/user-data?

So, does your deployment work with the template pointing to http://forman.example.com/userdata/ or does the host run into an error still?

@bastian-src, thanks for the information and additional explanation. I am not seeing anything in /var/log/httpd/foreman_access.log for GET userdata/user-data only the calls for GET /userdata/meta-data and calls for /pub/userdata/user-data when I point it to my static user-data file. The static file, which is my rendered template works fine. The template does not.

I have my IPXE pointed to s=http://forman.example.com/userdata/ as instructed, yet nothing is happening. When I provision my host it just comes to the normal install screen prompting for manual configuration.

My Ubuntu 22.04 LTS OS in the WebUI only has the following templates enabled:
iPXE template
User data template

Any and all help is appreciated.

Thanks,
Joe

Okay, so I guess we should check whether the userdata API works on your foreman properly.

Can you run a simple wget against http://foreman.example.com/userdata/user-data from your foreman itself?
In case you deployed an Ubuntu host successfully with your static template, can you also run the wget from that machine to http://foreman.example.com/userdata/user-data?

What are the responses of both wgets on the respective machine?

Do you see any of those requests in (a) your /var/log/foreman/production.log or (b) your /var/log/httpd/foreman_access.log?

If you have a look at the Ubuntu machine you want to deploy in your foreman. Does the host have the two templates you mentioned before (userdata and iPXE) assigned there? I mean at this tab in the UI:
host_templates

(just to check that your host has the right templates assigned)

@bastian-src,

wget against http://foreman.example.com/userdata/user-data from my foreman resolves no issue.
wget from the machine fails with a 404 error

We looked at net traffic and nothing is blocked and we also locally disable the firewall and ruled out FW as the cause. Any suggestions and help is appreciated. Not sure if there is something that needs configured?

The webui of my foreman does show both the my IPXE template and User data template as resolved for the system, so it is applying it to the host for provisioning the 404 to http://foreman.example.com/userdata/user-data from other systems seems to be the issue.

Thanks,
Joe

Just to avoid confusion of the hosts, I’ll use the following terminology:

  • foreman: your actual foreman machine
  • host A: your Ubuntu machine which fails to deploy using the “dynamic” user-data API
  • host B: your Ubuntu machine which you deployed successfully using the “static” user-data template at /pub

What about the two log files? Does the request from host A show up in any of the two files mentioned?

AFAIK, there shouldn’t be any configuration necessary. So, I assume there might be still an error with either (a) your host correctly reaching the foreman or (b) the foreman determining the template of your host correctly.


Checking (a)

In order to check (a), please have a look at the log files again as mentioned before. If the request shows up, the problem might be (b).

Did you also try to wget other APIs of the foreman from your host B to verify it can reach the foreman in general? This shouldn’t be an issue since that machine was already able to deploy using the template in /pub, but let’s just verify it can also successfully reach for example wget http://foreman.example.com (might wanna add a --no-check-certificate).

Checking (b)

In case you see the request of host A in any of your log files, we can try the following:

The foreman returns the userdata template depending on the sender IP in that respective request. Accordingly, only the actual host can retrieve its own template since only that host has the corresponding sender IP (let’s leave IP spoofing aside). But, the foreman also allows adding the MAC as an additional API parameter - this allows us to retrieve the userdata template of other hosts since the foreman does not evaluate the IP of the request if the MAC is given explicitly:

From your foreman, make a request including the MAC of host A (the MAC should show up in your webui right after deploying the machine even if the machine cannot be deployed successfully):

wget http://foreman.example.com/userdata/01:02:03:04:05:06/user-data

Does this request return anything? If yes, we can adapt the iPXE template and add the MAC to the request during the deployment of the host.

Hey @bastian-src I am struggling with the same issue Checking (a) @jch5625 is seeing with Ubuntu 22.04 provising with Foreman running on nginx specifically the userdata portion where I am getting 404 unless I specify the MAC address which according to multiple blogs and posts I have seen that no one is specifying.

If it is not a lot to ask, can you please outline the steps you took to get Ubuntu 22.04 working with Foreman??

I want to use the use the Foremans Dynamic user data API and we do not have static IPs and let dhcp handle those.

I also have foreman-proxy with tftp enabled running on foreman server itself, I have tried doing trusted_proxies under foremans settings.yml and also editing the proxy configs under foreman-proxy/settings.d/templates.yml and enabling tempales with speciying template_url on port 8000.

Any help is appreciated as I am grasing on straws here.

This is my issue if you are curious -Need help with PXE booting Ubuntu 22.04 on Foreman v 3.5 (nginx), getting 404 Not Found on userdata