Ubuntu host deployment

Problem:

Making slow progress with Deploying Ubuntu (22.04) from Foreman with Katello. I can deploy Ubuntu and enrol the new host into FreeIPA. But I get stuck trying to get it to use a Katello repo instead of an official Ubuntu repo. Hence I’m unable to get the Atix Foreman Client installed. But I’m doubting whether I need this as Foreman appears to be shifting to REX, which is fine by me. But this is where I then encounter issues, if I test REX by running something like “upload package profile for a host”. I always get an SSH error back. It appears that the SSH keys that are generated during the installation are rewritten after the first reboot. And thus the details in /var/lib/sss/pubconf/known_hosts are wrong.

Expected outcome:

Host is deployed, installed and REX works without needing manual intervention. And the Foreman client is installed, or REX just works without errors of missing executables.

Foreman and Proxy versions:

Foreman 3.13
Katello 4.15

Distribution and version:

CentOS 9 Stream

Other relevant data:

Ubuntu 22.04 LTS Operating System Templates:

  • Finish template: Preseed default finish
  • Host initial configuration template: Linux host_init_config default
  • iPXE template: Preseed default iPXE Autoinstall
  • PXEGrub2 template: Preseed default PXEGrub2 Autoinstall
  • PXELinux template: Preseed default PXELinux Autoinstall
  • User data template: Preseed Autoinstall cloud-init user data MM online

The User data template is cloned from Preseed Autoinstall cloud-init user data.

A couple of edits:

  • SSH pwd login disabled.
  • Changed root to a non-root user.
  • Removed the blocking of normal Ubuntu repos. By commenting out disable_components and disable_suites.
  • Removed i386 arch.
<%#
kind: user_data
name: Preseed Autoinstall cloud-init user data
model: ProvisioningTemplate
oses:
- ubuntu
test_on:
- ubuntu_autoinst4dhcp
description: |
  The provisioning template for Autoinstall based distributions. To customize the installation,
  modify the host parameters

  This template accepts the following parameters:
  - lang: string (default="en_US.UTF-8")
  - keyboard: string (default="us")
  - package_upgrade: boolean (default=false)
  - remote_execution_ssh_keys: string (default="")
  - username_to_create: string (default="root")
  - realname_to_create: string (default=username_to_create)
  - password_to_create: string (default=@host.root_pass)
-%>
<%-
username_to_create = host_param('username_to_create', 'local-admin')
realname_to_create = host_param('realname_to_create') || username_to_create
password_to_create = host_param('password_to_create') || @host.root_pass
enable_auto_update = (host_param_true?('package_upgrade') && !host_param('kt_activation_keys'))
os_major = @host.operatingsystem.major.to_i
os_minor = @host.operatingsystem.minor.to_i
-%>
#cloud-config
autoinstall:
  version: 1
  apt:
    geoip: false
    preserve_sources_list: false
    primary:
      - arches: [amd64]
        uri: http://archive.ubuntu.com/ubuntu
      - arches: [default]
        uri: http://ports.ubuntu.com/ubuntu-ports
<% unless enable_auto_update -%>
    disable_components: [] <%# [multiverse] %>
    disable_suites: [] <%# [backports,security,updates] %>
    fallback: offline-install
<% end -%>
<%= indent(4) { snippet_if_exists(template_name + " custom apt") } -%>
  user-data:
    disable_root: true
    fqdn: <%= @host.name %>
    users:
    - name: <%= username_to_create %>
      gecos: <%= realname_to_create %>
      lock_passwd: false
      hashed_passwd: <%= password_to_create %>
<% if !host_param('remote_execution_ssh_keys').blank? -%>
<%   if host_param('remote_execution_ssh_keys').is_a?(String) -%>
      ssh_authorized_keys: [<%= host_param('remote_execution_ssh_keys') %>]
<%   else -%>
      ssh_authorized_keys: <%= host_param('remote_execution_ssh_keys') %>
<%   end -%>
<% end -%>
  keyboard:
    layout: <%= host_param('keyboard', 'us') %>
    toggle: null
    variant: ''
  locale: <%= host_param('lang', 'en_US') %>.UTF-8
<%= snippet 'preseed_netplan_setup' -%>
  ssh:
    allow-pw: false
    install-server: true
  updates: security
<%= indent(2) { @host.diskLayout } %>
<%= indent(2) { snippet_if_exists(template_name + " custom root") } -%>
  late-commands:
<%= indent(2) { snippet 'preseed_autoinstall_clevis_tang_wrapper' if host_param('disk_enc_tang_servers') && os_major >= 22 && os_minor >= 3 } %>
  - wget -Y off <%= @static ? "'#{foreman_url('finish', static: 'true')}'" : foreman_url('finish') %> -O /target/tmp/finish.sh
  - curtin in-target -- chmod +x /tmp/finish.sh
  - curtin in-target -- /tmp/finish.sh

The content in /var/lib/sss/pubconf/known_hosts regularly changes and appears to have little bearing on whether Foreman can SSH to a host.

Job invocation error:

Error initializing command: RuntimeError - Could not establish connection to remote host using any available authentication method, tried publickey

Authentication method 'publickey' failed with:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:uG1FBXL6rF3GTNodZj+fsO2NGPfN07d3LTuEBQ4fGMs.
Please contact your system administrator.
Add correct host key in /usr/share/foreman-proxy/.ssh/known_hosts to get rid of this message.
Offending RSA key in /var/lib/sss/pubconf/known_hosts:3
Host key for sheri-bozic.gchar.mm.eu has changed and you have requested strict checking.
Host key verification failed.
Exit status: EXCEPTION
StandardError: Job execution failed

I removed the content from the known_hosts file and could then ssh from Foreman to this host, and Foreman could as well. But I’d prefer not to have to do manual edits of this file for each created host. And then at times Foreman appears to remove the lines independently, but still can’t ssh to a host.

It seems I may have found the REX issue: linux - Still getting “ssh failure: Offending RSA key in /var/lib/sss/pubconf/known_hosts” despite removing the specified key from “known_hosts” - Super User

This confirms my suspicion of the SSH keys being regenerated on first boot. I’ll now need to find which template is responsible for that as there’s no need for this when doing a clean install. And it causes the known_hosts file which is dynamically populated by sssd to contain the ssh keys from when the host was enrolled.

Manually updating the “SSH public keys” in IPA is a workaround for the known hosts issue for REX SSH connections. But I don’t get that while IPA sets the keys during enrolment, post-install subiquity regenerates the keys.

/var/log/installer/installer-journal.txt:Dec 24 13:07:03 saul-prokos.gchar.mm.eu subiquity_log.2036[28030]: Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
/var/log/installer/installer-journal.txt:Dec 24 13:07:03 saul-prokos.gchar.mm.eu subiquity_log.2036[28030]: Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
/var/log/installer/installer-journal.txt:Dec 24 13:07:03 saul-prokos.gchar.mm.eu subiquity_log.2036[28030]: Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
/var/log/cloud-init.log:2024-12-24 13:09:54,333 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_rsa_key.pub
/var/log/cloud-init.log:2024-12-24 13:09:54,333 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ecdsa_key
/var/log/cloud-init.log:2024-12-24 13:09:54,334 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ecdsa_key.pub
/var/log/cloud-init.log:2024-12-24 13:09:54,334 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ed25519_key
/var/log/cloud-init.log:2024-12-24 13:09:54,334 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_rsa_key
/var/log/cloud-init.log:2024-12-24 13:09:54,334 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ed25519_key.pub

Cloud-init regenerates the host keys on first boot and adding ssh_deletekeys: false to #cloud-config/autoinstall in Preseed Autoinstall cloud-init user data doesn’t seem to do anything for Ubuntu 22.04 deployments. I’ll try with Ubuntu 24.04 next.

Ubuntu 24.04 has the same issue. Is there anyone else who enrols their hosts into FreeIPA at deployment time?

What wasn’t clear to me from the cloud-init docs is exactly where the ssh_deletekeys statement needs to go.

After some trial and error I found it works under user-data on Ubuntu 24.04, I’ll be testing Ubuntu 22.04 next:

#cloud-config
autoinstall:
  version: 1
  ...
  user-data:
    ssh_deletekeys: false