Foreman Installed on Ubuntu 20.04 provision 22.04 version

gol19 · December 23, 2022, 12:22pm

Problem:
Cloud-init problem with Ubuntu 22.04

Expected outcome:

Foreman and Proxy versions:
3.4.1
Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

I have problem with Ubuntu22.04 provisioning through Foreman 3.4.1 installed on Ubuntu 20.04.
Installation stuck on cloud-init and getting below error, I highly suspect that this error is not in relaiton with foreman, but as I did not found solution anywhere just checking if anybody had anything similar

2022-12-23T10:32:20 [I] WEBrick::HTTPServer#start: pid=526 port=8443
2022-12-23T10:32:20 [I] Smart proxy has launched on 1 socket(s), waiting for requests
2022-12-23T11:03:58 21d02baf [I] Started GET /tftp/serverName
2022-12-23T11:03:58 21d02baf [I] Finished GET /tftp/serverName with 200 (6.29 ms)
2022-12-23T11:03:58 21d02baf [I] Started GET /dhcp/10.237.9.0/mac/56:6f:c0:c5:00:a8
2022-12-23T11:03:58 21d02baf [I] Finished GET /dhcp/10.237.9.0/mac/56:6f:c0:c5:00:a8 with 200 (1.45 ms)
2022-12-23T11:03:58 21d02baf [I] Started GET /dhcp/10.237.9.0/ip/10.237.9.132
2022-12-23T11:03:58 21d02baf [I] Finished GET /dhcp/10.237.9.0/ip/10.237.9.132 with 200 (0.92 ms)
2022-12-23T11:03:58 21d02baf [I] Started POST /tftp/PXELinux/56:6f:c0:c5:00:a8
2022-12-23T11:03:58 21d02baf [I] Finished POST /tftp/PXELinux/56:6f:c0:c5:00:a8 with 200 (12.11 ms)
2022-12-23T11:03:58 21d02baf [I] Started POST /tftp/PXEGrub2/56:6f:c0:c5:00:a8
2022-12-23T11:03:58 21d02baf [I] Finished POST /tftp/PXEGrub2/56:6f:c0:c5:00:a8 with 200 (3.83 ms)
2022-12-23T11:03:58 21d02baf [I] Started POST /tftp/PXEGrub/56:6f:c0:c5:00:a8
2022-12-23T11:03:58 21d02baf [I] Finished POST /tftp/PXEGrub/56:6f:c0:c5:00:a8 with 200 (2.66 ms)
2022-12-23T11:03:58 21d02baf [I] Started POST /tftp/iPXE/56:6f:c0:c5:00:a8
2022-12-23T11:03:58 21d02baf [I] Finished POST /tftp/iPXE/56:6f:c0:c5:00:a8 with 200 (0.81 ms)
2022-12-23T11:04:04 fe36c89c [I] Started GET /v2/features
2022-12-23T11:04:04 fe36c89c [I] Finished GET /v2/features with 200 (4.76 ms)
2022-12-23T11:04:07 0623ff53 [I] Started GET /tftp/serverName
2022-12-23T11:04:07 0623ff53 [I] Finished GET /tftp/serverName with 200 (0.49 ms)
2022-12-23T11:04:07 0623ff53 [I] Started GET /dhcp/10.237.9.0/mac/56:6f:c0:c5:00:a8
2022-12-23T11:04:07 0623ff53 [I] Finished GET /dhcp/10.237.9.0/mac/56:6f:c0:c5:00:a8 with 200 (1.31 ms)
2022-12-23T11:04:07 0623ff53 [I] Started GET /dhcp/10.237.9.0/ip/10.237.9.132
2022-12-23T11:04:07 0623ff53 [I] Finished GET /dhcp/10.237.9.0/ip/10.237.9.132 with 200 (1.41 ms)
2022-12-23T11:04:07 0623ff53 [I] Started GET /dhcp/10.237.9.0/mac/56:6f:c0:c5:00:a8
2022-12-23T11:04:07 0623ff53 [I] Finished GET /dhcp/10.237.9.0/mac/56:6f:c0:c5:00:a8 with 200 (0.54 ms)
2022-12-23T11:04:07 0623ff53 [I] Started POST /tftp/PXELinux/56:6f:c0:c5:00:a8
2022-12-23T11:04:07 0623ff53 [I] Finished POST /tftp/PXELinux/56:6f:c0:c5:00:a8 with 200 (0.76 ms)
2022-12-23T11:04:07 0623ff53 [I] Started POST /tftp/fetch_boot_file
2022-12-23T11:04:07 0623ff53 [I] Finished POST /tftp/fetch_boot_file with 200 (5.23 ms)
2022-12-23T11:04:07 0623ff53 [I] [2125] Started task /usr/bin/curl\ –silent\ –show-error\ –connect-timeout\ 10\ –retry\ 3\ –retry-delay\ 10\ –max-time\ 3600\ –remote-time\ –time-cond\ /srv/tftp/boot/ubuntu2204-yJLsSoEw1b0Y-vmlinuz\ –write-out\ Task\ done,\ result:\ %{http_code},\ size\ downloaded:\ %{size_download}b,\ speed:\ %{speed_download}b/s,\ time:\ %{time_total}ms\ –output\ /srv/tftp/boot/ubuntu2204-yJLsSoEw1b0Y-vmlinuz\ –location\ http://10.237.9.31/pub/installation_media/ubuntu/22.04-x86_64//casper/vmlinuz
2022-12-23T11:04:07 0623ff53 [I] Started POST /tftp/fetch_boot_file
2022-12-23T11:04:07 0623ff53 [I] Finished POST /tftp/fetch_boot_file with 200 (0.89 ms)
2022-12-23T11:04:07 0623ff53 [I] [2127] Started task /usr/bin/curl\ –silent\ –show-error\ –connect-timeout\ 10\ –retry\ 3\ –retry-delay\ 10\ –max-time\ 3600\ –remote-time\ –time-cond\ /srv/tftp/boot/ubuntu2204-yJLsSoEw1b0Y-initrd\ –write-out\ Task\ done,\ result:\ %{http_code},\ size\ downloaded:\ %{size_download}b,\ speed:\ %{speed_download}b/s,\ time:\ %{time_total}ms\ –output\ /srv/tftp/boot/ubuntu2204-yJLsSoEw1b0Y-initrd\ –location\ http://10.237.9.31/pub/installation_media/ubuntu/22.04-x86_64//casper/initrd

Dann · January 17, 2023, 6:04pm

Hi @gol19

i just had the exact same problem and by going in the shell, i noticed a OOM for cloud-init
my VM had 4Gb, i upped it to 8Gb and it solved the issue

you could try that

good luck

gol19 · January 17, 2023, 7:50pm

I did same, 5GB was enough, do you know maybe if there is any option for ubuntu installation with lower memory

Dann · January 17, 2023, 8:13pm

not that i know, maybe by playing with some cloud-init options if that even possible…

(looks like ubuntu team if making every effort to change everything at every release, i hate this distribution)

i have another question of my own, do you know if we can get unattented installation with 22.04 and foreman ? the default preseed doesn’t seem to work…

thanks

gol19 · January 18, 2023, 9:09am

Hi, I installed it without many problem after pass that cloud init issue.
can you just tell me what version of foreman you are using. on last version 3.4.1 i succeded it.

Dann · January 18, 2023, 12:36pm

it’s also version 3.4.1 i’m using, but i guess i must have messed up the install somehow

first of all i reinstall foreman after having installing it once, and now the SSL certificates are no longer recognized, and more importantly i guess my issue is with the /userdata access from the pxelinux.cfg file

i have this :
APPEND ip=dhcp url=http://foreman.me-corp.lan:80/pub/installation_media/ubuntu/22.04-x86_64.iso autoinstall ds=nocloud-net;s=http://foreman.me-corp.lan/userdata/ root=/dev/ram0 ramdisk_size=1500000 fsck.mode=skip

but when i tried to reach this : http://foreman.me-corp.lan/userdata/ from a browser, i’m getting a 404 error.
ActionController::RoutingError (No route matches [GET] "/userdata"):
in the foreman production.log

i’m sure this is a stupid error but i can’t find it yet if you have an idea i’ll take it

thanks

gol19 · January 18, 2023, 1:32pm

can you open in apache new website like below example and restart apache2
Also please follow guide for other steps
https://docs.orcharhino.com/or/docs/sources/guides/ubuntu/provisioning_hosts/configuring_provisioning_resources.html
Inform me if you stuck somewhere else

root@foreman:/etc/apache2# cat sites-available/06-pub.conf
Alias /pub /var/www/html/pub

<Location /pub>

PassengerEnabled off

Options +FollowSymLinks +Indexes
Require all granted

then enable the site, and the alias and autoindex apache2 mods

a2enmod alias
a2enmod autoindex
a2ensite 06-pub.conf

systemctl restart apache2

Dann · January 18, 2023, 6:23pm

i already made the site for /pub which works fine (i can start the install ubuntu 22 with no issue)

my problem is the autoinstall doesn’t work, i arrive to the dialog like i was using the ISO manually…

gol19 · January 19, 2023, 10:35am

I beleive you stuck at the point where it ask you for language, so if you use latest foreman 3.4.1
I am using below templates
Finish Template: Preseed default finish
PXELinux Template: Preseed default PXELinux Autoinstall
User data template: Preseed Autoinstall cloud-init

Also use autoinstall partition table. For me worked everything with this out of box. then you can tweak it to be good for your situation.
If this will not work try review what each template doing for specifed VM, send me if you stuck somewhere else
try replace foreman.me-corp.local in foreman settings with IP address

wrender · January 19, 2023, 6:19pm

I agree it sounds like you could be missing User data template: Preseed Autoinstall cloud-init

When I tested this template it provisioned Ubuntu 22.04 perfectly for me.

I would make sure to add this template to your Ubuntu 22.04 Operating System, delete your host entry and then re-create it.

Another thing is make sure to have an IP assigned to your host. I think it will only generate the userdata if you have assigned an IP address. For a while I was only putting MAC address in and it would generate the pxe files but not the userdata.

Dann · January 23, 2023, 2:57pm

Hi @gol19 @wrender

sadly i’m not missing anything you asked me to check

it seems the error i’m getting is that the host i’m trying to install doesn’t receive the template

i found on another thread this command that i tried to run from my host which indeed get stopped asking me the language (meaning not auto provisioning) :

curl -D - http://theforeman.xxxxxxx.xxx/userdata/meta-data

this command works fine on the foreman server itself and return this (meaning it finds the template i guess) :

HTTP/1.1 200 OK
Date: Mon, 23 Jan 2023 13:58:18 GMT
Server: Apache
Content-Type: text/plain; charset=utf-8
ETag: W/"97e82732b6a2aee20dae3492a5f20e8a"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: 5d5c4ed3-f9a3-4a80-bd0c-b10056b67556
X-Runtime: 0.042104
Via: 1.1 theforeman.xxxxxxxx.xxx
Transfer-Encoding: chunked

instance-id: i-356a192b7913b04c54
hostname: theforeman.xxxxxxxx.xxx
mac: ea:36:9a:5a:3b:cc
local-ipv4: 10.155.6.201
local-hostname: theforeman.xxxxxxx.xxx

but on the host i’m trying to install (i open a shell just after seeing the language option), i’m getting a 404, so i guess the problem is there…
only problem is, i have no idea where to solve it.

root@ubuntu-server:~# curl -D - -H theforeman.xxxxx.xx http://theforeman.xxxxxxx.xxx/userdata/meta-data
HTTP/1.1 404 Not Found
Date: Mon, 23 Jan 2023 13:58:00 GMT
Server: Apache
Content-Type: text/plain; charset=utf-8
Cache-Control: no-cache
X-Request-Id: 92ae4fc4-eb54-4f7d-b260-eb352f631792
X-Runtime: 0.013141
Via: 1.1 theforeman.xxxxxxxx.xxx
Transfer-Encoding: chunked

Could not find host for request 10.155.6.50

i tried to add the smart proxy in the trusted proxy in /etc/foreman/settings.yml, it didn’t change anything :

:trusted_proxies:
- '10.155.6.201/32'

this is what i get in the production.log itself about this error :


2023-01-23T14:06:00 [I|app|7b274a99] Started GET "/userdata/meta-data" for 10.155.6.50 at 2023-01-23 14:06:00 +0000
2023-01-23T14:06:00 [I|app|7b274a99] Processing by UserdataController#metadata as TEXT
2023-01-23T14:06:00 [E|app|7b274a99] Could not find host for request 10.155.6.50
2023-01-23T14:06:00 [I|app|7b274a99]   Rendered text template (Duration: 0.0ms | Allocations: 2)
2023-01-23T14:06:00 [I|app|7b274a99] Filter chain halted as :find_host rendered or redirected
2023-01-23T14:06:00 [I|app|7b274a99] Completed 404 Not Found in 10ms (Views: 0.4ms | ActiveRecord: 2.7ms | Allocations: 2333)

thanks for your help, this is infuriating that you said it worked out of the box for you

gol19 · January 24, 2023, 9:51am

can you add in trusted proxies also
:1 and 127.0.0.1

Can you review userdata if is generated correctly on template tab, and try create simple userdata template for test like below, to see if vm will pull it at all

#cloud-config
autoinstall:
version: 1
identity:
hostname: ubuntu-server
password: “$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0”
username: ubuntu

Dann · January 25, 2023, 11:44am

i’ll try the simple userdata, but i’m not very optimistic, as in my opinion foreman doesn’t even communicate with the thing…
i’m also trying to add the trusted proxy but for some reason now foreman-installer detects issues in configuration…

anyway on some other point, what foreman-install options/scenarios did you use when you said you installed ubuntu 22.04 out of the box ?

thanks

gol19 · January 25, 2023, 12:04pm

Good lick with this,

I used DHCP BMC and TFTP, it was enough for me as we are using foreman only for provisioning

Dann · January 25, 2023, 12:12pm

any chance you still have the foreman-installer command line ?

i’m considering reinstalling a third time…

gol19 · January 25, 2023, 12:34pm

on ubuntu is painful tshoot foreman-installer error

I do not have foreman installer, but add dhvp and tftp it should be no error,

add like this in trusted proxies
trusted_proxies:

127.0.0.1
“::1”
10.237.9.31 - your ip

Dann · January 30, 2023, 9:06am

it seems i got confused with the foreman-installer switches, using --foreman-proxy-trusted-hosts was not the same as using the line “trusted_proxies” directly in /etc/foreman-proxy/settings.yml

i’m not exactly sure what happened, after reinstalling foreman on centOS (as it seems to be preferred) and simply running foreman-installer, adding afterwards the config options i needed, also by not specifying/modifying the template URL in the preseed, it finally worked

thanks a lof for all your help, it seems the deciding key was the trusted_proxies in the settings, that you have to manually add
the documentation is not really clear on this point, and also i find it weird to have to specifically “trust” your own ip address where the foreman UI and the foreman-proxy resides… seems like an overkill

anyway, thanks

Dann · January 30, 2023, 12:23pm

replying to myself (cause i can’t edit twice), the trusted_proxies needs to be in foreman/settings.yaml, not foreman-proxy

foreman_user264 · January 25, 2024, 8:41pm

Hey @Dann I am struggling a lot with getting Ubuntu 22.04 provising with Foreman running on nginx specifically the userdata portion where I am getting 404 unless I specify the MAC address - If it is not a lot to ask, can you please outline the steps you took to get Ubuntu 22.04 working with Foreman??

I also have foreman-proxy with tftp enabled running on foreman server itself, I have tried doing trusted_proxies under foremans settings.yml and also editing the proxy configs under foreman-proxy/settings.d/templates.yml and enabling tempales with speciying template_url on port 8000.

Any help is appreciated as I am grasing on straws here.

This is my issue if you are curious -Need help with PXE booting Ubuntu 22.04 on Foreman v 3.5 (nginx), getting 404 Not Found on userdata

Dann · January 26, 2024, 8:36am

Hi @foreman_user264

from memory, apart from the httpd.conf change (above in the thread answer from gol19) and the trusted_proxies settings, that was the defining change for me.

i think i also had to modify some templates, but i’m not sure it was needed for basic setup.

i don’t have the config files near me anymore but i can try to find out, it won’t be before sunday though sorry

good luck in the meantime, i know i struggled a lot with foreman, but once you get it running, it’s pretty fun

++