How to to BMC provisioning MAAS style?

Problem:
MAAS has very interesting workflow where it makes sure that BMC is provisioned (with custom user and password) before operating system provisioning takes place. This is very useful, because when large batch of servers comes you just plug your machines and it automatically provisions your BMC, auto discovers facts and the powers off machine through BMC till the time when admin is ready to deploy operating system on it (in our case this could be weeks) and then it even automatically sets boot to pxe. You can even run hardware tests BEFORE installing operating system to make sure machines are not faulty or DOA. It is not acceptable for machine to be powered on and sitting in discovery for weeks. It is not acceptable to install operating system right away because we don’t know fqdn for those machines till we need them. I can’t imagine how to do such workflow in foreman. It seems discovery image doesn’t do anything with BMC (except only discovering network details of already configured BMC interface) and there is no place in the workflow before installing operating system itself, when/where to configure BMC. Help? :slight_smile:

Expected outcome:
BMC is configured before operating system is installed. BMC ip should be taken from foremans ipam.

Foreman and Proxy versions:
1.23

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

Hey, I assume you mean Ubuntu MaaS (because MAAS is a generic term).

No, Foreman discovery does not support anything like that at the moment. As much as I would like to work on this, I have a different priority in regard to discovery. I will redesign it to utilize Foreman Remote Execution feature, so after the changes users will be able to run arbitrary scripts from Foreman via templates. This will enable the feature you mention as “run hardware tests BEFORE”.

Adding BMC handling into discovery should not be that hard, discovery OS currently detects IPMI and reports it back, BMC interface is even created for a discovered host. What’s missing is username and password and functionality that would actually allow turning them on or off. Patches welcome.

there is a potential different flow, where a user just inputs a bmc
username / password and a ipmi endpoint (or maybe redfish) and that would
ask foreman to turn the server on, run discovery on it (via pxe or perhaps
virtual media option?) and start the process this way…?

1 Like

This is bad because it requires user intervention by default and assumes that BMC is configured in any way from factory - this assumption may not always be correct. Automation is god, user should only be required to intervene in exceptional circumstances.

MaaS workflow is closer to being ideal. First it “enlists” machine (this is BMC provisioning step). It uses ipmi by default. If that fails, enlisting enters failed state. From gui there are several predefined settings for some common exceptions from different manufacturers (providing manual configuration is also possible) - you can configure one of these and try enlisting again - all this can be scripted using their cli - really nice! Now MaaS controls machine power - it can reboot/power off machine after every step below (sometimes useful especially after automatic firmware update.
Second step is commisioning (in foremans terms this is auto-discovery equivalent).
Third steps (all optional): hardware test/firmware update/hba configuration (eg hardware raid volume creation).
Fourth step is system image deployment.
Fifth step is machine decomissioning with optional secure erase of disks (really nice feature).

Ideally login credentials should be generated, stored and periodically updated automatically (eg using something like Hashicorp Vault or some other secret management service). Some simpler solution like default user and pass for all BMCs would be acceptable (it could always be changed later per host using configuration management system).

1 Like

We know that Foreman has quite limited BMC and discovery features, on the other hand we beat most of the other projects in literally every other aspect. Foreman is very flexible, can do lots of things and it’s not based only around image-based provisioning.

In Foreman BMC are not separate entities, they are attributes of a host (a nic specifically). So this makes the pre-provisioning and discovery phase quite limiting. On the other hand, there are many users who are building on commodity hardware without BMC cards and that’s our main focus group. In extreme, Foreman users actually don’t need BMC cards because Foreman manages DHCP/TFTP/PXE and can effectively reprovision systems and this can be fully automated.

I believe with the new discovery design which should arrive in early 2020 the following workflow should be possible:

  1. A host is discovered and reports to Foreman
  2. Operator defined a post-discovery template which executes over SSH
  3. The template renders to a script which tries to detect IPMI device on the discovered node and tries “expected” username/passwords and sets up new password if needed
  4. Once a match is found, BMC NIC attached to the node is updated with credentials
  5. Optionally hardware test can be performed or firmware can be updated (anything it’s a shell script)
  6. The host stays powered on until provisioning (*)
  7. Another arbitrary SSH script can be called at any point (manually or from cron) and/or during provisioning

The last step could be solved by implementing BMC actions for discovered hosts (power off, power on) and this should not be too complicated as the infrastructure is already in place (Subnet - BMC Proxy - Foreman Proxy - ipmitool/freeipmi - discovered node).

I believe this is quite close to expected scenario, it’s not perfect but a viable workflow I think. Do you miss any important things there?

That mostly sounds good, but one of the big things with Ubuntu MAAS is that it changes the BMC credentials and gives you a unique pair per server. This improves (or rather can improve) security a fair bit.

You could do it of course. We do have a random string generation in our template context and it can be stored in a NIC (probably need a new macro to be exposed).

We know that Foreman has quite limited BMC and discovery features, on the other hand we beat most of the other projects in literally every other aspect. Foreman is very flexible, can do lots of things and it’s not based only around image-based provisioning.

I agree with all of the above. I just want Foreman to fit a little bit better into my use case.

On the other hand, there are many users who are building on commodity hardware without BMC cards and that’s our main focus group.

That’s fair.

I believe this is quite close to expected scenario, it’s not perfect but a viable workflow I think. Do you miss any important things there?

Post-discovery template seems ok, but how will I know if/where it failed? Logs from this post-discovery run should be available somewhere.

Absolutely, our Remote Execution framework have extensive reporting capabilities.

Alright, thank you for your time. Can’t wait to test it out :smiley: