Unfinished build puts host into reboot loop

Hi,

After I made a modification to the post-install part of our provisioning
template that appears to be invalid.

The deployment fails and reboots. And because the build never finished
and we have the servers set to boot from PXE, the host starts the
install again and then fails again, then reboots again.

Is there a way to stop it from rebooting and send the error to the
foreman? Or can we hook it up to a logserver during install?

Continuous rebooting puts unnecessary strain on our server components
and additional load on our management infrastructure.

Kind regards,

Jorick Astrego

Met vriendelijke groet, With kind regards,

Jorick Astrego

Netbulae Virtualization Experts

··· ----------------
Tel: 053 20 30 270 	info@netbulae.eu 	Staalsteden 4-3A 	KvK 08198180
Fax: 053 20 30 271 	www.netbulae.eu 	7547 TA Enschede 	BTW NL821234584B01

sounds like a feature request to me, would you mind creating one?

thanks,
Ohad

··· On Wed, Dec 17, 2014 at 12:57 PM, Jorick Astrego wrote: > > Is there a way to stop it from rebooting and send the error to the > foreman? Or can we hook it up to a logserver during install? >

> Hi,
>
> After I made a modification to the post-install part of our provisioning
> template that appears to be invalid.
>
> The deployment fails and reboots. And because the build never finished
> and we have the servers set to boot from PXE, the host starts the
> install again and then fails again, then reboots again.
>
> Is there a way to stop it from rebooting and send the error to the
> foreman? Or can we hook it up to a logserver during install?

The only way Foreman knows to stop the build is if the call to the built
URL is made (one of the last steps in the templates). Anaconda doesn't
have any feature like "in case install fails, do this" that I know of.

Maybe we could have some option like turn off build mode after last
needed template is requested, maybe an issue you could open on redmine.

··· On Wed, Dec 17, 2014 at 11:57:25AM +0100, Jorick Astrego wrote:

Continuous rebooting puts unnecessary strain on our server components
and additional load on our management infrastructure.

Kind regards,

Jorick Astrego

Met vriendelijke groet, With kind regards,

Jorick Astrego

Netbulae Virtualization Experts


Tel: 053 20 30 270 info@netbulae.eu Staalsteden 4-3A KvK 08198180
Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01



You received this message because you are subscribed to the Google Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.


Stephen Benjamin


Red Hat GmbH | http://de.redhat.com/ | Sitz: Grasbrunn
Handelsregister: Amtsgericht München, HRB 153243
Geschäftsführer: Charles Cachera, Michael Cunningham,
Michael O’Neill, Charles Peters

The best solution is a provisioning state machine within Foreman, so
that you can detect things like "this host has requested
/unattended/provision 3 times in a row without ever finishing the
built, it probably has a problem". You can then take actions like
removing the pxelinux template for failed hosts, or redirecting them
to another boot image, and notifying users etc. It's a lot of work
though, I looked at it a while ago - DynFlow will help, probably.

Greg

··· On 17 December 2014 at 13:59, Stephen Benjamin wrote: > On Wed, Dec 17, 2014 at 11:57:25AM +0100, Jorick Astrego wrote: >> Hi, >> >> After I made a modification to the post-install part of our provisioning >> template that appears to be invalid. >> >> The deployment fails and reboots. And because the build never finished >> and we have the servers set to boot from PXE, the host starts the >> install again and then fails again, then reboots again. >> >> Is there a way to stop it from rebooting and send the error to the >> foreman? Or can we hook it up to a logserver during install? > > The only way Foreman knows to stop the build is if the call to the built > URL is made (one of the last steps in the templates). Anaconda doesn't > have any feature like "in case install fails, do this" that I know of.

I opened a feature request here Feature #8780: Add a way to prevent reboot on failed deploy - Foreman

When I think of a way to implement it easily, I'll update the issue :wink:

Kind regards,

Jorick Astrego

Met vriendelijke groet, With kind regards,

Jorick Astrego

Netbulae Virtualization Experts

··· On 12/17/2014 03:11 PM, Greg Sutcliffe wrote: > On 17 December 2014 at 13:59, Stephen Benjamin wrote: >> On Wed, Dec 17, 2014 at 11:57:25AM +0100, Jorick Astrego wrote: >>> Hi, >>> >>> After I made a modification to the post-install part of our provisioning >>> template that appears to be invalid. >>> >>> The deployment fails and reboots. And because the build never finished >>> and we have the servers set to boot from PXE, the host starts the >>> install again and then fails again, then reboots again. >>> >>> Is there a way to stop it from rebooting and send the error to the >>> foreman? Or can we hook it up to a logserver during install? >> The only way Foreman knows to stop the build is if the call to the built >> URL is made (one of the last steps in the templates). Anaconda doesn't >> have any feature like "in case install fails, do this" that I know of. > The best solution is a provisioning state machine within Foreman, so > that you can detect things like "this host has requested > /unattended/provision 3 times in a row without ever finishing the > built, it probably has a problem". You can then take actions like > removing the pxelinux template for failed hosts, or redirecting them > to another boot image, and notifying users etc. It's a lot of work > though, I looked at it a while ago - DynFlow will help, probably. > > Greg > ----------------
Tel: 053 20 30 270 	info@netbulae.eu 	Staalsteden 4-3A 	KvK 08198180
Fax: 053 20 30 271 	www.netbulae.eu 	7547 TA Enschede 	BTW NL821234584B01