Upgrades with Kafo :-(

The strategy for upgrading Katello has been to basically yum update, run
component service migrations, and re-execute Katello installer. The
latter step is neccessary - we may introduce new components or adjust
configuration. For 2.1 it seems to work relatively well, I've done a
dozen upgrades or so by now. I think we're pretty much in the clear
for 1.7/2.1 but this is going to hit us hard for 2.2.

We rely on the answers file to be able to rerun katello-installer with
the user's previous selections. However, we make incompatible changes to
puppet modules - for example, Foreman will be releasing 3.0 of
puppet-foreman and puppet-foreman_capsule. These are neccessary but
painful. Here's an example.

Users running a Foreman 1.8 Smart Proxy have enabled services on their
capsules like dhcp, dns, etc. However, we've now introduced the
provisining template plugin – and at least in RHEL, Anaconda is limited
to kickstart over HTTP. So we've had to introduce a whole concept of
allowing some plugins (templates) to run on HTTP but not others (dhcp,
dns, etc).

This required a bunch of parameter name and value changes in the puppet
module-- which will cause problems with the existing answers files out
in the world.

Eric and I briefly talked on IRC about it and it sounds like we're going
to need some kind of tool to "migrate" answer files. Anyone else have
any suggestions or ideas about how to handle such scenarios?

··· -- Best Regards,

Stephen Benjamin
Red Hat Engineering

In the past I used to maintain a foreman installation which managed
itself and several smart proxies using the installer puppet modules by
writing a small wrapper class that was somewhere between an role and a
profile. Every now and then I updated the modules to their latest
version and looked at the changes introduced.

You want something very similar, but more automated. In the installer
situation the answers file equates to my wrapper class. So the task is
to migrate the answers file to get the installer to reach the same goal.

Currently the answers file is AFAIK complete, which means every value is
stored even if the user has not changed it. That was chosen so that even
after upgrading the defaults in the module, you would still end up with
the same system. The difficulty with that is when you do want defaults
to be updated. For example, when we phased out the passenger_scl
parameter, it was kept in for compatibility. Now every upgrading user
will get a deprecation warning even if the value was never changed.

Another approach would be to only store user changed values. It does
have risks that things change which would surprise the user, but
assuming the user doesn't change many values and we are conservative
with changing defaults (which we are!), you mostly need to worry about
removed parameters.

We have had a few discussions about this in the past, but that's spread
over various PRs. IIRC it was lzap, iNecas, Dominic, gwmngilfen and I
who were mostly involved in the discussions, but I'm sure I'm forgetting
someone.

··· On Fri, Jan 16, 2015 at 04:27:17PM +0100, Stephen Benjamin wrote: > The strategy for upgrading Katello has been to basically yum update, run > component service migrations, and re-execute Katello installer. The > latter step is neccessary - we may introduce new components or adjust > configuration. For 2.1 it seems to work relatively well, I've done a > dozen upgrades or so by now. I think we're pretty much in the clear > for 1.7/2.1 but this is going to hit us hard for 2.2. > > We rely on the answers file to be able to rerun katello-installer with > the user's previous selections. However, we make incompatible changes to > puppet modules - for example, Foreman will be releasing 3.0 of > puppet-foreman and puppet-foreman_capsule. These are neccessary but > painful. Here's an example. > > Users running a Foreman 1.8 Smart Proxy have enabled services on their > capsules like dhcp, dns, etc. However, we've now introduced the > provisining template plugin -- and at least in RHEL, Anaconda is limited > to kickstart over HTTP. So we've had to introduce a whole concept of > allowing some plugins (templates) to run on HTTP but not others (dhcp, > dns, etc). > > This required a bunch of parameter name and value changes in the puppet > module-- which will cause problems with the existing answers files out > in the world. > > Eric and I briefly talked on IRC about it and it sounds like we're going > to need some kind of tool to "migrate" answer files. Anyone else have > any suggestions or ideas about how to handle such scenarios?

> > The strategy for upgrading Katello has been to basically yum update, run
> > component service migrations, and re-execute Katello installer. The
> > latter step is neccessary - we may introduce new components or adjust
> > configuration. For 2.1 it seems to work relatively well, I've done a
> > dozen upgrades or so by now. I think we're pretty much in the clear
> > for 1.7/2.1 but this is going to hit us hard for 2.2.
> >
> > We rely on the answers file to be able to rerun katello-installer with
> > the user's previous selections. However, we make incompatible changes to
> > puppet modules - for example, Foreman will be releasing 3.0 of
> > puppet-foreman and puppet-foreman_capsule. These are neccessary but
> > painful. Here's an example.
> >
> > Users running a Foreman 1.8 Smart Proxy have enabled services on their
> > capsules like dhcp, dns, etc. However, we've now introduced the
> > provisining template plugin – and at least in RHEL, Anaconda is limited
> > to kickstart over HTTP. So we've had to introduce a whole concept of
> > allowing some plugins (templates) to run on HTTP but not others (dhcp,
> > dns, etc).
> >
> > This required a bunch of parameter name and value changes in the puppet
> > module-- which will cause problems with the existing answers files out
> > in the world.
> >
> > Eric and I briefly talked on IRC about it and it sounds like we're going
> > to need some kind of tool to "migrate" answer files. Anyone else have
> > any suggestions or ideas about how to handle such scenarios?
>
> In the past I used to maintain a foreman installation which managed
> itself and several smart proxies using the installer puppet modules by
> writing a small wrapper class that was somewhere between an role and a
> profile. Every now and then I updated the modules to their latest
> version and looked at the changes introduced.
>
> You want something very similar, but more automated. In the installer
> situation the answers file equates to my wrapper class. So the task is
> to migrate the answers file to get the installer to reach the same goal.
>
> Currently the answers file is AFAIK complete, which means every value is
> stored even if the user has not changed it. That was chosen so that even
> after upgrading the defaults in the module, you would still end up with
> the same system. The difficulty with that is when you do want defaults
> to be updated. For example, when we phased out the passenger_scl
> parameter, it was kept in for compatibility. Now every upgrading user
> will get a deprecation warning even if the value was never changed.
>
> Another approach would be to only store user changed values. It does
> have risks that things change which would surprise the user, but
> assuming the user doesn't change many values and we are conservative
> with changing defaults (which we are!), you mostly need to worry about
> removed parameters.

This would make the problem easier to handle - does Kafo have an option
for this or is it something that needs implementing?

> We have had a few discussions about this in the past, but that's spread
> over various PRs. IIRC it was lzap, iNecas, Dominic, gwmngilfen and I
> who were mostly involved in the discussions, but I'm sure I'm forgetting
> someone.

Was there any outcome out of that? As long as Foreman follows semver and
changes parameters sparingly, it's not that big of a problem, but it is
one we probably need to consider – especially since Foreman docs also
now reccomend running foreman-installer post-upgrade.

It's a valueable thing to have – for example, when encrypted NoVNC
connections were enabled, if users were consistently running
foreman-installer, they'd have gotten that nice feature, but I imagine
most people who had older Foreman instances that were upgraded through
to 1.6, don't know that's available now.

··· On Fri, Jan 16, 2015 at 05:07:41PM +0100, Ewoud Kohl van Wijngaarden wrote: > On Fri, Jan 16, 2015 at 04:27:17PM +0100, Stephen Benjamin wrote:


You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Best Regards,

Stephen Benjamin
Red Hat Engineering

> > > The strategy for upgrading Katello has been to basically yum update, run
> > > component service migrations, and re-execute Katello installer. The
> > > latter step is neccessary - we may introduce new components or adjust
> > > configuration. For 2.1 it seems to work relatively well, I've done a
> > > dozen upgrades or so by now. I think we're pretty much in the clear
> > > for 1.7/2.1 but this is going to hit us hard for 2.2.
> > >
> > > We rely on the answers file to be able to rerun katello-installer with
> > > the user's previous selections. However, we make incompatible changes to
> > > puppet modules - for example, Foreman will be releasing 3.0 of
> > > puppet-foreman and puppet-foreman_capsule. These are neccessary but
> > > painful. Here's an example.
> > >
> > > Users running a Foreman 1.8 Smart Proxy have enabled services on their
> > > capsules like dhcp, dns, etc. However, we've now introduced the
> > > provisining template plugin – and at least in RHEL, Anaconda is limited
> > > to kickstart over HTTP. So we've had to introduce a whole concept of
> > > allowing some plugins (templates) to run on HTTP but not others (dhcp,
> > > dns, etc).
> > >
> > > This required a bunch of parameter name and value changes in the puppet
> > > module-- which will cause problems with the existing answers files out
> > > in the world.
> > >
> > > Eric and I briefly talked on IRC about it and it sounds like we're going
> > > to need some kind of tool to "migrate" answer files. Anyone else have
> > > any suggestions or ideas about how to handle such scenarios?
> >
> > In the past I used to maintain a foreman installation which managed
> > itself and several smart proxies using the installer puppet modules by
> > writing a small wrapper class that was somewhere between an role and a
> > profile. Every now and then I updated the modules to their latest
> > version and looked at the changes introduced.
> >
> > You want something very similar, but more automated. In the installer
> > situation the answers file equates to my wrapper class. So the task is
> > to migrate the answers file to get the installer to reach the same goal.
> >
> > Currently the answers file is AFAIK complete, which means every value is
> > stored even if the user has not changed it. That was chosen so that even
> > after upgrading the defaults in the module, you would still end up with
> > the same system. The difficulty with that is when you do want defaults
> > to be updated. For example, when we phased out the passenger_scl
> > parameter, it was kept in for compatibility. Now every upgrading user
> > will get a deprecation warning even if the value was never changed.
> >
> > Another approach would be to only store user changed values. It does
> > have risks that things change which would surprise the user, but
> > assuming the user doesn't change many values and we are conservative
> > with changing defaults (which we are!), you mostly need to worry about
> > removed parameters.
>
> This would make the problem easier to handle - does Kafo have an option
> for this or is it something that needs implementing?

AFAIK it needs implementing, though maybe kafo already understands
partial answer files.

> > We have had a few discussions about this in the past, but that's spread
> > over various PRs. IIRC it was lzap, iNecas, Dominic, gwmngilfen and I
> > who were mostly involved in the discussions, but I'm sure I'm forgetting
> > someone.
>
> Was there any outcome out of that? As long as Foreman follows semver and
> changes parameters sparingly, it's not that big of a problem, but it is
> one we probably need to consider – especially since Foreman docs also
> now reccomend running foreman-installer post-upgrade.

The current situation was the outcome, but IIRC there never was a clear
majority.

> It's a valueable thing to have – for example, when encrypted NoVNC
> connections were enabled, if users were consistently running
> foreman-installer, they'd have gotten that nice feature, but I imagine
> most people who had older Foreman instances that were upgraded through
> to 1.6, don't know that's available now.

You don't have to convince me :slight_smile:

··· On Mon, Jan 19, 2015 at 12:39:45PM +0100, Stephen Benjamin wrote: > On Fri, Jan 16, 2015 at 05:07:41PM +0100, Ewoud Kohl van Wijngaarden wrote: > > On Fri, Jan 16, 2015 at 04:27:17PM +0100, Stephen Benjamin wrote:

> > > > The strategy for upgrading Katello has been to basically yum update,
> > > > run
> > > > component service migrations, and re-execute Katello installer. The
> > > > latter step is neccessary - we may introduce new components or adjust
> > > > configuration. For 2.1 it seems to work relatively well, I've done a
> > > > dozen upgrades or so by now. I think we're pretty much in the clear
> > > > for 1.7/2.1 but this is going to hit us hard for 2.2.
> > > >
> > > > We rely on the answers file to be able to rerun katello-installer with
> > > > the user's previous selections. However, we make incompatible changes
> > > > to
> > > > puppet modules - for example, Foreman will be releasing 3.0 of
> > > > puppet-foreman and puppet-foreman_capsule. These are neccessary but
> > > > painful. Here's an example.
> > > >
> > > > Users running a Foreman 1.8 Smart Proxy have enabled services on their
> > > > capsules like dhcp, dns, etc. However, we've now introduced the
> > > > provisining template plugin – and at least in RHEL, Anaconda is
> > > > limited
> > > > to kickstart over HTTP. So we've had to introduce a whole concept of
> > > > allowing some plugins (templates) to run on HTTP but not others (dhcp,
> > > > dns, etc).
> > > >
> > > > This required a bunch of parameter name and value changes in the puppet
> > > > module-- which will cause problems with the existing answers files out
> > > > in the world.
> > > >
> > > > Eric and I briefly talked on IRC about it and it sounds like we're
> > > > going
> > > > to need some kind of tool to "migrate" answer files. Anyone else have
> > > > any suggestions or ideas about how to handle such scenarios?
> > >
> > > In the past I used to maintain a foreman installation which managed
> > > itself and several smart proxies using the installer puppet modules by
> > > writing a small wrapper class that was somewhere between an role and a
> > > profile. Every now and then I updated the modules to their latest
> > > version and looked at the changes introduced.
> > >
> > > You want something very similar, but more automated. In the installer
> > > situation the answers file equates to my wrapper class. So the task is
> > > to migrate the answers file to get the installer to reach the same goal.
> > >
> > > Currently the answers file is AFAIK complete, which means every value is
> > > stored even if the user has not changed it. That was chosen so that even
> > > after upgrading the defaults in the module, you would still end up with
> > > the same system. The difficulty with that is when you do want defaults
> > > to be updated. For example, when we phased out the passenger_scl
> > > parameter, it was kept in for compatibility. Now every upgrading user
> > > will get a deprecation warning even if the value was never changed.
> > >
> > > Another approach would be to only store user changed values. It does
> > > have risks that things change which would surprise the user, but
> > > assuming the user doesn't change many values and we are conservative
> > > with changing defaults (which we are!), you mostly need to worry about
> > > removed parameters.
> >
> > This would make the problem easier to handle - does Kafo have an option
> > for this or is it something that needs implementing?
>
> AFAIK it needs implementing, though maybe kafo already understands
> partial answer files.

AFAIK kafo understands partial answer files (that's how it works with the
default answer files coming from the packages). However, now it makes it complete
after the first installer run.

I've been talking briefly about this some time ago with Marek and storing
just the values that the user specified explicitly was an option that popped up,
but perhaps might not even be needed.

As an admin, what I would like to see is some kind of migration tool to run
against my old answer file and ideally keeping the new answer file in a format
that the diff against the new and old would give some useful outputs (especially
keeping the ordering of the parameters).

The problem with partial answer files is that one might rely on some default values
that might change though the versions and not see them coming. So it's useful to have
both data about customized values as well as the complete list of parameters used.
When generating the new answer file, we would take the user-specific answer file
and run against the new installer, so that we would get new complete answer file and
let the user to compare that to the old one.

– Ivan

··· ----- Original Message ----- > On Mon, Jan 19, 2015 at 12:39:45PM +0100, Stephen Benjamin wrote: > > On Fri, Jan 16, 2015 at 05:07:41PM +0100, Ewoud Kohl van Wijngaarden wrote: > > > On Fri, Jan 16, 2015 at 04:27:17PM +0100, Stephen Benjamin wrote:

We have had a few discussions about this in the past, but that’s spread
over various PRs. IIRC it was lzap, iNecas, Dominic, gwmngilfen and I
who were mostly involved in the discussions, but I’m sure I’m forgetting
someone.

Was there any outcome out of that? As long as Foreman follows semver and
changes parameters sparingly, it’s not that big of a problem, but it is
one we probably need to consider – especially since Foreman docs also
now reccomend running foreman-installer post-upgrade.

The current situation was the outcome, but IIRC there never was a clear
majority.

It’s a valueable thing to have – for example, when encrypted NoVNC
connections were enabled, if users were consistently running
foreman-installer, they’d have gotten that nice feature, but I imagine
most people who had older Foreman instances that were upgraded through
to 1.6, don’t know that’s available now.

You don’t have to convince me :slight_smile:


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

>
> > > > > The strategy for upgrading Katello has been to basically yum update,
> > > > > run
> > > > > component service migrations, and re-execute Katello installer. The
> > > > > latter step is neccessary - we may introduce new components or
> > > > > adjust
> > > > > configuration. For 2.1 it seems to work relatively well, I've done
> > > > > a
> > > > > dozen upgrades or so by now. I think we're pretty much in the clear
> > > > > for 1.7/2.1 but this is going to hit us hard for 2.2.
> > > > >
> > > > > We rely on the answers file to be able to rerun katello-installer
> > > > > with
> > > > > the user's previous selections. However, we make incompatible
> > > > > changes
> > > > > to
> > > > > puppet modules - for example, Foreman will be releasing 3.0 of
> > > > > puppet-foreman and puppet-foreman_capsule. These are neccessary but
> > > > > painful. Here's an example.
> > > > >
> > > > > Users running a Foreman 1.8 Smart Proxy have enabled services on
> > > > > their
> > > > > capsules like dhcp, dns, etc. However, we've now introduced the
> > > > > provisining template plugin – and at least in RHEL, Anaconda is
> > > > > limited
> > > > > to kickstart over HTTP. So we've had to introduce a whole concept
> > > > > of
> > > > > allowing some plugins (templates) to run on HTTP but not others
> > > > > (dhcp,
> > > > > dns, etc).
> > > > >
> > > > > This required a bunch of parameter name and value changes in the
> > > > > puppet
> > > > > module-- which will cause problems with the existing answers files
> > > > > out
> > > > > in the world.
> > > > >
> > > > > Eric and I briefly talked on IRC about it and it sounds like we're
> > > > > going
> > > > > to need some kind of tool to "migrate" answer files. Anyone else
> > > > > have
> > > > > any suggestions or ideas about how to handle such scenarios?
> > > >
> > > > In the past I used to maintain a foreman installation which managed
> > > > itself and several smart proxies using the installer puppet modules by
> > > > writing a small wrapper class that was somewhere between an role and a
> > > > profile. Every now and then I updated the modules to their latest
> > > > version and looked at the changes introduced.
> > > >
> > > > You want something very similar, but more automated. In the installer
> > > > situation the answers file equates to my wrapper class. So the task is
> > > > to migrate the answers file to get the installer to reach the same
> > > > goal.
> > > >
> > > > Currently the answers file is AFAIK complete, which means every value
> > > > is
> > > > stored even if the user has not changed it. That was chosen so that
> > > > even
> > > > after upgrading the defaults in the module, you would still end up
> > > > with
> > > > the same system. The difficulty with that is when you do want defaults
> > > > to be updated. For example, when we phased out the passenger_scl
> > > > parameter, it was kept in for compatibility. Now every upgrading user
> > > > will get a deprecation warning even if the value was never changed.
> > > >
> > > > Another approach would be to only store user changed values. It does
> > > > have risks that things change which would surprise the user, but
> > > > assuming the user doesn't change many values and we are conservative
> > > > with changing defaults (which we are!), you mostly need to worry about
> > > > removed parameters.
> > >
> > > This would make the problem easier to handle - does Kafo have an option
> > > for this or is it something that needs implementing?
> >
> > AFAIK it needs implementing, though maybe kafo already understands
> > partial answer files.
>
> AFAIK kafo understands partial answer files (that's how it works with the
> default answer files coming from the packages). However, now it makes it
> complete after the first installer run.

Correct

> I've been talking briefly about this some time ago with Marek and storing
> just the values that the user specified explicitly was an option that popped
> up, but perhaps might not even be needed.
>
> As an admin, what I would like to see is some kind of migration tool to run
> against my old answer file and ideally keeping the new answer file in a
> format that the diff against the new and old would give some useful outputs
> (especially keeping the ordering of the parameters).
>
> The problem with partial answer files is that one might rely on some default
> values that might change though the versions and not see them coming. So
> it's useful to have both data about customized values as well as the
> complete list of parameters used. When generating the new answer file, we
> would take the user-specific answer file and run against the new installer,
> so that we would get new complete answer file and let the user to compare
> that to the old one.

Very well described. I think this would be possible with work on scenarios
feature that Martin is working on. We should take similar approach to an issue
when you use first scenario (e.g. Only Foreman) and then you run another
scenario (e.g. Foreman + Katello). While their answer files may be compatible
there may be some changes so we should let user to see diff and select which
values to use, which should be removed etc.

··· On Tuesday 20 of January 2015 03:30:26 Ivan Necas wrote: > ----- Original Message ----- > > On Mon, Jan 19, 2015 at 12:39:45PM +0100, Stephen Benjamin wrote: > > > On Fri, Jan 16, 2015 at 05:07:41PM +0100, Ewoud Kohl van Wijngaarden wrote: > > > > On Fri, Jan 16, 2015 at 04:27:17PM +0100, Stephen Benjamin wrote:


Marek

– Ivan

We have had a few discussions about this in the past, but that’s
spread
over various PRs. IIRC it was lzap, iNecas, Dominic, gwmngilfen and I
who were mostly involved in the discussions, but I’m sure I’m
forgetting
someone.

Was there any outcome out of that? As long as Foreman follows semver and
changes parameters sparingly, it’s not that big of a problem, but it is
one we probably need to consider – especially since Foreman docs also
now reccomend running foreman-installer post-upgrade.

The current situation was the outcome, but IIRC there never was a clear
majority.

It’s a valueable thing to have – for example, when encrypted NoVNC
connections were enabled, if users were consistently running
foreman-installer, they’d have gotten that nice feature, but I imagine
most people who had older Foreman instances that were upgraded through
to 1.6, don’t know that’s available now.

You don’t have to convince me :slight_smile:


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.