Big provisioning libraries update and test


#1

Hello,

I’ve heard that for fixing GCE support, we’ll need to bump fog gem to >= 2.0. Also Azure support would need it. That will potentially have an impact on all compute resources. At the same time, I proposed to update fog-aws and I’ve seen fog-libvirt was recently updated. It seems like there’s a lot going on at the same time.

I think we should merge all the changes ASAP as it’s shortly after the branching. I also spoke with our QE and they volunteered to help us testing all compute resources on nightlies. In order to coordinate all of this, I’m starting this thread where I’d like to keep the list of related PRs and testing status.

If you have some related work to this, please update the main post or let me know to update it.

Planned changes:
update fog to fog-core 2.1.0 and stop using fog itself https://github.com/theforeman/foreman/pull/6475
update fog-aws https://github.com/theforeman/foreman/pull/6240
update fog-libvirt https://github.com/theforeman/foreman/pull/6447
update fog stack in packaging PR - contains fog-google 1.8.2, fog-openstack 1.0.8, fog-vsphere 3.0.0, fog-aws 3.3.0
update fog-ovirt https://github.com/fog/fog-ovirt/pull/55 and https://github.com/theforeman/foreman/pull/6485
fix GCE support @kgaikwad
update fog-vsphere to 3.0 in core
finish Azure provisioning @apuntamb

Test:

  • VMware network image bootdisk
  • oVirt network image
  • libvirt network image
  • openstack image
  • aws image
  • GCE image
  • azure image

I hope to get it all :slight_smile:

The suggested order that we’ll start with tomorrow (19.3.), me and @ekohl will be working on this:
release new foreman-docker for “remove only release”
get the Katello dependency removal merged or release 5.0.1 restoring the compatibility with Katello so they can be coinstalled
merge the new version of foreman-docker in foreman-packaging rpm PR deb PR
merge the gce update on core PR
merge the vsphere update on core PR
rebase openstack PR and merge that
open a PR with fog-openstack bump (part of the big PR below)
merge foreman-packaging PRs for aws and gce and the fog-openstack as a single PR
merge the big packaging PR


Google cloud compute ressource: header field value cannot include CR/LF
Foreman OpenNebula plugin sunsetting
Foreman 1.22 schedule and planning
#2

what about https://github.com/tristanrobert/foreman_fog_proxmox (Additionally, I would suggest to move this plugin to theforeman project at github)


#3

We also ship rackware compute resource in core, no idea if it is working or not but the fog gem seems not very maintained - https://github.com/fog/fog-rackspace - so i think we need to either take it over and update it or extract it from core to a plugin.
regarding proxmox - @Bernhard_Suttner do you have contact with the author? we need their cooperation to move it to the foreman organization, as well as making sure a couple of other details are handled, iirc making sure there is packaging for it and giving theforeman user ownership on rubygems (@ekohl or @Gwmngilfen might recall if there are other requirements).


#4

(maybe it would be a good point in time to move foreman_fog_proxmox to theforeman project - @tristanrobert)


#5

Good news!


#6

We’ll need @tristanrobert’s help to move the repo (which could be forked, but obviously better not to) and to have community access to the gem (which has no other workaround). If you’re interested @tristanrobert, then I can let you know what’s needed :slight_smile:


#7

I have tried to transfer ownership’s repo to theforeman organization, but I don’t have permissions in this organization.


#8

Indeed, we have a process. I’ve sent you an invite to the Transfer team on GitHub. Once transferred, I can make a team and add you as an admin.

For the gem, could you add theforeman.rubygems@gmail.com as an author of the gem? It’s only in case of “bus-factor” :slight_smile:


#9

I have transferred repo to theforeman/core team.


#10

Thanks, I’ve created the team, added you to it, and given it admin rights. You should basically be able to do anything you could before :slight_smile:. Just need the gem authors now.

Any volunteers for being on the GitHub team?


#11

Thanks. I will update authors gem soon.


#12

I am not sure if someone is still using Foreman with Compute Resource Opennebula, but our Managed Service team has switched to OpenStack, so will probably not test and fix it.


#13

Gem is published with foreman team author added.


#14

An update on current status: we bumped fog-aws in develop branch, which now requires fog-core 2.1.0. It turned out there are few compute resources which requires fog-core to be < 2.1.0. We’re now investigating if fog_azure_rm and fog-google could be update to work with fog-core 2.1.0. Details can be found at https://github.com/theforeman/foreman/pull/6475. The other option to revert fog-aws, though adding more compute resources using old version of fog seem as adding more work for future. Other affected compute resources are foreman_docker and foreman_kubevirt.


#15

So we settled on fog-core 2.1.0. Here’s the first packaging update that should bring newer fog-aws while using versions compatible with upcoming fog-google and fog_azure_rm. These will be added as separate PRs.


#16

I filed OpenStack fog upgrade, did some minor updates and I can confirm that 1.0.8 works fine against OpenStack v2 API. I have no access to v3 instance tho.


#17

I’ve updated the first post with the order of things we plan to do tomorrow with @ekohl. There may be some nightly issues caused by this, but we’ll try to minimize it and fix asap. If there’s something you know we missed, please let us know, I plan to start tomorrow morning.


#18

Given our today jenkins outage, we have to postpone.


#19

We are now starting with the upgrade. We expect to have all merged by end of today. Results will be posted here.


#20

Upgrade is done. Thanks to @evgeni, @ekohl and @tbrisker we have now packaging up to date. As part of repoclosure testing we realized we need to remove foreman_one which does not seem to be used and hopefully just temporarily disable foreman_fog_proxmox which is not ready for fog-core 2.1.0.

The repoclosure is now green, meaning nightly repositories should have new versions soon. We’ll do a sanity check and confirm here once that happens. Then we can start testing provisioning on all affected compute resources.