Foreman 2.2 -> 2.3 upgrade update_repository_metadata failed & hosts cannot be found

>
> I may have several things going on, and I'll try and keep it as concise as
> possible.
>
> The upgrade instructions are straightforward, I haven't deviated from
> them.
> When we get here in the katello-installer, the metadata update fails:
>
> Upgrade Step: update_repository_metadata (this may take a while) …
> Upgrade step update_repository_metadata failed. Check logs for more
> information.
>
> If I check the log, and then the task which has a Read lock, here is what I
> see:
>
> Actions::ElasticSearch::Repository::IndexContent
>
> PGError: ERROR: duplicate key value violates unique constraint
> "index_katello_errata_on_uuid"
> DETAIL: Key (uuid)=(225b2724-7897-42d7-b446-cded5344f060) already exists.
> : INSERT INTO "katello_errata" ("created_at", "description", "errata_id",
> "errata_type", "issued", "reboot_suggested", "severity", "solution",
> "summary", "title", "updated", "updated_at", "uuid") VALUES ($1, $2, $3,
> $4, $5, $6, $7, $8, $9, $10, $11, $12, $13) RETURNING "id"

This should've been fixed since the even the first RC's:
http://projects.theforeman.org/issues/8586

What's the version of ruby193-rubygem-katello package? You ran yum
update prior to running the upgrade?

>
>
> I could post the full output, but there are 500+ lines - if that is
> necessary, I can. This is on EPEL for EL5 x86_64 repository
>
> I restart the task and it finishes, but with this output:
>
>
> undefined method update_index' for #<Katello::Repository:0x007f0fe4650608> > > Actions::ElasticSearch::Reindex > Input: > > {"id"=>305, > "class_name"=>"Katello::Repository", > "remote_user"=>"admin", > "remote_cp_user"=>"foreman_admin", > "locale"=>"en"} > > Output: > > {} > > Exception: > > NoMethodError: undefined methodupdate_index' for
> #<Katello::Repository:0x007f0fe4650608>
>
>
>
> I re-run katello-installer --upgrade, wash, rinse, and repeat until the
> upgrade completes. The upgrade seems to be successful, but hosts cannot
> check in:
>
> Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
> Failed when searching for node puppetclienthost.myorg.org: Failed to find
> puppetclienthost.myorg.org via exec: Execution of '/etc/puppet/node.rb
> puppetclienthost.myorg.org' returned 1:
> Warning: Not using cache on failed catalog
> Error: Could not retrieve catalog; skipping run

And in /var/log/foreman/production.log?

The web UI loads?

··· On Tue, Aug 25, 2015 at 03:17:01PM -0700, Eric du Toit wrote:

I hope to answer my own questions & solve my own problem tomorrow and post
an update, but if this has been seen before, if I should change something,
or if debugging output is wanted, I can work on that.


You received this message because you are subscribed to the Google Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.


Best Regards,

Stephen Benjamin
Red Hat Engineering

>
>
>
> This should've been fixed since the even the first RC's:
> Bug #8586: Race condition in indexing errata - Katello - Foreman
>
> What's the version of ruby193-rubygem-katello package? You ran yum
> update prior to running the upgrade?
>
>
I've resolved some issues. I'm using a VM snapshot so I can go back to
pre-upgrade attempt if necessary. What I found is that there were several
repositories which had not properly sync'd (Sync Incomplete) and needed
their tasks restarted. I did restart their tasks and then
My steps :

1.) yum update (host patches, etc…) & reboot
2.) update katello / foreman release packages
3.) clean all / yum update
4.) katello-installer --upgrade

ruby193-rubygem-katello-2.3.0-6.el7.noarch is the package I have installed
now.

katello-installer --upgrade worked without error this time.

The error I see now:

/etc/puppet/node.rb foremanhost.myorg.org

Could not send facts to Foreman: SSL_connect returned=1 errno=0 state=SSLv3
read server session ticket A: tlsv1 alert unknown ca

>
> And in /var/log/foreman/production.log?
>
> The web UI loads?
>

WebUI is fine, nothing in the foreman production log when I run the node.rb.

It seems that my upgrade stalling mid-stream was resolved by fixing the
Sync Incomplete repositories, but now I still have an SSL error.

··· On Wednesday, August 26, 2015 at 8:07:44 AM UTC-5, stephen wrote:

Also - I have restarted katello-services and rebooted the host.

··· On Wednesday, August 26, 2015 at 3:59:20 PM UTC-5, Eric du Toit wrote: > > > > The error I see now: > > # /etc/puppet/node.rb foremanhost.myorg.org > Could not send facts to Foreman: SSL_connect returned=1 errno=0 > state=SSLv3 read server session ticket A: tlsv1 alert unknown ca > >

maybe its related to this? see
https://groups.google.com/d/msg/foreman-users/YxlGVHZ6LA4/Ei0K30MLJQAJ
If yes please respond because this ssl crtr issue could be a critical issue
for others as well…

··· Am Mittwoch, 26. August 2015 23:00:15 UTC+2 schrieb Eric du Toit: > > > > On Wednesday, August 26, 2015 at 3:59:20 PM UTC-5, Eric du Toit wrote: >> >> >> >> The error I see now: >> >> # /etc/puppet/node.rb foremanhost.myorg.org >> Could not send facts to Foreman: SSL_connect returned=1 errno=0 >> state=SSLv3 read server session ticket A: tlsv1 alert unknown ca >> >> > > Also - I have restarted katello-services and rebooted the host. >

Ya - that is it. I am using DigiCert signed certificates if that makes
any difference. I had some trouble at first, but I've been through at
least 2 upgrades without issue.
https://groups.google.com/d/msg/foreman-users/YVvOdrbwirM/BgCyLSxDwX0J

If I can help more - let me know.

··· On Wednesday, August 26, 2015 at 4:15:12 PM UTC-5, ehar...@gmail.com wrote: > > maybe its related to this? see > https://groups.google.com/d/msg/foreman-users/YxlGVHZ6LA4/Ei0K30MLJQAJ > If yes please respond because this ssl crtr issue could be a critical > issue for others as well.... > >

Good to hear that you solved the issue with my workaround, I will put the
comments to Bug #10210

··· On Thursday, August 27, 2015 at 5:11:26 AM UTC+2, Eric du Toit wrote: > > > > On Wednesday, August 26, 2015 at 4:15:12 PM UTC-5, ehar...@gmail.com > wrote: >> >> maybe its related to this? see >> https://groups.google.com/d/msg/foreman-users/YxlGVHZ6LA4/Ei0K30MLJQAJ >> If yes please respond because this ssl crtr issue could be a critical >> issue for others as well.... >> >> > > Ya - that is it. I am using DigiCert signed certificates if that makes > any difference. I had some trouble at first, but I've been through at > least 2 upgrades without issue. > https://groups.google.com/d/msg/foreman-users/YVvOdrbwirM/BgCyLSxDwX0J > > If I can help more - let me know. >