[Katello] New packages are not available to Content hosts

We don't have any Content views defined and the content hosts are assigned
the 'Default Organizational View', when the content host attempts to
install a package, the follow (among other) package error occurs (yum
install sssd):

Error: Package: libldb-1.1.20-1.el7_2.2.x86_64 (rhel-7-server-rpms)

       Requires: libtalloc(x86-64) >= 2.1.2

       Installed: libtalloc-2.1.1-1.el7.x86_64 (@rhel-7-server-rpms)

           libtalloc(x86-64) = 2.1.1-1.el7

       Available: libtalloc-2.0.8-4.el7.x86_64 (rhel-7-server-rpms)

           libtalloc(x86-64) = 2.0.8-4.el7

If I go to Content -> Lifecycle Environments -> Library -> Packages,
(Select 'Default Organization View'), Searching for libtalloc shows several
versions including version 2.1.2 so we have version 2.1.2 in our package
repository.

This is confirmed by checking /var/lib/pulp/content/rpm/libtalloc, all
expected versions are found there. But
in /opt/data/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/os
version 2.1.2 is not listed in here.

Is there an extra step that needs to occur for the latest version of all
packages to appear in the 7Server repositories and become available to
content hosts?

Thanks
Francois

> We don't have any Content views defined and the content hosts are
> assigned the 'Default Organizational View', when the content host
> attempts to install a package, the follow (among other) package error
> occurs (yum install sssd):
>
> >
>
> Error:Package:libldb-1.1.20-1.el7_2.2.x86_64 (rhel-7-server-rpms)
>
>
> Requires:libtalloc(x86-64)>=2.1.2
>
>
> Installed:libtalloc-2.1.1-1.el7.x86_64 (@rhel-7-server-rpms)
>
>
> libtalloc(x86-64)=2.1.1-1.el7
>
>
> Available:libtalloc-2.0.8-4.el7.x86_64 (rhel-7-server-rpms)
>
>
> libtalloc(x86-64)=2.0.8-4.el7
>
>
> >
>
> If I go to Content -> Lifecycle Environments -> Library -> Packages,
> (Select 'Default Organization View'), Searching for libtalloc shows
> several versions including version 2.1.2 so we have version 2.1.2 in
> our package repository.
>
> This is confirmed by checking /var/lib/pulp/content/rpm/libtalloc, all
> expected versions are found there. But
> in /opt/data/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/os
> version 2.1.2 is not listed in here.
>
> Is there an extra step that needs to occur for the latest version of
> all packages to appear in the 7Server repositories and become
> available to content hosts?
It sounds like maybe the repo publish step of the sync failed in pulp.
Would you be able to go to Monitor > Tasks and find the sync task for
that repository, click on the 'raw' tab and copy and paste the output?

Its possible there was an error there?

-Justin

··· On 01/13/2016 08:12 PM, Francois Herbert wrote:

Thanks
Francois

You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com
mailto:foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com
mailto:foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Hi Justin

Might take some time to find the failed task, I'm filtering with 'sync' and
there are 15794 tasks…
Todays sync tasks have all finished successfully. Should I try and find
some earlier ones which have failed?

Thanks
Francois

··· On Thursday, January 14, 2016 at 3:17:23 PM UTC+13, jsherril wrote: > > On 01/13/2016 08:12 PM, Francois Herbert wrote: > > We don't have any Content views defined and the content hosts are assigned > the 'Default Organizational View', when the content host attempts to > install a package, the follow (among other) package error occurs (yum > install sssd): > > Error: Package: libldb-1.1.20-1.el7_2.2.x86_64 (rhel-7-server-rpms) > > Requires: libtalloc(x86-64) >= 2.1.2 > > Installed: libtalloc-2.1.1-1.el7.x86_64 (@rhel-7-server-rpms) > > libtalloc(x86-64) = 2.1.1-1.el7 > > Available: libtalloc-2.0.8-4.el7.x86_64 (rhel-7-server-rpms) > > libtalloc(x86-64) = 2.0.8-4.el7 > > > If I go to Content -> Lifecycle Environments -> Library -> Packages, > (Select 'Default Organization View'), Searching for libtalloc shows several > versions including version 2.1.2 so we have version 2.1.2 in our package > repository. > > This is confirmed by checking /var/lib/pulp/content/rpm/libtalloc, all > expected versions are found there. But > in /opt/data/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/os > version 2.1.2 is not listed in here. > > Is there an extra step that needs to occur for the latest version of all > packages to appear in the 7Server repositories and become available to > content hosts? > > It sounds like maybe the repo publish step of the sync failed in pulp. > Would you be able to go to Monitor > Tasks and find the sync task for that > repository, click on the 'raw' tab and copy and paste the output? > > Its possible there was an error there? > > -Justin > > > > Thanks > Francois > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-user...@googlegroups.com . > To post to this group, send email to forema...@googlegroups.com > . > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > > >

Errors: PLP0000: Importer indicated a failed response
The raw data (only raw input) is:

{"repository"=>
{"id"=>4,
"name"=>"Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server",
"label"=>"Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server"},
"product"=>
{"id"=>7,
"name"=>"Red Hat Enterprise Linux Server",
"label"=>"Red_Hat_Enterprise_Linux_Server",
"cp_id"=>"69"},
"provider"=>{"id"=>2, "name"=>"Red Hat"},
"organization"=>
{"id"=>1, "name"=>"Massey University", "label"=>"Default_Organization"},
"id"=>4,
"sync_result"=>Step(3).output,
"user_id"=>1,
"locale"=>"en"}

One thing that may be the cause of this issue is the file system that the
pulp repos are stored on did run out of space a while ago.

··· On Thursday, January 14, 2016 at 4:26:40 PM UTC+13, Francois Herbert wrote: > > Hi Justin > > Might take some time to find the failed task, I'm filtering with 'sync' > and there are 15794 tasks... > Todays sync tasks have all finished successfully. Should I try and find > some earlier ones which have failed? > > Thanks > Francois > > On Thursday, January 14, 2016 at 3:17:23 PM UTC+13, jsherril wrote: >> >> On 01/13/2016 08:12 PM, Francois Herbert wrote: >> >> We don't have any Content views defined and the content hosts are >> assigned the 'Default Organizational View', when the content host attempts >> to install a package, the follow (among other) package error occurs (yum >> install sssd): >> >> Error: Package: libldb-1.1.20-1.el7_2.2.x86_64 (rhel-7-server-rpms) >> >> Requires: libtalloc(x86-64) >= 2.1.2 >> >> Installed: libtalloc-2.1.1-1.el7.x86_64 (@rhel-7-server-rpms) >> >> libtalloc(x86-64) = 2.1.1-1.el7 >> >> Available: libtalloc-2.0.8-4.el7.x86_64 (rhel-7-server-rpms) >> >> libtalloc(x86-64) = 2.0.8-4.el7 >> >> >> If I go to Content -> Lifecycle Environments -> Library -> Packages, >> (Select 'Default Organization View'), Searching for libtalloc shows several >> versions including version 2.1.2 so we have version 2.1.2 in our package >> repository. >> >> This is confirmed by checking /var/lib/pulp/content/rpm/libtalloc, all >> expected versions are found there. But >> in /opt/data/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/os >> version 2.1.2 is not listed in here. >> >> Is there an extra step that needs to occur for the latest version of all >> packages to appear in the 7Server repositories and become available to >> content hosts? >> >> It sounds like maybe the repo publish step of the sync failed in pulp. >> Would you be able to go to Monitor > Tasks and find the sync task for that >> repository, click on the 'raw' tab and copy and paste the output? >> >> Its possible there was an error there? >> >> -Justin >> >> >> >> Thanks >> Francois >> -- >> >> I found this task with a warning, and a message Output: Forbidden,

>
>
>
> Hi Justin
>
> Might take some time to find the failed task, I'm filtering with
> 'sync' and there are 15794 tasks…
> Todays sync tasks have all finished successfully. Should I try and
> find some earlier ones which have failed?
>
> Thanks
> Francois
>
>
>> We don't have any Content views defined and the content hosts
>> are assigned the 'Default Organizational View', when the
>> content host attempts to install a package, the follow (among
>> other) package error occurs (yum install sssd):
>>
>> >
>>
>> Error:Package:libldb-1.1.20-1.el7_2.2.x86_64 (rhel-7-server-rpms)
>>
>>
>> Requires:libtalloc(x86-64)>=2.1.2
>>
>>
>> Installed:libtalloc-2.1.1-1.el7.x86_64 (@rhel-7-server-rpms)
>>
>>
>> libtalloc(x86-64)=2.1.1-1.el7
>>
>>
>> Available:libtalloc-2.0.8-4.el7.x86_64 (rhel-7-server-rpms)
>>
>>
>> libtalloc(x86-64)=2.0.8-4.el7
>>
>>
>> >
>>
>> If I go to Content -> Lifecycle Environments -> Library ->
>> Packages, (Select 'Default Organization View'), Searching for
>> libtalloc shows several versions including version 2.1.2 so
>> we have version 2.1.2 in our package repository.
>>
>> This is confirmed by
>> checking /var/lib/pulp/content/rpm/libtalloc, all expected
>> versions are found there. But
>> in /opt/data/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/os
>> version 2.1.2 is not listed in here.
>>
>> Is there an extra step that needs to occur for the latest
>> version of all packages to appear in the 7Server repositories
>> and become available to content hosts?
> It sounds like maybe the repo publish step of the sync failed
> in pulp. Would you be able to go to Monitor > Tasks and find
> the sync task for that repository, click on the 'raw' tab and
> copy and paste the output?
>
> Its possible there was an error there?
>
> -Justin
>
>
>>
>> Thanks
>> Francois
>> –
>
> I found this task with a warning, and a message Output: Forbidden,
> Errors: PLP0000: Importer indicated a failed response
> The raw data (only raw input) is:
> >
> {"repository"=>
> {"id"=>4,
> "name"=>"Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server",
> "label"=>"Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server"},
> "product"=>
> {"id"=>7,
> "name"=>"Red Hat Enterprise Linux Server",
> "label"=>"Red_Hat_Enterprise_Linux_Server",
> "cp_id"=>"69"},
> "provider"=>{"id"=>2, "name"=>"Red Hat"},
> "organization"=>
> {"id"=>1, "name"=>"Massey University", "label"=>"Default_Organization"},
> "id"=>4,
> "sync_result"=>Step(3).output,
> "user_id"=>1,
> "locale"=>"en"}
> >
>
> One thing that may be the cause of this issue is the file system that
> the pulp repos are stored on did run out of space a while ago.

How long was a while ago?

What version of katello are you on? There was a bug in 2.3
(Bug #12397: Actions::Katello::Repository::Sync reports success regardless errors - Katello - Foreman) where sync failures were
not properly reported for syncs.

Also that output doesn't look like what i was expecting (notice the
sync_result isn't filled out which i've not seen before). You might
jump to the dynflow console and examine the this particular task for
more details. Depending on what version of katello you are on we can
export that task list and I can take look as well (assuming you send me
the export).

-Justin

··· On 01/13/2016 10:36 PM, Francois Herbert wrote: > On Thursday, January 14, 2016 at 4:26:40 PM UTC+13, Francois Herbert > wrote: > On Thursday, January 14, 2016 at 3:17:23 PM UTC+13, jsherril wrote: > On 01/13/2016 08:12 PM, Francois Herbert wrote:


You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com
mailto:foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com
mailto:foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

>
>
> One thing that may be the cause of this issue is the file system that the
> pulp repos are stored on did run out of space a while ago.
>
>
> How long was a while ago?
>
> What version of katello are you on? There was a bug in 2.3 (
> Bug #12397: Actions::Katello::Repository::Sync reports success regardless errors - Katello - Foreman) where sync failures were not
> properly reported for syncs.
>
> Also that output doesn't look like what i was expecting (notice the
> sync_result isn't filled out which i've not seen before). You might jump
> to the dynflow console and examine the this particular task for more
> details. Depending on what version of katello you are on we can export
> that task list and I can take look as well (assuming you send me the
> export).
>
> -Justin
>
> At least 2 months ago, I was away and when I came back the volume was full
:frowning: We are running katello-2.3.0-1.201503310129git07a6e29.el7.noarch.
No problem sending you the export.

Francois

I've checked the date of the file concerned (libtalloc) - 18 Nov 19 11:11
and checked the tasks for this date, there is an error on this day, but
when I first looked at the system after I returned, I had to manually
resume some locked tasks in order to get the syncs working again and this
looks like it may have been one of those tasks. This task has the following
data info: Started at: 2015-08-26 22:04:39 UTC Ended at: 2015-12-14
19:47:17 UTC, and a lot more info in the raw input and output but the task
result says success.

··· On Thursday, January 14, 2016 at 4:51:43 PM UTC+13, Francois Herbert wrote: > > >> One thing that may be the cause of this issue is the file system that the >> pulp repos are stored on did run out of space a while ago. >> >> >> How long was a while ago? >> >> What version of katello are you on? There was a bug in 2.3 ( >> http://projects.theforeman.org/issues/12397) where sync failures were >> not properly reported for syncs. >> >> Also that output doesn't look like what i was expecting (notice the >> sync_result isn't filled out which i've not seen before). You might jump >> to the dynflow console and examine the this particular task for more >> details. Depending on what version of katello you are on we can export >> that task list and I can take look as well (assuming you send me the >> export). >> >> -Justin >> >> At least 2 months ago, I was away and when I came back the volume was > full :-( We are running katello-2.3.0-1.201503310129git07a6e29.el7.noarch. > No problem sending you the export. > > Francois >

I ended up fixing this by unselecting the 'Red Hat Enterprise Linux 7
Server RPMs x86_64 7Server' repository from the Content -> Red Hat
Repositories page, then selecting it again. Since all rpms were already
downloaded, they didn't need to be downloaded again, just the repo
publishing needed to be run again.