Smart proxy sync not complete although showing that it successfully completed

[daniejstriata]

I have 2 Foreman servers. One is a Proxy getting its repos from the main foreman. When I publish new versions of Content Views they are successfully synced across to the Foreman Proxy but I do not have access to the new packages on hosts assigned to the proxy. I have to do a Complete Sync each time before the Proxy serves the updates packages.

I want to find out why the Quick sync is not working as expected but don’t see any errors in production log and no error on frontend as the GUI states the quick sync was successfully completed.

Foreman and Proxy versions:
2.2.2

Foreman and Proxy plugin versions:

Distribution and version:
Centos 7

Other relevant data:
I cannot see any errors in the production.log and the quick sync completes without errors.

[iballou]

Hi @daniejstriata,![Screenshot from 2021-01-08 12-03-32|516x500]

Can you show us the Dynflow console plans for your quick sync and your complete sync? You will find it by clicking the “Dynflow console” button on each of the “Synchronize smart proxy” tasks.

The plan in the Dynflow console will look like this:

Also, can you show us the “Services” tab on your main (not remote) smart proxy? I’d like to see if you’re running Pulp 2 or Pulp 3.

The page looks like this:

[daniejstriata]

Thanks for getting back to me.

Here are the quick sync tasks:

Mine is not like yours

The complete sync looks a lot longer over many pages.

Are these normal?

[iballou]

Would you mind showing me the “Services” tab for your smart proxy that isn’t the remote one as well? That would be the smart proxy that says “Pulp” instead of “Pulp node” on the “Services” tab. I can tell that your remote proxy doesn’t have Pulp 3 (which is expected) but I’m still not 100% sure about your main Katello server’s smart proxy.

Was this Katello server a fresh install for Foreman 2.2.2 or did you upgrade from a previous version?

[daniejstriata]

Hi,

I’ve been upgrading this Foreman install of mine since version 2.0.

Here is the services screenshot from the main Foreman. It shows I’m still using pulp2. I thought I migrated to 3 prevoiously.

[iballou]

Hi @daniejstriata,

I’m guessing you did the migration to Pulp 3 back when it was just for docker and file content. That’s fine that you’re on Pulp 2 still for yum content, it’ll just have to be migrated again before you update to Katello 4.0 in the future.

Is it just yum content that you’re syncing to your smart proxy? I’m going to see if I can reproduce the issue in my environment.

[iballou]

@daniejstriata,

Can you give us your Katello version? Some of the smart proxy code changed between 3.17 and 3.18. I’d like to be sure which version you’re on.

[daniejstriata]

I’m on katello-3.17.1-1.el7.noarch for both the servers. Should I upgrade to Foreman 2.3 and see if problem goes away?

[iballou]

I’m not yet if upgrading would fix your issue.

If you expand the single Actions::Pulp::Consumer::SyncCapsule action in your optimized sync task’s Dynflow console, there should be at least one line that contains - pulp:repository:<repo id>. Can I see what that says?

I think you should see a Actions::Pulp::Consumer::SyncCapsule for each repo that needs to be synced. Are you expecting more than 1 repo to be synced to your smart proxy?

Oh also just to be sure, when you said quick sync, did you mean optimized sync?

[iballou]

@iballou

If you expand the single Actions::Pulp::Consumer::SyncCapsule action in your optimized sync task’s Dynflow console, there should be at least one line that contains - pulp:repository:<repo id>. Can I see what that says?

There should be a “Synchronize smart proxy” task that pops up right after you promote your content view, I’d be curious to see the - pulp:repository:<repo id> for that task specifically (if it’s not the same one that you started showing me above).

[daniejstriata]

Hi,

Just to make sure I’m understanding you. Here is the steps as I see it.

1. I create a new version of my CV but have not promoted any Lifecycle Environments. It is just built Library.
2. When it completed to create a task was kicked off called
   Synchronize smart proxy 'xxx'
3. I waited for this task to complete.
4. I clicked on Dynflow console and this is what I see ( I don’t see what you’re asking for)
   
   

[Thulium-Drake]

Perhaps related to: Bug: Smart Proxies do not sync (Katello 3.15 through 3.18 RC2) - #6 by Thulium-Drake

Check your logs for any python stacktraces.

[daniejstriata]

Thank you for this! It is very possible.

I have a different log from what is in the reported bug though

[iballou]

@daniejstriata,

I was expecting a smart proxy sync task to run automatically after you promote a new content view version to one of the lifecycle environments that are attached to your smart proxy. Since you’re manually syncing your smart proxy after content view publishing, it leads me to think that isn’t happening for some reason.

So we’re on the same page, what lifecycle environments are you syncing with your smart proxy? It would be good to know if Library is among them.

In your newest Dynflow screen shot, I see many SyncCapsule actions (should be one per repo synced), which makes me think this smart proxy sync you did actually updated your smart proxy. Did this work? Was it an optimized sync or complete? Your original screen shot earlier only had one SyncCapsule task, so I’d like to know the difference.

Can you to expand one of the Dynflow SyncCapsule actions for a smart proxy sync task that didn’t update packages on your smart proxy and show all of the text inside?

[daniejstriata]

Hi,

The screenshot is from the automatically created sync as soon as I created the new version for the CV.

I’m expecting all the lifecycle environments to sync across and expect it to happen automatically.

I don’t under stand your question about “smart proxy sync you did actually updated your smart proxy”

I believe I attached what you asked for:
Dynflow Console.tar.gz (11.6 KB)

[iballou]

@daniejstriata

I don’t under stand your question about “smart proxy sync you did actually updated your smart proxy”

I wasn’t sure if the smart proxy sync you just showed me worked or not. Were packages on your smart proxy updated?

If yes, they were updated, then I need to see the same logs for a smart proxy sync that didn’t work.

If no, the smart proxy sync failed to sync new packages, then I’d like to see the following:

1. Determine which repository did not get an update but should’ve
2. Browse to the repository and find the numeric ID that’s at the end of the url for the repo (url should look something like https://katello.com/products/1/repositories/5)
3. Open the foreman console with sudo foreman-rake console
4. Show me the output of:

::Katello::Repository.find(<ID HERE>).pulp_id

I’m trying to find out if the missing packages are happening because there is a missing SyncCapsule action or if there is a correct SyncCapsule action that is silently doing nothing. By giving me the Pulp ID of a repository that failed to sync, I can check your logs to see if the repository is mentioned in the SyncCapsule action.

[daniejstriata]

Hi,

I have many products but I chose one specifically because I know it was missing from hosts served by the proxy. Its ID is 1367.

irb(main):002:0> ::Katello::Repository.find(1367).pulp_id
=> "a06330c6-1a54-4733-9d6b-483a16670232"

The images I sent you are for the Automatic sync. When I do complete syncs the data is available but the entries are much longer from Dynflow. If you want I can share the complete sync’s dynflow task with you.

[iballou]

@daniejstriata,

Okay, I didn’t find that pulp_id in the logs for the automatic sync that you showed me. You’re sure that the repository with ID 1367 was in your content view before you published the new content view version?

I’m not yet seeing in the code how a repository could go missing like that if it was in your content view. I also wasn’t able to reproduce the problem in the quick tests I tried, hm.

Has this issue only been occurring since you upgraded to Katello 3.17?

[daniejstriata]

Yes. It’s been a product I setup for a long time now. I did not have this issue on version 2.0 (I always use the Katello version matching the Foreman version). I may have noticed it happen a couple of times on 2.1 but since 2.2 it is consistently not auto syncing.

Another thing I noticed (not sure related) about syncing the repos I found is that I cannot use any of a new version’s packages. If I try to update/install a package on the proxy first the yum would tell me it knows about the new package but it fails to download. When I download the package in question first on a host managed by the main foreman instance it seems to make it available for the proxy’s host too as once that is done I can go to a host managed by the proxy and then the package will download from the repo hosted by the proxy. All my repos are set to: Mirror on Sync - Yes but previously this also was not an issue.

[iballou]

For the content view that we’ve been discussing (the one from this post), can you find the ID from the url and show me the following in the console?

::Katello::ContentView.find(<content view ID here>).repositories.pluck(:pulp_id)

I’d like to see how many of the repositories on your content view match up to the log you sent.

@daniejstriata

When I download the package in question first on a host managed by the main foreman instance it seems to make it available for the proxy’s host too as once that is done I can go to a host managed by the proxy and then the package will download from the repo hosted by the proxy.

I have to think more about this behavior, but I don’t suppose it has to do with the remote smart proxy being set to download “on demand” instead of “immediate”? That would be a bug and this would be a workaround, but it might be worth trying if your smart proxy has the disk space available.