[Katello 2.4] Capsule Synchronization hangs at 75%

Katello 2.4 / Foreman 1.10.1 on CentOS 7.2

My capsule synchronization never seems to complete. It gets to 75% and
then hangs indefinitely.

This is what I see when I attempt it (it will stay like this for hours
until i Ctrl-C it)

------- errors ------
[root@katello1~]# hammer -u admin -p <password> capsule content synchronize
–id 4
[…
] [75%]^
------- errors ------

If I attempt to promote content through the web interface, the synchronize
task that gets spawned fails :

------- errors ------

Host did not respond within 20 seconds. Sync has been cancelled. Is katello-agent installed and goferd running on the Host?

------- errors ------

The goferd service is defintely running :

---- troubleshooting --------
[root@katello1 ~]# systemctl | grep goferd

goferd.service
loaded active running Gofer Agent
------ troubleshooting ------

Also, I did a network capture, and there is definitely a SSL handshake
going on between the 2 servers and some data transfer, but regardless, the
content never actually shows up on the capsule :

No. Time Source Destination Protocol
Length Info
1 0.000000 10.178.0.110 10.128.17.110 TCP
491 5646 → 43052 [PSH, ACK] Seq=1 Ack=1 Win=1432 Len=425 TSval=15011609
TSecr=14701871
2 0.000656 10.128.17.110 10.178.0.110 TCP
485 43052 → 5646 [PSH, ACK] Seq=1 Ack=426 Win=1424 Len=419
TSval=14702871 TSecr=15011609
3 0.000676 10.178.0.110 10.128.17.110 TCP
66 5646 → 43052 [ACK] Seq=426 Ack=420 Win=1432 Len=0 TSval=15011610
TSecr=14702871
4 0.000683 10.128.17.110 10.178.0.110 TCP
140 43052 → 5646 [PSH, ACK] Seq=420 Ack=426 Win=1424 Len=74
TSval=14702872 TSecr=15011609
5 0.000688 10.178.0.110 10.128.17.110 TCP
66 5646 → 43052 [ACK] Seq=426 Ack=494 Win=1432 Len=0 TSval=15011610
TSecr=14702872
6 0.000912 10.178.0.110 10.128.17.110 TCP
140 5646 → 43052 [PSH, ACK] Seq=426 Ack=494 Win=1432 Len=74
TSval=15011610 TSecr=14702872
7 0.040612 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=494 Ack=500 Win=1424 Len=0 TSval=14702912
TSecr=15011610
8 1.000939 10.178.0.110 10.128.17.110 TCP
491 5646 → 43052 [PSH, ACK] Seq=500 Ack=494 Win=1432 Len=425
TSval=15012610 TSecr=14702912
9 1.001197 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=494 Ack=925 Win=1424 Len=0 TSval=14703872
TSecr=15012610
10 1.001549 10.128.17.110 10.178.0.110 TCP
140 43052 → 5646 [PSH, ACK] Seq=494 Ack=925 Win=1424 Len=74
TSval=14703872 TSecr=15012610
11 1.032061 10.128.17.110 10.178.0.110 TCP
103 45857 → 5647 [PSH, ACK] Seq=1 Ack=1 Win=1424 Len=37 TSval=14703903
TSecr=15008637
12 1.032070 10.178.0.110 10.128.17.110 TCP
66 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=0 TSval=15012641
TSecr=14703903
13 1.040637 10.178.0.110 10.128.17.110 TCP
66 5646 → 43052 [ACK] Seq=925 Ack=568 Win=1432 Len=0 TSval=15012650
TSecr=14703872
14 2.001423 10.128.17.110 10.178.0.110 TCP
485 43052 → 5646 [PSH, ACK] Seq=568 Ack=925 Win=1424 Len=419
TSval=14704872 TSecr=15012650
15 2.001444 10.178.0.110 10.128.17.110 TCP
66 5646 → 43052 [ACK] Seq=925 Ack=987 Win=1432 Len=0 TSval=15013610
TSecr=14704872
16 2.001552 10.178.0.110 10.128.17.110 TCP
491 5646 → 43052 [PSH, ACK] Seq=925 Ack=987 Win=1432 Len=425
TSval=15013610 TSecr=14704872
17 2.001742 10.178.0.110 10.128.17.110 TCP
140 5646 → 43052 [PSH, ACK] Seq=1350 Ack=987 Win=1432 Len=74
TSval=15013611 TSecr=14704872
18 2.002011 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=987 Ack=1424 Win=1424 Len=0 TSval=14704873
TSecr=15013610
19 2.002019 10.128.17.110 10.178.0.110 TCP
140 43052 → 5646 [PSH, ACK] Seq=987 Ack=1424 Win=1424 Len=74
TSval=14704873 TSecr=15013610
20 2.041643 10.178.0.110 10.128.17.110 TCP
66 5646 → 43052 [ACK] Seq=1424 Ack=1061 Win=1432 Len=0 TSval=15013651
TSecr=14704873
21 3.002182 10.178.0.110 10.128.17.110 TCP
491 5646 → 43052 [PSH, ACK] Seq=1424 Ack=1061 Win=1432 Len=425
TSval=15014611 TSecr=14704873
22 3.002605 10.128.17.110 10.178.0.110 TCP
485 43052 → 5646 [PSH, ACK] Seq=1061 Ack=1849 Win=1424 Len=419
TSval=14705873 TSecr=15014611
23 3.002623 10.178.0.110 10.128.17.110 TCP
66 5646 → 43052 [ACK] Seq=1849 Ack=1480 Win=1432 Len=0 TSval=15014611
TSecr=14705873
24 3.002828 10.128.17.110 10.178.0.110 TCP
140 43052 → 5646 [PSH, ACK] Seq=1480 Ack=1849 Win=1424 Len=74
TSval=14705874 TSecr=15014611
25 3.002833 10.178.0.110 10.128.17.110 TCP
66 5646 → 43052 [ACK] Seq=1849 Ack=1554 Win=1432 Len=0 TSval=15014612
TSecr=14705874
26 3.002865 10.178.0.110 10.128.17.110 TCP
140 5646 → 43052 [PSH, ACK] Seq=1849 Ack=1554 Win=1432 Len=74
TSval=15014612 TSecr=14705874
27 3.042608 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=1554 Ack=1923 Win=1424 Len=0 TSval=14705914
TSecr=15014612
28 3.479979 10.178.0.110 10.128.17.110 TCP
2962 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=2896 TSval=15015089
TSecr=14703903
29 3.479991 10.178.0.110 10.128.17.110 TCP
2962 5647 → 45857 [ACK] Seq=2897 Ack=38 Win=710 Len=2896 TSval=15015089
TSecr=14703903
30 3.479994 10.178.0.110 10.128.17.110 TCP
2962 5647 → 45857 [ACK] Seq=5793 Ack=38 Win=710 Len=2896 TSval=15015089
TSecr=14703903
31 3.480091 10.178.0.110 10.128.17.110 TCP
2962 5647 → 45857 [ACK] Seq=8689 Ack=38 Win=710 Len=2896 TSval=15015089
TSecr=14703903
32 3.480101 10.178.0.110 10.128.17.110 TCP
2962 5647 → 45857 [ACK] Seq=11585 Ack=38 Win=710 Len=2896 TSval=15015089
TSecr=14703903
33 3.480584 10.128.17.110 10.178.0.110 TCP
66 45857 → 5647 [ACK] Seq=38 Ack=14481 Win=1424 Len=0 TSval=14706351
TSecr=15015089
34 3.480597 10.178.0.110 10.128.17.110 TCP
2535 5647 → 45857 [PSH, ACK] Seq=14481 Ack=38 Win=710 Len=2469
TSval=15015089 TSecr=14706351
35 3.481230 10.128.17.110 10.178.0.110 TCP
66 45857 → 5647 [ACK] Seq=38 Ack=16950 Win=1424 Len=0 TSval=14706352
TSecr=15015089
36 3.482661 10.128.17.110 10.178.0.110 TCP
220 45857 → 5647 [PSH, ACK] Seq=38 Ack=16950 Win=1424 Len=154
TSval=14706354 TSecr=15015089
37 3.482671 10.178.0.110 10.128.17.110 TCP
66 5647 → 45857 [ACK] Seq=16950 Ack=192 Win=733 Len=0 TSval=15015092
TSecr=14706354
38 3.484424 10.178.0.110 10.128.17.110 TCP
232 5647 → 45857 [PSH, ACK] Seq=16950 Ack=192 Win=733 Len=166
TSval=15015093 TSecr=14706354
39 3.485299 10.128.17.110 10.178.0.110 TCP
468 45857 → 5647 [PSH, ACK] Seq=192 Ack=17116 Win=1424 Len=402
TSval=14706356 TSecr=15015093
40 3.485985 10.178.0.110 10.128.17.110 TCP
131 5647 → 45857 [PSH, ACK] Seq=17116 Ack=594 Win=755 Len=65
TSval=15015095 TSecr=14706356
41 3.525616 10.128.17.110 10.178.0.110 TCP
66 45857 → 5647 [ACK] Seq=594 Ack=17181 Win=1424 Len=0 TSval=14706397
TSecr=15015095
42 4.002777 10.178.0.110 10.128.17.110 TCP
491 5646 → 43052 [PSH, ACK] Seq=1923 Ack=1554 Win=1432 Len=425
TSval=15015612 TSecr=14705914
43 4.003036 10.128.17.110 10.178.0.110 TCP
485 43052 → 5646 [PSH, ACK] Seq=1554 Ack=1923 Win=1424 Len=419
TSval=14706874 TSecr=15014612
44 4.003043 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=1973 Ack=2348 Win=1424 Len=0 TSval=14706874
TSecr=15015612
45 4.003482 10.128.17.110 10.178.0.110 TCP
140 43052 → 5646 [PSH, ACK] Seq=1973 Ack=2348 Win=1424 Len=74
TSval=14706874 TSecr=15015612
46 4.003552 10.178.0.110 10.128.17.110 TCP
66 5646 → 43052 [ACK] Seq=2348 Ack=2047 Win=1432 Len=0 TSval=15015612
TSecr=14706874
47 4.003750 10.178.0.110 10.128.17.110 TCP
140 5646 → 43052 [PSH, ACK] Seq=2348 Ack=2047 Win=1432 Len=74
TSval=15015613 TSecr=14706874
48 4.043644 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=2047 Ack=2422 Win=1424 Len=0 TSval=14706915
TSecr=15015613
49 4.076622 10.178.0.110 10.128.17.110 TCP
133 5647 → 45857 [PSH, ACK] Seq=17181 Ack=594 Win=755 Len=67
TSval=15015685 TSecr=14706397
50 4.076864 10.128.17.110 10.178.0.110 TCP
66 45857 → 5647 [ACK] Seq=594 Ack=17248 Win=1424 Len=0 TSval=14706948
TSecr=15015685
51 4.078152 10.128.17.110 10.178.0.110 TCP
119 45857 → 5647 [PSH, ACK] Seq=594 Ack=17248 Win=1424 Len=53
TSval=14706949 TSecr=15015685
52 4.078234 10.178.0.110 10.128.17.110 TCP
119 5647 → 45857 [PSH, ACK] Seq=17248 Ack=647 Win=755 Len=53
TSval=15015687 TSecr=14706949
53 4.082247 10.128.17.110 10.178.0.110 TCP
172 45857 → 5647 [PSH, ACK] Seq=647 Ack=17301 Win=1424 Len=106
TSval=14706953 TSecr=15015687
54 4.121634 10.178.0.110 10.128.17.110 TCP
66 5647 → 45857 [ACK] Seq=17301 Ack=753 Win=755 Len=0 TSval=15015731
TSecr=14706953
55 5.003648 10.178.0.110 10.128.17.110 TCP
491 5646 → 43052 [PSH, ACK] Seq=2422 Ack=2047 Win=1432 Len=425
TSval=15016613 TSecr=14706915
56 5.003926 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=2047 Ack=2847 Win=1424 Len=0 TSval=14707875
TSecr=15016613
57 5.004452 10.128.17.110 10.178.0.110 TCP
530 43052 → 5646 [PSH, ACK] Seq=2047 Ack=2847 Win=1424 Len=464
TSval=14707875 TSecr=15016613
58 5.004702 10.178.0.110 10.128.17.110 TCP
140 5646 → 43052 [PSH, ACK] Seq=2847 Ack=2511 Win=1432 Len=74
TSval=15016614 TSecr=14707875
59 5.044893 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=2511 Ack=2921 Win=1424 Len=0 TSval=14707916
TSecr=15016614
60 6.005076 10.178.0.110 10.128.17.110 TCP
491 5646 → 43052 [PSH, ACK] Seq=2921 Ack=2511 Win=1432 Len=425
TSval=15017614 TSecr=14707916
61 6.005370 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=2511 Ack=3346 Win=1424 Len=0 TSval=14708876
TSecr=15017614
62 6.005714 10.128.17.110 10.178.0.110 TCP
530 43052 → 5646 [PSH, ACK] Seq=2511 Ack=3346 Win=1424 Len=464
TSval=14708877 TSecr=15017614
63 6.005929 10.178.0.110 10.128.17.110 TCP
140 5646 → 43052 [PSH, ACK] Seq=3346 Ack=2975 Win=1432 Len=74
TSval=15017615 TSecr=14708877
64 6.045606 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=2975 Ack=3420 Win=1424 Len=0 TSval=14708917
TSecr=15017615
65 7.005640 10.128.17.110 10.178.0.110 TCP
485 43052 → 5646 [PSH, ACK] Seq=2975 Ack=3420 Win=1424 Len=419
TSval=14709877 TSecr=15017615
66 7.006071 10.178.0.110 10.128.17.110 TCP
536 5646 → 43052 [PSH, ACK] Seq=3420 Ack=3394 Win=1432 Len=470
TSval=15018615 TSecr=14709877
67 7.006273 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=3394 Ack=3890 Win=1424 Len=0 TSval=14709877
TSecr=15018615
68 7.006534 10.128.17.110 10.178.0.110 TCP
140 43052 → 5646 [PSH, ACK] Seq=3394 Ack=3890 Win=1424 Len=74
TSval=14709877 TSecr=15018615
69 7.045642 10.178.0.110 10.128.17.110 TCP
66 5646 → 43052 [ACK] Seq=3890 Ack=3468 Win=1432 Len=0 TSval=15018655
TSecr=14709877
70 8.006236 10.178.0.110 10.128.17.110 TCP
491 5646 → 43052 [PSH, ACK] Seq=3890 Ack=3468 Win=1432 Len=425
TSval=15019615 TSecr=14709877
71 8.007656 10.128.17.110 10.178.0.110 TCP
530 43052 → 5646 [PSH, ACK] Seq=3468 Ack=4315 Win=1424 Len=464
TSval=14710878 TSecr=15019615
72 8.007668 10.178.0.110 10.128.17.110 TCP
66 5646 → 43052 [ACK] Seq=4315 Ack=3932 Win=1432 Len=0 TSval=15019617
TSecr=14710878
73 8.007899 10.178.0.110 10.128.17.110 TCP
140 5646 → 43052 [PSH, ACK] Seq=4315 Ack=3932 Win=1432 Len=74
TSval=15019617 TSecr=14710878
74 8.047677 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=3932 Ack=4389 Win=1424 Len=0 TSval=14710919
TSecr=15019617
75 8.086902 10.128.17.110 10.178.0.110 TCP
103 45857 → 5647 [PSH, ACK] Seq=753 Ack=17301 Win=1424 Len=37
TSval=14710958 TSecr=15015731
76 8.086919 10.178.0.110 10.128.17.110 TCP
66 5647 → 45857 [ACK] Seq=17301 Ack=790 Win=755 Len=0 TSval=15019696
TSecr=14710958
77 9.007011 10.178.0.110 10.128.17.110 TCP
491 5646 → 43052 [PSH, ACK] Seq=4389 Ack=3932 Win=1432 Len=425
TSval=15020616 TSecr=14710919
78 9.007473 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=3932 Ack=4814 Win=1424 Len=0 TSval=14711878
TSecr=15020616
79 9.008270 10.128.17.110 10.178.0.110 TCP
530 43052 → 5646 [PSH, ACK] Seq=3932 Ack=4814 Win=1424 Len=464
TSval=14711879 TSecr=15020616
80 9.008459 10.178.0.110 10.128.17.110 TCP
140 5646 → 43052 [PSH, ACK] Seq=4814 Ack=4396 Win=1432 Len=74
TSval=15020617 TSecr=14711879
81 9.048720 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=4396 Ack=4888 Win=1424 Len=0 TSval=14711920
TSecr=15020617
82 10.007365 10.178.0.110 10.128.17.110 TCP
491 5646 → 43052 [PSH, ACK] Seq=4888 Ack=4396 Win=1432 Len=425
TSval=15021616 TSecr=14711920
83 10.007704 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=4396 Ack=5313 Win=1424 Len=0 TSval=14712878
TSecr=15021616
84 10.008836 10.128.17.110 10.178.0.110 TCP
530 43052 → 5646 [PSH, ACK] Seq=4396 Ack=5313 Win=1424 Len=464
TSval=14712880 TSecr=15021616
85 10.009095 10.178.0.110 10.128.17.110 TCP
140 5646 → 43052 [PSH, ACK] Seq=5313 Ack=4860 Win=1432 Len=74
TSval=15021618 TSecr=14712880
86 10.224636 10.178.0.110 10.128.17.110 TCP
140 [TCP Retransmission] 5646 → 43052 [PSH, ACK] Seq=5313 Ack=4860
Win=1432 Len=74 TSval=15021834 TSecr=14712880
87 10.224968 10.128.17.110 10.178.0.110 TCP
78 43052 → 5646 [ACK] Seq=4860 Ack=5387 Win=1424 Len=0 TSval=14713096
TSecr=15021834 SLE=5313 SRE=5387
88 11.008404 10.178.0.110 10.128.17.110 TCP
491 5646 → 43052 [PSH, ACK] Seq=5387 Ack=4860 Win=1432 Len=425
TSval=15022617 TSecr=14713096
89 11.009072 10.128.17.110 10.178.0.110 TCP
66 43052 → 5646 [ACK] Seq=4860 Ack=5812 Win=1424 Len=0 TSval=14713880
TSecr=15022617
90 11.009568 10.128.17.110 10.178.0.110 TCP
530 43052 → 5646 [PSH, ACK] Seq=4860 Ack=5812 Win=1424 Len=464
TSval=14713880 TSecr=15022617
91 11.009777 10.178.0.110 10.128.17.110 TCP
140 5646 → 43052 [PSH,

So this is still not working. Has no one seriously ever tested a capsule
before? This seems like a pretty core feature to have broken so badly.

I've now setup multiple capsules on different subnets, including on the
same subnet as the main Katello server (same switch etc), and none of them
ever finish synchronizing under any circumstances.

The one exception is if I assign a capsule to NO lifecycle environments,
its reaches 100% synchronization instantly and returns to the shell
prompt. If I add even a single view to the lifecycle environment (even if
that view only has like 10 packages in it), it will always freeze up and
not finish, not matter which capsule on which network I try to sync to.
The amount it freezes up at seems inconsistent though. Sometimes it's 99%,
sometimes 75%, sometimes 95%.

Here is console output of several different tests with different lifecycle
environments available to the capsules. The final 100% was only with no
lifecycle environments assigned to the capsule and obviously that is not an
option as it transfers zero content to the capsule. In each case, I wait
at least 3 hours before ctrl-c and scan the disk free space on the capsule
to ensure that nothing is being transferred.

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…
]
[99%]^[…
] [99%]

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…
]
[75%]^[…
] [75%]

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…
]
[99%]^[…
] [99%]

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…]
[100%]

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…
] [95%]

··· On Tuesday, February 2, 2016 at 3:51:21 PM UTC-8, Nathan Peters wrote: > > Katello 2.4 / Foreman 1.10.1 on CentOS 7.2 > > My capsule synchronization never seems to complete. It gets to 75% and > then hangs indefinitely. > > This is what I see when I attempt it (it will stay like this for hours > until i Ctrl-C it) > > ------- errors ------ > [root@katello1~]# hammer -u admin -p capsule content > synchronize --id 4 > [............................................................................... > ] [75%]^ > ------- errors ------ > > If I attempt to promote content through the web interface, the synchronize > task that gets spawned fails : > > ------- errors ------ > > Host did not respond within 20 seconds. Sync has been cancelled. Is katello-agent installed and goferd running on the Host? > > ------- errors ------ > > > The goferd service is defintely running : > > ---- troubleshooting -------- > [root@katello1 ~]# systemctl | grep goferd > > goferd.service > loaded active running Gofer Agent > ------ troubleshooting ------ > > Also, I did a network capture, and there is definitely a SSL handshake > going on between the 2 servers and some data transfer, but regardless, the > content never actually shows up on the capsule : > > No. Time Source Destination > Protocol Length Info > 1 0.000000 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1 Ack=1 Win=1432 Len=425 > TSval=15011609 TSecr=14701871 > 2 0.000656 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1 Ack=426 Win=1424 Len=419 > TSval=14702871 TSecr=15011609 > 3 0.000676 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=426 Ack=420 Win=1432 Len=0 > TSval=15011610 TSecr=14702871 > 4 0.000683 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=420 Ack=426 Win=1424 Len=74 > TSval=14702872 TSecr=15011609 > 5 0.000688 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=426 Ack=494 Win=1432 Len=0 > TSval=15011610 TSecr=14702872 > 6 0.000912 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=426 Ack=494 Win=1432 Len=74 > TSval=15011610 TSecr=14702872 > 7 0.040612 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=494 Ack=500 Win=1424 Len=0 > TSval=14702912 TSecr=15011610 > 8 1.000939 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=500 Ack=494 Win=1432 Len=425 > TSval=15012610 TSecr=14702912 > 9 1.001197 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=494 Ack=925 Win=1424 Len=0 > TSval=14703872 TSecr=15012610 > 10 1.001549 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=494 Ack=925 Win=1424 Len=74 > TSval=14703872 TSecr=15012610 > 11 1.032061 10.128.17.110 10.178.0.110 > TCP 103 45857 → 5647 [PSH, ACK] Seq=1 Ack=1 Win=1424 Len=37 > TSval=14703903 TSecr=15008637 > 12 1.032070 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=0 > TSval=15012641 TSecr=14703903 > 13 1.040637 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=925 Ack=568 Win=1432 Len=0 > TSval=15012650 TSecr=14703872 > 14 2.001423 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=568 Ack=925 Win=1424 Len=419 > TSval=14704872 TSecr=15012650 > 15 2.001444 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=925 Ack=987 Win=1432 Len=0 > TSval=15013610 TSecr=14704872 > 16 2.001552 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=925 Ack=987 Win=1432 Len=425 > TSval=15013610 TSecr=14704872 > 17 2.001742 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=1350 Ack=987 Win=1432 Len=74 > TSval=15013611 TSecr=14704872 > 18 2.002011 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=987 Ack=1424 Win=1424 Len=0 > TSval=14704873 TSecr=15013610 > 19 2.002019 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=987 Ack=1424 Win=1424 Len=74 > TSval=14704873 TSecr=15013610 > 20 2.041643 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1424 Ack=1061 Win=1432 Len=0 > TSval=15013651 TSecr=14704873 > 21 3.002182 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1424 Ack=1061 Win=1432 Len=425 > TSval=15014611 TSecr=14704873 > 22 3.002605 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1061 Ack=1849 Win=1424 Len=419 > TSval=14705873 TSecr=15014611 > 23 3.002623 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1480 Win=1432 Len=0 > TSval=15014611 TSecr=14705873 > 24 3.002828 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=1480 Ack=1849 Win=1424 Len=74 > TSval=14705874 TSecr=15014611 > 25 3.002833 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1554 Win=1432 Len=0 > TSval=15014612 TSecr=14705874 > 26 3.002865 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=1849 Ack=1554 Win=1432 Len=74 > TSval=15014612 TSecr=14705874 > 27 3.042608 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=1554 Ack=1923 Win=1424 Len=0 > TSval=14705914 TSecr=15014612 > 28 3.479979 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 29 3.479991 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=2897 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 30 3.479994 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=5793 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 31 3.480091 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=8689 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 32 3.480101 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=11585 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 33 3.480584 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=38 Ack=14481 Win=1424 Len=0 > TSval=14706351 TSecr=15015089 > 34 3.480597 10.178.0.110 10.128.17.110 > TCP 2535 5647 → 45857 [PSH, ACK] Seq=14481 Ack=38 Win=710 Len=2469 > TSval=15015089 TSecr=14706351 > 35 3.481230 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=38 Ack=16950 Win=1424 Len=0 > TSval=14706352 TSecr=15015089 > 36 3.482661 10.128.17.110 10.178.0.110 > TCP 220 45857 → 5647 [PSH, ACK] Seq=38 Ack=16950 Win=1424 Len=154 > TSval=14706354 TSecr=15015089 > 37 3.482671 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=16950 Ack=192 Win=733 Len=0 > TSval=15015092 TSecr=14706354 > 38 3.484424 10.178.0.110 10.128.17.110 > TCP 232 5647 → 45857 [PSH, ACK] Seq=16950 Ack=192 Win=733 Len=166 > TSval=15015093 TSecr=14706354 > 39 3.485299 10.128.17.110 10.178.0.110 > TCP 468 45857 → 5647 [PSH, ACK] Seq=192 Ack=17116 Win=1424 Len=402 > TSval=14706356 TSecr=15015093 > 40 3.485985 10.178.0.110 10.128.17.110 > TCP 131 5647 → 45857 [PSH, ACK] Seq=17116 Ack=594 Win=755 Len=65 > TSval=15015095 TSecr=14706356 > 41 3.525616 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17181 Win=1424 Len=0 > TSval=14706397 TSecr=15015095 > 42 4.002777 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1923 Ack=1554 Win=1432 Len=425 > TSval=15015612 TSecr=14705914 > 43 4.003036 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1554 Ack=1923 Win=1424 Len=419 > TSval=14706874 TSecr=15014612 > 44 4.003043 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=1973 Ack=2348 Win=1424 Len=0 > TSval=14706874 TSecr=15015612 > 45 4.003482 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=1973 Ack=2348 Win=1424 Len=74 > TSval=14706874 TSecr=15015612 > 46 4.003552 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=2348 Ack=2047 Win=1432 Len=0 > TSval=15015612 TSecr=14706874 > 47 4.003750 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=2348 Ack=2047 Win=1432 Len=74 > TSval=15015613 TSecr=14706874 > 48 4.043644 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2047 Ack=2422 Win=1424 Len=0 > TSval=14706915 TSecr=15015613 > 49 4.076622 10.178.0.110 10.128.17.110 > TCP 133 5647 → 45857 [PSH, ACK] Seq=17181 Ack=594 Win=755 Len=67 > TSval=15015685 TSecr=14706397 > 50 4.076864 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17248 Win=1424 Len=0 > TSval=14706948 TSecr=15015685 > 51 4.078152 10.128.17.110 10.178.0.110 > TCP 119 45857 → 5647 [PSH, ACK] Seq=594 Ack=17248 Win=1424 Len=53 > TSval=14706949 TSecr=15015685 > 52 4.078234 10.178.0.110 10.128.17.110 > TCP 119 5647 → 45857 [PSH, ACK] Seq=17248 Ack=647 Win=755 Len=53 > TSval=15015687 TSecr=14706949 > 53 4.082247 10.128.17.110 10.178.0.110 > TCP 172 45857 → 5647 [PSH, ACK] Seq=647 Ack=17301 Win=1424 Len=106 > TSval=14706953 TSecr=15015687 > 54 4.121634 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=17301 Ack=753 Win=755 Len=0 > TSval=15015731 TSecr=14706953 > 55 5.003648 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=2422 Ack=2047 Win=1432 Len=425 > TSval=15016613 TSecr=14706915 > 56 5.003926 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2047 Ack=2847 Win=1424 Len=0 > TSval=14707875 TSecr=15016613 > 57 5.004452 10.128.17.110 10.178.0.110 > TCP 530 43052 → 5646 [PSH, ACK] Seq=2047 Ack=2847 Win=1424 Len=464 > TSval=14707875 TSecr=15016613 > 58 5.004702 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=2847 Ack=2511 Win=1432 Len=74 > TSval=15016614 TSecr=14707875 > 59 5.044893 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2511 Ack=2921 Win=1424 Len=0 > TSval=14707916 TSecr=15016614 > 60 6.005076 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=2921 Ack=2511 Win=1432 Len=425 > TSval=15017614 TSecr=14707916 > 61 6.005370 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2511 Ack=3346 Win=1424 Len=0 > TSval=14708876 TSecr=15017614 > 62 6.005714 10.128.17.110 10.178.0.110 > TCP 530 43052 → 5646 [PSH, ACK] Seq=2511 Ack=3346 Win=1424 Len=464 > TSval=14708877 TSecr=15017614 > 63 6.005929 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=3346 Ack=2975 Win=1432 Len=74 > TSval=15017615 TSecr=14708877 > 64 6.045606 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2975 Ack=3420 Win=1424 Len=0 > TSval=14708917 TSecr=15017615 > 65 7.005640 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=2975 Ack=3420 Win=1424 Len=419 > TSval=14709877 TSecr=15017615 > 66 7.006071 10.178.0.110 10.128.17.110 > TCP 536 5646 → 43052 [PSH, ACK] Seq=3420 Ack=3394 Win=1432 Len=470 > TSval=15018615 TSecr=14709877 > 67 7.006273 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=3394 Ack=3890 Win=1424 Len=0 > TSval=14709877 TSecr=15018615 > 68 7.006534 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=3394 Ack=3890 Win=1424 Len=74 > TSval=14709877 TSecr=15018615 > 69 7.045642 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=3890 Ack=3468 Win=1432 Len=0 > TSval=15018655 TSecr=14709877 > 70 8.006236 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=3890 Ack=3468 Win=1432 Len=425 > TSval=15019615 TSecr=14709877 > 71 8.007656 10.128.17.110 10.178.0.110 > TCP 530 43052 → 5646 [PSH, ACK] Seq=3468 Ack=4315 Win=1424 Len=464 > TSval=14710878 TSecr=15019615 > 72 8.007668 10.178.0.110 10.128.17.110

If you look at the Capsule logs, are there errors in /var/log/messages? You
should be able to look there and see Pulp kick up when the sync begins and
trace it down to find if it errored out. Another possibility is the Capsule
becomes bogged down and isn't able to respond which indicates to the task
something is wrong all the while it actually finished. What are the specs
of your capsules memory wise?

··· On Wed, Feb 10, 2016 at 1:00 PM, Nathan Peters wrote:

So this is still not working. Has no one seriously ever tested a capsule
before? This seems like a pretty core feature to have broken so badly.

I’ve now setup multiple capsules on different subnets, including on the
same subnet as the main Katello server (same switch etc), and none of them
ever finish synchronizing under any circumstances.

The one exception is if I assign a capsule to NO lifecycle environments,
its reaches 100% synchronization instantly and returns to the shell
prompt. If I add even a single view to the lifecycle environment (even if
that view only has like 10 packages in it), it will always freeze up and
not finish, not matter which capsule on which network I try to sync to.
The amount it freezes up at seems inconsistent though. Sometimes it’s 99%,
sometimes 75%, sometimes 95%.

Here is console output of several different tests with different lifecycle
environments available to the capsules. The final 100% was only with no
lifecycle environments assigned to the capsule and obviously that is not an
option as it transfers zero content to the capsule. In each case, I wait
at least 3 hours before ctrl-c and scan the disk free space on the capsule
to ensure that nothing is being transferred.

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…
]
[99%]^[…
] [99%]

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…
]
[75%]^[…
] [75%]

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…
]
[99%]^[…
] [99%]

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…]
[100%]

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…
] [95%]

On Tuesday, February 2, 2016 at 3:51:21 PM UTC-8, Nathan Peters wrote:

Katello 2.4 / Foreman 1.10.1 on CentOS 7.2

My capsule synchronization never seems to complete. It gets to 75% and
then hangs indefinitely.

This is what I see when I attempt it (it will stay like this for hours
until i Ctrl-C it)

------- errors ------
[root@katello1~]# hammer -u admin -p capsule content
synchronize --id 4
[…
] [75%]^
------- errors ------

If I attempt to promote content through the web interface, the
synchronize task that gets spawned fails :

------- errors ------

Host did not respond within 20 seconds. Sync has been cancelled. Is katello-agent installed and goferd running on the Host?

------- errors ------

The goferd service is defintely running :

---- troubleshooting --------
[root@katello1 ~]# systemctl | grep goferd

goferd.service
loaded active running Gofer Agent
------ troubleshooting ------

Also, I did a network capture, and there is definitely a SSL handshake
going on between the 2 servers and some data transfer, but regardless, the
content never actually shows up on the capsule :

No. Time Source Destination
Protocol Length Info
1 0.000000 10.178.0.110 10.128.17.110
TCP 491 5646 → 43052 [PSH, ACK] Seq=1 Ack=1 Win=1432 Len=425
TSval=15011609 TSecr=14701871
2 0.000656 10.128.17.110 10.178.0.110
TCP 485 43052 → 5646 [PSH, ACK] Seq=1 Ack=426 Win=1424 Len=419
TSval=14702871 TSecr=15011609
3 0.000676 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=426 Ack=420 Win=1432 Len=0
TSval=15011610 TSecr=14702871
4 0.000683 10.128.17.110 10.178.0.110
TCP 140 43052 → 5646 [PSH, ACK] Seq=420 Ack=426 Win=1424 Len=74
TSval=14702872 TSecr=15011609
5 0.000688 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=426 Ack=494 Win=1432 Len=0
TSval=15011610 TSecr=14702872
6 0.000912 10.178.0.110 10.128.17.110
TCP 140 5646 → 43052 [PSH, ACK] Seq=426 Ack=494 Win=1432 Len=74
TSval=15011610 TSecr=14702872
7 0.040612 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=494 Ack=500 Win=1424 Len=0
TSval=14702912 TSecr=15011610
8 1.000939 10.178.0.110 10.128.17.110
TCP 491 5646 → 43052 [PSH, ACK] Seq=500 Ack=494 Win=1432 Len=425
TSval=15012610 TSecr=14702912
9 1.001197 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=494 Ack=925 Win=1424 Len=0
TSval=14703872 TSecr=15012610
10 1.001549 10.128.17.110 10.178.0.110
TCP 140 43052 → 5646 [PSH, ACK] Seq=494 Ack=925 Win=1424 Len=74
TSval=14703872 TSecr=15012610
11 1.032061 10.128.17.110 10.178.0.110
TCP 103 45857 → 5647 [PSH, ACK] Seq=1 Ack=1 Win=1424 Len=37
TSval=14703903 TSecr=15008637
12 1.032070 10.178.0.110 10.128.17.110
TCP 66 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=0
TSval=15012641 TSecr=14703903
13 1.040637 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=925 Ack=568 Win=1432 Len=0
TSval=15012650 TSecr=14703872
14 2.001423 10.128.17.110 10.178.0.110
TCP 485 43052 → 5646 [PSH, ACK] Seq=568 Ack=925 Win=1424 Len=419
TSval=14704872 TSecr=15012650
15 2.001444 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=925 Ack=987 Win=1432 Len=0
TSval=15013610 TSecr=14704872
16 2.001552 10.178.0.110 10.128.17.110
TCP 491 5646 → 43052 [PSH, ACK] Seq=925 Ack=987 Win=1432 Len=425
TSval=15013610 TSecr=14704872
17 2.001742 10.178.0.110 10.128.17.110
TCP 140 5646 → 43052 [PSH, ACK] Seq=1350 Ack=987 Win=1432 Len=74
TSval=15013611 TSecr=14704872
18 2.002011 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=987 Ack=1424 Win=1424 Len=0
TSval=14704873 TSecr=15013610
19 2.002019 10.128.17.110 10.178.0.110
TCP 140 43052 → 5646 [PSH, ACK] Seq=987 Ack=1424 Win=1424 Len=74
TSval=14704873 TSecr=15013610
20 2.041643 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=1424 Ack=1061 Win=1432 Len=0
TSval=15013651 TSecr=14704873
21 3.002182 10.178.0.110 10.128.17.110
TCP 491 5646 → 43052 [PSH, ACK] Seq=1424 Ack=1061 Win=1432 Len=425
TSval=15014611 TSecr=14704873
22 3.002605 10.128.17.110 10.178.0.110
TCP 485 43052 → 5646 [PSH, ACK] Seq=1061 Ack=1849 Win=1424 Len=419
TSval=14705873 TSecr=15014611
23 3.002623 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1480 Win=1432 Len=0
TSval=15014611 TSecr=14705873
24 3.002828 10.128.17.110 10.178.0.110
TCP 140 43052 → 5646 [PSH, ACK] Seq=1480 Ack=1849 Win=1424 Len=74
TSval=14705874 TSecr=15014611
25 3.002833 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1554 Win=1432 Len=0
TSval=15014612 TSecr=14705874
26 3.002865 10.178.0.110 10.128.17.110
TCP 140 5646 → 43052 [PSH, ACK] Seq=1849 Ack=1554 Win=1432 Len=74
TSval=15014612 TSecr=14705874
27 3.042608 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=1554 Ack=1923 Win=1424 Len=0
TSval=14705914 TSecr=15014612
28 3.479979 10.178.0.110 10.128.17.110
TCP 2962 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=2896
TSval=15015089 TSecr=14703903
29 3.479991 10.178.0.110 10.128.17.110
TCP 2962 5647 → 45857 [ACK] Seq=2897 Ack=38 Win=710 Len=2896
TSval=15015089 TSecr=14703903
30 3.479994 10.178.0.110 10.128.17.110
TCP 2962 5647 → 45857 [ACK] Seq=5793 Ack=38 Win=710 Len=2896
TSval=15015089 TSecr=14703903
31 3.480091 10.178.0.110 10.128.17.110
TCP 2962 5647 → 45857 [ACK] Seq=8689 Ack=38 Win=710 Len=2896
TSval=15015089 TSecr=14703903
32 3.480101 10.178.0.110 10.128.17.110
TCP 2962 5647 → 45857 [ACK] Seq=11585 Ack=38 Win=710 Len=2896
TSval=15015089 TSecr=14703903
33 3.480584 10.128.17.110 10.178.0.110
TCP 66 45857 → 5647 [ACK] Seq=38 Ack=14481 Win=1424 Len=0
TSval=14706351 TSecr=15015089
34 3.480597 10.178.0.110 10.128.17.110
TCP 2535 5647 → 45857 [PSH, ACK] Seq=14481 Ack=38 Win=710 Len=2469
TSval=15015089 TSecr=14706351
35 3.481230 10.128.17.110 10.178.0.110
TCP 66 45857 → 5647 [ACK] Seq=38 Ack=16950 Win=1424 Len=0
TSval=14706352 TSecr=15015089
36 3.482661 10.128.17.110 10.178.0.110
TCP 220 45857 → 5647 [PSH, ACK] Seq=38 Ack=16950 Win=1424 Len=154
TSval=14706354 TSecr=15015089
37 3.482671 10.178.0.110 10.128.17.110
TCP 66 5647 → 45857 [ACK] Seq=16950 Ack=192 Win=733 Len=0
TSval=15015092 TSecr=14706354
38 3.484424 10.178.0.110 10.128.17.110
TCP 232 5647 → 45857 [PSH, ACK] Seq=16950 Ack=192 Win=733 Len=166
TSval=15015093 TSecr=14706354
39 3.485299 10.128.17.110 10.178.0.110
TCP 468 45857 → 5647 [PSH, ACK] Seq=192 Ack=17116 Win=1424 Len=402
TSval=14706356 TSecr=15015093
40 3.485985 10.178.0.110 10.128.17.110
TCP 131 5647 → 45857 [PSH, ACK] Seq=17116 Ack=594 Win=755 Len=65
TSval=15015095 TSecr=14706356
41 3.525616 10.128.17.110 10.178.0.110
TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17181 Win=1424 Len=0
TSval=14706397 TSecr=15015095
42 4.002777 10.178.0.110 10.128.17.110
TCP 491 5646 → 43052 [PSH, ACK] Seq=1923 Ack=1554 Win=1432 Len=425
TSval=15015612 TSecr=14705914
43 4.003036 10.128.17.110 10.178.0.110
TCP 485 43052 → 5646 [PSH, ACK] Seq=1554 Ack=1923 Win=1424 Len=419
TSval=14706874 TSecr=15014612
44 4.003043 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=1973 Ack=2348 Win=1424 Len=0
TSval=14706874 TSecr=15015612
45 4.003482 10.128.17.110 10.178.0.110
TCP 140 43052 → 5646 [PSH, ACK] Seq=1973 Ack=2348 Win=1424 Len=74
TSval=14706874 TSecr=15015612
46 4.003552 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=2348 Ack=2047 Win=1432 Len=0
TSval=15015612 TSecr=14706874
47 4.003750 10.178.0.110 10.128.17.110
TCP 140 5646 → 43052 [PSH, ACK] Seq=2348 Ack=2047 Win=1432 Len=74
TSval=15015613 TSecr=14706874
48 4.043644 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=2047 Ack=2422 Win=1424 Len=0
TSval=14706915 TSecr=15015613
49 4.076622 10.178.0.110 10.128.17.110
TCP 133 5647 → 45857 [PSH, ACK] Seq=17181 Ack=594 Win=755 Len=67
TSval=15015685 TSecr=14706397
50 4.076864 10.128.17.110 10.178.0.110
TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17248 Win=1424 Len=0
TSval=14706948 TSecr=15015685
51 4.078152 10.128.17.110 10.178.0.110
TCP 119 45857 → 5647 [PSH, ACK] Seq=594 Ack=17248 Win=1424 Len=53
TSval=14706949 TSecr=15015685
52 4.078234 10.178.0.110 10.128.17.110
TCP 119 5647 → 45857 [PSH, ACK] Seq=17248 Ack=647 Win=755 Len=53
TSval=15015687 TSecr=14706949
53 4.082247 10.128.17.110 10.178.0.110
TCP 172 45857 → 5647 [PSH, ACK] Seq=647 Ack=17301 Win=1424 Len=106
TSval=14706953 TSecr=15015687
54 4.121634 10.178.0.110 10.128.17.110
TCP 66 5647 → 45857 [ACK] Seq=17301 Ack=753 Win=755 Len=0
TSval=15015731 TSecr=14706953
55 5.003648 10.178.0.110 10.128.17.110
TCP 491 5646 → 43052 [PSH, ACK] Seq=2422 Ack=2047 Win=1432 Len=425
TSval=15016613 TSecr=14706915
56 5.003926 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=2047 Ack=2847 Win=1424 Len=0
TSval=14707875 TSecr=15016613
57 5.004452 10.128.17.110 10.178.0.110
TCP 530 43052 → 5646 [PSH, ACK] Seq=2047 Ack=2847 Win=1424 Len=464
TSval=14707875 TSecr=15016613
58 5.004702 10.178.0.110 10.128.17.110
TCP 140 5646 → 43052 [PSH, ACK] Seq=2847 Ack=2511 Win=1432 Len=74
TSval=15016614 TSecr=14707875
59 5.044893 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=2511 Ack=2921 Win=1424 Len=0
TSval=14707916 TSecr=15016614
60 6.005076 10.178.0.110 10.128.17.110
TCP 491 5646 → 43052 [PSH, ACK] Seq=2921 Ack=2511 Win=1432 Len=425
TSval=15017614 TSecr=14707916
61 6.005370 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=2511 Ack=3346 Win=1424 Len=0
TSval=14708876 TSecr=15017614
62 6.005714 10.128.17.110 10.178.0.110
TCP 530 43052 → 5646 [PSH, ACK] Seq=2511 Ack=3346 Win=1424 Len=464
TSval=14708877 TSecr=15017614
63 6.005929 1

On the capsule, /var/log/messages literally has no entries during or after
the attempt The last entry in it is my user login from ssh before I begin
the sync attempt. Neither does journalctl -f. /var/log/pulp has no files
in it. /var/log/foreman-proxy/production.log is 0 bytes.

Also, one capsule stalls at 75% with no log entries, and the other capsule
apparently doesn't respond although it is clearly listening.

[root@katello1~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…
]
[75%]^[…
] [75%]

[root@katello1~]# hammer -u admin capsule list
[Foreman] Password for admin:
—|-------------------------------------------|-------------------------------------------------------

ID NAME URL
1 katello1 https://katello1.mydomain.net:9090
4 katello2 https://katello2.mydomain.net:9090
6 katello3 https://katello3.mydomain.net:9090
------------------------------------------- -------------------------------------------------------

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 4
[Foreman] Password for admin:
[…]
[100%]
Host did not respond within 20 seconds. Sync has been cancelled. Is
katello-agent installed and goferd running on the Host?
[root@katello1 ~]# nmap katello2.mydomain.net -p 9090

Starting Nmap 6.40 ( http://nmap.org ) at 2016-02-11 09:38 PST
Nmap scan report for katello2.mydomain.net (10.128.17.110)
Host is up (0.00084s latency).
PORT STATE SERVICE
9090/tcp open zeus-admin

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds

---- then on the host that is 'not responding' —

[root@katello2 log]# netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 0.0.0.0:9090 0.0.0.0:*
LISTEN 21522/ruby
[root@katello2 log]# systemctl | grep "gofer"

goferd.service
loaded active running Gofer Agent

··· On Thursday, February 11, 2016 at 5:12:47 AM UTC-8, Eric Helms wrote: > > If you look at the Capsule logs, are there errors in /var/log/messages? > You should be able to look there and see Pulp kick up when the sync begins > and trace it down to find if it errored out. Another possibility is the > Capsule becomes bogged down and isn't able to respond which indicates to > the task something is wrong all the while it actually finished. What are > the specs of your capsules memory wise? > > On Wed, Feb 10, 2016 at 1:00 PM, Nathan Peters > wrote: > > So this is still not working. Has no one seriously ever tested a capsule > before? This seems like a pretty core feature to have broken so badly. > > I've now setup multiple capsules on different subnets, including on the > same subnet as the main Katello server (same switch etc), and none of them > ever finish synchronizing under any circumstances. > > The one exception is if I assign a capsule to NO lifecycle environments, > its reaches 100% synchronization instantly and returns to the shell > prompt. If I add even a single view to the lifecycle environment (even if > that view only has like 10 packages in it), it will *always* freeze up and > not finish, not matter which capsule on which network I try to sync to. > The amount it freezes up at seems inconsistent though. Sometimes it's 99%, > sometimes 75%, sometimes 95%. > > Here is console output of several different tests with different lifecycle > environments available to the capsules. The final 100% was only with no > lifecycle environments assigned to the capsule and obviously that is not an > option as it transfers zero content to the capsule. In each case, I wait > at least 3 hours before ctrl-c and scan the disk free space on the capsule > to ensure that nothing is being transferred. > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [..................................................................................................................................................................................... > ] > [99%]^[..................................................................................................................................................................................... > ] [99%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [........................................................................................................................................ > ] > [75%]^[........................................................................................................................................ > ] [75%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [..................................................................................................................................................................................... > ] > [99%]^[..................................................................................................................................................................................... > ] [99%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [.....................................................................................................................................................................................] > [100%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [............................................................................................................................................................................ > ] [95%] > > > On Tuesday, February 2, 2016 at 3:51:21 PM UTC-8, Nathan Peters wrote: > > Katello 2.4 / Foreman 1.10.1 on CentOS 7.2 > > My capsule synchronization never seems to complete. It gets to 75% and > then hangs indefinitely. > > This is what I see when I attempt it (it will stay like this for hours > until i Ctrl-C it) > > ------- errors ------ > [root@katello1~]# hammer -u admin -p capsule content > synchronize --id 4 > [............................................................................... > ] [75%]^ > ------- errors ------ > > If I attempt to promote content through the web interface, the synchronize > task that gets spawned fails : > > ------- errors ------ > > Host did not respond within 20 seconds. Sync has been cancelled. Is katello-agent installed and goferd running on the Host? > > ------- errors ------ > > > The goferd service is defintely running : > > ---- troubleshooting -------- > [root@katello1 ~]# systemctl | grep goferd > > goferd.service > loaded active running Gofer Agent > ------ troubleshooting ------ > > Also, I did a network capture, and there is definitely a SSL handshake > going on between the 2 servers and some data transfer, but regardless, the > content never actually shows up on the capsule : > > No. Time Source Destination > Protocol Length Info > 1 0.000000 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1 Ack=1 Win=1432 Len=425 > TSval=15011609 TSecr=14701871 > 2 0.000656 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1 Ack=426 Win=1424 Len=419 > TSval=14702871 TSecr=15011609 > 3 0.000676 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=426 Ack=420 Win=1432 Len=0 > TSval=15011610 TSecr=14702871 > 4 0.000683 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=420 Ack=426 Win=1424 Len=74 > TSval=14702872 TSecr=15011609 > 5 0.000688 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=426 Ack=494 Win=1432 Len=0 > TSval=15011610 TSecr=14702872 > 6 0.000912 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=426 Ack=494 Win=1432 Len=74 > TSval=15011610 TSecr=14702872 > 7 0.040612 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=494 Ack=500 Win=1424 Len=0 > TSval=14702912 TSecr=15011610 > 8 1.000939 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=500 Ack=494 Win=1432 Len=425 > TSval=15012610 TSecr=14702912 > 9 1.001197 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=494 Ack=925 Win=1424 Len=0 > TSval=14703872 TSecr=15012610 > 10 1.001549 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=494 Ack=925 Win=1424 Len=74 > TSval=14703872 TSecr=15012610 > 11 1.032061 10.128.17.110 10.178.0.110 > TCP 103 45857 → 5647 [PSH, ACK] Seq=1 Ack=1 Win=1424 Len=37 > TSval=14703903 TSecr=15008637 > 12 1.032070 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=0 > TSval=15012641 TSecr=14703903 > 13 1.040637 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=925 Ack=568 Win=1432 Len=0 > TSval=15012650 TSecr=14703872 > 14 2.001423 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=568 Ack=925 Win=1424 Len=419 > TSval=14704872 TSecr=15012650 > 15 2.001444 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=925 Ack=987 Win=1432 Len=0 > TSval=15013610 TSecr=14704872 > 16 2.001552 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=925 Ack=987 Win=1432 Len=425 > TSval=15013610 TSecr=14704872 > 17 2.001742 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=1350 Ack=987 Win=1432 Len=74 > TSval=15013611 TSecr=14704872 > 18 2.002011 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=987 Ack=1424 Win=1424 Len=0 > TSval=14704873 TSecr=15013610 > 19 2.002019 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=987 Ack=1424 Win=1424 Len=74 > TSval=14704873 TSecr=15013610 > 20 2.041643 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1424 Ack=1061 Win=1432 Len=0 > TSval=15013651 TSecr=14704873 > 21 3.002182 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1424 Ack=1061 Win=1432 Len=425 > TSval=15014611 TSecr=14704873 > 22 3.002605 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1061 Ack=1849 Win=1424 Len=419 > TSval=14705873 TSecr=15014611 > 23 3.002623 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1480 Win=1432 Len=0 > TSval=15014611 TSecr=14705873 > 24 3.002828 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=1480 Ack=1849 Win=1424 Len=74 > TSval=14705874 TSecr=15014611 > 25 3.002833 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1554 Win=1432 Len=0 > TSval=15014612 TSecr=14705874 > 26 3.002865 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=1849 Ack=1554 Win=1432 Len=74 > TSval=15014612 TSecr=14705874 > 27 3.042608 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=1554 Ack=1923 Win=1424 Len=0 > TSval=14705914 TSecr=15014612 > 28 3.479979 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 29 3.479991 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=2897 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 30 3.479994 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=5793 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 31 3.480091 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=8689 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 32 3.480101 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=11585 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 33 3.480584 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=38 Ack=14481 Win=1424 Len=0 > TSval=14706351 TSecr=15015089 > 34 3.480597 10.178.0.110 10.128.17.110 > TCP 2535 5647 → 45857 [PSH, ACK] Seq=14481 Ack=38 Win=710 Len=2469 > TSval=15015089 TSecr=14706351 > 35 3.481230 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=38 Ack=16950 Win=1424 Len=0 > TSval=14706352 TSecr=15015089 > 36 3.482661 10.128.17.110 10.178.0.110 > TCP 220 45857 → 5647 [PSH, ACK] Seq=38 Ack=16950 Win=1424 Len=154 > TSval=14706354 TSecr=15015089 > 37 3.482671 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=16950 Ack=192 Win=733 Len=0 > TSval=15015092 TSecr=14706354 > 38 3.484424 10.178.0.110 10.128.17.110 > TCP 232 5647 → 45857 [PSH, ACK] Seq=16950 Ack=192 Win=733 Len=166 > TSval=15015093 TSecr=14706354 > 39 3.485299 10.128.17.110 10.178.0.110 > TCP 468 45857 → 5647 [PSH, ACK] Seq=192 Ack=17116 Win=1424 Len=402 > TSval=14706356 TSecr=15015093 > 40 3.485985 10.178.0.110 10.128.17.110 > TCP 131 5647 → 45857 [PSH, ACK] Seq=17116 Ack=594 Win=755 Len=65 > TSval=15015095 TSecr=14706356 > 41 3.525616 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17181 Win=1424 Len=0 > TSval=14706397 TSecr=15015095 > 42 4.002777 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1923 Ack=1554 Win=1432 Len=425 > TSval=15015612 TSecr=14705914 > 43 4.003036 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1554 Ack=1923 Win=1424 Len=419 > TSval=14706874 TSecr=15014612 > 44 4.003043 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=1973 Ack=2348 Win=1424 Len=0 > TSval=14706874 TSecr=15015612 > 45 4.003482 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=1973 Ack=2348 Win=1424 Len=74 > TSval=14706874 TSecr=15015612 > 46 4.003552 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=2348 Ack=2047 Win=1432 Len=0 > TSval=15015612 TSecr=14706874 > 47 4.003750 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=2348 Ack=2047 Win=1432 Len=74 > TSval=15015613 TSecr=14706874 > 48 4.043644 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2047 Ack=2422 Win=1424 Len=0 > TSval=14706915 TSecr=15015613 > 49 4.076622 10.178.0.110 10.128.17.110 > TCP 133 5647 → 45857 [PSH, ACK] Seq=17181 Ack=594 Win=755 Len=67 > TSval=15015685 TSecr=14706397 > 50 4.076864 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17248 Win=1424 Len=0 > TSval=14706948 TSecr=15015685 > 51 4.078152 10.128.17.110 10.178.0.110 > TCP 119 45857 → 5647 [PSH, ACK] Seq=594 Ack=17248 Win=1424 Len=53 > TSval=14706949 TSecr=15015685 > 52 4.078234 10.178.0.110 10.128.17.110 > TCP 119 5647 → 45857 [PSH, ACK] Seq=17248 Ack=647 Win=755 Len=53 > TSval=15015687 TSecr=14706949 > 53 4.082247 10.128.17.110 10.178.0.110 > TCP 172 45857 → 5647 [PSH, ACK] Seq=647 Ack=17301 Win=1424 Len=106 > TSval=14706953 TSecr=15015687 > 54 4.121634 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=17301 Ack=753 Win=755 Len=0 > TSval=15015731 TSecr=14706953 > 55 5.003648 10.178.0.110

More troubleshooting :

Here is the production.log from the katello master during a sync attempt
that freezes up at 75%. You can see some log entries at the start, and
then it just seems to go into a permanent loop grabbing the status so that
it can refresh that percentage display bar I have showing in the other
window.

75% freeze up and ctrl-c logs (/var/log/messages and journalctl both report
the same line) from master
Feb 11 09:50:28 katello1 pulp: kombu.transport.qpid:INFO: Connected to qpid
with SASL mechanism PLAIN

20 second timeout logs from (/var/log/messages and journalctl both report
the same line) katello master
Feb 11 09:51:38 katello1 pulp: kombu.transport.qpid:INFO: Connected to qpid
with SASL mechanism PLAIN
Feb 11 09:51:59 katello1.dev-mydomain.net pulp[14928]:
pulp.server.async.tasks:INFO: Task canceled:
e885d2ec-b6b9-4a36-9b7f-7f232c07f787.

[root@katello1 foreman]# tail -f production.log
2016-02-11 09:54:21 [app] [I] Completed 200 OK in 71ms (Views: 0.7ms |
ActiveRecord: 7.2ms)
2016-02-11 09:54:21 [app] [I]
>
> Started POST "/api/reports" for 10.128.17.110 at 2016-02-11 09:54:21
-0800
2016-02-11 09:54:21 [app] [I] Processing by
Api::V2::ReportsController#create as JSON
2016-02-11 09:54:21 [app] [I] Parameters: {"report"=>"[FILTERED]",
"apiv"=>"v2"}
2016-02-11 09:54:21 [app] [I] processing report for katello2.mydomain.net
2016-02-11 09:54:21 [app] [I] Imported report for katello2.mydomain.net in
0.01 seconds
2016-02-11 09:54:21 [app] [I] Rendered api/v2/reports/create.json.rabl
(6.0ms)
2016-02-11 09:54:21 [app] [I] Completed 201 Created in 37ms (Views: 6.2ms |
ActiveRecord: 7.0ms)
2016-02-11 09:55:00 [app] [I]
>
> Started POST "/katello/api/capsules/6/content/sync" for ::1 at
2016-02-11 09:55:00 -0800
2016-02-11 09:55:00 [app] [I] Processing by
Katello::Api::V2::CapsuleContentController#sync as
application/json;version=2
2016-02-11 09:55:00 [app] [I] Parameters: {"api_version"=>"v2",
"id"=>"6", "capsule_content"=>{}}
2016-02-11 09:55:00 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:00 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-2.4.0/app/views/katello/api/v2/common/async.json.rabl
within katello/api/v2/layouts/resource (17.4ms)
2016-02-11 09:55:00 [app] [I] Completed 202 Accepted in 233ms (Views:
14.7ms | ActiveRecord: 3.3ms)
2016-02-11 09:55:00 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:00 -0800
2016-02-11 09:55:00 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:00 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:00 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:00 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(16.5ms)
2016-02-11 09:55:00 [app] [I] Completed 200 OK in 39ms (Views: 14.2ms |
ActiveRecord: 7.0ms)
2016-02-11 09:55:02 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:02 -0800
2016-02-11 09:55:02 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:02 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:02 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:02 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(17.3ms)
2016-02-11 09:55:02 [app] [I] Completed 200 OK in 40ms (Views: 15.0ms |
ActiveRecord: 7.7ms)
2016-02-11 09:55:04 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:04 -0800
2016-02-11 09:55:04 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:04 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:04 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:04 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(17.4ms)
2016-02-11 09:55:04 [app] [I] Completed 200 OK in 40ms (Views: 15.1ms |
ActiveRecord: 7.7ms)
2016-02-11 09:55:06 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:06 -0800
2016-02-11 09:55:06 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:06 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:06 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:07 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(18.9ms)
2016-02-11 09:55:07 [app] [I] Completed 200 OK in 44ms (Views: 16.6ms |
ActiveRecord: 8.1ms)
2016-02-11 09:55:08 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:09 -0800
2016-02-11 09:55:09 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:09 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:09 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:09 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(18.0ms)
2016-02-11 09:55:09 [app] [I] Completed 200 OK in 42ms (Views: 15.6ms |
ActiveRecord: 7.9ms)
2016-02-11 09:55:11 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:11 -0800
2016-02-11 09:55:11 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:11 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:11 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:11 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(17.4ms)
2016-02-11 09:55:11 [app] [I] Completed 200 OK in 45ms (Views: 15.1ms |
ActiveRecord: 12.0ms)
2016-02-11 09:55:12 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:13 -0800
2016-02-11 09:55:13 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:13 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:13 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:13 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(17.8ms)
2016-02-11 09:55:13 [app] [I] Completed 200 OK in 42ms (Views: 15.5ms |
ActiveRecord: 8.7ms)
2016-02-11 09:55:15 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:15 -0800
2016-02-11 09:55:15 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:15 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:15 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:15 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(18.0ms)
2016-02-11 09:55:15 [app] [I] Completed 200 OK in 42ms (Views: 15.7ms |
ActiveRecord: 7.7ms)
2016-02-11 09:55:17 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:17 -0800
2016-02-11 09:55:17 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:17 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:17 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:17 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(17.1ms)
2016-02-11 09:55:17 [app] [I] Completed 200 OK in 41ms (Views: 14.7ms |
ActiveRecord: 8.0ms)
2016-02-11 09:55:17 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:19 -0800
2016-02-11 09:55:19 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:19 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:19 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:19 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(18.3ms)
2016-02-11 09:55:19 [app] [I] Completed 200 OK in 43ms (Views: 16.0ms |
ActiveRecord: 7.8ms)
2016-02-11 09:55:19 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:21 -0800
2016-02-11 09:55:21 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:21 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:21 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:21 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(17.7ms)
2016-02-11 09:55:21 [app] [I] Completed 200 OK in 41ms (Views: 15.2ms |
ActiveRecord: 8.4ms)
2016-02-11 09:55:23 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:23 -0800
2016-02-11 09:55:23 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:23 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:23 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:23 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(17.5ms)
2016-02-11 09:55:23 [app] [I] Completed 200 OK in 41ms (Views: 15.1ms |
ActiveRecord: 7.8ms)
2016-02-11 09:55:25 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:25 -0800
2016-02-11 09:55:25 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:25 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:25 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:25 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(17.6ms)
2016-02-11 09:55:25 [app] [I] Completed 200 OK in 41ms (Views: 15.3ms |
ActiveRecord: 7.8ms)
2016-02-11 09:55:26 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:27 -0800
2016-02-11 09:55:27 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:27 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:27 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:27 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(18.5ms)
2016-02-11 09:55:27 [app] [I] Completed 200 OK in 43ms (Views: 16.2ms |
ActiveRecord: 7.8ms)
2016-02-11 09:55:29 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:29 -0800
2016-02-11 09:55:29 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:29 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:29 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:29 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(18.6ms)
2016-02-11 09:55:29 [app] [I] Completed 200 OK in 44ms (Views: 16.2ms |
ActiveRecord: 8.2ms)
2016-02-11 09:55:31 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:31 -0800
2016-02-11 09:55:31 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:31 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:31 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:31 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(17.7ms)
2016-02-11 09:55:31 [app] [I] Completed 200 OK in 42ms (Views: 15.5ms |
ActiveRecord: 7.9ms)
2016-02-11 09:55:33 [app] [I]
>
> Started GET
"/foreman_tasks/api/tasks/86cce0bd-d58e-4b42-b569-9b502e454479" for ::1 at
2016-02-11 09:55:33 -0800
2016-02-11 09:55:33 [app] [I] Processing by
ForemanTasks::Api::TasksController#show as application/json;version=2
2016-02-11 09:55:33 [app] [I] Parameters:
{"id"=>"86cce0bd-d58e-4b42-b569-9b502e454479", "task"=>{}}
2016-02-11 09:55:33 [app] [I] Authorized user admin(Admin User)
2016-02-11 09:55:33 [app] [I] Rendered
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/api/tasks/show.json.rabl
(18.3ms)
2016-02-11 09:55:33 [app] [I] Completed 200 OK in 43ms (Views: 16.1ms |
ActiveRecord: 7.9ms)
^C

··· On Thursday, February 11, 2016 at 5:12:47 AM UTC-8, Eric Helms wrote: > > If you look at the Capsule logs, are there errors in /var/log/messages? > You should be able to look there and see Pulp kick up when the sync begins > and trace it down to find if it errored out. Another possibility is the > Capsule becomes bogged down and isn't able to respond which indicates to > the task something is wrong all the while it actually finished. What are > the specs of your capsules memory wise? > > On Wed, Feb 10, 2016 at 1:00 PM, Nathan Peters > wrote: > > So this is still not working. Has no one seriously ever tested a capsule > before? This seems like a pretty core feature to have broken so badly. > > I've now setup multiple capsules on different subnets, including on the > same subnet as the main Katello server (same switch etc), and none of them > ever finish synchronizing under any circumstances. > > The one exception is if I assign a capsule to NO lifecycle environments, > its reaches 100% synchronization instantly and returns to the shell > prompt. If I add even a single view to the lifecycle environment (even if > that view only has like 10 packages in it), it will *always* freeze up and > not finish, not matter which capsule on which network I try to sync to. > The amount it freezes up at seems inconsistent though. Sometimes it's 99%, > sometimes 75%, sometimes 95%. > > Here is console output of several different tests with different lifecycle > environments available to the capsules. The final 100% was only with no > lifecycle environments assigned to the capsule and obviously that is not an > option as it transfers zero content to the capsule. In each case, I wait > at least 3 hours before ctrl-c and scan the disk free space on the capsule > to ensure that nothing is being transferred. > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [..................................................................................................................................................................................... > ] > [99%]^[..................................................................................................................................................................................... > ] [99%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [........................................................................................................................................ > ] > [75%]^[........................................................................................................................................ > ] [75%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [..................................................................................................................................................................................... > ] > [99%]^[..................................................................................................................................................................................... > ] [99%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [.....................................................................................................................................................................................] > [100%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [............................................................................................................................................................................ > ] [95%] > > > On Tuesday, February 2, 2016 at 3:51:21 PM UTC-8, Nathan Peters wrote: > > Katello 2.4 / Foreman 1.10.1 on CentOS 7.2 > > My capsule synchronization never seems to complete. It gets to 75% and > then hangs indefinitely. > > This is what I see when I

Sorry forgot to answer your question about memory.

All Katello servers are 16GB RAM, 4 core XEON processor, 2TB disk space.

What's strange about this is that the TCP captures at the start seems to
match up with the logs.

You can see the QPID SASL negotion go through in the logs, and you can see
some TLS stuff happen on port 443 during the TCP dump, but then in the
logs, it just goes into a loop getting status and not doing anything, and
the tcp dump just shows a session that turns into a bunch of pushes and
acks with no actual traffic or data after the handshake.

··· On Thursday, February 11, 2016 at 5:12:47 AM UTC-8, Eric Helms wrote: > > If you look at the Capsule logs, are there errors in /var/log/messages? > You should be able to look there and see Pulp kick up when the sync begins > and trace it down to find if it errored out. Another possibility is the > Capsule becomes bogged down and isn't able to respond which indicates to > the task something is wrong all the while it actually finished. What are > the specs of your capsules memory wise? > > On Wed, Feb 10, 2016 at 1:00 PM, Nathan Peters > wrote: > > So this is still not working. Has no one seriously ever tested a capsule > before? This seems like a pretty core feature to have broken so badly. > > I've now setup multiple capsules on different subnets, including on the > same subnet as the main Katello server (same switch etc), and none of them > ever finish synchronizing under any circumstances. > > The one exception is if I assign a capsule to NO lifecycle environments, > its reaches 100% synchronization instantly and returns to the shell > prompt. If I add even a single view to the lifecycle environment (even if > that view only has like 10 packages in it), it will *always* freeze up and > not finish, not matter which capsule on which network I try to sync to. > The amount it freezes up at seems inconsistent though. Sometimes it's 99%, > sometimes 75%, sometimes 95%. > > Here is console output of several different tests with different lifecycle > environments available to the capsules. The final 100% was only with no > lifecycle environments assigned to the capsule and obviously that is not an > option as it transfers zero content to the capsule. In each case, I wait > at least 3 hours before ctrl-c and scan the disk free space on the capsule > to ensure that nothing is being transferred. > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [..................................................................................................................................................................................... > ] > [99%]^[..................................................................................................................................................................................... > ] [99%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [........................................................................................................................................ > ] > [75%]^[........................................................................................................................................ > ] [75%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [..................................................................................................................................................................................... > ] > [99%]^[..................................................................................................................................................................................... > ] [99%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [.....................................................................................................................................................................................] > [100%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [............................................................................................................................................................................ > ] [95%] > > > On Tuesday, February 2, 2016 at 3:51:21 PM UTC-8, Nathan Peters wrote: > > Katello 2.4 / Foreman 1.10.1 on CentOS 7.2 > > My capsule synchronization never seems to complete. It gets to 75% and > then hangs indefinitely. > > This is what I see when I attempt it (it will stay like this for hours > until i Ctrl-C it) > > ------- errors ------ > [root@katello1~]# hammer -u admin -p capsule content > synchronize --id 4 > [............................................................................... > ] [75%]^ > ------- errors ------ > > If I attempt to promote content through the web interface, the synchronize > task that gets spawned fails : > > ------- errors ------ > > Host did not respond within 20 seconds. Sync has been cancelled. Is katello-agent installed and goferd running on the Host? > > ------- errors ------ > > > The goferd service is defintely running : > > ---- troubleshooting -------- > [root@katello1 ~]# systemctl | grep goferd > > goferd.service > loaded active running Gofer Agent > ------ troubleshooting ------ > > Also, I did a network capture, and there is definitely a SSL handshake > going on between the 2 servers and some data transfer, but regardless, the > content never actually shows up on the capsule : > > No. Time Source Destination > Protocol Length Info > 1 0.000000 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1 Ack=1 Win=1432 Len=425 > TSval=15011609 TSecr=14701871 > 2 0.000656 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1 Ack=426 Win=1424 Len=419 > TSval=14702871 TSecr=15011609 > 3 0.000676 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=426 Ack=420 Win=1432 Len=0 > TSval=15011610 TSecr=14702871 > 4 0.000683 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=420 Ack=426 Win=1424 Len=74 > TSval=14702872 TSecr=15011609 > 5 0.000688 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=426 Ack=494 Win=1432 Len=0 > TSval=15011610 TSecr=14702872 > 6 0.000912 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=426 Ack=494 Win=1432 Len=74 > TSval=15011610 TSecr=14702872 > 7 0.040612 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=494 Ack=500 Win=1424 Len=0 > TSval=14702912 TSecr=15011610 > 8 1.000939 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=500 Ack=494 Win=1432 Len=425 > TSval=15012610 TSecr=14702912 > 9 1.001197 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=494 Ack=925 Win=1424 Len=0 > TSval=14703872 TSecr=15012610 > 10 1.001549 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=494 Ack=925 Win=1424 Len=74 > TSval=14703872 TSecr=15012610 > 11 1.032061 10.128.17.110 10.178.0.110 > TCP 103 45857 → 5647 [PSH, ACK] Seq=1 Ack=1 Win=1424 Len=37 > TSval=14703903 TSecr=15008637 > 12 1.032070 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=0 > TSval=15012641 TSecr=14703903 > 13 1.040637 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=925 Ack=568 Win=1432 Len=0 > TSval=15012650 TSecr=14703872 > 14 2.001423 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=568 Ack=925 Win=1424 Len=419 > TSval=14704872 TSecr=15012650 > 15 2.001444 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=925 Ack=987 Win=1432 Len=0 > TSval=15013610 TSecr=14704872 > 16 2.001552 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=925 Ack=987 Win=1432 Len=425 > TSval=15013610 TSecr=14704872 > 17 2.001742 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=1350 Ack=987 Win=1432 Len=74 > TSval=15013611 TSecr=14704872 > 18 2.002011 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=987 Ack=1424 Win=1424 Len=0 > TSval=14704873 TSecr=15013610 > 19 2.002019 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=987 Ack=1424 Win=1424 Len=74 > TSval=14704873 TSecr=15013610 > 20 2.041643 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1424 Ack=1061 Win=1432 Len=0 > TSval=15013651 TSecr=14704873 > 21 3.002182 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1424 Ack=1061 Win=1432 Len=425 > TSval=15014611 TSecr=14704873 > 22 3.002605 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1061 Ack=1849 Win=1424 Len=419 > TSval=14705873 TSecr=15014611 > 23 3.002623 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1480 Win=1432 Len=0 > TSval=15014611 TSecr=14705873 > 24 3.002828 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=1480 Ack=1849 Win=1424 Len=74 > TSval=14705874 TSecr=15014611 > 25 3.002833 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1554 Win=1432 Len=0 > TSval=15014612 TSecr=14705874 > 26 3.002865 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=1849 Ack=1554 Win=1432 Len=74 > TSval=15014612 TSecr=14705874 > 27 3.042608 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=1554 Ack=1923 Win=1424 Len=0 > TSval=14705914 TSecr=15014612 > 28 3.479979 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 29 3.479991 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=2897 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 30 3.479994 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=5793 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 31 3.480091 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=8689 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 32 3.480101 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=11585 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 33 3.480584 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=38 Ack=14481 Win=1424 Len=0 > TSval=14706351 TSecr=15015089 > 34 3.480597 10.178.0.110 10.128.17.110 > TCP 2535 5647 → 45857 [PSH, ACK] Seq=14481 Ack=38 Win=710 Len=2469 > TSval=15015089 TSecr=14706351 > 35 3.481230 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=38 Ack=16950 Win=1424 Len=0 > TSval=14706352 TSecr=15015089 > 36 3.482661 10.128.17.110 10.178.0.110 > TCP 220 45857 → 5647 [PSH, ACK] Seq=38 Ack=16950 Win=1424 Len=154 > TSval=14706354 TSecr=15015089 > 37 3.482671 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=16950 Ack=192 Win=733 Len=0 > TSval=15015092 TSecr=14706354 > 38 3.484424 10.178.0.110 10.128.17.110 > TCP 232 5647 → 45857 [PSH, ACK] Seq=16950 Ack=192 Win=733 Len=166 > TSval=15015093 TSecr=14706354 > 39 3.485299 10.128.17.110 10.178.0.110 > TCP 468 45857 → 5647 [PSH, ACK] Seq=192 Ack=17116 Win=1424 Len=402 > TSval=14706356 TSecr=15015093 > 40 3.485985 10.178.0.110 10.128.17.110 > TCP 131 5647 → 45857 [PSH, ACK] Seq=17116 Ack=594 Win=755 Len=65 > TSval=15015095 TSecr=14706356 > 41 3.525616 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17181 Win=1424 Len=0 > TSval=14706397 TSecr=15015095 > 42 4.002777 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1923 Ack=1554 Win=1432 Len=425 > TSval=15015612 TSecr=14705914 > 43 4.003036 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1554 Ack=1923 Win=1424 Len=419 > TSval=14706874 TSecr=15014612 > 44 4.003043 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=1973 Ack=2348 Win=1424 Len=0 > TSval=14706874 TSecr=15015612 > 45 4.003482 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=1973 Ack=2348 Win=1424 Len=74 > TSval=14706874 TSecr=15015612 > 46 4.003552 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=2348 Ack=2047 Win=1432 Len=0 > TSval=15015612 TSecr=14706874 > 47 4.003750 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=2348 Ack=2047 Win=1432 Len=74 > TSval=15015613 TSecr=14706874 > 48 4.043644 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2047 Ack=2422 Win=1424 Len=0 > TSval=14706915 TSecr=15015613 > 49 4.076622 10.178.0.110 10.128.17.110 > TCP 133 5647 → 45857 [PSH, ACK] Seq=17181 Ack=594 Win=755 Len=67 > TSval=15015685 TSecr=14706397 > 50 4.076864 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17248 Win=1424 Len=0 > TSval=14706948 TSecr=15015685 > 51 4.078152 10.128.17.110 10.178.0.110 > TCP 119 45857 → 5647 [PSH, ACK] Seq=594 Ack=17248 Win=1424 Len=53 > TSval=14706949 TSecr=15015685 > 52 4.078234 10.178.0.110 10.128.17.110 > TCP 119 5647 → 45857 [PSH, ACK] Seq=17248 Ack=647 Win=755 Len=53 > TSval=15015687 TSecr=14706949 > 53 4.082247 10.128.17.110 10.178.0.110 > TCP 172 45857 → 5647 [PSH, ACK] Seq=647 Ack=17301 Win=1424 Len=106 > TSval=14706953 TSecr=15015687 > 54 4.121634 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=17301 Ack=753 Win=755 Len=0 > TSval=15015731 TSecr=14706953 > 55 5.003648 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=2422 Ack=2047 Win=1432 Len=425 > TSval=15016613 TSecr=14706915 > 56 5.003926 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2047 Ack=2847 Win=1424 Len=0 > TSval=14707875 TSecr=15016613 > 57 5.004452 10.128.17.110 10.178.0.110 > TCP 530 43052 → 5646 [PSH, ACK] Seq=2047 Ack=2847 Win=1424 Len=464 > TSval=14707875 TSecr=15016613 > 58 5.004702 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=2847 Ack=2511 Win=1432 Len=74 > TSval=15016614 TSecr=14707875 > 59 5.044893 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2511 Ack=2921 Win=1424 Len=0 > TSval=14707916 TSecr=15016614 > 60 6.005076 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=2921 Ack=2511 Win=1432 Len=425 > TSval=15017614 TSecr=14707916 > 61 6.005370 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2511 Ack=3346 Win=1424 Len=0 > TSval=14708876 TSecr=15017614 > 62 6.005714 10.128.17.110 10.178.0.110 > TCP 530 43052 → 5646 [PSH, ACK] Seq=2511 Ack=3346 Win=1424 Len=464 > TSval=14708877 TSecr=15017614 > 63 6.005929 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=3346 Ack=2975 Win=1432 Len=74 > TSval=15017615 TSecr=14708877 > 64 6.045606 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2975 Ack=3420 Win=1424 Len=0 > TSval=14708917 TSecr=1501761

Nathan,

If you haven't already, can you try restarting goferd on the capsule and
check /var/log/messages again?

-John

··· On 02/11/2016 01:14 PM, Nathan Peters wrote: > Sorry forgot to answer your question about memory. > > All Katello servers are 16GB RAM, 4 core XEON processor, 2TB disk space. > > What's strange about this is that the TCP captures at the start seems > to match up with the logs. > > You can see the QPID SASL negotion go through in the logs, and you can > see some TLS stuff happen on port 443 during the TCP dump, but then in > the logs, it just goes into a loop getting status and not doing > anything, and the tcp dump just shows a session that turns into a > bunch of pushes and acks with no actual traffic or data after the > handshake. > > > On Thursday, February 11, 2016 at 5:12:47 AM UTC-8, Eric Helms wrote: > > If you look at the Capsule logs, are there errors in > /var/log/messages? You should be able to look there and see Pulp > kick up when the sync begins and trace it down to find if it > errored out. Another possibility is the Capsule becomes bogged > down and isn't able to respond which indicates to the task > something is wrong all the while it actually finished. What are > the specs of your capsules memory wise? > > On Wed, Feb 10, 2016 at 1:00 PM, Nathan Peters > <nathanp...@gmail.com > wrote: > > So this is still not working. Has no one seriously ever > tested a capsule before? This seems like a pretty core > feature to have broken so badly. > > I've now setup multiple capsules on different subnets, > including on the same subnet as the main Katello server (same > switch etc), and none of them ever finish synchronizing under > any circumstances. > > The one exception is if I assign a capsule to NO lifecycle > environments, its reaches 100% synchronization instantly and > returns to the shell prompt. If I add even a single view to > the lifecycle environment (even if that view only has like 10 > packages in it), it will *always* freeze up and not finish, > not matter which capsule on which network I try to sync to. > The amount it freezes up at seems inconsistent though. > Sometimes it's 99%, sometimes 75%, sometimes 95%. > > Here is console output of several different tests with > different lifecycle environments available to the capsules. > The final 100% was only with no lifecycle environments > assigned to the capsule and obviously that is not an option as > it transfers zero content to the capsule. In each case, I > wait at least 3 hours before ctrl-c and scan the disk free > space on the capsule to ensure that nothing is being transferred. > > [root@katello1 ~]# hammer -u admin capsule content synchronize > --id 6 > [Foreman] Password for admin: > [..................................................................................................................................................................................... > ] > [99%]^[..................................................................................................................................................................................... > ] [99%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize > --id 6 > [Foreman] Password for admin: > [........................................................................................................................................ > ] > [75%]^[........................................................................................................................................ > ] [75%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize > --id 6 > [Foreman] Password for admin: > [..................................................................................................................................................................................... > ] > [99%]^[..................................................................................................................................................................................... > ] [99%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize > --id 6 > [Foreman] Password for admin: > [.....................................................................................................................................................................................] > [100%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize > --id 6 > [Foreman] Password for admin: > [............................................................................................................................................................................ > ] [95%] > > > On Tuesday, February 2, 2016 at 3:51:21 PM UTC-8, Nathan > Peters wrote: > > Katello 2.4 / Foreman 1.10.1 on CentOS 7.2 > > My capsule synchronization never seems to complete. It > gets to 75% and then hangs indefinitely. > > This is what I see when I attempt it (it will stay like > this for hours until i Ctrl-C it) > > ------- errors ------ > [root@katello1~]# hammer -u admin -p capsule > content synchronize --id 4 > [............................................................................... > ] [75%]^ > ------- errors ------ > > If I attempt to promote content through the web interface, > the synchronize task that gets spawned fails : > > ------- errors ------ > > Host did not respond within 20 seconds. Sync has been cancelled. Is katello-agent installed and goferd running on the Host? > > ------- errors ------ > > > The goferd service is defintely running : > > ---- troubleshooting -------- > [root@katello1 ~]# systemctl | grep goferd > goferd.service loaded active running Gofer Agent > ------ troubleshooting ------ > > Also, I did a network capture, and there is definitely a > SSL handshake going on between the 2 servers and some data > transfer, but regardless, the content never actually shows > up on the capsule : > > No. Time Source Destination > Protocol Length Info > 1 0.000000 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1 Ack=1 > Win=1432 Len=425 TSval=15011609 TSecr=14701871 > 2 0.000656 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1 Ack=426 > Win=1424 Len=419 TSval=14702871 TSecr=15011609 > 3 0.000676 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=426 Ack=420 > Win=1432 Len=0 TSval=15011610 TSecr=14702871 > 4 0.000683 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=420 Ack=426 > Win=1424 Len=74 TSval=14702872 TSecr=15011609 > 5 0.000688 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=426 Ack=494 > Win=1432 Len=0 TSval=15011610 TSecr=14702872 > 6 0.000912 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=426 Ack=494 > Win=1432 Len=74 TSval=15011610 TSecr=14702872 > 7 0.040612 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=494 Ack=500 > Win=1424 Len=0 TSval=14702912 TSecr=15011610 > 8 1.000939 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=500 Ack=494 > Win=1432 Len=425 TSval=15012610 TSecr=14702912 > 9 1.001197 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=494 Ack=925 > Win=1424 Len=0 TSval=14703872 TSecr=15012610 > 10 1.001549 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=494 Ack=925 > Win=1424 Len=74 TSval=14703872 TSecr=15012610 > 11 1.032061 10.128.17.110 10.178.0.110 > TCP 103 45857 → 5647 [PSH, ACK] Seq=1 Ack=1 > Win=1424 Len=37 TSval=14703903 TSecr=15008637 > 12 1.032070 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 > Len=0 TSval=15012641 TSecr=14703903 > 13 1.040637 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=925 Ack=568 > Win=1432 Len=0 TSval=15012650 TSecr=14703872 > 14 2.001423 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=568 Ack=925 > Win=1424 Len=419 TSval=14704872 TSecr=15012650 > 15 2.001444 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=925 Ack=987 > Win=1432 Len=0 TSval=15013610 TSecr=14704872 > 16 2.001552 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=925 Ack=987 > Win=1432 Len=425 TSval=15013610 TSecr=14704872 > 17 2.001742 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=1350 Ack=987 > Win=1432 Len=74 TSval=15013611 TSecr=14704872 > 18 2.002011 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=987 Ack=1424 > Win=1424 Len=0 TSval=14704873 TSecr=15013610 > 19 2.002019 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=987 Ack=1424 > Win=1424 Len=74 TSval=14704873 TSecr=15013610 > 20 2.041643 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1424 Ack=1061 > Win=1432 Len=0 TSval=15013651 TSecr=14704873 > 21 3.002182 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1424 Ack=1061 > Win=1432 Len=425 TSval=15014611 TSecr=14704873 > 22 3.002605 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1061 Ack=1849 > Win=1424 Len=419 TSval=14705873 TSecr=15014611 > 23 3.002623 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1480 > Win=1432 Len=0 TSval=15014611 TSecr=14705873 > 24 3.002828 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=1480 Ack=1849 > Win=1424 Len=74 TSval=14705874 TSecr=15014611 > 25 3.002833 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1554 > Win=1432 Len=0 TSval=15014612 TSecr=14705874 > 26 3.002865 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=1849 Ack=1554 > Win=1432 Len=74 TSval=15014612 TSecr=14705874 > 27 3.042608 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=1554 Ack=1923 > Win=1424 Len=0 TSval=14705914 TSecr=15014612 > 28 3.479979 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 > Len=2896 TSval=15015089 TSecr=14703903 > 29 3.479991 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=2897 Ack=38 Win=710 > Len=2896 TSval=15015089 TSecr=14703903 > 30 3.479994 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=5793 Ack=38 Win=710 > Len=2896 TSval=15015089 TSecr=14703903 > 31 3.480091 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=8689 Ack=38 Win=710 > Len=2896 TSval=15015089 TSecr=14703903 > 32 3.480101 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=11585 Ack=38 > Win=710 Len=2896 TSval=15015089 TSecr=14703903 > 33 3.480584 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=38 Ack=14481 > Win=1424 Len=0 TSval=14706351 TSecr=15015089 > 34 3.480597 10.178.0.110 10.128.17.110 > TCP 2535 5647 → 45857 [PSH, ACK] Seq=14481 Ack=38 > Win=710 Len=2469 TSval=15015089 TSecr=14706351 > 35 3.481230 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=38 Ack=16950 > Win=1424 Len=0 TSval=14706352 TSecr=15015089 > 36 3.482661 10.128.17.110 10.178.0.110 > TCP 220 45857 → 5647 [PSH, ACK] Seq=38 Ack=16950 > Win=1424 Len=154 TSval=14706354 TSecr=15015089 > 37 3.482671 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=16950 Ack=192 > Win=733 Len=0 TSval=15015092 TSecr=14706354 > 38 3.484424 10.178.0.110 10.128.17.110 > TCP 232 5647 → 45857 [PSH, ACK] Seq=16950 Ack=192 > Win=733 Len=166 TSval=15015093 TSecr=14706354 > 39 3.485299 10.128.17.110 10.178.0.110 > TCP 468 45857 → 5647 [PSH, ACK] Seq=192 Ack=17116 > Win=1424 Len=402 TSval=14706356 TSecr=15015093 > 40 3.485985 10.178.0.110 10.128.17.110 > TCP 131 5647 → 45857 [PSH, ACK] Seq=17116 Ack=594 > Win=755 Len=65 TSval=15015095 TSecr=14706356 > 41 3.525616 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17181 > Win=1424 Len=0 TSval=14706397 TSecr=15015095 > 42 4.002777 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1923 Ack=1554 > Win=1432 Len=425 TSval=15015612 TSecr=14705914 > 43 4.003036 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1554 Ack=1923 > Win=1424 Len=419 TSval=14706874 TSecr=15014612 > 44 4.003043 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=1973 Ack=2348 > Win=1424 Len=0 TSval=14706874 TSecr=15015612 > 45 4.003482 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=1973 Ack=2348 > Win=1424 Len=74 TSval=14706874 TSecr=15015612 > 46 4.003552 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=2348 Ack=2047 > Win=1432 Len=0 TSval=15015612 TSecr=14706874 > 47 4.003750 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=2348 Ack=2047 > Win=1432 Len=74 TSval=15015613 TSecr=14706874 > 48 4.043644 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2047 Ack=2422 > Win=1424 Len=0 TSval=14706915 TSecr=15015613 > 49 4.076622 10.178.0.110 10.128.17.110 > TCP 133 5647 → 45857 [PSH, ACK] Seq=17181 Ack=594 > Win=755 Len=67 TSval=15015685 TSecr=14706397 > 50 4.076864 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17248 > Win=1424 Len=0 TSval=14706948 TSecr=15015685 > 51 4.078152 10.128.17.110 10.178.0.110 > TCP 119 45857 → 5647 [PSH, ACK] Seq=594 Ack=17248 > Win=1424 Len=53 TSval=14706949 TSecr=15015685 > 52 4.078234 10.178.0.110 10.128.17.110 > TCP 119 5647 → 45857 [PSH, ACK] Seq=17248 Ack=647 > Win=755 Len=53 TSval=15015687 TSecr=14706949 > 53 4.082247 10.128.17.110 10.178.0.110 > TCP 172 45857 → 56

Have already restarted the capsule more than 10 times during trying to
troubleshoot this…

Nothing special in /var/log/message after any restart.

··· On Friday, February 12, 2016 at 3:01:55 PM UTC-8, John Mitsch wrote: > > Nathan, > > If you haven't already, can you try restarting goferd on the capsule and > check /var/log/messages again? > > -John > > On 02/11/2016 01:14 PM, Nathan Peters wrote: > > Sorry forgot to answer your question about memory. > > All Katello servers are 16GB RAM, 4 core XEON processor, 2TB disk space. > > What's strange about this is that the TCP captures at the start seems to > match up with the logs. > > You can see the QPID SASL negotion go through in the logs, and you can see > some TLS stuff happen on port 443 during the TCP dump, but then in the > logs, it just goes into a loop getting status and not doing anything, and > the tcp dump just shows a session that turns into a bunch of pushes and > acks with no actual traffic or data after the handshake. > > > On Thursday, February 11, 2016 at 5:12:47 AM UTC-8, Eric Helms wrote: > > If you look at the Capsule logs, are there errors in /var/log/messages? > You should be able to look there and see Pulp kick up when the sync begins > and trace it down to find if it errored out. Another possibility is the > Capsule becomes bogged down and isn't able to respond which indicates to > the task something is wrong all the while it actually finished. What are > the specs of your capsules memory wise? > > On Wed, Feb 10, 2016 at 1:00 PM, Nathan Peters > wrote: > > So this is still not working. Has no one seriously ever tested a capsule > before? This seems like a pretty core feature to have broken so badly. > > I've now setup multiple capsules on different subnets, including on the > same subnet as the main Katello server (same switch etc), and none of them > ever finish synchronizing under any circumstances. > > The one exception is if I assign a capsule to NO lifecycle environments, > its reaches 100% synchronization instantly and returns to the shell > prompt. If I add even a single view to the lifecycle environment (even if > that view only has like 10 packages in it), it will *always* freeze up and > not finish, not matter which capsule on which network I try to sync to. > The amount it freezes up at seems inconsistent though. Sometimes it's 99%, > sometimes 75%, sometimes 95%. > > Here is console output of several different tests with different lifecycle > environments available to the capsules. The final 100% was only with no > lifecycle environments assigned to the capsule and obviously that is not an > option as it transfers zero content to the capsule. In each case, I wait > at least 3 hours before ctrl-c and scan the disk free space on the capsule > to ensure that nothing is being transferred. > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [..................................................................................................................................................................................... > ] > [99%]^[..................................................................................................................................................................................... > ] [99%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [........................................................................................................................................ > ] > [75%]^[........................................................................................................................................ > ] [75%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [..................................................................................................................................................................................... > ] > [99%]^[..................................................................................................................................................................................... > ] [99%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [.....................................................................................................................................................................................] > [100%] > > [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 > [Foreman] Password for admin: > [............................................................................................................................................................................ > ] [95%] > > > On Tuesday, February 2, 2016 at 3:51:21 PM UTC-8, Nathan Peters wrote: > > Katello 2.4 / Foreman 1.10.1 on CentOS 7.2 > > My capsule synchronization never seems to complete. It gets to 75% and > then hangs indefinitely. > > This is what I see when I attempt it (it will stay like this for hours > until i Ctrl-C it) > > ------- errors ------ > [root@katello1~]# hammer -u admin -p capsule content > synchronize --id 4 > [............................................................................... > ] [75%]^ > ------- errors ------ > > If I attempt to promote content through the web interface, the synchronize > task that gets spawned fails : > > ------- errors ------ > > Host did not respond within 20 seconds. Sync has been cancelled. Is katello-agent installed and goferd running on the Host? > > ------- errors ------ > > > The goferd service is defintely running : > > ---- troubleshooting -------- > [root@katello1 ~]# systemctl | grep goferd > > goferd.service > loaded active running Gofer Agent > ------ troubleshooting ------ > > Also, I did a network capture, and there is definitely a SSL handshake > going on between the 2 servers and some data transfer, but regardless, the > content never actually shows up on the capsule : > > No. Time Source Destination > Protocol Length Info > 1 0.000000 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1 Ack=1 Win=1432 Len=425 > TSval=15011609 TSecr=14701871 > 2 0.000656 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1 Ack=426 Win=1424 Len=419 > TSval=14702871 TSecr=15011609 > 3 0.000676 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=426 Ack=420 Win=1432 Len=0 > TSval=15011610 TSecr=14702871 > 4 0.000683 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=420 Ack=426 Win=1424 Len=74 > TSval=14702872 TSecr=15011609 > 5 0.000688 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=426 Ack=494 Win=1432 Len=0 > TSval=15011610 TSecr=14702872 > 6 0.000912 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=426 Ack=494 Win=1432 Len=74 > TSval=15011610 TSecr=14702872 > 7 0.040612 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=494 Ack=500 Win=1424 Len=0 > TSval=14702912 TSecr=15011610 > 8 1.000939 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=500 Ack=494 Win=1432 Len=425 > TSval=15012610 TSecr=14702912 > 9 1.001197 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=494 Ack=925 Win=1424 Len=0 > TSval=14703872 TSecr=15012610 > 10 1.001549 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=494 Ack=925 Win=1424 Len=74 > TSval=14703872 TSecr=15012610 > 11 1.032061 10.128.17.110 10.178.0.110 > TCP 103 45857 → 5647 [PSH, ACK] Seq=1 Ack=1 Win=1424 Len=37 > TSval=14703903 TSecr=15008637 > 12 1.032070 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=0 > TSval=15012641 TSecr=14703903 > 13 1.040637 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=925 Ack=568 Win=1432 Len=0 > TSval=15012650 TSecr=14703872 > 14 2.001423 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=568 Ack=925 Win=1424 Len=419 > TSval=14704872 TSecr=15012650 > 15 2.001444 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=925 Ack=987 Win=1432 Len=0 > TSval=15013610 TSecr=14704872 > 16 2.001552 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=925 Ack=987 Win=1432 Len=425 > TSval=15013610 TSecr=14704872 > 17 2.001742 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=1350 Ack=987 Win=1432 Len=74 > TSval=15013611 TSecr=14704872 > 18 2.002011 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=987 Ack=1424 Win=1424 Len=0 > TSval=14704873 TSecr=15013610 > 19 2.002019 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=987 Ack=1424 Win=1424 Len=74 > TSval=14704873 TSecr=15013610 > 20 2.041643 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1424 Ack=1061 Win=1432 Len=0 > TSval=15013651 TSecr=14704873 > 21 3.002182 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1424 Ack=1061 Win=1432 Len=425 > TSval=15014611 TSecr=14704873 > 22 3.002605 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1061 Ack=1849 Win=1424 Len=419 > TSval=14705873 TSecr=15014611 > 23 3.002623 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1480 Win=1432 Len=0 > TSval=15014611 TSecr=14705873 > 24 3.002828 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=1480 Ack=1849 Win=1424 Len=74 > TSval=14705874 TSecr=15014611 > 25 3.002833 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1554 Win=1432 Len=0 > TSval=15014612 TSecr=14705874 > 26 3.002865 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=1849 Ack=1554 Win=1432 Len=74 > TSval=15014612 TSecr=14705874 > 27 3.042608 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=1554 Ack=1923 Win=1424 Len=0 > TSval=14705914 TSecr=15014612 > 28 3.479979 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 29 3.479991 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=2897 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 30 3.479994 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=5793 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 31 3.480091 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=8689 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 32 3.480101 10.178.0.110 10.128.17.110 > TCP 2962 5647 → 45857 [ACK] Seq=11585 Ack=38 Win=710 Len=2896 > TSval=15015089 TSecr=14703903 > 33 3.480584 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=38 Ack=14481 Win=1424 Len=0 > TSval=14706351 TSecr=15015089 > 34 3.480597 10.178.0.110 10.128.17.110 > TCP 2535 5647 → 45857 [PSH, ACK] Seq=14481 Ack=38 Win=710 Len=2469 > TSval=15015089 TSecr=14706351 > 35 3.481230 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=38 Ack=16950 Win=1424 Len=0 > TSval=14706352 TSecr=15015089 > 36 3.482661 10.128.17.110 10.178.0.110 > TCP 220 45857 → 5647 [PSH, ACK] Seq=38 Ack=16950 Win=1424 Len=154 > TSval=14706354 TSecr=15015089 > 37 3.482671 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=16950 Ack=192 Win=733 Len=0 > TSval=15015092 TSecr=14706354 > 38 3.484424 10.178.0.110 10.128.17.110 > TCP 232 5647 → 45857 [PSH, ACK] Seq=16950 Ack=192 Win=733 Len=166 > TSval=15015093 TSecr=14706354 > 39 3.485299 10.128.17.110 10.178.0.110 > TCP 468 45857 → 5647 [PSH, ACK] Seq=192 Ack=17116 Win=1424 Len=402 > TSval=14706356 TSecr=15015093 > 40 3.485985 10.178.0.110 10.128.17.110 > TCP 131 5647 → 45857 [PSH, ACK] Seq=17116 Ack=594 Win=755 Len=65 > TSval=15015095 TSecr=14706356 > 41 3.525616 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17181 Win=1424 Len=0 > TSval=14706397 TSecr=15015095 > 42 4.002777 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=1923 Ack=1554 Win=1432 Len=425 > TSval=15015612 TSecr=14705914 > 43 4.003036 10.128.17.110 10.178.0.110 > TCP 485 43052 → 5646 [PSH, ACK] Seq=1554 Ack=1923 Win=1424 Len=419 > TSval=14706874 TSecr=15014612 > 44 4.003043 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=1973 Ack=2348 Win=1424 Len=0 > TSval=14706874 TSecr=15015612 > 45 4.003482 10.128.17.110 10.178.0.110 > TCP 140 43052 → 5646 [PSH, ACK] Seq=1973 Ack=2348 Win=1424 Len=74 > TSval=14706874 TSecr=15015612 > 46 4.003552 10.178.0.110 10.128.17.110 > TCP 66 5646 → 43052 [ACK] Seq=2348 Ack=2047 Win=1432 Len=0 > TSval=15015612 TSecr=14706874 > 47 4.003750 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=2348 Ack=2047 Win=1432 Len=74 > TSval=15015613 TSecr=14706874 > 48 4.043644 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2047 Ack=2422 Win=1424 Len=0 > TSval=14706915 TSecr=15015613 > 49 4.076622 10.178.0.110 10.128.17.110 > TCP 133 5647 → 45857 [PSH, ACK] Seq=17181 Ack=594 Win=755 Len=67 > TSval=15015685 TSecr=14706397 > 50 4.076864 10.128.17.110 10.178.0.110 > TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17248 Win=1424 Len=0 > TSval=14706948 TSecr=15015685 > 51 4.078152 10.128.17.110 10.178.0.110 > TCP 119 45857 → 5647 [PSH, ACK] Seq=594 Ack=17248 Win=1424 Len=53 > TSval=14706949 TSecr=15015685 > 52 4.078234 10.178.0.110 10.128.17.110 > TCP 119 5647 → 45857 [PSH, ACK] Seq=17248 Ack=647 Win=755 Len=53 > TSval=15015687 TSecr=14706949 > 53 4.082247 10.128.17.110 10.178.0.110 > TCP 172 45857 → 5647 [PSH, ACK] Seq=647 Ack=17301 Win=1424 Len=106 > TSval=14706953 TSecr=15015687 > 54 4.121634 10.178.0.110 10.128.17.110 > TCP 66 5647 → 45857 [ACK] Seq=17301 Ack=753 Win=755 Len=0 > TSval=15015731 TSecr=14706953 > 55 5.003648 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=2422 Ack=2047 Win=1432 Len=425 > TSval=15016613 TSecr=14706915 > 56 5.003926 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2047 Ack=2847 Win=1424 Len=0 > TSval=14707875 TSecr=15016613 > 57 5.004452 10.128.17.110 10.178.0.110 > TCP 530 43052 → 5646 [PSH, ACK] Seq=2047 Ack=2847 Win=1424 Len=464 > TSval=14707875 TSecr=15016613 > 58 5.004702 10.178.0.110 10.128.17.110 > TCP 140 5646 → 43052 [PSH, ACK] Seq=2847 Ack=2511 Win=1432 Len=74 > TSval=15016614 TSecr=14707875 > 59 5.044893 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2511 Ack=2921 Win=1424 Len=0 > TSval=14707916 TSecr=15016614 > 60 6.005076 10.178.0.110 10.128.17.110 > TCP 491 5646 → 43052 [PSH, ACK] Seq=2921 Ack=2511 Win=1432 Len=425 > TSval=15017614 TSecr=14707916 > 61 6.005370 10.128.17.110 10.178.0.110 > TCP 66 43052 → 5646 [ACK] Seq=2511 Ack=3346 Win=1424 Len=0 > TSval=14708876 TSecr=15017614 > 62 6.005714 10.128.17.110 10.178.0.110 > TCP 530 43052 → 5646 [PSH, ACK

I have had similar issues with my capsules. Goferd constantly is getting
diconnected and I have to bounce it in order for capsule sync to occur.
Katello 2.3 here.

··· On Tuesday, February 16, 2016 at 4:35:26 PM UTC-5, Nathan Peters wrote: > > Have already restarted the capsule more than 10 times during trying to > troubleshoot this... > > Nothing special in /var/log/message after any restart. > > On Friday, February 12, 2016 at 3:01:55 PM UTC-8, John Mitsch wrote: >> >> Nathan, >> >> If you haven't already, can you try restarting goferd on the capsule and >> check /var/log/messages again? >> >> -John >> >> On 02/11/2016 01:14 PM, Nathan Peters wrote: >> >> Sorry forgot to answer your question about memory. >> >> All Katello servers are 16GB RAM, 4 core XEON processor, 2TB disk space. >> >> What's strange about this is that the TCP captures at the start seems to >> match up with the logs. >> >> You can see the QPID SASL negotion go through in the logs, and you can >> see some TLS stuff happen on port 443 during the TCP dump, but then in the >> logs, it just goes into a loop getting status and not doing anything, and >> the tcp dump just shows a session that turns into a bunch of pushes and >> acks with no actual traffic or data after the handshake. >> >> >> On Thursday, February 11, 2016 at 5:12:47 AM UTC-8, Eric Helms wrote: >> >> If you look at the Capsule logs, are there errors in /var/log/messages? >> You should be able to look there and see Pulp kick up when the sync begins >> and trace it down to find if it errored out. Another possibility is the >> Capsule becomes bogged down and isn't able to respond which indicates to >> the task something is wrong all the while it actually finished. What are >> the specs of your capsules memory wise? >> >> On Wed, Feb 10, 2016 at 1:00 PM, Nathan Peters >> wrote: >> >> So this is still not working. Has no one seriously ever tested a capsule >> before? This seems like a pretty core feature to have broken so badly. >> >> I've now setup multiple capsules on different subnets, including on the >> same subnet as the main Katello server (same switch etc), and none of them >> ever finish synchronizing under any circumstances. >> >> The one exception is if I assign a capsule to NO lifecycle environments, >> its reaches 100% synchronization instantly and returns to the shell >> prompt. If I add even a single view to the lifecycle environment (even if >> that view only has like 10 packages in it), it will *always* freeze up and >> not finish, not matter which capsule on which network I try to sync to. >> The amount it freezes up at seems inconsistent though. Sometimes it's 99%, >> sometimes 75%, sometimes 95%. >> >> Here is console output of several different tests with different >> lifecycle environments available to the capsules. The final 100% was only >> with no lifecycle environments assigned to the capsule and obviously that >> is not an option as it transfers zero content to the capsule. In each >> case, I wait at least 3 hours before ctrl-c and scan the disk free space on >> the capsule to ensure that nothing is being transferred. >> >> [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 >> [Foreman] Password for admin: >> [..................................................................................................................................................................................... >> ] >> [99%]^[..................................................................................................................................................................................... >> ] [99%] >> >> [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 >> [Foreman] Password for admin: >> [........................................................................................................................................ >> ] >> [75%]^[........................................................................................................................................ >> ] [75%] >> >> [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 >> [Foreman] Password for admin: >> [..................................................................................................................................................................................... >> ] >> [99%]^[..................................................................................................................................................................................... >> ] [99%] >> >> [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 >> [Foreman] Password for admin: >> [.....................................................................................................................................................................................] >> [100%] >> >> [root@katello1 ~]# hammer -u admin capsule content synchronize --id 6 >> [Foreman] Password for admin: >> [............................................................................................................................................................................ >> ] [95%] >> >> >> On Tuesday, February 2, 2016 at 3:51:21 PM UTC-8, Nathan Peters wrote: >> >> Katello 2.4 / Foreman 1.10.1 on CentOS 7.2 >> >> My capsule synchronization never seems to complete. It gets to 75% and >> then hangs indefinitely. >> >> This is what I see when I attempt it (it will stay like this for hours >> until i Ctrl-C it) >> >> ------- errors ------ >> [root@katello1~]# hammer -u admin -p capsule content >> synchronize --id 4 >> [............................................................................... >> ] [75%]^ >> ------- errors ------ >> >> If I attempt to promote content through the web interface, the >> synchronize task that gets spawned fails : >> >> ------- errors ------ >> >> Host did not respond within 20 seconds. Sync has been cancelled. Is katello-agent installed and goferd running on the Host? >> >> ------- errors ------ >> >> >> The goferd service is defintely running : >> >> ---- troubleshooting -------- >> [root@katello1 ~]# systemctl | grep goferd >> >> goferd.service >> loaded active running Gofer Agent >> ------ troubleshooting ------ >> >> Also, I did a network capture, and there is definitely a SSL handshake >> going on between the 2 servers and some data transfer, but regardless, the >> content never actually shows up on the capsule : >> >> No. Time Source Destination >> Protocol Length Info >> 1 0.000000 10.178.0.110 10.128.17.110 >> TCP 491 5646 → 43052 [PSH, ACK] Seq=1 Ack=1 Win=1432 Len=425 >> TSval=15011609 TSecr=14701871 >> 2 0.000656 10.128.17.110 10.178.0.110 >> TCP 485 43052 → 5646 [PSH, ACK] Seq=1 Ack=426 Win=1424 Len=419 >> TSval=14702871 TSecr=15011609 >> 3 0.000676 10.178.0.110 10.128.17.110 >> TCP 66 5646 → 43052 [ACK] Seq=426 Ack=420 Win=1432 Len=0 >> TSval=15011610 TSecr=14702871 >> 4 0.000683 10.128.17.110 10.178.0.110 >> TCP 140 43052 → 5646 [PSH, ACK] Seq=420 Ack=426 Win=1424 Len=74 >> TSval=14702872 TSecr=15011609 >> 5 0.000688 10.178.0.110 10.128.17.110 >> TCP 66 5646 → 43052 [ACK] Seq=426 Ack=494 Win=1432 Len=0 >> TSval=15011610 TSecr=14702872 >> 6 0.000912 10.178.0.110 10.128.17.110 >> TCP 140 5646 → 43052 [PSH, ACK] Seq=426 Ack=494 Win=1432 Len=74 >> TSval=15011610 TSecr=14702872 >> 7 0.040612 10.128.17.110 10.178.0.110 >> TCP 66 43052 → 5646 [ACK] Seq=494 Ack=500 Win=1424 Len=0 >> TSval=14702912 TSecr=15011610 >> 8 1.000939 10.178.0.110 10.128.17.110 >> TCP 491 5646 → 43052 [PSH, ACK] Seq=500 Ack=494 Win=1432 Len=425 >> TSval=15012610 TSecr=14702912 >> 9 1.001197 10.128.17.110 10.178.0.110 >> TCP 66 43052 → 5646 [ACK] Seq=494 Ack=925 Win=1424 Len=0 >> TSval=14703872 TSecr=15012610 >> 10 1.001549 10.128.17.110 10.178.0.110 >> TCP 140 43052 → 5646 [PSH, ACK] Seq=494 Ack=925 Win=1424 Len=74 >> TSval=14703872 TSecr=15012610 >> 11 1.032061 10.128.17.110 10.178.0.110 >> TCP 103 45857 → 5647 [PSH, ACK] Seq=1 Ack=1 Win=1424 Len=37 >> TSval=14703903 TSecr=15008637 >> 12 1.032070 10.178.0.110 10.128.17.110 >> TCP 66 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=0 >> TSval=15012641 TSecr=14703903 >> 13 1.040637 10.178.0.110 10.128.17.110 >> TCP 66 5646 → 43052 [ACK] Seq=925 Ack=568 Win=1432 Len=0 >> TSval=15012650 TSecr=14703872 >> 14 2.001423 10.128.17.110 10.178.0.110 >> TCP 485 43052 → 5646 [PSH, ACK] Seq=568 Ack=925 Win=1424 Len=419 >> TSval=14704872 TSecr=15012650 >> 15 2.001444 10.178.0.110 10.128.17.110 >> TCP 66 5646 → 43052 [ACK] Seq=925 Ack=987 Win=1432 Len=0 >> TSval=15013610 TSecr=14704872 >> 16 2.001552 10.178.0.110 10.128.17.110 >> TCP 491 5646 → 43052 [PSH, ACK] Seq=925 Ack=987 Win=1432 Len=425 >> TSval=15013610 TSecr=14704872 >> 17 2.001742 10.178.0.110 10.128.17.110 >> TCP 140 5646 → 43052 [PSH, ACK] Seq=1350 Ack=987 Win=1432 Len=74 >> TSval=15013611 TSecr=14704872 >> 18 2.002011 10.128.17.110 10.178.0.110 >> TCP 66 43052 → 5646 [ACK] Seq=987 Ack=1424 Win=1424 Len=0 >> TSval=14704873 TSecr=15013610 >> 19 2.002019 10.128.17.110 10.178.0.110 >> TCP 140 43052 → 5646 [PSH, ACK] Seq=987 Ack=1424 Win=1424 Len=74 >> TSval=14704873 TSecr=15013610 >> 20 2.041643 10.178.0.110 10.128.17.110 >> TCP 66 5646 → 43052 [ACK] Seq=1424 Ack=1061 Win=1432 Len=0 >> TSval=15013651 TSecr=14704873 >> 21 3.002182 10.178.0.110 10.128.17.110 >> TCP 491 5646 → 43052 [PSH, ACK] Seq=1424 Ack=1061 Win=1432 Len=425 >> TSval=15014611 TSecr=14704873 >> 22 3.002605 10.128.17.110 10.178.0.110 >> TCP 485 43052 → 5646 [PSH, ACK] Seq=1061 Ack=1849 Win=1424 Len=419 >> TSval=14705873 TSecr=15014611 >> 23 3.002623 10.178.0.110 10.128.17.110 >> TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1480 Win=1432 Len=0 >> TSval=15014611 TSecr=14705873 >> 24 3.002828 10.128.17.110 10.178.0.110 >> TCP 140 43052 → 5646 [PSH, ACK] Seq=1480 Ack=1849 Win=1424 Len=74 >> TSval=14705874 TSecr=15014611 >> 25 3.002833 10.178.0.110 10.128.17.110 >> TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1554 Win=1432 Len=0 >> TSval=15014612 TSecr=14705874 >> 26 3.002865 10.178.0.110 10.128.17.110 >> TCP 140 5646 → 43052 [PSH, ACK] Seq=1849 Ack=1554 Win=1432 Len=74 >> TSval=15014612 TSecr=14705874 >> 27 3.042608 10.128.17.110 10.178.0.110 >> TCP 66 43052 → 5646 [ACK] Seq=1554 Ack=1923 Win=1424 Len=0 >> TSval=14705914 TSecr=15014612 >> 28 3.479979 10.178.0.110 10.128.17.110 >> TCP 2962 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=2896 >> TSval=15015089 TSecr=14703903 >> 29 3.479991 10.178.0.110 10.128.17.110 >> TCP 2962 5647 → 45857 [ACK] Seq=2897 Ack=38 Win=710 Len=2896 >> TSval=15015089 TSecr=14703903 >> 30 3.479994 10.178.0.110 10.128.17.110 >> TCP 2962 5647 → 45857 [ACK] Seq=5793 Ack=38 Win=710 Len=2896 >> TSval=15015089 TSecr=14703903 >> 31 3.480091 10.178.0.110 10.128.17.110 >> TCP 2962 5647 → 45857 [ACK] Seq=8689 Ack=38 Win=710 Len=2896 >> TSval=15015089 TSecr=14703903 >> 32 3.480101 10.178.0.110 10.128.17.110 >> TCP 2962 5647 → 45857 [ACK] Seq=11585 Ack=38 Win=710 Len=2896 >> TSval=15015089 TSecr=14703903 >> 33 3.480584 10.128.17.110 10.178.0.110 >> TCP 66 45857 → 5647 [ACK] Seq=38 Ack=14481 Win=1424 Len=0 >> TSval=14706351 TSecr=15015089 >> 34 3.480597 10.178.0.110 10.128.17.110 >> TCP 2535 5647 → 45857 [PSH, ACK] Seq=14481 Ack=38 Win=710 Len=2469 >> TSval=15015089 TSecr=14706351 >> 35 3.481230 10.128.17.110 10.178.0.110 >> TCP 66 45857 → 5647 [ACK] Seq=38 Ack=16950 Win=1424 Len=0 >> TSval=14706352 TSecr=15015089 >> 36 3.482661 10.128.17.110 10.178.0.110 >> TCP 220 45857 → 5647 [PSH, ACK] Seq=38 Ack=16950 Win=1424 Len=154 >> TSval=14706354 TSecr=15015089 >> 37 3.482671 10.178.0.110 10.128.17.110 >> TCP 66 5647 → 45857 [ACK] Seq=16950 Ack=192 Win=733 Len=0 >> TSval=15015092 TSecr=14706354 >> 38 3.484424 10.178.0.110 10.128.17.110 >> TCP 232 5647 → 45857 [PSH, ACK] Seq=16950 Ack=192 Win=733 Len=166 >> TSval=15015093 TSecr=14706354 >> 39 3.485299 10.128.17.110 10.178.0.110 >> TCP 468 45857 → 5647 [PSH, ACK] Seq=192 Ack=17116 Win=1424 Len=402 >> TSval=14706356 TSecr=15015093 >> 40 3.485985 10.178.0.110 10.128.17.110 >> TCP 131 5647 → 45857 [PSH, ACK] Seq=17116 Ack=594 Win=755 Len=65 >> TSval=15015095 TSecr=14706356 >> 41 3.525616 10.128.17.110 10.178.0.110 >> TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17181 Win=1424 Len=0 >> TSval=14706397 TSecr=15015095 >> 42 4.002777 10.178.0.110 10.128.17.110 >> TCP 491 5646 → 43052 [PSH, ACK] Seq=1923 Ack=1554 Win=1432 Len=425 >> TSval=15015612 TSecr=14705914 >> 43 4.003036 10.128.17.110 10.178.0.110 >> TCP 485 43052 → 5646 [PSH, ACK] Seq=1554 Ack=1923 Win=1424 Len=419 >> TSval=14706874 TSecr=15014612 >> 44 4.003043 10.128.17.110 10.178.0.110 >> TCP 66 43052 → 5646 [ACK] Seq=1973 Ack=2348 Win=1424 Len=0 >> TSval=14706874 TSecr=15015612 >> 45 4.003482 10.128.17.110 10.178.0.110 >> TCP 140 43052 → 5646 [PSH, ACK] Seq=1973 Ack=2348 Win=1424 Len=74 >> TSval=14706874 TSecr=15015612 >> 46 4.003552 10.178.0.110 10.128.17.110 >> TCP 66 5646 → 43052 [ACK] Seq=2348 Ack=2047 Win=1432 Len=0 >> TSval=15015612 TSecr=14706874 >> 47 4.003750 10.178.0.110 10.128.17.110 >> TCP 140 5646 → 43052 [PSH, ACK] Seq=2348 Ack=2047 Win=1432 Len=74 >> TSval=15015613 TSecr=14706874 >> 48 4.043644 10.128.17.110 10.178.0.110 >> TCP 66 43052 → 5646 [ACK] Seq=2047 Ack=2422 Win=1424 Len=0 >> TSval=14706915 TSecr=15015613 >> 49 4.076622 10.178.0.110 10.128.17.110 >> TCP 133 5647 → 45857 [PSH, ACK] Seq=17181 Ack=594 Win=755 Len=67 >> TSval=15015685 TSecr=14706397 >> 50 4.076864 10.128.17.110 10.178.0.110 >> TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17248 Win=1424 Len=0 >> TSval=14706948 TSecr=15015685 >> 51 4.078152 10.128.17.110 10.178.0.110 >> TCP 119 45857 → 5647 [PSH, ACK] Seq=594 Ack=17248 Win=1424 Len=53 >> TSval=14706949 TSecr=15015685 >> 52 4.078234 10.178.0.110 10.128.17.110 >> TCP 119 5647 → 45857 [PSH, ACK] Seq=17248 Ack=647 Win=755 Len=53 >> TSval=15015687 TSecr=14706949 >> 53 4.082247 10.128.17.110 10.178.0.110 >> TCP 172 45857 → 5647 [PSH, ACK] Seq=647 Ack=17301 Win=1424 Len=106 >> TSval=14706953 TSecr=15015687 >> 54 4.121634 10.178.0.110 10.128.17.110 >> TCP 66 5647 → 45857 [ACK] Seq=17301 Ack=753 Win=755 Len=0 >> TSval=15015731 TSecr=14706953 >> 55 5.003648 10.178.0.110 10.128.17.110 >> TCP 491 5646 → 43052 [PSH, ACK] Seq=2422 Ack=2047 Win=1432 Len=425 >> TSval=15016613 TSecr=14706915 >> 56 5.003926 10.128.17.110 10.178.0.110 >> TCP 66 43052 → 5646 [ACK] Seq=2047 Ack=2847 Win=1424 Len=0 >> TSval=14707875 TSecr=15016613 >> 57 5.004452 10.128.17.110 10.178.0.110 >> TCP 530 43052 → 5646 [PSH, ACK] Seq=2047 Ack=2847 Win=1424 Len=464 >> TSval=14707875 TSecr=15016613 >> 58 5.004702 10.178.0.110 10.128.17.110 >> TCP 140 5646 → 43052 [PSH, ACK] Seq=2847 Ack=2511 Win=1432 Len=74 >> TSval=15016614 TSecr=14707875 >> 59 5.044893 10.128.17.110 10.178.0.110 >> T

We have found this a hard area to reproduce and debug. We could try pushing
out new qpid packages to see if they alleviate these issues for 2.4.Z.

For Katello 3.0, the upcoming release, we have switched our paradigm for
management of content on a capsule to use the Pulp APIs directly instead of
going through gofer. This should alleviate issues that users encounter with
gofer for syncing content out.

Eric

··· On Sat, Feb 20, 2016 at 8:04 PM, Travis Camechis wrote:

I have had similar issues with my capsules. Goferd constantly is getting
diconnected and I have to bounce it in order for capsule sync to occur.
Katello 2.3 here.

On Tuesday, February 16, 2016 at 4:35:26 PM UTC-5, Nathan Peters wrote:

Have already restarted the capsule more than 10 times during trying to
troubleshoot this…

Nothing special in /var/log/message after any restart.

On Friday, February 12, 2016 at 3:01:55 PM UTC-8, John Mitsch wrote:

Nathan,

If you haven’t already, can you try restarting goferd on the capsule and
check /var/log/messages again?

-John

On 02/11/2016 01:14 PM, Nathan Peters wrote:

Sorry forgot to answer your question about memory.

All Katello servers are 16GB RAM, 4 core XEON processor, 2TB disk space.

What’s strange about this is that the TCP captures at the start seems to
match up with the logs.

You can see the QPID SASL negotion go through in the logs, and you can
see some TLS stuff happen on port 443 during the TCP dump, but then in the
logs, it just goes into a loop getting status and not doing anything, and
the tcp dump just shows a session that turns into a bunch of pushes and
acks with no actual traffic or data after the handshake.

On Thursday, February 11, 2016 at 5:12:47 AM UTC-8, Eric Helms wrote:

If you look at the Capsule logs, are there errors in /var/log/messages?
You should be able to look there and see Pulp kick up when the sync begins
and trace it down to find if it errored out. Another possibility is the
Capsule becomes bogged down and isn’t able to respond which indicates to
the task something is wrong all the while it actually finished. What are
the specs of your capsules memory wise?

On Wed, Feb 10, 2016 at 1:00 PM, Nathan Peters nathanp...@gmail.com >>> wrote:

So this is still not working. Has no one seriously ever tested a
capsule before? This seems like a pretty core feature to have broken so
badly.

I’ve now setup multiple capsules on different subnets, including on the
same subnet as the main Katello server (same switch etc), and none of them
ever finish synchronizing under any circumstances.

The one exception is if I assign a capsule to NO lifecycle environments,
its reaches 100% synchronization instantly and returns to the shell
prompt. If I add even a single view to the lifecycle environment (even if
that view only has like 10 packages in it), it will always freeze up and
not finish, not matter which capsule on which network I try to sync to.
The amount it freezes up at seems inconsistent though. Sometimes it’s 99%,
sometimes 75%, sometimes 95%.

Here is console output of several different tests with different
lifecycle environments available to the capsules. The final 100% was only
with no lifecycle environments assigned to the capsule and obviously that
is not an option as it transfers zero content to the capsule. In each
case, I wait at least 3 hours before ctrl-c and scan the disk free space on
the capsule to ensure that nothing is being transferred.

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…
]
[99%]^[…
] [99%]

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…
]
[75%]^[…
] [75%]

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…
]
[99%]^[…
] [99%]

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…]
[100%]

[root@katello1 ~]# hammer -u admin capsule content synchronize --id 6
[Foreman] Password for admin:
[…
] [95%]

On Tuesday, February 2, 2016 at 3:51:21 PM UTC-8, Nathan Peters wrote:

Katello 2.4 / Foreman 1.10.1 on CentOS 7.2

My capsule synchronization never seems to complete. It gets to 75% and
then hangs indefinitely.

This is what I see when I attempt it (it will stay like this for hours
until i Ctrl-C it)

------- errors ------
[root@katello1~]# hammer -u admin -p capsule content
synchronize --id 4
[…
] [75%]^
------- errors ------

If I attempt to promote content through the web interface, the
synchronize task that gets spawned fails :

------- errors ------

Host did not respond within 20 seconds. Sync has been cancelled. Is katello-agent installed and goferd running on the Host?

------- errors ------

The goferd service is defintely running :

---- troubleshooting --------
[root@katello1 ~]# systemctl | grep goferd

goferd.service
loaded active running Gofer Agent
------ troubleshooting ------

Also, I did a network capture, and there is definitely a SSL handshake
going on between the 2 servers and some data transfer, but regardless, the
content never actually shows up on the capsule :

No. Time Source Destination
Protocol Length Info
1 0.000000 10.178.0.110 10.128.17.110
TCP 491 5646 → 43052 [PSH, ACK] Seq=1 Ack=1 Win=1432 Len=425
TSval=15011609 TSecr=14701871
2 0.000656 10.128.17.110 10.178.0.110
TCP 485 43052 → 5646 [PSH, ACK] Seq=1 Ack=426 Win=1424 Len=419
TSval=14702871 TSecr=15011609
3 0.000676 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=426 Ack=420 Win=1432 Len=0
TSval=15011610 TSecr=14702871
4 0.000683 10.128.17.110 10.178.0.110
TCP 140 43052 → 5646 [PSH, ACK] Seq=420 Ack=426 Win=1424 Len=74
TSval=14702872 TSecr=15011609
5 0.000688 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=426 Ack=494 Win=1432 Len=0
TSval=15011610 TSecr=14702872
6 0.000912 10.178.0.110 10.128.17.110
TCP 140 5646 → 43052 [PSH, ACK] Seq=426 Ack=494 Win=1432 Len=74
TSval=15011610 TSecr=14702872
7 0.040612 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=494 Ack=500 Win=1424 Len=0
TSval=14702912 TSecr=15011610
8 1.000939 10.178.0.110 10.128.17.110
TCP 491 5646 → 43052 [PSH, ACK] Seq=500 Ack=494 Win=1432 Len=425
TSval=15012610 TSecr=14702912
9 1.001197 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=494 Ack=925 Win=1424 Len=0
TSval=14703872 TSecr=15012610
10 1.001549 10.128.17.110 10.178.0.110
TCP 140 43052 → 5646 [PSH, ACK] Seq=494 Ack=925 Win=1424 Len=74
TSval=14703872 TSecr=15012610
11 1.032061 10.128.17.110 10.178.0.110
TCP 103 45857 → 5647 [PSH, ACK] Seq=1 Ack=1 Win=1424 Len=37
TSval=14703903 TSecr=15008637
12 1.032070 10.178.0.110 10.128.17.110
TCP 66 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=0
TSval=15012641 TSecr=14703903
13 1.040637 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=925 Ack=568 Win=1432 Len=0
TSval=15012650 TSecr=14703872
14 2.001423 10.128.17.110 10.178.0.110
TCP 485 43052 → 5646 [PSH, ACK] Seq=568 Ack=925 Win=1424 Len=419
TSval=14704872 TSecr=15012650
15 2.001444 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=925 Ack=987 Win=1432 Len=0
TSval=15013610 TSecr=14704872
16 2.001552 10.178.0.110 10.128.17.110
TCP 491 5646 → 43052 [PSH, ACK] Seq=925 Ack=987 Win=1432 Len=425
TSval=15013610 TSecr=14704872
17 2.001742 10.178.0.110 10.128.17.110
TCP 140 5646 → 43052 [PSH, ACK] Seq=1350 Ack=987 Win=1432 Len=74
TSval=15013611 TSecr=14704872
18 2.002011 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=987 Ack=1424 Win=1424 Len=0
TSval=14704873 TSecr=15013610
19 2.002019 10.128.17.110 10.178.0.110
TCP 140 43052 → 5646 [PSH, ACK] Seq=987 Ack=1424 Win=1424 Len=74
TSval=14704873 TSecr=15013610
20 2.041643 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=1424 Ack=1061 Win=1432 Len=0
TSval=15013651 TSecr=14704873
21 3.002182 10.178.0.110 10.128.17.110
TCP 491 5646 → 43052 [PSH, ACK] Seq=1424 Ack=1061 Win=1432 Len=425
TSval=15014611 TSecr=14704873
22 3.002605 10.128.17.110 10.178.0.110
TCP 485 43052 → 5646 [PSH, ACK] Seq=1061 Ack=1849 Win=1424 Len=419
TSval=14705873 TSecr=15014611
23 3.002623 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1480 Win=1432 Len=0
TSval=15014611 TSecr=14705873
24 3.002828 10.128.17.110 10.178.0.110
TCP 140 43052 → 5646 [PSH, ACK] Seq=1480 Ack=1849 Win=1424 Len=74
TSval=14705874 TSecr=15014611
25 3.002833 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=1849 Ack=1554 Win=1432 Len=0
TSval=15014612 TSecr=14705874
26 3.002865 10.178.0.110 10.128.17.110
TCP 140 5646 → 43052 [PSH, ACK] Seq=1849 Ack=1554 Win=1432 Len=74
TSval=15014612 TSecr=14705874
27 3.042608 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=1554 Ack=1923 Win=1424 Len=0
TSval=14705914 TSecr=15014612
28 3.479979 10.178.0.110 10.128.17.110
TCP 2962 5647 → 45857 [ACK] Seq=1 Ack=38 Win=710 Len=2896
TSval=15015089 TSecr=14703903
29 3.479991 10.178.0.110 10.128.17.110
TCP 2962 5647 → 45857 [ACK] Seq=2897 Ack=38 Win=710 Len=2896
TSval=15015089 TSecr=14703903
30 3.479994 10.178.0.110 10.128.17.110
TCP 2962 5647 → 45857 [ACK] Seq=5793 Ack=38 Win=710 Len=2896
TSval=15015089 TSecr=14703903
31 3.480091 10.178.0.110 10.128.17.110
TCP 2962 5647 → 45857 [ACK] Seq=8689 Ack=38 Win=710 Len=2896
TSval=15015089 TSecr=14703903
32 3.480101 10.178.0.110 10.128.17.110
TCP 2962 5647 → 45857 [ACK] Seq=11585 Ack=38 Win=710 Len=2896
TSval=15015089 TSecr=14703903
33 3.480584 10.128.17.110 10.178.0.110
TCP 66 45857 → 5647 [ACK] Seq=38 Ack=14481 Win=1424 Len=0
TSval=14706351 TSecr=15015089
34 3.480597 10.178.0.110 10.128.17.110
TCP 2535 5647 → 45857 [PSH, ACK] Seq=14481 Ack=38 Win=710 Len=2469
TSval=15015089 TSecr=14706351
35 3.481230 10.128.17.110 10.178.0.110
TCP 66 45857 → 5647 [ACK] Seq=38 Ack=16950 Win=1424 Len=0
TSval=14706352 TSecr=15015089
36 3.482661 10.128.17.110 10.178.0.110
TCP 220 45857 → 5647 [PSH, ACK] Seq=38 Ack=16950 Win=1424 Len=154
TSval=14706354 TSecr=15015089
37 3.482671 10.178.0.110 10.128.17.110
TCP 66 5647 → 45857 [ACK] Seq=16950 Ack=192 Win=733 Len=0
TSval=15015092 TSecr=14706354
38 3.484424 10.178.0.110 10.128.17.110
TCP 232 5647 → 45857 [PSH, ACK] Seq=16950 Ack=192 Win=733 Len=166
TSval=15015093 TSecr=14706354
39 3.485299 10.128.17.110 10.178.0.110
TCP 468 45857 → 5647 [PSH, ACK] Seq=192 Ack=17116 Win=1424 Len=402
TSval=14706356 TSecr=15015093
40 3.485985 10.178.0.110 10.128.17.110
TCP 131 5647 → 45857 [PSH, ACK] Seq=17116 Ack=594 Win=755 Len=65
TSval=15015095 TSecr=14706356
41 3.525616 10.128.17.110 10.178.0.110
TCP 66 45857 → 5647 [ACK] Seq=594 Ack=17181 Win=1424 Len=0
TSval=14706397 TSecr=15015095
42 4.002777 10.178.0.110 10.128.17.110
TCP 491 5646 → 43052 [PSH, ACK] Seq=1923 Ack=1554 Win=1432 Len=425
TSval=15015612 TSecr=14705914
43 4.003036 10.128.17.110 10.178.0.110
TCP 485 43052 → 5646 [PSH, ACK] Seq=1554 Ack=1923 Win=1424 Len=419
TSval=14706874 TSecr=15014612
44 4.003043 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=1973 Ack=2348 Win=1424 Len=0
TSval=14706874 TSecr=15015612
45 4.003482 10.128.17.110 10.178.0.110
TCP 140 43052 → 5646 [PSH, ACK] Seq=1973 Ack=2348 Win=1424 Len=74
TSval=14706874 TSecr=15015612
46 4.003552 10.178.0.110 10.128.17.110
TCP 66 5646 → 43052 [ACK] Seq=2348 Ack=2047 Win=1432 Len=0
TSval=15015612 TSecr=14706874
47 4.003750 10.178.0.110 10.128.17.110
TCP 140 5646 → 43052 [PSH, ACK] Seq=2348 Ack=2047 Win=1432 Len=74
TSval=15015613 TSecr=14706874
48 4.043644 10.128.17.110 10.178.0.110
TCP 66 43052 → 5646 [ACK] Seq=2047 Ack=2422 Win=1424 Len=0
TSval=14706915 TSecr=15015613
49 4.076622 10.178.0.110 10.128.17.110
TCP 133 5647 → 45857 [PSH, ACK] Seq=17181 Ack=594 Win=755 Len=67
TSval=15015685 TSecr=3