I had to brought Koji services offline, I expect them to come back in
an hour, we are trying to move the data to different EC2 instance.
This is just a test and if the new instance will work fine, we will do
the switchover during the week.
Sorry for inconvenience, please resubmit your jobs if you have problems.
···
On Sun, Jun 4, 2017 at 1:34 PM, Lukas Zapletal wrote:
> Hey,
>
> I had to brought Koji services offline, I expect them to come back in
> an hour, we are trying to move the data to different EC2 instance.
> This is just a test and if the new instance will work fine, we will do
> the switchover during the week.
>
> Sorry for inconvenience, please resubmit your jobs if you have problems.
>
> --
> Later,
> Lukas @lzap Zapletal
This appears to be active now, but some newRepo tasks are failing with
segfaults. I am able to log into the new host and it appears to be out
of memory - dmesg contains OOM messages, killing mergerepo_c. Could you
add more memory or swap please?
dmesg snippet:
Out of memory: Kill process 25711 (mergerepo_c) score 406 or sacrifice child
Killed process 25711, UID 0, (mergerepo_c) total-vm:1738484kB,
anon-rss:1619816kB, file-rss:72kB
···
On 04/06/17 12:34, Lukas Zapletal wrote:
> I had to brought Koji services offline, I expect them to come back in
> an hour, we are trying to move the data to different EC2 instance.
> This is just a test and if the new instance will work fine, we will do
> the switchover during the week.
we are suffering from capacity issues - builder3 (that's the only
extra builder we currently have) was spawned on hypervisor which is
under stress, therefore kernel reports load of 3 and kojid does not
schedule any builds there since this exceeds the capacity. The
instance is simply idling, I need to figure out how to solve this by
either increasing capacity or restarting it on different hypervisor.
LZ
···
On Sun, Jun 4, 2017 at 1:34 PM, Lukas Zapletal wrote:
> Hey,
>
> I had to brought Koji services offline, I expect them to come back in
> an hour, we are trying to move the data to different EC2 instance.
> This is just a test and if the new instance will work fine, we will do
> the switchover during the week.
>
> Sorry for inconvenience, please resubmit your jobs if you have problems.
>
> --
> Later,
> Lukas @lzap Zapletal
I was wondering, if someone can put the /etc/hosts entry to jenkins, I
assume it's a change somewhere in puppet or foreman-infra?
Thanks
···
On Sun, Jun 4, 2017 at 5:59 PM, Lukas Zapletal wrote:
> The koji was re-spinned as a new instance with new IP 54.163.225.45,
> please change your /etc/hosts until DNS gets populated:
>
> 54.163.225.45 koji.katello.org
>
> The very last job successful that was processed on the old koji was:
>
> http://koji.katello.org/koji/taskinfo?taskID=644141
>
> Just for the record.
>
> On Sun, Jun 4, 2017 at 1:34 PM, Lukas Zapletal wrote:
>> Hey,
>>
>> I had to brought Koji services offline, I expect them to come back in
>> an hour, we are trying to move the data to different EC2 instance.
>> This is just a test and if the new instance will work fine, we will do
>> the switchover during the week.
>>
>> Sorry for inconvenience, please resubmit your jobs if you have problems.
>>
>> --
>> Later,
>> Lukas @lzap Zapletal
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
Yeah we are in process of adding new builder, did not go well
yesterday, Bryan what is the status? Can you bring one up? Make sure
it's the same specs, then I will enable it and remove repocreate tasks
from builder1.
Unfortunately, I can't make swap, there's not make enough room for it.
Bryan, did you not create volume for swap on purpose? It used to be
the /dev/xvdj2 on the old system, now I can see that /dev/xvdj1 spans
across whole volume and there is no room for it. In the meantime, I
created /swapfile of size 1GB, that's all I can do for now.
Resubmitted your task, but I don't believe one extra gig will do it.
We need to wait for Bryan.
Wait I have access to AWS now, checking. The builder is not corrrectly
configured, it's in wrong security group. Let me spawn new one and
configure it.
···
On Wed, Jun 7, 2017 at 10:10 PM, Dominic Cleal wrote:
> On 04/06/17 12:34, Lukas Zapletal wrote:
>> I had to brought Koji services offline, I expect them to come back in
>> an hour, we are trying to move the data to different EC2 instance.
>> This is just a test and if the new instance will work fine, we will do
>> the switchover during the week.
>
> This appears to be active now, but some newRepo tasks are failing with
> segfaults. I am able to log into the new host and it appears to be out
> of memory - dmesg contains OOM messages, killing mergerepo_c. Could you
> add more memory or swap please?
>
> Example task: http://koji.katello.org/koji/taskinfo?taskID=644261
>
> dmesg snippet:
> Out of memory: Kill process 25711 (mergerepo_c) score 406 or sacrifice child
> Killed process 25711, UID 0, (mergerepo_c) total-vm:1738484kB,
> anon-rss:1619816kB, file-rss:72kB
>
> --
> Dominic Cleal
> dominic@cleal.org
>
> --
> You received this message because you are subscribed to the Google Groups "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
After I increased capacity it started to pick tasks, but all were
stuck, I tracked it down to NFS hard connection which was stuck, I
rebooted the builder and it does not come up.
If I won't fix this in a reasonable time, I will shutdown htttp on our
new koji to prevent more tasks to queue up. NFS was always huge pain
of our deployment and I think we can simply launch koji on new but
bigger instance with enough memory and CPUs so we can handle all the
load with single instance without NFS dance.
LZ
···
On Fri, Jun 9, 2017 at 10:17 AM, Lukas Zapletal wrote:
> Hey,
>
> we are suffering from capacity issues - builder3 (that's the only
> extra builder we currently have) was spawned on hypervisor which is
> under stress, therefore kernel reports load of 3 and kojid does not
> schedule any builds there since this exceeds the capacity. The
> instance is simply idling, I need to figure out how to solve this by
> either increasing capacity or restarting it on different hypervisor.
>
> LZ
>
> On Sun, Jun 4, 2017 at 1:34 PM, Lukas Zapletal wrote:
>> Hey,
>>
>> I had to brought Koji services offline, I expect them to come back in
>> an hour, we are trying to move the data to different EC2 instance.
>> This is just a test and if the new instance will work fine, we will do
>> the switchover during the week.
>>
>> Sorry for inconvenience, please resubmit your jobs if you have problems.
>>
>> --
>> Later,
>> Lukas @lzap Zapletal
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
during migration I noticed we are running out of space on the data
volume, I am going to remove Fedora 22 and 23 external repositories
tomorrow. If you still need them, speak up!
LZ
···
On Sun, Jun 4, 2017 at 6:11 PM, Lukas Zapletal wrote:
> I was wondering, if someone can put the /etc/hosts entry to jenkins, I
> assume it's a change somewhere in puppet or foreman-infra?
>
> Thanks
>
> On Sun, Jun 4, 2017 at 5:59 PM, Lukas Zapletal wrote:
>> The koji was re-spinned as a new instance with new IP 54.163.225.45,
>> please change your /etc/hosts until DNS gets populated:
>>
>> 54.163.225.45 koji.katello.org
>>
>> The very last job successful that was processed on the old koji was:
>>
>> http://koji.katello.org/koji/taskinfo?taskID=644141
>>
>> Just for the record.
>>
>> On Sun, Jun 4, 2017 at 1:34 PM, Lukas Zapletal wrote:
>>> Hey,
>>>
>>> I had to brought Koji services offline, I expect them to come back in
>>> an hour, we are trying to move the data to different EC2 instance.
>>> This is just a test and if the new instance will work fine, we will do
>>> the switchover during the week.
>>>
>>> Sorry for inconvenience, please resubmit your jobs if you have problems.
>>>
>>> --
>>> Later,
>>> Lukas @lzap Zapletal
>>
>>
>>
>> --
>> Later,
>> Lukas @lzap Zapletal
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
Bryan, I terminated builder2 it was running in incorrect security
group. You can delete it. The one we need to use is called
"launch-wizard-1" - this is where both koji and builder3 which I
spawned today are. I set the rules to accept all TCP/UDP connections
in this group (ssh/https from outside only).
We are now running "standard" setup of koji master with kojibuilder1
(slow, only 3.5 GB RAM, no createrepo tasks there) and one builder
(kojibuilder3, 8GB RAM, 2 cores, can do all tasks).
I have updated our wiki with some changed commands. I had some
troubles with NFS when we moved external-repos to different volume.
Also added some permission changes.
Please give it a try now, it should be hopefully fine.
···
On Thu, Jun 8, 2017 at 11:17 AM, Lukas Zapletal wrote:
> Yeah we are in process of adding new builder, did not go well
> yesterday, Bryan what is the status? Can you bring one up? Make sure
> it's the same specs, then I will enable it and remove repocreate tasks
> from builder1.
>
> Unfortunately, I can't make swap, there's not make enough room for it.
> Bryan, did you not create volume for swap on purpose? It used to be
> the /dev/xvdj2 on the old system, now I can see that /dev/xvdj1 spans
> across whole volume and there is no room for it. In the meantime, I
> created /swapfile of size 1GB, that's all I can do for now.
> Resubmitted your task, but I don't believe one extra gig will do it.
> We need to wait for Bryan.
>
> Wait I have access to AWS now, checking. The builder is not corrrectly
> configured, it's in wrong security group. Let me spawn new one and
> configure it.
>
>
> On Wed, Jun 7, 2017 at 10:10 PM, Dominic Cleal wrote:
>> On 04/06/17 12:34, Lukas Zapletal wrote:
>>> I had to brought Koji services offline, I expect them to come back in
>>> an hour, we are trying to move the data to different EC2 instance.
>>> This is just a test and if the new instance will work fine, we will do
>>> the switchover during the week.
>>
>> This appears to be active now, but some newRepo tasks are failing with
>> segfaults. I am able to log into the new host and it appears to be out
>> of memory - dmesg contains OOM messages, killing mergerepo_c. Could you
>> add more memory or swap please?
>>
>> Example task: http://koji.katello.org/koji/taskinfo?taskID=644261
>>
>> dmesg snippet:
>> Out of memory: Kill process 25711 (mergerepo_c) score 406 or sacrifice child
>> Killed process 25711, UID 0, (mergerepo_c) total-vm:1738484kB,
>> anon-rss:1619816kB, file-rss:72kB
>>
>> --
>> Dominic Cleal
>> dominic@cleal.org
>>
>> --
>> You received this message because you are subscribed to the Google Groups "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
I have some bad news to share. I started on moving from our 2nd koji
to 3rd koji created as m1.xxlarge instance so we would avoid NFS and
could run on a single VM. I prepared everything and when I was
shutting down all the builders and test VMs I accidentaly stopped the
2nd koji. Unfortunately the original koji instance was stopped
yesterday as well. We lost all data from epheremal storage volumes,
these were /mnt/tmp holding most importantly Postgres database for
kojihub and not sure what else. Our deployment was dozen of symlinks
across two local volumes and EBS.
I found some backups of pgsql on EBS volume, but unfortunately these
stopped being dumped summer 2016
(koji.katello.org-kojidb-20160830-230505.dump is the newest one). Here
are the options we have:
Restore the DB and attempt to start services back - but this will
give us instance back with huge gap of almost one year. I am not sure
how much useful that would be - we will not be able to bu ild into
1.15/1.14 repos, missing tags, builds, logs.
Move to https://cbs.centos.org CentOS CBS, they run their own Koji
and start building there
Speed up Copr transition
Build whole new Koji from scratch
?
···
On Fri, Jun 9, 2017 at 11:42 AM, Lukas Zapletal wrote:
> After I increased capacity it started to pick tasks, but all were
> stuck, I tracked it down to NFS hard connection which was stuck, I
> rebooted the builder and it does not come up.
>
> If I won't fix this in a reasonable time, I will shutdown htttp on our
> new koji to prevent more tasks to queue up. NFS was always huge pain
> of our deployment and I think we can simply launch koji on new but
> bigger instance with enough memory and CPUs so we can handle all the
> load with single instance without NFS dance.
>
> LZ
>
> On Fri, Jun 9, 2017 at 10:17 AM, Lukas Zapletal wrote:
>> Hey,
>>
>> we are suffering from capacity issues - builder3 (that's the only
>> extra builder we currently have) was spawned on hypervisor which is
>> under stress, therefore kernel reports load of 3 and kojid does not
>> schedule any builds there since this exceeds the capacity. The
>> instance is simply idling, I need to figure out how to solve this by
>> either increasing capacity or restarting it on different hypervisor.
>>
>> LZ
>>
>> On Sun, Jun 4, 2017 at 1:34 PM, Lukas Zapletal wrote:
>>> Hey,
>>>
>>> I had to brought Koji services offline, I expect them to come back in
>>> an hour, we are trying to move the data to different EC2 instance.
>>> This is just a test and if the new instance will work fine, we will do
>>> the switchover during the week.
>>>
>>> Sorry for inconvenience, please resubmit your jobs if you have problems.
>>>
>>> --
>>> Later,
>>> Lukas @lzap Zapletal
>>
>>
>>
>> --
>> Later,
>> Lukas @lzap Zapletal
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
The kojibuilder was snapshotted live, so it is doing 20-minutes fsck
of root volume on first boot.
LZ
···
On Thu, Jun 8, 2017 at 1:59 PM, Bryan Kearney wrote:
> Thanks. Any idea why I could not ssh into the instance when it was launched
> in the old security group?
>
> -- bk
>
>
> On 06/08/2017 06:02 AM, Lukas Zapletal wrote:
>>
>> Bryan, I terminated builder2 it was running in incorrect security
>> group. You can delete it. The one we need to use is called
>> "launch-wizard-1" - this is where both koji and builder3 which I
>> spawned today are. I set the rules to accept all TCP/UDP connections
>> in this group (ssh/https from outside only).
>>
>> We are now running "standard" setup of koji master with kojibuilder1
>> (slow, only 3.5 GB RAM, no createrepo tasks there) and one builder
>> (kojibuilder3, 8GB RAM, 2 cores, can do all tasks).
>>
>> http://koji.katello.org/koji/hosts
>>
>> I have updated our wiki with some changed commands. I had some
>> troubles with NFS when we moved external-repos to different volume.
>> Also added some permission changes.
>>
>> http://projects.theforeman.org/projects/foreman/wiki/KojiBuilderSetup
>>
>> Please give it a try now, it should be hopefully fine.
>>
>> On Thu, Jun 8, 2017 at 11:17 AM, Lukas Zapletal wrote:
>>>
>>> Yeah we are in process of adding new builder, did not go well
>>> yesterday, Bryan what is the status? Can you bring one up? Make sure
>>> it's the same specs, then I will enable it and remove repocreate tasks
>>> from builder1.
>>>
>>> Unfortunately, I can't make swap, there's not make enough room for it.
>>> Bryan, did you not create volume for swap on purpose? It used to be
>>> the /dev/xvdj2 on the old system, now I can see that /dev/xvdj1 spans
>>> across whole volume and there is no room for it. In the meantime, I
>>> created /swapfile of size 1GB, that's all I can do for now.
>>> Resubmitted your task, but I don't believe one extra gig will do it.
>>> We need to wait for Bryan.
>>>
>>> Wait I have access to AWS now, checking. The builder is not corrrectly
>>> configured, it's in wrong security group. Let me spawn new one and
>>> configure it.
>>>
>>>
>>> On Wed, Jun 7, 2017 at 10:10 PM, Dominic Cleal wrote:
>>>>
>>>> On 04/06/17 12:34, Lukas Zapletal wrote:
>>>>>
>>>>> I had to brought Koji services offline, I expect them to come back in
>>>>> an hour, we are trying to move the data to different EC2 instance.
>>>>> This is just a test and if the new instance will work fine, we will do
>>>>> the switchover during the week.
>>>>
>>>>
>>>> This appears to be active now, but some newRepo tasks are failing with
>>>> segfaults. I am able to log into the new host and it appears to be out
>>>> of memory - dmesg contains OOM messages, killing mergerepo_c. Could you
>>>> add more memory or swap please?
>>>>
>>>> Example task: http://koji.katello.org/koji/taskinfo?taskID=644261
>>>>
>>>> dmesg snippet:
>>>> Out of memory: Kill process 25711 (mergerepo_c) score 406 or sacrifice
>>>> child
>>>> Killed process 25711, UID 0, (mergerepo_c) total-vm:1738484kB,
>>>> anon-rss:1619816kB, file-rss:72kB
>>>>
>>>> --
>>>> Dominic Cleal
>>>> dominic@cleal.org
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "foreman-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to foreman-dev+unsubscribe@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>>
>>> --
>>> Later,
>>> Lukas @lzap Zapletal
>>
>>
>>
>>
>
A variation here is to use a new DB on the instance we have. So, same
certs just a blank DB.
– bk
···
On 06/09/2017 08:27 AM, Lukas Zapletal wrote:
> I have some bad news to share. I started on moving from our 2nd koji
> to 3rd koji created as m1.xxlarge instance so we would avoid NFS and
> could run on a single VM. I prepared everything and when I was
> shutting down all the builders and test VMs I accidentaly stopped the
> 2nd koji. Unfortunately the original koji instance was stopped
> yesterday as well. We lost all data from epheremal storage volumes,
> these were /mnt/tmp holding most importantly Postgres database for
> kojihub and not sure what else. Our deployment was dozen of symlinks
> across two local volumes and EBS.
>
> I found some backups of pgsql on EBS volume, but unfortunately these
> stopped being dumped summer 2016
> (koji.katello.org-kojidb-20160830-230505.dump is the newest one). Here
> are the options we have:
>
> 1) Restore the DB and attempt to start services back - but this will
> give us instance back with huge gap of almost one year. I am not sure
> how much useful that would be - we will not be able to bu ild into
> 1.15/1.14 repos, missing tags, builds, logs.
>
> 2) Move to https://cbs.centos.org CentOS CBS, they run their own Koji
> and start building there
>
> 3) Speed up Copr transition
>
> 4) Build whole new Koji from scratch
Hmmm if I'd have to create new koji I'd rather start from scratch
reimporting our repos back in.
Current problems we had:
small vm with not enough cores/RAM to run builds
outdated system (RHEL6) and koji version
no use of LVM
data on ephemeral volumes
lots of symlinks which were added as we grew
If I'd need recreate Koji from scratch, I'd try to avoid all of these.
Local volumes should be only used as cache, swap and temp, external
repos should be on external NFS, everything else on dedicated block
device with regular backups.
···
On Fri, Jun 9, 2017 at 2:33 PM, Bryan Kearney wrote:
> On 06/09/2017 08:27 AM, Lukas Zapletal wrote:
>>
>> I have some bad news to share. I started on moving from our 2nd koji
>> to 3rd koji created as m1.xxlarge instance so we would avoid NFS and
>> could run on a single VM. I prepared everything and when I was
>> shutting down all the builders and test VMs I accidentaly stopped the
>> 2nd koji. Unfortunately the original koji instance was stopped
>> yesterday as well. We lost all data from epheremal storage volumes,
>> these were /mnt/tmp holding most importantly Postgres database for
>> kojihub and not sure what else. Our deployment was dozen of symlinks
>> across two local volumes and EBS.
>>
>> I found some backups of pgsql on EBS volume, but unfortunately these
>> stopped being dumped summer 2016
>> (koji.katello.org-kojidb-20160830-230505.dump is the newest one). Here
>> are the options we have:
>>
>> 1) Restore the DB and attempt to start services back - but this will
>> give us instance back with huge gap of almost one year. I am not sure
>> how much useful that would be - we will not be able to bu ild into
>> 1.15/1.14 repos, missing tags, builds, logs.
>>
>> 2) Move to https://cbs.centos.org CentOS CBS, they run their own Koji
>> and start building there
>>
>> 3) Speed up Copr transition
>>
>> 4) Build whole new Koji from scratch
>
>
> A variation here is to use a new DB on the instance we have. So, same certs
> just a blank DB.
>
> -- bk
···
On Fri, Jun 9, 2017 at 2:46 PM, Lukas Zapletal wrote:
> Hmmm if I'd have to create new koji I'd rather start from scratch
> reimporting our repos back in.
>
> Current problems we had:
>
> - small vm with not enough cores/RAM to run builds
> - outdated system (RHEL6) and koji version
> - no use of LVM
> - data on ephemeral volumes
> - lots of symlinks which were added as we grew
>
> If I'd need recreate Koji from scratch, I'd try to avoid all of these.
> Local volumes should be only used as cache, swap and temp, external
> repos should be on external NFS, everything else on dedicated block
> device with regular backups.
>
> On Fri, Jun 9, 2017 at 2:33 PM, Bryan Kearney wrote:
>> On 06/09/2017 08:27 AM, Lukas Zapletal wrote:
>>>
>>> I have some bad news to share. I started on moving from our 2nd koji
>>> to 3rd koji created as m1.xxlarge instance so we would avoid NFS and
>>> could run on a single VM. I prepared everything and when I was
>>> shutting down all the builders and test VMs I accidentaly stopped the
>>> 2nd koji. Unfortunately the original koji instance was stopped
>>> yesterday as well. We lost all data from epheremal storage volumes,
>>> these were /mnt/tmp holding most importantly Postgres database for
>>> kojihub and not sure what else. Our deployment was dozen of symlinks
>>> across two local volumes and EBS.
>>>
>>> I found some backups of pgsql on EBS volume, but unfortunately these
>>> stopped being dumped summer 2016
>>> (koji.katello.org-kojidb-20160830-230505.dump is the newest one). Here
>>> are the options we have:
>>>
>>> 1) Restore the DB and attempt to start services back - but this will
>>> give us instance back with huge gap of almost one year. I am not sure
>>> how much useful that would be - we will not be able to bu ild into
>>> 1.15/1.14 repos, missing tags, builds, logs.
>>>
>>> 2) Move to https://cbs.centos.org CentOS CBS, they run their own Koji
>>> and start building there
>>>
>>> 3) Speed up Copr transition
>>>
>>> 4) Build whole new Koji from scratch
>>
>>
>> A variation here is to use a new DB on the instance we have. So, same certs
>> just a blank DB.
>>
>> -- bk
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
I am planning to reverse-engineer from the composes since I have all the
builds loaded already. But, if there is an easy rebuild that would be great.
– bk
···
On 06/09/2017 05:15 PM, Lukas Zapletal wrote:
> Quick update. Bryan is working on clean koji installation creating
> tags from scratch.
>
> I have koji running from backup, last version built is 1.12. My plan is:
>
> - use our release engineering script to create new 1.13 tags
> - regen repos for 1.13 and rebuild all 1.13 packages using tito rebuild
> - make a compose and compare that against what we have on yum.theforeman.org
>
> If this is success, I will do the same for 1.15.
>
> New koji recovered from backup is at
> http://ec2-54-163-236-166.compute-1.amazonaws.com/koji/ and we
> requested DNS change already, hopefully it's ready on Monday at least
> for nightly builds.
>
> On Fri, Jun 9, 2017 at 2:46 PM, Lukas Zapletal wrote:
>> Hmmm if I'd have to create new koji I'd rather start from scratch
>> reimporting our repos back in.
>>
>> Current problems we had:
>>
>> - small vm with not enough cores/RAM to run builds
>> - outdated system (RHEL6) and koji version
>> - no use of LVM
>> - data on ephemeral volumes
>> - lots of symlinks which were added as we grew
>>
>> If I'd need recreate Koji from scratch, I'd try to avoid all of these.
>> Local volumes should be only used as cache, swap and temp, external
>> repos should be on external NFS, everything else on dedicated block
>> device with regular backups.
>>
>> On Fri, Jun 9, 2017 at 2:33 PM, Bryan Kearney wrote:
>>> On 06/09/2017 08:27 AM, Lukas Zapletal wrote:
>>>>
>>>> I have some bad news to share. I started on moving from our 2nd koji
>>>> to 3rd koji created as m1.xxlarge instance so we would avoid NFS and
>>>> could run on a single VM. I prepared everything and when I was
>>>> shutting down all the builders and test VMs I accidentaly stopped the
>>>> 2nd koji. Unfortunately the original koji instance was stopped
>>>> yesterday as well. We lost all data from epheremal storage volumes,
>>>> these were /mnt/tmp holding most importantly Postgres database for
>>>> kojihub and not sure what else. Our deployment was dozen of symlinks
>>>> across two local volumes and EBS.
>>>>
>>>> I found some backups of pgsql on EBS volume, but unfortunately these
>>>> stopped being dumped summer 2016
>>>> (koji.katello.org-kojidb-20160830-230505.dump is the newest one). Here
>>>> are the options we have:
>>>>
>>>> 1) Restore the DB and attempt to start services back - but this will
>>>> give us instance back with huge gap of almost one year. I am not sure
>>>> how much useful that would be - we will not be able to bu ild into
>>>> 1.15/1.14 repos, missing tags, builds, logs.
>>>>
>>>> 2) Move to https://cbs.centos.org CentOS CBS, they run their own Koji
>>>> and start building there
>>>>
>>>> 3) Speed up Copr transition
>>>>
>>>> 4) Build whole new Koji from scratch
>>>
>>>
>>> A variation here is to use a new DB on the instance we have. So, same certs
>>> just a blank DB.
>>>
>>> -- bk
>>
>>
>>
>> --
>> Later,
>> Lukas @lzap Zapletal
>
>
>
···
On 06/09/2017 05:26 PM, Bryan Kearney wrote:
> Is there a script to rebuild for nightlies? The tags and external repos
> are up and running for nightlies at
>
> http://34.226.218.207/koji/
>
> (cleankoji)
>
> I am planning to reverse-engineer from the composes since I have all the
> builds loaded already. But, if there is an easy rebuild that would be
> great.
>
> -- bk
>
> On 06/09/2017 05:15 PM, Lukas Zapletal wrote:
>> Quick update. Bryan is working on clean koji installation creating
>> tags from scratch.
>>
>> I have koji running from backup, last version built is 1.12. My plan is:
>>
>> - use our release engineering script to create new 1.13 tags
>> - regen repos for 1.13 and rebuild all 1.13 packages using tito rebuild
>> - make a compose and compare that against what we have on
>> yum.theforeman.org
>>
>> If this is success, I will do the same for 1.15.
>>
>> New koji recovered from backup is at
>> http://ec2-54-163-236-166.compute-1.amazonaws.com/koji/ and we
>> requested DNS change already, hopefully it's ready on Monday at least
>> for nightly builds.
>>
>> On Fri, Jun 9, 2017 at 2:46 PM, Lukas Zapletal wrote:
>>> Hmmm if I'd have to create new koji I'd rather start from scratch
>>> reimporting our repos back in.
>>>
>>> Current problems we had:
>>>
>>> - small vm with not enough cores/RAM to run builds
>>> - outdated system (RHEL6) and koji version
>>> - no use of LVM
>>> - data on ephemeral volumes
>>> - lots of symlinks which were added as we grew
>>>
>>> If I'd need recreate Koji from scratch, I'd try to avoid all of these.
>>> Local volumes should be only used as cache, swap and temp, external
>>> repos should be on external NFS, everything else on dedicated block
>>> device with regular backups.
>>>
>>> On Fri, Jun 9, 2017 at 2:33 PM, Bryan Kearney >>> wrote:
>>>> On 06/09/2017 08:27 AM, Lukas Zapletal wrote:
>>>>>
>>>>> I have some bad news to share. I started on moving from our 2nd koji
>>>>> to 3rd koji created as m1.xxlarge instance so we would avoid NFS and
>>>>> could run on a single VM. I prepared everything and when I was
>>>>> shutting down all the builders and test VMs I accidentaly stopped the
>>>>> 2nd koji. Unfortunately the original koji instance was stopped
>>>>> yesterday as well. We lost all data from epheremal storage volumes,
>>>>> these were /mnt/tmp holding most importantly Postgres database for
>>>>> kojihub and not sure what else. Our deployment was dozen of symlinks
>>>>> across two local volumes and EBS.
>>>>>
>>>>> I found some backups of pgsql on EBS volume, but unfortunately these
>>>>> stopped being dumped summer 2016
>>>>> (koji.katello.org-kojidb-20160830-230505.dump is the newest one). Here
>>>>> are the options we have:
>>>>>
>>>>> 1) Restore the DB and attempt to start services back - but this will
>>>>> give us instance back with huge gap of almost one year. I am not sure
>>>>> how much useful that would be - we will not be able to bu ild into
>>>>> 1.15/1.14 repos, missing tags, builds, logs.
>>>>>
>>>>> 2) Move to https://cbs.centos.org CentOS CBS, they run their own Koji
>>>>> and start building there
>>>>>
>>>>> 3) Speed up Copr transition
>>>>>
>>>>> 4) Build whole new Koji from scratch
>>>>
>>>>
>>>> A variation here is to use a new DB on the instance we have. So,
>>>> same certs
>>>> just a blank DB.
>>>>
>>>> -- bk
>>>
>>>
>>>
>>> --
>>> Later,
>>> Lukas @lzap Zapletal
>>
>>
>>
>
Basically you go to foreman-packaging rpm/develop and then:
git annex get (few won't download so delete them for now and skip them)
for each package directory (you can do in loop)
tito release koji-foreman
tito release koji-foreman-plugins
First build all packages without "foreman" in it, then regen
buildroots, then all with "foreman" in it and then solve all failed
builds. The order matters.
But I suggest to turn off repo regeneration after each build, I don't
remember how this can be done (anyone?) Then you regenerate at the end
of the build with:
Also tune your kojibuilder1, I set maxjobs to 7 and capacity to 999 so
it is under nice load of 5-7 concurrent jobs.
···
On Fri, Jun 9, 2017 at 11:32 PM, Bryan Kearney wrote:
> also.. which rhel7 build is used? scl or nonscl?
>
> -- bk
>
>
> On 06/09/2017 05:26 PM, Bryan Kearney wrote:
>>
>> Is there a script to rebuild for nightlies? The tags and external repos
>> are up and running for nightlies at
>>
>> http://34.226.218.207/koji/
>>
>> (cleankoji)
>>
>> I am planning to reverse-engineer from the composes since I have all the
>> builds loaded already. But, if there is an easy rebuild that would be great.
>>
>> -- bk
>>
>> On 06/09/2017 05:15 PM, Lukas Zapletal wrote:
>>>
>>> Quick update. Bryan is working on clean koji installation creating
>>> tags from scratch.
>>>
>>> I have koji running from backup, last version built is 1.12. My plan is:
>>>
>>> - use our release engineering script to create new 1.13 tags
>>> - regen repos for 1.13 and rebuild all 1.13 packages using tito rebuild
>>> - make a compose and compare that against what we have on
>>> yum.theforeman.org
>>>
>>> If this is success, I will do the same for 1.15.
>>>
>>> New koji recovered from backup is at
>>> http://ec2-54-163-236-166.compute-1.amazonaws.com/koji/ and we
>>> requested DNS change already, hopefully it's ready on Monday at least
>>> for nightly builds.
>>>
>>> On Fri, Jun 9, 2017 at 2:46 PM, Lukas Zapletal wrote:
>>>>
>>>> Hmmm if I'd have to create new koji I'd rather start from scratch
>>>> reimporting our repos back in.
>>>>
>>>> Current problems we had:
>>>>
>>>> - small vm with not enough cores/RAM to run builds
>>>> - outdated system (RHEL6) and koji version
>>>> - no use of LVM
>>>> - data on ephemeral volumes
>>>> - lots of symlinks which were added as we grew
>>>>
>>>> If I'd need recreate Koji from scratch, I'd try to avoid all of these.
>>>> Local volumes should be only used as cache, swap and temp, external
>>>> repos should be on external NFS, everything else on dedicated block
>>>> device with regular backups.
>>>>
>>>> On Fri, Jun 9, 2017 at 2:33 PM, Bryan Kearney >>>> wrote:
>>>>>
>>>>> On 06/09/2017 08:27 AM, Lukas Zapletal wrote:
>>>>>>
>>>>>>
>>>>>> I have some bad news to share. I started on moving from our 2nd koji
>>>>>> to 3rd koji created as m1.xxlarge instance so we would avoid NFS and
>>>>>> could run on a single VM. I prepared everything and when I was
>>>>>> shutting down all the builders and test VMs I accidentaly stopped the
>>>>>> 2nd koji. Unfortunately the original koji instance was stopped
>>>>>> yesterday as well. We lost all data from epheremal storage volumes,
>>>>>> these were /mnt/tmp holding most importantly Postgres database for
>>>>>> kojihub and not sure what else. Our deployment was dozen of symlinks
>>>>>> across two local volumes and EBS.
>>>>>>
>>>>>> I found some backups of pgsql on EBS volume, but unfortunately these
>>>>>> stopped being dumped summer 2016
>>>>>> (koji.katello.org-kojidb-20160830-230505.dump is the newest one). Here
>>>>>> are the options we have:
>>>>>>
>>>>>> 1) Restore the DB and attempt to start services back - but this will
>>>>>> give us instance back with huge gap of almost one year. I am not sure
>>>>>> how much useful that would be - we will not be able to bu ild into
>>>>>> 1.15/1.14 repos, missing tags, builds, logs.
>>>>>>
>>>>>> 2) Move to https://cbs.centos.org CentOS CBS, they run their own Koji
>>>>>> and start building there
>>>>>>
>>>>>> 3) Speed up Copr transition
>>>>>>
>>>>>> 4) Build whole new Koji from scratch
>>>>>
>>>>>
>>>>>
>>>>> A variation here is to use a new DB on the instance we have. So, same
>>>>> certs
>>>>> just a blank DB.
>>>>>
>>>>> -- bk
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Later,
>>>> Lukas @lzap Zapletal
>>>
>>>
>>>
>>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Since your clean koji has SSD I can set dmcache for you for the EBS
volume to speed up workload, but I am wrapping up for today, so
tomorrow.
The key is to turn off repo regen after each build, not sure how to do
this. I know it can be done somewhere in configs…
LZ
···
On Fri, Jun 9, 2017 at 11:43 PM, Lukas Zapletal wrote:
> Basically you go to foreman-packaging rpm/develop and then:
>
> git annex get (few won't download so delete them for now and skip them)
>
> for each package directory (you can do in loop)
> tito release koji-foreman
> tito release koji-foreman-plugins
>
> First build all packages without "foreman" in it, then regen
> buildroots, then all with "foreman" in it and then solve all failed
> builds. The order matters.
>
> But I suggest to turn off repo regeneration after each build, I don't
> remember how this can be done (anyone?) Then you regenerate at the end
> of the build with:
>
> koji list-targets --quiet | awk '{print $2}' | sort -u | egrep
> '(foreman|katello|pulp)' | grep nightly | xargs -n 1 koji regen-repo
>
> Also tune your kojibuilder1, I set maxjobs to 7 and capacity to 999 so
> it is under nice load of 5-7 concurrent jobs.
>
> On Fri, Jun 9, 2017 at 11:32 PM, Bryan Kearney wrote:
>> also.. which rhel7 build is used? scl or nonscl?
>>
>> -- bk
>>
>>
>> On 06/09/2017 05:26 PM, Bryan Kearney wrote:
>>>
>>> Is there a script to rebuild for nightlies? The tags and external repos
>>> are up and running for nightlies at
>>>
>>> http://34.226.218.207/koji/
>>>
>>> (cleankoji)
>>>
>>> I am planning to reverse-engineer from the composes since I have all the
>>> builds loaded already. But, if there is an easy rebuild that would be great.
>>>
>>> -- bk
>>>
>>> On 06/09/2017 05:15 PM, Lukas Zapletal wrote:
>>>>
>>>> Quick update. Bryan is working on clean koji installation creating
>>>> tags from scratch.
>>>>
>>>> I have koji running from backup, last version built is 1.12. My plan is:
>>>>
>>>> - use our release engineering script to create new 1.13 tags
>>>> - regen repos for 1.13 and rebuild all 1.13 packages using tito rebuild
>>>> - make a compose and compare that against what we have on
>>>> yum.theforeman.org
>>>>
>>>> If this is success, I will do the same for 1.15.
>>>>
>>>> New koji recovered from backup is at
>>>> http://ec2-54-163-236-166.compute-1.amazonaws.com/koji/ and we
>>>> requested DNS change already, hopefully it's ready on Monday at least
>>>> for nightly builds.
>>>>
>>>> On Fri, Jun 9, 2017 at 2:46 PM, Lukas Zapletal wrote:
>>>>>
>>>>> Hmmm if I'd have to create new koji I'd rather start from scratch
>>>>> reimporting our repos back in.
>>>>>
>>>>> Current problems we had:
>>>>>
>>>>> - small vm with not enough cores/RAM to run builds
>>>>> - outdated system (RHEL6) and koji version
>>>>> - no use of LVM
>>>>> - data on ephemeral volumes
>>>>> - lots of symlinks which were added as we grew
>>>>>
>>>>> If I'd need recreate Koji from scratch, I'd try to avoid all of these.
>>>>> Local volumes should be only used as cache, swap and temp, external
>>>>> repos should be on external NFS, everything else on dedicated block
>>>>> device with regular backups.
>>>>>
>>>>> On Fri, Jun 9, 2017 at 2:33 PM, Bryan Kearney >>>>> wrote:
>>>>>>
>>>>>> On 06/09/2017 08:27 AM, Lukas Zapletal wrote:
>>>>>>>
>>>>>>>
>>>>>>> I have some bad news to share. I started on moving from our 2nd koji
>>>>>>> to 3rd koji created as m1.xxlarge instance so we would avoid NFS and
>>>>>>> could run on a single VM. I prepared everything and when I was
>>>>>>> shutting down all the builders and test VMs I accidentaly stopped the
>>>>>>> 2nd koji. Unfortunately the original koji instance was stopped
>>>>>>> yesterday as well. We lost all data from epheremal storage volumes,
>>>>>>> these were /mnt/tmp holding most importantly Postgres database for
>>>>>>> kojihub and not sure what else. Our deployment was dozen of symlinks
>>>>>>> across two local volumes and EBS.
>>>>>>>
>>>>>>> I found some backups of pgsql on EBS volume, but unfortunately these
>>>>>>> stopped being dumped summer 2016
>>>>>>> (koji.katello.org-kojidb-20160830-230505.dump is the newest one). Here
>>>>>>> are the options we have:
>>>>>>>
>>>>>>> 1) Restore the DB and attempt to start services back - but this will
>>>>>>> give us instance back with huge gap of almost one year. I am not sure
>>>>>>> how much useful that would be - we will not be able to bu ild into
>>>>>>> 1.15/1.14 repos, missing tags, builds, logs.
>>>>>>>
>>>>>>> 2) Move to https://cbs.centos.org CentOS CBS, they run their own Koji
>>>>>>> and start building there
>>>>>>>
>>>>>>> 3) Speed up Copr transition
>>>>>>>
>>>>>>> 4) Build whole new Koji from scratch
>>>>>>
>>>>>>
>>>>>>
>>>>>> A variation here is to use a new DB on the instance we have. So, same
>>>>>> certs
>>>>>> just a blank DB.
>>>>>>
>>>>>> -- bk
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Later,
>>>>> Lukas @lzap Zapletal
>>>>
>>>>
>>>>
>>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscribe@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
Bryan I think I found out, turn off kojira service and then you should
not get regens on each successful build!
LZ
···
On Fri, Jun 9, 2017 at 11:46 PM, Lukas Zapletal wrote:
> Since your clean koji has SSD I can set dmcache for you for the EBS
> volume to speed up workload, but I am wrapping up for today, so
> tomorrow.
>
> The key is to turn off repo regen after each build, not sure how to do
> this. I know it can be done somewhere in configs...
>
> LZ
>
> On Fri, Jun 9, 2017 at 11:43 PM, Lukas Zapletal wrote:
>> Basically you go to foreman-packaging rpm/develop and then:
>>
>> git annex get (few won't download so delete them for now and skip them)
>>
>> for each package directory (you can do in loop)
>> tito release koji-foreman
>> tito release koji-foreman-plugins
>>
>> First build all packages without "foreman" in it, then regen
>> buildroots, then all with "foreman" in it and then solve all failed
>> builds. The order matters.
>>
>> But I suggest to turn off repo regeneration after each build, I don't
>> remember how this can be done (anyone?) Then you regenerate at the end
>> of the build with:
>>
>> koji list-targets --quiet | awk '{print $2}' | sort -u | egrep
>> '(foreman|katello|pulp)' | grep nightly | xargs -n 1 koji regen-repo
>>
>> Also tune your kojibuilder1, I set maxjobs to 7 and capacity to 999 so
>> it is under nice load of 5-7 concurrent jobs.
>>
>> On Fri, Jun 9, 2017 at 11:32 PM, Bryan Kearney wrote:
>>> also.. which rhel7 build is used? scl or nonscl?
>>>
>>> -- bk
>>>
>>>
>>> On 06/09/2017 05:26 PM, Bryan Kearney wrote:
>>>>
>>>> Is there a script to rebuild for nightlies? The tags and external repos
>>>> are up and running for nightlies at
>>>>
>>>> http://34.226.218.207/koji/
>>>>
>>>> (cleankoji)
>>>>
>>>> I am planning to reverse-engineer from the composes since I have all the
>>>> builds loaded already. But, if there is an easy rebuild that would be great.
>>>>
>>>> -- bk
>>>>
>>>> On 06/09/2017 05:15 PM, Lukas Zapletal wrote:
>>>>>
>>>>> Quick update. Bryan is working on clean koji installation creating
>>>>> tags from scratch.
>>>>>
>>>>> I have koji running from backup, last version built is 1.12. My plan is:
>>>>>
>>>>> - use our release engineering script to create new 1.13 tags
>>>>> - regen repos for 1.13 and rebuild all 1.13 packages using tito rebuild
>>>>> - make a compose and compare that against what we have on
>>>>> yum.theforeman.org
>>>>>
>>>>> If this is success, I will do the same for 1.15.
>>>>>
>>>>> New koji recovered from backup is at
>>>>> http://ec2-54-163-236-166.compute-1.amazonaws.com/koji/ and we
>>>>> requested DNS change already, hopefully it's ready on Monday at least
>>>>> for nightly builds.
>>>>>
>>>>> On Fri, Jun 9, 2017 at 2:46 PM, Lukas Zapletal wrote:
>>>>>>
>>>>>> Hmmm if I'd have to create new koji I'd rather start from scratch
>>>>>> reimporting our repos back in.
>>>>>>
>>>>>> Current problems we had:
>>>>>>
>>>>>> - small vm with not enough cores/RAM to run builds
>>>>>> - outdated system (RHEL6) and koji version
>>>>>> - no use of LVM
>>>>>> - data on ephemeral volumes
>>>>>> - lots of symlinks which were added as we grew
>>>>>>
>>>>>> If I'd need recreate Koji from scratch, I'd try to avoid all of these.
>>>>>> Local volumes should be only used as cache, swap and temp, external
>>>>>> repos should be on external NFS, everything else on dedicated block
>>>>>> device with regular backups.
>>>>>>
>>>>>> On Fri, Jun 9, 2017 at 2:33 PM, Bryan Kearney >>>>>> wrote:
>>>>>>>
>>>>>>> On 06/09/2017 08:27 AM, Lukas Zapletal wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> I have some bad news to share. I started on moving from our 2nd koji
>>>>>>>> to 3rd koji created as m1.xxlarge instance so we would avoid NFS and
>>>>>>>> could run on a single VM. I prepared everything and when I was
>>>>>>>> shutting down all the builders and test VMs I accidentaly stopped the
>>>>>>>> 2nd koji. Unfortunately the original koji instance was stopped
>>>>>>>> yesterday as well. We lost all data from epheremal storage volumes,
>>>>>>>> these were /mnt/tmp holding most importantly Postgres database for
>>>>>>>> kojihub and not sure what else. Our deployment was dozen of symlinks
>>>>>>>> across two local volumes and EBS.
>>>>>>>>
>>>>>>>> I found some backups of pgsql on EBS volume, but unfortunately these
>>>>>>>> stopped being dumped summer 2016
>>>>>>>> (koji.katello.org-kojidb-20160830-230505.dump is the newest one). Here
>>>>>>>> are the options we have:
>>>>>>>>
>>>>>>>> 1) Restore the DB and attempt to start services back - but this will
>>>>>>>> give us instance back with huge gap of almost one year. I am not sure
>>>>>>>> how much useful that would be - we will not be able to bu ild into
>>>>>>>> 1.15/1.14 repos, missing tags, builds, logs.
>>>>>>>>
>>>>>>>> 2) Move to https://cbs.centos.org CentOS CBS, they run their own Koji
>>>>>>>> and start building there
>>>>>>>>
>>>>>>>> 3) Speed up Copr transition
>>>>>>>>
>>>>>>>> 4) Build whole new Koji from scratch
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> A variation here is to use a new DB on the instance we have. So, same
>>>>>>> certs
>>>>>>> just a blank DB.
>>>>>>>
>>>>>>> -- bk
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Later,
>>>>>> Lukas @lzap Zapletal
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "foreman-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to foreman-dev+unsubscribe@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Later,
>> Lukas @lzap Zapletal
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
Also one thought - if your SSD VM is fast enough in building, just
send all packages in a loop and mark first and last task ID, and then
regen repos and resubmit all tasks once again. Those which were built
will not build again (will be red tho but skipped) and those missing
dependencies will build. Repeat several times, N^2 in the worst case
LZ
···
On Fri, Jun 9, 2017 at 11:50 PM, Lukas Zapletal wrote:
> Bryan I think I found out, turn off kojira service and then you should
> not get regens on each successful build!
>
> LZ
>
> On Fri, Jun 9, 2017 at 11:46 PM, Lukas Zapletal wrote:
>> Since your clean koji has SSD I can set dmcache for you for the EBS
>> volume to speed up workload, but I am wrapping up for today, so
>> tomorrow.
>>
>> The key is to turn off repo regen after each build, not sure how to do
>> this. I know it can be done somewhere in configs...
>>
>> LZ
>>
>> On Fri, Jun 9, 2017 at 11:43 PM, Lukas Zapletal wrote:
>>> Basically you go to foreman-packaging rpm/develop and then:
>>>
>>> git annex get (few won't download so delete them for now and skip them)
>>>
>>> for each package directory (you can do in loop)
>>> tito release koji-foreman
>>> tito release koji-foreman-plugins
>>>
>>> First build all packages without "foreman" in it, then regen
>>> buildroots, then all with "foreman" in it and then solve all failed
>>> builds. The order matters.
>>>
>>> But I suggest to turn off repo regeneration after each build, I don't
>>> remember how this can be done (anyone?) Then you regenerate at the end
>>> of the build with:
>>>
>>> koji list-targets --quiet | awk '{print $2}' | sort -u | egrep
>>> '(foreman|katello|pulp)' | grep nightly | xargs -n 1 koji regen-repo
>>>
>>> Also tune your kojibuilder1, I set maxjobs to 7 and capacity to 999 so
>>> it is under nice load of 5-7 concurrent jobs.
>>>
>>> On Fri, Jun 9, 2017 at 11:32 PM, Bryan Kearney wrote:
>>>> also.. which rhel7 build is used? scl or nonscl?
>>>>
>>>> -- bk
>>>>
>>>>
>>>> On 06/09/2017 05:26 PM, Bryan Kearney wrote:
>>>>>
>>>>> Is there a script to rebuild for nightlies? The tags and external repos
>>>>> are up and running for nightlies at
>>>>>
>>>>> http://34.226.218.207/koji/
>>>>>
>>>>> (cleankoji)
>>>>>
>>>>> I am planning to reverse-engineer from the composes since I have all the
>>>>> builds loaded already. But, if there is an easy rebuild that would be great.
>>>>>
>>>>> -- bk
>>>>>
>>>>> On 06/09/2017 05:15 PM, Lukas Zapletal wrote:
>>>>>>
>>>>>> Quick update. Bryan is working on clean koji installation creating
>>>>>> tags from scratch.
>>>>>>
>>>>>> I have koji running from backup, last version built is 1.12. My plan is:
>>>>>>
>>>>>> - use our release engineering script to create new 1.13 tags
>>>>>> - regen repos for 1.13 and rebuild all 1.13 packages using tito rebuild
>>>>>> - make a compose and compare that against what we have on
>>>>>> yum.theforeman.org
>>>>>>
>>>>>> If this is success, I will do the same for 1.15.
>>>>>>
>>>>>> New koji recovered from backup is at
>>>>>> http://ec2-54-163-236-166.compute-1.amazonaws.com/koji/ and we
>>>>>> requested DNS change already, hopefully it's ready on Monday at least
>>>>>> for nightly builds.
>>>>>>
>>>>>> On Fri, Jun 9, 2017 at 2:46 PM, Lukas Zapletal wrote:
>>>>>>>
>>>>>>> Hmmm if I'd have to create new koji I'd rather start from scratch
>>>>>>> reimporting our repos back in.
>>>>>>>
>>>>>>> Current problems we had:
>>>>>>>
>>>>>>> - small vm with not enough cores/RAM to run builds
>>>>>>> - outdated system (RHEL6) and koji version
>>>>>>> - no use of LVM
>>>>>>> - data on ephemeral volumes
>>>>>>> - lots of symlinks which were added as we grew
>>>>>>>
>>>>>>> If I'd need recreate Koji from scratch, I'd try to avoid all of these.
>>>>>>> Local volumes should be only used as cache, swap and temp, external
>>>>>>> repos should be on external NFS, everything else on dedicated block
>>>>>>> device with regular backups.
>>>>>>>
>>>>>>> On Fri, Jun 9, 2017 at 2:33 PM, Bryan Kearney >>>>>>> wrote:
>>>>>>>>
>>>>>>>> On 06/09/2017 08:27 AM, Lukas Zapletal wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have some bad news to share. I started on moving from our 2nd koji
>>>>>>>>> to 3rd koji created as m1.xxlarge instance so we would avoid NFS and
>>>>>>>>> could run on a single VM. I prepared everything and when I was
>>>>>>>>> shutting down all the builders and test VMs I accidentaly stopped the
>>>>>>>>> 2nd koji. Unfortunately the original koji instance was stopped
>>>>>>>>> yesterday as well. We lost all data from epheremal storage volumes,
>>>>>>>>> these were /mnt/tmp holding most importantly Postgres database for
>>>>>>>>> kojihub and not sure what else. Our deployment was dozen of symlinks
>>>>>>>>> across two local volumes and EBS.
>>>>>>>>>
>>>>>>>>> I found some backups of pgsql on EBS volume, but unfortunately these
>>>>>>>>> stopped being dumped summer 2016
>>>>>>>>> (koji.katello.org-kojidb-20160830-230505.dump is the newest one). Here
>>>>>>>>> are the options we have:
>>>>>>>>>
>>>>>>>>> 1) Restore the DB and attempt to start services back - but this will
>>>>>>>>> give us instance back with huge gap of almost one year. I am not sure
>>>>>>>>> how much useful that would be - we will not be able to bu ild into
>>>>>>>>> 1.15/1.14 repos, missing tags, builds, logs.
>>>>>>>>>
>>>>>>>>> 2) Move to https://cbs.centos.org CentOS CBS, they run their own Koji
>>>>>>>>> and start building there
>>>>>>>>>
>>>>>>>>> 3) Speed up Copr transition
>>>>>>>>>
>>>>>>>>> 4) Build whole new Koji from scratch
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> A variation here is to use a new DB on the instance we have. So, same
>>>>>>>> certs
>>>>>>>> just a blank DB.
>>>>>>>>
>>>>>>>> -- bk
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Later,
>>>>>>> Lukas @lzap Zapletal
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "foreman-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to foreman-dev+unsubscribe@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> Later,
>>> Lukas @lzap Zapletal
>>
>>
>>
>> --
>> Later,
>> Lukas @lzap Zapletal
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
