Our Koji is currently down from a web perspective and ssh access. Please
don't merge anything further to -packaging until we've resolved this. All
actions requiring Koji repositories for testing or actions in Koji cannot
be performed.
If Bryan or Lukas (since I am not sure who has AWS access to the box) could
investigate for us please.
Likely a hardware failure according to notification, our instance is
not responding. We are trying restart first.
···
***
EC2 has detected degradation of the underlying hardware hosting your
Amazon EC2 instance associated with this event in the us-east-1
region. Due to this degradation, your instance could already be
unreachable. After 2017-10-30 00:00 UTC your instance, which has an
EBS volume as the root device, will be stopped.
How does this affect you?
Your instance’s root device is an EBS volume and the instance will be
stopped after the specified retirement date. You can start it again at
any time. Note that if you have EC2 instance store volumes attached to
the instance, any data on these volumes will be lost when the instance
is stopped or terminated as these volumes are physically attached to
the host computer
What do you need to do?
You may still be able to access the instance. We recommend that you
replace the instance by creating an AMI of your instance and launch a
new instance from the AMI. For more information please see Amazon
Machine Images (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html)
in the EC2 User Guide. In case of difficulties stopping your
EBS-backed instance, please see the Instance FAQ
(http://aws.amazon.com/instance-help/#ebs-stuck-stopping).
Why retirement?
AWS may schedule instances for retirement in cases where there is an
unrecoverable issue with the underlying hardware. For more information
about scheduled retirement events please see the EC2 user guide
(http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-retirement.html).
To avoid single points of failure within critical applications, please
refer to our architecture center for more information on implementing
fault-tolerant architectures: http://aws.amazon.com/architecture
Our Koji is currently down from a web perspective and ssh access. Please
don’t merge anything further to -packaging until we’ve resolved this. All
actions requiring Koji repositories for testing or actions in Koji cannot be
performed.
If Bryan or Lukas (since I am not sure who has AWS access to the box) could
investigate for us please.
Image which is now pending completion is called ami-2ecd6b54 and it
has both root EBS volume and data EBS volume (900 GB) which is the
reason why this is so slow. Then we can start new AMI to see if it
boots up. The instance type was i3.xlarge for the record, we want the
same one which was the best performance/price ratio for Koji workload.
After new koji boots up we want to recreate /mnt/tmp folder structure
and swap. Open /mnt/fstab to see the mountpoints, the i3.xlarge has
like 950 GB ephemeral storage, but it was unused (we had 400 GB swap
and 400 GB /mnt/tmp). In the /mnt/tmp directory there was just few
directories where koji was doing building locally. More CPU intensive
flavours were more expensive so we had this IO intensive one instead
which stills delivers 4 cores and 32 GB RAM which is good.
On the main EBS volume (900 GB one) there is a backup directory and in
this directory we should have a backup of the directory structure.
There is a cron job that does this daily. It was not backing up
temporary files, just directories. This should be enough to get koji
daemons back online. There should be a daily backup of postgre
database as well.
The EBS volume snapshot is ongoing, it is required to do snapshot
first and then you can create new AMI from it. I have some family
business in an hour, so I am writing this summary so someone else form
US timezone can carry on from here. Next step would be - start new
instance, let it boot (there might be ext4 file system check - not
sure if we use XFS or ext4 for the data volume - see the AMI console
for boot) and then find the /mnt/tmp backups, restore the directory
structure, restart (rather whole system than just koji) and it should
show up. Last thing would be to associate the elastic IP.
···
On Thu, Oct 26, 2017 at 2:56 PM, Lukas Zapletal wrote:
> Likely a hardware failure according to notification, our instance is
> not responding. We are trying restart first.
>
> ***
>
> EC2 has detected degradation of the underlying hardware hosting your
> Amazon EC2 instance associated with this event in the us-east-1
> region. Due to this degradation, your instance could already be
> unreachable. After 2017-10-30 00:00 UTC your instance, which has an
> EBS volume as the root device, will be stopped.
>
> You can see more information on your instances that are scheduled for
> retirement in the AWS Management Console
> (https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Events)
>
> * How does this affect you?
> Your instance's root device is an EBS volume and the instance will be
> stopped after the specified retirement date. You can start it again at
> any time. Note that if you have EC2 instance store volumes attached to
> the instance, any data on these volumes will be lost when the instance
> is stopped or terminated as these volumes are physically attached to
> the host computer
>
> * What do you need to do?
> You may still be able to access the instance. We recommend that you
> replace the instance by creating an AMI of your instance and launch a
> new instance from the AMI. For more information please see Amazon
> Machine Images (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html)
> in the EC2 User Guide. In case of difficulties stopping your
> EBS-backed instance, please see the Instance FAQ
> (http://aws.amazon.com/instance-help/#ebs-stuck-stopping).
>
> * Why retirement?
> AWS may schedule instances for retirement in cases where there is an
> unrecoverable issue with the underlying hardware. For more information
> about scheduled retirement events please see the EC2 user guide
> (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-retirement.html).
> To avoid single points of failure within critical applications, please
> refer to our architecture center for more information on implementing
> fault-tolerant architectures: http://aws.amazon.com/architecture
>
> LZ
>
> On Thu, Oct 26, 2017 at 1:51 PM, Eric D Helms wrote:
>> Our Koji is currently down from a web perspective and ssh access. Please
>> don't merge anything further to -packaging until we've resolved this. All
>> actions requiring Koji repositories for testing or actions in Koji cannot be
>> performed.
>>
>> If Bryan or Lukas (since I am not sure who has AWS access to the box) could
>> investigate for us please.
>>
>> --
>> Eric D. Helms
>> Red Hat Engineering
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscribe@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
Image which is now pending completion is called ami-2ecd6b54 and it
has both root EBS volume and data EBS volume (900 GB) which is the
reason why this is so slow. Then we can start new AMI to see if it
boots up. The instance type was i3.xlarge for the record, we want the
same one which was the best performance/price ratio for Koji workload.
After new koji boots up we want to recreate /mnt/tmp folder structure
and swap. Open /mnt/fstab to see the mountpoints, the i3.xlarge has
like 950 GB ephemeral storage, but it was unused (we had 400 GB swap
and 400 GB /mnt/tmp). In the /mnt/tmp directory there was just few
directories where koji was doing building locally. More CPU intensive
flavours were more expensive so we had this IO intensive one instead
which stills delivers 4 cores and 32 GB RAM which is good.
On the main EBS volume (900 GB one) there is a backup directory and in
this directory we should have a backup of the directory structure.
There is a cron job that does this daily. It was not backing up
temporary files, just directories. This should be enough to get koji
daemons back online. There should be a daily backup of postgre
database as well.
The EBS volume snapshot is ongoing, it is required to do snapshot
first and then you can create new AMI from it. I have some family
business in an hour, so I am writing this summary so someone else form
US timezone can carry on from here. Next step would be - start new
instance, let it boot (there might be ext4 file system check - not
sure if we use XFS or ext4 for the data volume - see the AMI console
for boot) and then find the /mnt/tmp backups, restore the directory
structure, restart (rather whole system than just koji) and it should
show up. Last thing would be to associate the elastic IP.
On Thu, Oct 26, 2017 at 2:56 PM, Lukas Zapletal lzap@redhat.com wrote:
Likely a hardware failure according to notification, our instance is
not responding. We are trying restart first.
EC2 has detected degradation of the underlying hardware hosting your
Amazon EC2 instance associated with this event in the us-east-1
region. Due to this degradation, your instance could already be
unreachable. After 2017-10-30 00:00 UTC your instance, which has an
EBS volume as the root device, will be stopped.
How does this affect you?
Your instance’s root device is an EBS volume and the instance will be
stopped after the specified retirement date. You can start it again at
any time. Note that if you have EC2 instance store volumes attached to
the instance, any data on these volumes will be lost when the instance
is stopped or terminated as these volumes are physically attached to
the host computer
What do you need to do?
You may still be able to access the instance. We recommend that you
replace the instance by creating an AMI of your instance and launch a
new instance from the AMI. For more information please see Amazon
Machine Images (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.
html)
in the EC2 User Guide. In case of difficulties stopping your
EBS-backed instance, please see the Instance FAQ
(http://aws.amazon.com/instance-help/#ebs-stuck-stopping).
Why retirement?
AWS may schedule instances for retirement in cases where there is an
unrecoverable issue with the underlying hardware. For more information
about scheduled retirement events please see the EC2 user guide
(http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/
instance-retirement.html).
To avoid single points of failure within critical applications, please
refer to our architecture center for more information on implementing
fault-tolerant architectures: http://aws.amazon.com/architecture
It looks like the rsync service isn't started causing our promotion
pipeline to fail. Could you have a look?
···
On Thu, Oct 26, 2017 at 03:31:28PM +0200, Lukas Zapletal wrote:
>Here is an update. Restart did not help, we stopped the instance and I
>am following this guide to create new AMI and start it:
>
>http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html
>
>Image which is now pending completion is called ami-2ecd6b54 and it
>has both root EBS volume and data EBS volume (900 GB) which is the
>reason why this is so slow. Then we can start new AMI to see if it
>boots up. The instance type was i3.xlarge for the record, we want the
>same one which was the best performance/price ratio for Koji workload.
>
>After new koji boots up we want to recreate /mnt/tmp folder structure
>and swap. Open /mnt/fstab to see the mountpoints, the i3.xlarge has
>like 950 GB ephemeral storage, but it was unused (we had 400 GB swap
>and 400 GB /mnt/tmp). In the /mnt/tmp directory there was just few
>directories where koji was doing building locally. More CPU intensive
>flavours were more expensive so we had this IO intensive one instead
>which stills delivers 4 cores and 32 GB RAM which is good.
>
>On the main EBS volume (900 GB one) there is a backup directory and in
>this directory we should have a backup of the directory structure.
>There is a cron job that does this daily. It was not backing up
>temporary files, just directories. This should be enough to get koji
>daemons back online. There should be a daily backup of postgre
>database as well.
>
>The EBS volume snapshot is ongoing, it is required to do snapshot
>first and then you can create new AMI from it. I have some family
>business in an hour, so I am writing this summary so someone else form
>US timezone can carry on from here. Next step would be - start new
>instance, let it boot (there might be ext4 file system check - not
>sure if we use XFS or ext4 for the data volume - see the AMI console
>for boot) and then find the /mnt/tmp backups, restore the directory
>structure, restart (rather whole system than just koji) and it should
>show up. Last thing would be to associate the elastic IP.
>
>
>
>On Thu, Oct 26, 2017 at 2:56 PM, Lukas Zapletal wrote:
>> Likely a hardware failure according to notification, our instance is
>> not responding. We are trying restart first.
>>
>> ***
>>
>> EC2 has detected degradation of the underlying hardware hosting your
>> Amazon EC2 instance associated with this event in the us-east-1
>> region. Due to this degradation, your instance could already be
>> unreachable. After 2017-10-30 00:00 UTC your instance, which has an
>> EBS volume as the root device, will be stopped.
>>
>> You can see more information on your instances that are scheduled for
>> retirement in the AWS Management Console
>> (https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Events)
>>
>> * How does this affect you?
>> Your instance's root device is an EBS volume and the instance will be
>> stopped after the specified retirement date. You can start it again at
>> any time. Note that if you have EC2 instance store volumes attached to
>> the instance, any data on these volumes will be lost when the instance
>> is stopped or terminated as these volumes are physically attached to
>> the host computer
>>
>> * What do you need to do?
>> You may still be able to access the instance. We recommend that you
>> replace the instance by creating an AMI of your instance and launch a
>> new instance from the AMI. For more information please see Amazon
>> Machine Images (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html)
>> in the EC2 User Guide. In case of difficulties stopping your
>> EBS-backed instance, please see the Instance FAQ
>> (http://aws.amazon.com/instance-help/#ebs-stuck-stopping).
>>
>> * Why retirement?
>> AWS may schedule instances for retirement in cases where there is an
>> unrecoverable issue with the underlying hardware. For more information
>> about scheduled retirement events please see the EC2 user guide
>> (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-retirement.html).
>> To avoid single points of failure within critical applications, please
>> refer to our architecture center for more information on implementing
>> fault-tolerant architectures: http://aws.amazon.com/architecture
>>
>> LZ
>>
>> On Thu, Oct 26, 2017 at 1:51 PM, Eric D Helms wrote:
>>> Our Koji is currently down from a web perspective and ssh access. Please
>>> don't merge anything further to -packaging until we've resolved this. All
>>> actions requiring Koji repositories for testing or actions in Koji cannot be
>>> performed.
>>>
>>> If Bryan or Lukas (since I am not sure who has AWS access to the box) could
>>> investigate for us please.
>>>
>>> --
>>> Eric D. Helms
>>> Red Hat Engineering
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "foreman-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to foreman-dev+unsubscribe@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Later,
>> Lukas @lzap Zapletal
>
>
>
>--
>Later,
> Lukas @lzap Zapletal
>
>--
>You received this message because you are subscribed to the Google Groups "foreman-dev" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.
Turns out it was started in a new security group which didn't allow
rsync. lzap fixed that now.
···
On Sat, Oct 28, 2017 at 08:09:39PM +0200, Ewoud Kohl van Wijngaarden wrote:
>It looks like the rsync service isn't started causing our promotion
>pipeline to fail. Could you have a look?
>
>On Thu, Oct 26, 2017 at 03:31:28PM +0200, Lukas Zapletal wrote:
>>Here is an update. Restart did not help, we stopped the instance and I
>>am following this guide to create new AMI and start it:
>>
>>http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html
>>
>>Image which is now pending completion is called ami-2ecd6b54 and it
>>has both root EBS volume and data EBS volume (900 GB) which is the
>>reason why this is so slow. Then we can start new AMI to see if it
>>boots up. The instance type was i3.xlarge for the record, we want the
>>same one which was the best performance/price ratio for Koji workload.
>>
>>After new koji boots up we want to recreate /mnt/tmp folder structure
>>and swap. Open /mnt/fstab to see the mountpoints, the i3.xlarge has
>>like 950 GB ephemeral storage, but it was unused (we had 400 GB swap
>>and 400 GB /mnt/tmp). In the /mnt/tmp directory there was just few
>>directories where koji was doing building locally. More CPU intensive
>>flavours were more expensive so we had this IO intensive one instead
>>which stills delivers 4 cores and 32 GB RAM which is good.
>>
>>On the main EBS volume (900 GB one) there is a backup directory and in
>>this directory we should have a backup of the directory structure.
>>There is a cron job that does this daily. It was not backing up
>>temporary files, just directories. This should be enough to get koji
>>daemons back online. There should be a daily backup of postgre
>>database as well.
>>
>>The EBS volume snapshot is ongoing, it is required to do snapshot
>>first and then you can create new AMI from it. I have some family
>>business in an hour, so I am writing this summary so someone else form
>>US timezone can carry on from here. Next step would be - start new
>>instance, let it boot (there might be ext4 file system check - not
>>sure if we use XFS or ext4 for the data volume - see the AMI console
>>for boot) and then find the /mnt/tmp backups, restore the directory
>>structure, restart (rather whole system than just koji) and it should
>>show up. Last thing would be to associate the elastic IP.
>>
>>
>>
>>On Thu, Oct 26, 2017 at 2:56 PM, Lukas Zapletal wrote:
>>>Likely a hardware failure according to notification, our instance is
>>>not responding. We are trying restart first.
>>>
>>>***
>>>
>>>EC2 has detected degradation of the underlying hardware hosting your
>>>Amazon EC2 instance associated with this event in the us-east-1
>>>region. Due to this degradation, your instance could already be
>>>unreachable. After 2017-10-30 00:00 UTC your instance, which has an
>>>EBS volume as the root device, will be stopped.
>>>
>>>You can see more information on your instances that are scheduled for
>>>retirement in the AWS Management Console
>>>(https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Events)
>>>
>>>* How does this affect you?
>>>Your instance's root device is an EBS volume and the instance will be
>>>stopped after the specified retirement date. You can start it again at
>>>any time. Note that if you have EC2 instance store volumes attached to
>>>the instance, any data on these volumes will be lost when the instance
>>>is stopped or terminated as these volumes are physically attached to
>>>the host computer
>>>
>>>* What do you need to do?
>>>You may still be able to access the instance. We recommend that you
>>>replace the instance by creating an AMI of your instance and launch a
>>>new instance from the AMI. For more information please see Amazon
>>>Machine Images (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html)
>>>in the EC2 User Guide. In case of difficulties stopping your
>>>EBS-backed instance, please see the Instance FAQ
>>>(http://aws.amazon.com/instance-help/#ebs-stuck-stopping).
>>>
>>>* Why retirement?
>>>AWS may schedule instances for retirement in cases where there is an
>>>unrecoverable issue with the underlying hardware. For more information
>>>about scheduled retirement events please see the EC2 user guide
>>>(http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-retirement.html).
>>>To avoid single points of failure within critical applications, please
>>>refer to our architecture center for more information on implementing
>>>fault-tolerant architectures: http://aws.amazon.com/architecture
>>>
>>>LZ
>>>
>>>On Thu, Oct 26, 2017 at 1:51 PM, Eric D Helms wrote:
>>>>Our Koji is currently down from a web perspective and ssh access. Please
>>>>don't merge anything further to -packaging until we've resolved this. All
>>>>actions requiring Koji repositories for testing or actions in Koji cannot be
>>>>performed.
>>>>
>>>>If Bryan or Lukas (since I am not sure who has AWS access to the box) could
>>>>investigate for us please.
>>>>
>>>>--
>>>>Eric D. Helms
>>>>Red Hat Engineering
>>>>
>>>>--
>>>>You received this message because you are subscribed to the Google Groups
>>>>"foreman-dev" group.
>>>>To unsubscribe from this group and stop receiving emails from it, send an
>>>>email to foreman-dev+unsubscribe@googlegroups.com.
>>>>For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>>--
>>>Later,
>>> Lukas @lzap Zapletal
>>
>>
>>
>>--
>>Later,
>> Lukas @lzap Zapletal
>>
>>--
>>You received this message because you are subscribed to the Google Groups "foreman-dev" group.
>>To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
>>For more options, visit https://groups.google.com/d/optout.
>
>--
>You received this message because you are subscribed to the Google Groups "foreman-dev" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.
I can confirm that rsync port was not in the SG, added and Ewoud
tested this. Thanks.
What happened is in a summary in a separate thread. Let's carry on
discussion there. But the good thing about this failure is we have
been able to recover quite fast. Next time this happens I think we are
able to spin off in dozens of minutes now. (Most of the time we were
waiting for a snapshot of the data EBS volume just to have a copy.)
LZ
···
On Mon, Oct 30, 2017 at 9:33 AM, Ewoud Kohl van Wijngaarden wrote:
> Turns out it was started in a new security group which didn't allow rsync.
> lzap fixed that now.
>
>
> On Sat, Oct 28, 2017 at 08:09:39PM +0200, Ewoud Kohl van Wijngaarden wrote:
>>
>> It looks like the rsync service isn't started causing our promotion
>> pipeline to fail. Could you have a look?
>>
>> On Thu, Oct 26, 2017 at 03:31:28PM +0200, Lukas Zapletal wrote:
>>>
>>> Here is an update. Restart did not help, we stopped the instance and I
>>> am following this guide to create new AMI and start it:
>>>
>>>
>>> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html
>>>
>>> Image which is now pending completion is called ami-2ecd6b54 and it
>>> has both root EBS volume and data EBS volume (900 GB) which is the
>>> reason why this is so slow. Then we can start new AMI to see if it
>>> boots up. The instance type was i3.xlarge for the record, we want the
>>> same one which was the best performance/price ratio for Koji workload.
>>>
>>> After new koji boots up we want to recreate /mnt/tmp folder structure
>>> and swap. Open /mnt/fstab to see the mountpoints, the i3.xlarge has
>>> like 950 GB ephemeral storage, but it was unused (we had 400 GB swap
>>> and 400 GB /mnt/tmp). In the /mnt/tmp directory there was just few
>>> directories where koji was doing building locally. More CPU intensive
>>> flavours were more expensive so we had this IO intensive one instead
>>> which stills delivers 4 cores and 32 GB RAM which is good.
>>>
>>> On the main EBS volume (900 GB one) there is a backup directory and in
>>> this directory we should have a backup of the directory structure.
>>> There is a cron job that does this daily. It was not backing up
>>> temporary files, just directories. This should be enough to get koji
>>> daemons back online. There should be a daily backup of postgre
>>> database as well.
>>>
>>> The EBS volume snapshot is ongoing, it is required to do snapshot
>>> first and then you can create new AMI from it. I have some family
>>> business in an hour, so I am writing this summary so someone else form
>>> US timezone can carry on from here. Next step would be - start new
>>> instance, let it boot (there might be ext4 file system check - not
>>> sure if we use XFS or ext4 for the data volume - see the AMI console
>>> for boot) and then find the /mnt/tmp backups, restore the directory
>>> structure, restart (rather whole system than just koji) and it should
>>> show up. Last thing would be to associate the elastic IP.
>>>
>>>
>>>
>>> On Thu, Oct 26, 2017 at 2:56 PM, Lukas Zapletal wrote:
>>>>
>>>> Likely a hardware failure according to notification, our instance is
>>>> not responding. We are trying restart first.
>>>>
>>>> ***
>>>>
>>>> EC2 has detected degradation of the underlying hardware hosting your
>>>> Amazon EC2 instance associated with this event in the us-east-1
>>>> region. Due to this degradation, your instance could already be
>>>> unreachable. After 2017-10-30 00:00 UTC your instance, which has an
>>>> EBS volume as the root device, will be stopped.
>>>>
>>>> You can see more information on your instances that are scheduled for
>>>> retirement in the AWS Management Console
>>>> (https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Events)
>>>>
>>>> * How does this affect you?
>>>> Your instance's root device is an EBS volume and the instance will be
>>>> stopped after the specified retirement date. You can start it again at
>>>> any time. Note that if you have EC2 instance store volumes attached to
>>>> the instance, any data on these volumes will be lost when the instance
>>>> is stopped or terminated as these volumes are physically attached to
>>>> the host computer
>>>>
>>>> * What do you need to do?
>>>> You may still be able to access the instance. We recommend that you
>>>> replace the instance by creating an AMI of your instance and launch a
>>>> new instance from the AMI. For more information please see Amazon
>>>> Machine Images
>>>> (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html)
>>>> in the EC2 User Guide. In case of difficulties stopping your
>>>> EBS-backed instance, please see the Instance FAQ
>>>> (http://aws.amazon.com/instance-help/#ebs-stuck-stopping).
>>>>
>>>> * Why retirement?
>>>> AWS may schedule instances for retirement in cases where there is an
>>>> unrecoverable issue with the underlying hardware. For more information
>>>> about scheduled retirement events please see the EC2 user guide
>>>>
>>>> (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-retirement.html).
>>>> To avoid single points of failure within critical applications, please
>>>> refer to our architecture center for more information on implementing
>>>> fault-tolerant architectures: http://aws.amazon.com/architecture
>>>>
>>>> LZ
>>>>
>>>> On Thu, Oct 26, 2017 at 1:51 PM, Eric D Helms >>>> wrote:
>>>>>
>>>>> Our Koji is currently down from a web perspective and ssh access.
>>>>> Please
>>>>> don't merge anything further to -packaging until we've resolved this.
>>>>> All
>>>>> actions requiring Koji repositories for testing or actions in Koji
>>>>> cannot be
>>>>> performed.
>>>>>
>>>>> If Bryan or Lukas (since I am not sure who has AWS access to the box)
>>>>> could
>>>>> investigate for us please.
>>>>>
>>>>> --
>>>>> Eric D. Helms
>>>>> Red Hat Engineering
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups
>>>>> "foreman-dev" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an
>>>>> email to foreman-dev+unsubscribe@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Later,
>>>> Lukas @lzap Zapletal
>>>
>>>
>>>
>>>
>>> --
>>> Later,
>>> Lukas @lzap Zapletal
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "foreman-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to foreman-dev+unsubscribe@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscribe@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
