Empowering Media: Customer Service Blog

Kernel Update on CentOS 5 & 6 Instances

Zachary — Wed, 25 Apr 2012 11:10:13 +0000

We are performing a kernel update for all CentOS 5 & 6 instances. The upgrade will occur on April 28th from 10p – 11p EDT. It is expected each instance will be unavailable for less than 15 minutes each. We need to perform this update to ensure the security of the kernel.

Level3 Response to Outage

Larry — Thu, 12 Apr 2012 18:44:01 +0000

Here is the official response from Level3 regarding the outages we experienced at their data center.

Summary

Start Time: 3/18/12 14:37 GMT
End Time: 3/18/12 15:09 GMT
Impacted For: 32 Minutes

On March 18th, 2012 at 14:37 GMT Level 3 IP NOC detected alarms indicating both uplinks in Garden City, NY were down isolating the device from the network. Transport NOC was engaged to investigate the isolation.
Transport NOC investigated and identified a previous outage on February 28th, 2012 at this location where these uplinks were down. At that time the equipment vendor identified a Kbyte bug active on the device and recommended restarting the node. Transport NOC performed a warm restart of the node again which restored services. The Technical Service Center confirmed restoration with the impacted customers and the node was monitored for stability.

Tier III and the equipment vendor were engaged to identify a permanent fix. It was determined there were 2 failing cards in the node which caused the services to fail on both working and protect requiring a restart to restore temporarily. Replacement cards were ordered to be replaced on arrival 3/22/12. Before the cards arrived on site a third outage took place on 3/22/12. Transport NOC reported the cards were in route to the site at that time and performed a warm restart of the node to restore services. Upon arrival of the cards the field technician replaced both modules to permanently restore services.

Timeline

18-Mar-2012 14:37 GMT – Level 3 IP NOC detects alarms in Garden City, NJ indicating both uplinks down isolating the Garden City, NJ site. Transport NOC is engaged to investigate the equipment.

15:00 GMT – Transport NOC performs a warm restart on the node to clear alarms.

15:09 GMT – Transport NOC reports the node has restarted and is up and stable at this time. Services are confirmed restored with customers by the Technical Service Center.

15:22 GMT – Tier III is engaged to investigate the failures on the node and identify a permanent fix action.

21-Mar-2012 13:12 GMT – Tier III working with the equipment vendor identify 2 failing modules in the device and replacement parts are ordered to affect permanent resolution. The modules will be replaced upon arrival 3/22/12.

22-Mar-2012 00:03 GMT – IP NOC reports both uplinks in Garden City, NJ are down. Transport NOC is engaged to investigate.

00:33 GMT – Transport NOC indicated this is a known issue and replacement cards are in route to the site at this time.

01:01 GMT – Transport NOC performed a warm restart of the device to restore services until the replacement cards are on site.

06:46 GMT – The first card arrives on site and is replaced by the field technician.

09:32 GMT – The second card arrives and is replaced by the field technician. The technical service center confirms restoration with the impacted customers.

DC1 Unavailable

Larry — Thu, 22 Mar 2012 00:18:48 +0000

DC1 (Level3) routes appear to be not functioning again. We are aware of the issue and in contact with our ISP to determine why this happened, and why has this occurred again. We will update this post when we get new information.

Update on CentOS 5 Instances

Zachary — Mon, 19 Mar 2012 16:48:04 +0000

We are performing a update for all CentOS 5 instances. The upgrade will occur on March 24th from 10p – 11p EDT. It is expected each instance will be unavailable for less than 15 minutes each. We need to perform this update to ensure the security of the kernel.

Kernel Update on CentOS 6 Instances

Zachary — Thu, 08 Mar 2012 18:09:18 +0000

We are performing a kernel update for all CentOS 6 instances. The upgrade will occur on March 10th from 10p – 11p EDT. It is expected each instance will be unavailable for less than 15 minutes each. We need to perform this update to ensure the security of the kernel.

Server Migration For Corporate Sites

Zachary — Fri, 02 Mar 2012 18:56:43 +0000

Sunday, March 4th 2012 from 7pm – 11pm EST we will be migrating our corporate accounts to a new server.

Supportem.com – Customer Service Center
empoweringmedia.com – Cloud Computing Management

The sites not only include our web sites, but also includes corporate E-mail and our helpdesk. All will be unavailable during this time period. If during this time you need to contact us, you can contact us via the following methods:

Twitter: @empoweringmedia
Facebook: EmpoweringMedia
Telephone: 1-516-741-0181 (option #2 and leave a voicemail)

During this downtime, no customer sites will be affected.

Outage with DC1/Level3

Larry — Tue, 28 Feb 2012 14:04:52 +0000

This morning starting from approximately 8AM EDT Level3, our primary provider with our data center experienced a massive routing outage within the North East corridor of the United States. Our data center should now be accessible within their network. We are in communication with Level3 to determine the cause of the issue. If you are still experiencing any problems please contact us via our helpdesk, Facebook, or Twitter.

Hostcube.com Site Migration

Zachary — Thu, 23 Feb 2012 13:12:01 +0000

Saturday February 25th 2012 from 10pm – 11pm EST we will be migrating Hostcube.com to a new server.
During this downtime, no customer sites will be affected.

Node01.dc1 crash

Larry — Wed, 15 Feb 2012 01:56:55 +0000

Node01.dc1 crashed approximately at 8:40pm EST. The node is currently in the process of checking file systems.

If this issue occurs again we will replace the hardware. I will update this post when all instances are back online.

Pontus Upgrade

Zachary — Tue, 14 Feb 2012 14:28:40 +0000

We will be upgrading the diskspace on Pontus Saturday February 18th from 10P – 11P EST