[Completed] Savvis Issues
Posted on Friday August 24th, 2007 - 10:38 PM by staff member Adam
Hey,
It appears that our company website and all servers located in the Savvis datacenter are have packet loss and are unable to connect. We are aware of the issue and we are working on it with Savvis.
Staff Updates:
We are currently experiencing issues upstream from our Savvis. Savvis was performing un-scheduled maintainence on a backbone devices which has caused a peering issue and sever or 100% packet loss for most hosts on the Savvis network.
We are working with Savvis now to get the issue resolved as quickly as possible and will update everyone with further details as they are found.
--
I am still waiting on a call back from Savvis. Will be a few more mins before I get an update and ETA for resolution.
---
Savvis has responded and is in the process of restoring previous configurations on the affected devices. The devices will be reloaded and brought back online in hopefully the next 15-20 mins.
---
Current Savvis ETA: 45mins - 1 hour
I am waiting on a call back from Savvis NOC as more details are known and will update this post with them.
Packet loss appears to be at 0% now. Repeated tests of 200-300+ pings show 0% loss in the past 15 mins.
— 209.67.208.178 ping statistics —
300 packets transmitted, 300 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.507/1.692/2.008/0.083 ms
Savvis has not closed out the issue or stated it is 100% resolved so some issues may re-appear until the issue is fully resolved.
Savvis has reported 1 of the 3 routers has been fully resolved and is back online. They are now making changes to the remaining 2 that are showing issues. There will be brief lapses of 100% packet loss or heavy packet loss while the last 2 are being repaired.
---
Savvis is still working on the issues with 2 affected backbone routers. They are stating it will be a new ETA of 1-2 Hours from now. The current issue appears to only be affecting some ranges at the IDC and some inbound peers at this time
=============
Starting at or around 10:00PM on August 24th, 2007, EastSolid's Network Operations Center picked up some network issues with servers with in the Savvis Datacenter. EastSolid first picked up small packet loss which showed only about 5% loss. Shortly after we found problems with packet loss, the packet loss increased to 100%. Our NOC Technicians quickly started to fix the problem. Shortly after packet loss increased, we called our upstream provider for the Savvis Datacenter and found out, they were performing an un-scheduled maintenance on a few backbone devices.
We alerted Savvis the problems we were facing with the un-scheduled maintenance. We received a phone call back from Savvis at or around 11:50PM on August 24th, 2007 stating they are in the process of restoring previous configurations on the affected devices. The affected devices are core backbone routers that supply most of the datacenter internet access. We are still unsure why an un-scheduled maintenance was being performed on these core routers but can assume it was on an emergency bases.
Because the core routers were down, most if not all internet access to the Savvis datacenter was halted. This issue even took down our own company website, www.EastSolid.com. From our past experience in web hosting and running a top notch network, we formed an Off Site Emergency Status Page, www.CentralSolid.com. This website is hosted in our Los Angeles, CA, USA office away from the network that hosts any of your servers. We were updating clients of this issue in real time while our website was down.
At or around 12:50AM on August 25th, 2007, the Savvis network was slowly lowering its packet loss from 100%. One of the three core routers was being fixed.
At or around 1:30AM on August 25th, 2007 all servers with in the Savvis datacenter started to appear online from different locations outside EastSolid's Network Operations Center.
Savvis officially declared all core routers up and running at 1:48AM on August 25th, 2007. Savvis is now monitoring these devices for any further issues.
[Resolved]
[Completed] Databank Network Maintenance
Posted on Friday April 27th, 2007 - 08:17 AM by staff member Adam
**Maintenance being performed by DataBank - our Upstream provider - tonight from midnight WST, thur 6:00am Saturday 04/28/07**
All:
Our upstream provider is performing maintenance early in the morning ours this morning; primarily between 2:00am and 6:00am. This could possibly affect all servers in the Databank network.
During these hours, you could experience latency, brief loss of connection, high ping times / traceroute times, and/or a period of general "lag' on the network at this facility.
Our upstream provider is now providing us with two more provider links, and two new 2GigE connections, in addition to our current connections. This should serve to improve our service on the entire network at this facility, and help to alleviate the periods of latency caused by not having these links previously.
If you experience any issues that begin during this window and either do not resolve, or become worse after the maintenance window has ended at 6:00AM CST (-6 GMT), please open a ticket.
Our team will notify me if we have any network-wide issues develop, and I will then update you all here.
Staff Updates:
-------------------------------------------------------------------------------------
Saturday, April 28th, 2007 1:00am
~~~
Still working on upgrading our network...
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Saturday, April 28th, 2007 3:00am
~~~
Everything should be up and running....
-------------------------------------------------------------------------------------
[Completed] Los Angeles NOC Loss of Power
Posted on Thursday April 12th, 2007 - 03:00 AM by staff member Adam
**Refresh the page to get an updated status**
Hey,
We are currently have lossed power in our Los Angeles, CA, USA office where we answer all tickets and support request. Due to the high winds (60+ miles per hour) 100,000 DWP (Department of Water and Power) customers (including us) have lost power.
We are currently running on UPS systems, however we will soon be powering downing.
More information can be found on DWP's website.
Staff Updates:
-------------------------------------------------------------------------------------
Friday, April 13th, 2007 10:15am
~~~
We are still with out power. We are currently working out of a cyber cafe and answering emergency tickets are this time...
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Friday, April 13th, 2007 3:42pm
~~~
We have gotten word that power has been restored in our area! Only 24+ hours with out power. We will be answering tickets threw out the night so please stand by and we will address them ASAP. Normal support funtions should be resolved with in 24 hours...
-------------------------------------------------------------------------------------
[Completed] Reboot Card Installation
Posted on Sunday March 18th, 2007 - 08:10 AM by staff member Adam
**Refresh the page to get an updated status**
Hey,
The planned outage for installing our reboot card only takes 20min - 60min to install. If you need to contact support, please click the "Contact Us" link at the top of the page.
Server:
ChallengerEstimate Downtime: 60min
----
Started at:
10:50amEnded at:
11:12am----
Total Downtime:
22min
-------------------------------------------------------------------------------------
Friday, March 18th, 2007 11:20am
~~~
All services should be up and running...
-------------------------------------------------------------------------------------
[Completed] Recompile PHP4 and PHP5
Posted on Friday February 16th, 2007 - 08:27 PM by staff member Adam
**Refresh the page to get an updated status**
Hey,
The planned outage for recompiling of Apache, PHP4 and PHP5 has started now. Out will last no longer then 4 hours and no sooner then 1 hour. We are hoping everything will go smooth and quick.
If you need to contact support, please click the "Contact Us" link at the top of the page.
~~~~~~~~~~~~~~~~
To Do List:
~~~~~~~~~~~~~~~~
Recompile Apache
Recompile PHP 4.4.4 --> PHP 4.4.5
Recompile PHP 5.2.0 --> PHP 5.2.1
Server:
ChallengerEstimate Downtime: 240min
----
Started at:
8:20pmEnded at:
9:00pm----
Total Downtime:
40min
-------------------------------------------------------------------------------------
Friday, February 16th, 2007 08:31pm
~~~
Apache and been taken off-line while we start the recompiling. We will update you once we are done or have any more information.
-------------------------------------------------------------------------------------
Friday, February 16th, 2007 09:00pm
~~~
We are done with recompile of Apache, PHP4 and PHP5
http://72.232.195.146/info.php
http://72.232.195.146/info.php5
Support request should now be handled back in our support desk.
[Completed] Challenger Apache - New Mods
Posted on Sunday February 04th, 2007 - 02:19 PM by staff member Adam
We are currently testing out some new mods with Apache. We have turn all sites on that server to a default white page while we run test with Apache and try some new mods.
Server:
ChallengerEstimate Downtime: 120min
----
Started at:
1:00pmEnded at:
3:00pm----
Total Downtime:
120min
------
Sunday February 4th, 2007 - 3:00 pm
Sites are up...
[Completed] Challenger Hard Drive DOA - [fsck]
Posted on Friday February 02nd, 2007 - 07:32 AM by staff member Adam
This server is currently having a problem with its hard drive. READY ONLY SYSTEM FILES. We are having our DC look into it and replacing the HD. Currently NO DATA HAS BEEN LOST!
Server:
ChallengerEstimate Downtime: ?min
----
Started at:
6:00amEnded at:
10:00am----
Total Downtime:
240 min
------
Friday February 2nd, 2007 - 8:20 AM
We are running fsck to clean up any errors with the hard dive. We will update you when its done...
------
Friday February 2nd, 2007 - 9:02 AM
Still running fsck....
------
Friday February 2nd, 2007 - 10:00 AM
Server up and runnng.
[Completed] Challenger Kenerl Panic
Posted on Thursday January 25th, 2007 - 07:13 AM by staff member Adam
This server is currently having a problem with its kernel and is unable to reboot. We currently have our datacenter team looking into this to get it up and running.
Server:
ChallengerEstimate Downtime: ?min
----
Started at:
7:00amEnded at:
7:20am----
Total Downtime:
20min
------
Thursday Janurary 25th, 2006 - 7:20 PM
Server up and running :)
[Completed] Challenger PHP Update
Posted on Friday December 01st, 2006 - 09:21 PM by staff member Adam
We are currently updating PHP from 4.4.4 --> 4.4.4 and PHP 5.1.6 --> 5.2.0.
Server:
ChallengerEstimate Downtime: 120min
----
Started at:
9:00pmEnded at:
10:40pm----
Total Downtime:
80min
------
Friday December 1st, 2006 - 9:20 PM
Starting to build PHP 4.4.4
------
Friday December 1st, 2006 - 9:40 PM
Finished building PHP 4.4.4
------
Friday December 1st, 2006 - 9:45PM
Starting to build PHP 5.2.0
------
Friday December 1st, 2006 - 10:40PM
Finished building PHP 5.2.0
[Completed] Server Restore
Posted on Wednesday November 22nd, 2006 - 05:28 PM by staff member Adam
Hey,
We are working on the server. No backups were lost. (Last night backups are safe and getting accounts up and running...
Staff Updates:
Wednesday November 22, 2006 - 7:00pm
Accounts are being restored. Should be up and running in the next 80 min or less..
-----
Wednesday November 22, 2006 - 7:50pm
We are currently 25% done restoring accounts. Only 75% more to go. 
-----
Wednesday November 22, 2006 - 8:23pm
We are currently 50% done restoring accounts...
-----
Wednesday November 22, 2006 - 9:07pm
We are currently 100% done restoring accounts...
If you have any problems, send us an email...
[Completedl] Challenger Down
Posted on Sunday November 19th, 2006 - 08:39 PM by staff member Adam
We are currently testing some software on challenger, which requires us to shut everything off but SSH. Please standby.
Server:
ChallengerEstimate Downtime: 20min
----
Started at:
8:30pmEnded at:
8:45pm----
Total Downtime:
15min
[Completed] Challenger Down
Posted on Monday November 06th, 2006 - 07:01 AM by staff member Adam
Looks like challenger was having some problems with Apache so we restarted it a few times and then noticed the cache size for the server was in the GB's so we did a nice restart however the server is still sleeping (aka did not come back online after the restart request)
We have contacted our datacenter for them to reboot the server and see what caused the problem...
Server:
ChallengerEstimate Downtime: 30min
----
Started at:
7:00amEnded at:
7:21am----
Total Downtime:
21min
-------
Datacenter is now looking into this...
--------
Server online!!!
[Completed] Apache Update
Posted on Sunday October 15th, 2006 - 01:47 PM by staff member Adam
Hey,
We are in the process of updating Apache. We found that patching a few bugs in the old version of Apache and Updating PHP did not do the trick. Services NOT effected:
Mail
FTP
SSH
Ruby on Rails
MySQL
Server:
ChallengerEstimate Downtime: 60min
----
Started at:
1:40pmEnded at:
2:25pm----
Total Downtime:
45min
Sunday October 15th, 2006 - 2:00 PM
Doing Apache and server configuration...
----
Sunday October 15th, 2006 - 2:10 PM
The main Apache configuration is done, we now to get the accounts up and working...
----
Sunday October 15th, 2006 - 3:16 PM
Apache and your website should be up and running. However some other options like Zend and Ioncube does not seem to be loading. We are looking into this...
----
Sunday October 15th, 2006 - 10:45 PM
Done...
[Completed] PHP Update
Posted on Sunday October 15th, 2006 - 12:48 AM by staff member Adam
--------------------------------------------------------------
All Accounts
[CustomerName]
[CustomerEmail]
Email Ticket ID:991342ZJD20
--------------------------------------------------------------
Dear [CustomerName],
We are planning a server outage for your domain [Domain]. This outage will effect PHP and Apache. FTP, MySQL, Mail and SSH will be up and running during this outage. This outage will also effect our support site, www.support.eastsolid.com. If you feel the need to contact support
during this plan outage, or simply want to find out minute by minute updates, please visit our Off Site Emergency Status Page.
Outage Agenda
- Recompile PHP4 and PHP5
- Update small Apache bugs
- Monitor server for the next 6 hours
Getting information when our site is down
As stated in the welcome email, we have a Off Site Emergency Status Page which is a backup server to see what's up with your server. We also offer Live Support, via AIM, Yahoo and MSN when our site is out, as well as email. For more information please visit the Off Site Emergency Status Page.
From,
The Late Night Support Staff/Upgrade Team (everyone)
Sunday October 15th, 2006 - 1:00 AM
We are planning and making sure on our ends and the servers for errors or any errors that might come up...
----
Sunday October 15th, 2006 - 3:50 AM
Apache being turned off...
----
Sunday October 15th, 2006 - 4:00 AM
Starting to update PHP4 and PHP5
----
Sunday October 15th, 2006 - 4:30 AM
PHP4 and PHP5 are now up todate...
