As of early this morning external generator power has been restored to the data cetner.
6:30am - Effective power restored to the data center
7:30am - intial email restored to working order
8:00am - reresurgence.com back online
8:30am - Campus sites and Wordpress sites back online
10:23am - Message from Internap:
“At this time all cooling and power equipment servicing the SEF facility continues to operate normally from the interim power solution in place. There is an 8 hour refueling rotation to keep generators online and providing power to the facility. The latest update from the Fisher Building staff and Internap Field Operations staff shows no indications of problems since the restoration of services.
As previous notifications have outlined the facility is currently being powered by mobile generators while the facility power infrastructure is being examined further to determine what the correct course of repair should be. Fisher does not anticipate any further interruptions in power
at this time, and as previously noted the Fisher engineering staff and various vendor representatives will be on site at SEF throughout the day to closely monitor the current situation. Internap Facility engineers will also be monitoring the situation closely and are available to assist customers as needed. Please contact the Internap NOC to request Remote Hands services if necessary.”
Currently working to restore marshillchurch.org, inside.marshillchurch.org, relit.org, resound.org.
12:40pm -
Marshillchurch.org os now back online and the front end is completely fuctional.
There was an unexpected outage where our public sites and email systems are hosted. The outage began around 12:45am July 3rd. Currently the duration of the outage is unknown. The phone systems will all function normally, but email and public websites will be inaccessible. We’ll keep this site updated as things progress.
News:
http://www.seattlepi.com/local/6420ap_wa_fisher_plaza_fire.html?source=mypi
http://www.techcrunch.com/2009/07/03/authorizenet-goes-under-e-commerce-vendors-left-hanging/
Follow details with this Twitter search:
https://twitter.com/#search?q=%23fisherfire
A list of affected sites:
http://blog.kylemulka.com/2009/07/list-of-sites-affected-by-fisher-plaza-data-center-fire/
4:50pm-
Fisher Plaza Press Release
http://www.marketwire.com/press-release/Fisher-Communications-Inc-NASDAQ-FSCI-1012899.html
11:51pm -
More details from KOMO news
http://www.komonews.com/news/local/49873927.html
Current estimate from host is 5pm
11:35am -
The datacenter had a fire in the utility switching gear. Backup generators are under water.
10:40am -
AdHost now reports that the building engineering team says it hopes to have power restored by 1:30 pm Pacific/4:30 pm Eastern.
“The best information I am aware of is that there was a failure in the main/generator transfer switch which resulted in a fire. The sprinkler system activated. From speaking to the fire battalion chief, I am under the impression that Seattle Fire did use water on the fire as well, but I am unsure of this.
Given the failure location, generator power was not available, and cooling failed. UPS power to systems continued, and I can personally vouch that they held out for well over an hour. When we were able to access our equipment, ambient air temps were well over 100 degrees in the room our equipment is located in.
At least some, if not many circuits were affected. Several large
co-location providers and other datacenters are located in the facility, these facilities have no power.
As this was the main/generator switch, and it is now highly damaged, the circuits in the area are damaged, and the entire area is doused in water, a rapid restoration of power does not seem likely. Fisher Plaza’s phone numbers now result in fast-busy signals, so I have no recent update from them directly.”
10:00am - servers restarted
10:08am - servers back online
Some www.marshillchurch.org content pages were unavailable from 2:25pm to 2:35pm on Thurs. Apr 23. There was an issue while updating the site.
8:30pm - Major outage at our datacenter, being investigated. All sites down.
This outage is also effecting our exchange server, blackberry server, and voice mail server, meaning mars hill mail is also down as well as the main mars hill phone number.
11:00pm - Issue identified by host. Loss of SAN. Resolution in process.
11:30pm - Partial site restoration, system taken back down
12:00am - Most web system back up and functional. marshillchurch.org and subdomains
12:30am - Resurgence fully restored
12:45am - Exchange Server back onlin
1:30am - Phone System back online
10:07pm All updates are complete.
8:42pm Backups complete, starting updates.
8:30pm Backups of the server have begun.
11:24am - marshillchurchorg, and all subdomains may be experiencing intermittent issues with database connections at the moment
11:40am - All sites should be back up and running, after this Database outage.
Thursday 03/18/09
All FTP services at mhcfiles.com are down and unavailable. This includes ftp.mhcfiles.com and bandref.mhcfiles.com
Saturday 03/21/09
All FTP services for mhcfiles.com have been restored, but some data loss as occurred. A backup from around three weeks ago has been restored back to the server.
9:38 AM - Access to discussion topics is not working properly. We are working with The City for a resolution. Updates to follow.
1:42 PM - This issue has been resolved.
