MHC Outage

Blog & theresurgence.com outage

Fri, 30 Oct 2009 02:20:06 -0400

We will be doing some maintenance on all the blog sites and theresurgence.com tonight October 29th starting at 11:30pm PST. The outage may last to 1:00am PST October 30th.

Unexpected Financial Edge Outage

Mon, 05 Oct 2009 14:34:00 -0400

Issue: Financial Edges gives some users an error when logging in

Sunday 4 Ocober

9am - Financial Edge allows some users to connect while other users are given an error upon login

9pm - Server is restarted

Monday 5 October

9am - issue is retested with several users and errors persist

Wednesday 7 October

12pm - issue resolved. all services back to normal

Unexpected Ballard Public Datastore Outage

Mon, 10 Aug 2009 20:26:00 -0400

Monday 8/10

5:22pm - server restarted

5:29pm - server online. service restored.

5:40pm - server offline for second restart.

5:47pm - server online, but shared folders are unavailable.

8:00pm - 11:00pm worked to restore service to the storage device. The storage device has a failed disk, and the storage devices interface is unavailable to access the backups we have been doing.

Tuesday - 8/11

9:00am - All Staff Outage notification set out addressing the issues.

12:00pm - Clone of backup volume has started.

Wednesday - 8/12

Failed Harddrives have been sent off for recovery.

Unexpected Finance Server Outage

Wed, 05 Aug 2009 13:58:00 -0400

10:00am - The Finance Server has stop responding. We are working to restore services as quick as possile.

6:29pm - Restoring data from backups - this has taken the majority of the day.

8:50pm - Server has been restore.

Unexpected 2x Outage

Sun, 02 Aug 2009 14:50:37 -0400

9:45am - 2x determined to be down
11:20am - 2x server restarted
11:25am - 2x server fully operational

Scheduled 2X Outage

Thu, 16 Jul 2009 21:49:00 -0400

6:49PM - Starting update process.

7:15PM - Restarting Server

7:40PM - Server is back up and upgrade process is complete.

Scheduled Email Outage

Fri, 10 Jul 2009 01:01:00 -0400

Today at 9:45pm updates are being run to the server. The total outage should last 30 minutes.

10:18pm - Server is being restarted.

10:22pm - Server is restarted and all services are restarted.

Scheduled outage for marshillchurch.org

Thu, 09 Jul 2009 20:30:00 -0400

Marshillchurch.org will be doing maintenance from 8:00pm to 9:00pm PST. Tonight (July 9th). This is expected to only impact http://www.marshillchurch.org and feeds.marshillchurch.org.

The Resurgence and Blog Outage

Sun, 05 Jul 2009 10:57:00 -0400

4:00pm -
All sites should once again be functionl All of the sites neded to be restored from a 4am Thursday backup.

12:00pm -
Database repairs are being run on theresurgence.com, perioding errors may result on the site.

10:30am -
Still suffering data corruption issues on some parts of theresurgence.com and blog sites.

8:00am -
theresurgence.com is now back available, we are still working through issues with the blog and camps sites.

6:00am -
The Resurgence, the Mars Hill Blog and Campus sites are experiencing db error. Being investigated.

Systems operational

Sat, 04 Jul 2009 16:13:36 -0400

All systems impacted by the fire at the data center are operational again. Email is operational, TheResurgence.com, blog.marshillchurch.org and the campus blogs were operational around 9:00am. www.marshillchurch.org and relit.org were back by 12:30. There is still some cleanup to be done (possible scheduled outages).

Site Restoration in Progress

Sat, 04 Jul 2009 12:14:00 -0400

As of early this morning external generator power has been restored to the data cetner.

6:30am - Effective power restored to the data center

7:30am - intial email restored to working order

8:00am - reresurgence.com back online

8:30am - Campus sites and Wordpress sites back online

10:23am - Message from Internap:

“At this time all cooling and power equipment servicing the SEF facility continues to operate normally from the interim power solution in place. There is an 8 hour refueling rotation to keep generators online and providing power to the facility. The latest update from the Fisher Building staff and Internap Field Operations staff shows no indications of problems since the restoration of services.

As previous notifications have outlined the facility is currently being powered by mobile generators while the facility power infrastructure is being examined further to determine what the correct course of repair should be. Fisher does not anticipate any further interruptions in power

at this time, and as previously noted the Fisher engineering staff and various vendor representatives will be on site at SEF throughout the day to closely monitor the current situation. Internap Facility engineers will also be monitoring the situation closely and are available to assist customers as needed. Please contact the Internap NOC to request Remote Hands services if necessary.”

Currently working to restore marshillchurch.org, inside.marshillchurch.org, relit.org, resound.org.

12:40pm -
Marshillchurch.org os now back online and the front end is completely fuctional.

Unexpected outage of just about everything

Fri, 03 Jul 2009 04:32:00 -0400

There was an unexpected outage where our public sites and email systems are hosted. The outage began around 12:45am July 3rd. Currently the duration of the outage is unknown. The phone systems will all function normally, but email and public websites will be inaccessible. We’ll keep this site updated as things progress.

News:
http://www.seattlepi.com/local/6420ap_wa_fisher_plaza_fire.html?source=mypi

http://www.techcrunch.com/2009/07/03/authorizenet-goes-under-e-commerce-vendors-left-hanging/

Follow details with this Twitter search:
https://twitter.com/#search?q=%23fisherfire

A list of affected sites:
http://blog.kylemulka.com/2009/07/list-of-sites-affected-by-fisher-plaza-data-center-fire/

4:50pm-
Fisher Plaza Press Release
http://www.marketwire.com/press-release/Fisher-Communications-Inc-NASDAQ-FSCI-1012899.html

11:51pm -
More details from KOMO news
http://www.komonews.com/news/local/49873927.html
Current estimate from host is 5pm

11:35am -
The datacenter had a fire in the utility switching gear. Backup generators are under water.

10:40am -
AdHost now reports that the building engineering team says it hopes to have power restored by 1:30 pm Pacific/4:30 pm Eastern.

“The best information I am aware of is that there was a failure in the main/generator transfer switch which resulted in a fire. The sprinkler system activated. From speaking to the fire battalion chief, I am under the impression that Seattle Fire did use water on the fire as well, but I am unsure of this.

Given the failure location, generator power was not available, and cooling failed. UPS power to systems continued, and I can personally vouch that they held out for well over an hour. When we were able to access our equipment, ambient air temps were well over 100 degrees in the room our equipment is located in.

At least some, if not many circuits were affected. Several large
co-location providers and other datacenters are located in the facility, these facilities have no power.

As this was the main/generator switch, and it is now highly damaged, the circuits in the area are damaged, and the entire area is doused in water, a rapid restoration of power does not seem likely. Fisher Plaza’s phone numbers now result in fast-busy signals, so I have no recent update from them directly.”

Restarting 2x and Financial Database Servers

Tue, 30 Jun 2009 12:58:00 -0400

10:00am - servers restarted

10:08am - servers back online

Unexpected Site Outage - RESOLVED

Sun, 17 May 2009 14:33:00 -0400

11:45am: Resolved.

11:30am: Sites are currently not responding to http requests. Issue being investigated.

http://www.marshillchurch.org/
http://theresurgence.com/
http://campus.marshillchurch.org/
http://blog.marshillchurch.org/

www.marshillchurch.org content pages

Thu, 23 Apr 2009 17:36:13 -0400

Some www.marshillchurch.org content pages were unavailable from 2:25pm to 2:35pm on Thurs. Apr 23. There was an issue while updating the site.

Major Unexpected Systems Outage (restored)

Thu, 09 Apr 2009 23:58:00 -0400

8:30pm - Major outage at our datacenter, being investigated. All sites down.

This outage is also effecting our exchange server, blackberry server, and voice mail server, meaning mars hill mail is also down as well as the main mars hill phone number.

11:00pm - Issue identified by host. Loss of SAN. Resolution in process.

11:30pm - Partial site restoration, system taken back down

12:00am - Most web system back up and functional. marshillchurch.org and subdomains

12:30am - Resurgence fully restored

12:45am - Exchange Server back onlin

1:30am - Phone System back online

Financial Server Planned Outage

Thu, 02 Apr 2009 23:31:00 -0400

10:07pm All updates are complete.

8:42pm Backups complete, starting updates.

8:30pm Backups of the server have begun.

Website Connection Issues

Thu, 19 Mar 2009 14:24:00 -0400

11:24am - marshillchurchorg, and all subdomains may be experiencing intermittent issues with database connections at the moment

11:40am - All sites should be back up and running, after this Database outage.

Unplanned Outage - FTP Services on mhcfiles.com

Thu, 19 Mar 2009 13:12:00 -0400

Thursday 03/18/09

All FTP services at mhcfiles.com are down and unavailable. This includes ftp.mhcfiles.com and bandref.mhcfiles.com

Saturday 03/21/09

All FTP services for mhcfiles.com have been restored, but some data loss as occurred. A backup from around three weeks ago has been restored back to the server.

Unexpected The City outage

Tue, 17 Mar 2009 12:30:00 -0400

9:38 AM - Access to discussion topics is not working properly. We are working with The City for a resolution. Updates to follow.

1:42 PM - This issue has been resolved.