Blog & theresurgence.com outage
We will be doing some maintenance on all the blog sites and theresurgence.com tonight October 29th starting at 11:30pm PST. The outage may last to 1:00am PST October 30th.
MHC Outage
Unexpected Financial Edge Outage
Issue: Financial Edges gives some users an error when logging in
Sunday 4 Ocober
9am - Financial Edge allows some users to connect while other users are given an error upon login
9pm - Server is restarted
Monday 5 October
9am - issue is retested with several users and errors persist
Wednesday 7 October
12pm - issue resolved. all services back to normal
Unexpected Ballard Public Datastore Outage
Monday 8/10
5:22pm - server restarted
5:29pm - server online. service restored.
5:40pm - server offline for second restart.
5:47pm - server online, but shared folders are unavailable.
8:00pm - 11:00pm worked to restore service to the storage device. The storage device has a failed disk, and the storage devices interface is unavailable to access the backups we have been doing.
Tuesday - 8/11
9:00am - All Staff Outage notification set out addressing the issues.
12:00pm - Clone of backup volume has started.
Wednesday - 8/12
Failed Harddrives have been sent off for recovery.
Unexpected Finance Server Outage
10:00am - The Finance Server has stop responding. We are working to restore services as quick as possile.
6:29pm - Restoring data from backups - this has taken the majority of the day.
8:50pm - Server has been restore.
Unexpected 2x Outage
9:45am - 2x determined to be down
11:20am - 2x server restarted
11:25am - 2x server fully operational
Scheduled 2X Outage
6:49PM - Starting update process.
7:15PM - Restarting Server
7:40PM - Server is back up and upgrade process is complete.
Scheduled Email Outage
Today at 9:45pm updates are being run to the server. The total outage should last 30 minutes.
10:18pm - Server is being restarted.
10:22pm - Server is restarted and all services are restarted.
Scheduled outage for marshillchurch.org
Marshillchurch.org will be doing maintenance from 8:00pm to 9:00pm PST. Tonight (July 9th). This is expected to only impact http://www.marshillchurch.org and feeds.marshillchurch.org.
The Resurgence and Blog Outage
4:00pm -
All sites should once again be functionl All of the sites neded to be restored from a 4am Thursday backup.
12:00pm -
Database repairs are being run on theresurgence.com, perioding errors may result on the site.
10:30am -
Still suffering data corruption issues on some parts of theresurgence.com and blog sites.
8:00am -
theresurgence.com is now back available, we are still working through issues with the blog and camps sites.
6:00am -
The Resurgence, the Mars Hill Blog and Campus sites are experiencing db error. Being investigated.
Systems operational
All systems impacted by the fire at the data center are operational again. Email is operational, TheResurgence.com, blog.marshillchurch.org and the campus blogs were operational around 9:00am. www.marshillchurch.org and relit.org were back by 12:30. There is still some cleanup to be done (possible scheduled outages).
