At approximately 2:25pm local time we lost connection with the severs at our co-location while working on a file system upgrade.  Our techs have been dispatched and are working on the issue.

Update 11:02 AM – We’ve rebooted the problematic server and service should be back to normal.  Thank you for your patience.

At approximately 10:05 AM Mountain Time we began experiencing a problem with a file server that’s causing an interruption of service on the ArtFire website.  Techs have been dispatched to investigate and resolve the problem, we’ll keep you updated.

At approximately 2:20PM local time ArtFire was taken offline to facilitate a necessary repair.  We expect the  duration of downtime to last no more than 30 min.

At approximately 11am local time the front end servers for ArtFire became unresponsive.  We’ve dispatched our tech team to our co-location facility to investigate and resolve the problem and will keep you updated on our progress.

ArtFire experienced a brief site interruption beginning at 9:30 AM Local Time (MST)   The interruption was followed by a cacheing problem that caused some 404 pages to be cached by our content delivery network, causing some users to see unscheduled maintenance or 404 errors for pages that are in fact live.

If you’re experiencing problems accessing ArtFire or your account restarting your web browser should fix the prolblem.

ArtFire experienced a service interruption beginning just before 12.00PM local time (Arizona).  Service was partially restored by 1:10PM and an issue affecting images was finally resolved just after 2:00PM.

The cause of this downtime was a connectivity problem between internal servers; specifically there was an issue preventing a file server from connecting to our client servers.  Other than the connectivity problem there was/is no danger to ArtFire’s files (we utilize redundant storage and backups to safeguard our users information and data).

Our development team is exploring potential upgrades to our file server infrastructure to prevent interruptions like this from occurring in the future.

7: 30 AM PDT Currently the ArtFire site servers are not responding properly to connection attempts.  We have contacted the technical team but do not have an initial diagnosis yet.   We will update as more information becomes available.

8:03 AM PDT – Remote log in not possible.  Tech team is on the way to the colocation facility.

8:20 AM PDT –  Site Service resotored.  We are investigating a potential power loss issue as remote access to all servers, hardware and systems was disabled.   For now, tech team remains on site.

One of our user generated content servers ( the servers that store the local copy of photos and images) crashed this morning.  The secondary fail over servers which mirror that content did not properly connect with our web servers.  Currently, our tech team is manually connecting them and restarting the original servers.

8:40 – Servers are restarted but not properly connecting with the web servers still.  We are investigating.

8:53 –  Out techs are unable to connect the servers remotely.  We have someone en route to the colocation facility to work on the problem.

Historically, this is a much lower traffic day for ArtFire, our team is in the office today working on optimization and database maintenance projects that are more complex.

We hope to minimize any downtime today, but there is the potential for a few minutes of site outage from time to time during this process.

At approximately 3 PM AZ time the ArtFire database servers became unresponsive.  Our CTO attempted to log in remotely to investigate the issue but was unable to access or remotely reboot the database servers.    He traveled to our co-location facility in Tucson and has been working on site since approximately 3:30 PM to determine the cause of the issue.  As part of the repair process he is duplicating data on a failed hard drive.  Due to the drive’s technical failure, this is taking longer than normal.  When we have more information we will update this blog.  As it is Sunday and most staff is out of the office,  the blog and our forums will be the only two channels updated today.

Update 5:27 PM AZ Time – The data recovery process was completed approximately 10 minutes ago, our CTO is troubleshooting the reboot of these servers.  There is an as yet unknown technical problem preventing the full restoration of the server stack.

Update 5:37 PM AZ Time – All data is secure.  However, the product table continues to “lock” ( this is something that normally happens only during full re-indexes or DB changes we make, not usually on its own).  Our CTO will continue to troubleshoot.  However it may require migrating one of the database servers to a back-up server to resolve. You may see  the site “up” intermittently, but if the table lock is not resolved it will not be stable.

Update: 6 PM AZ Time – The site has returned to normal operation.  The “locking” issues seems to be related to the referral link tracking we use to track in bound referral sources for seller items and shops in our internal stats structure.  Specifically the size of that table was quite large, and holding that data long term, seems to have caused resource slowing and the eventual lock.  We have the data backed-up but will be flushing all referrals prior to mid 2011 in the live database to alleviate the problem.

A full post-mortem and planning to avoid future repeats of the issue will take place on Monday with the technical team.