ArtFire experienced a brief site interruption beginning at 9:30 AM Local Time (MST)   The interruption was followed by a cacheing problem that caused some 404 pages to be cached by our content delivery network, causing some users to see unscheduled maintenance or 404 errors for pages that are in fact live.

If you’re experiencing problems accessing ArtFire or your account restarting your web browser should fix the prolblem.

ArtFire experienced a service interruption beginning just before 12.00PM local time (Arizona).  Service was partially restored by 1:10PM and an issue affecting images was finally resolved just after 2:00PM.

The cause of this downtime was a connectivity problem between internal servers; specifically there was an issue preventing a file server from connecting to our client servers.  Other than the connectivity problem there was/is no danger to ArtFire’s files (we utilize redundant storage and backups to safeguard our users information and data).

Our development team is exploring potential upgrades to our file server infrastructure to prevent interruptions like this from occurring in the future.

7: 30 AM PDT Currently the ArtFire site servers are not responding properly to connection attempts.  We have contacted the technical team but do not have an initial diagnosis yet.   We will update as more information becomes available.

8:03 AM PDT – Remote log in not possible.  Tech team is on the way to the colocation facility.

8:20 AM PDT –  Site Service resotored.  We are investigating a potential power loss issue as remote access to all servers, hardware and systems was disabled.   For now, tech team remains on site.

One of our user generated content servers ( the servers that store the local copy of photos and images) crashed this morning.  The secondary fail over servers which mirror that content did not properly connect with our web servers.  Currently, our tech team is manually connecting them and restarting the original servers.

8:40 – Servers are restarted but not properly connecting with the web servers still.  We are investigating.

8:53 –  Out techs are unable to connect the servers remotely.  We have someone en route to the colocation facility to work on the problem.

Historically, this is a much lower traffic day for ArtFire, our team is in the office today working on optimization and database maintenance projects that are more complex.

We hope to minimize any downtime today, but there is the potential for a few minutes of site outage from time to time during this process.

At approximately 3 PM AZ time the ArtFire database servers became unresponsive.  Our CTO attempted to log in remotely to investigate the issue but was unable to access or remotely reboot the database servers.    He traveled to our co-location facility in Tucson and has been working on site since approximately 3:30 PM to determine the cause of the issue.  As part of the repair process he is duplicating data on a failed hard drive.  Due to the drive’s technical failure, this is taking longer than normal.  When we have more information we will update this blog.  As it is Sunday and most staff is out of the office,  the blog and our forums will be the only two channels updated today.

Update 5:27 PM AZ Time – The data recovery process was completed approximately 10 minutes ago, our CTO is troubleshooting the reboot of these servers.  There is an as yet unknown technical problem preventing the full restoration of the server stack.

Update 5:37 PM AZ Time – All data is secure.  However, the product table continues to “lock” ( this is something that normally happens only during full re-indexes or DB changes we make, not usually on its own).  Our CTO will continue to troubleshoot.  However it may require migrating one of the database servers to a back-up server to resolve. You may see  the site “up” intermittently, but if the table lock is not resolved it will not be stable.

Update: 6 PM AZ Time – The site has returned to normal operation.  The “locking” issues seems to be related to the referral link tracking we use to track in bound referral sources for seller items and shops in our internal stats structure.  Specifically the size of that table was quite large, and holding that data long term, seems to have caused resource slowing and the eventual lock.  We have the data backed-up but will be flushing all referrals prior to mid 2011 in the live database to alleviate the problem.

A full post-mortem and planning to avoid future repeats of the issue will take place on Monday with the technical team.

This week at ArtFire we are excited to announce the launch of our newest top level marketplace on ArtFire, the Commercial category. The new Commercial category joins the other top level marketplaces: Handmade, Fine Art, Media, Design, Vintage, and Craft Supplies, getting its own link in the marketplace box, a landing page, and it’s own category structure.  The Commercial category creates a space on ArtFire for items that are not made or produced by the shop owner and that do not meet the definition of for Vintage or Craft Supplies.

The Early Bird Special – Pre-Pay Today and Save!

To help jump-start our newest marketplace we’re offering a limited time discount to new and existing shop owners:  A full year of ArtFire for $100; that’s a discount of over 30% off the current month-to-month rate!  While the Early Bird Special was launched in tandem with the new Commercial Marketplace,  we are offering the discount to all new and existing members that would like to save by pre-paying for the next year.  Whether you’re a subscriber at the $9.95 or $11.95 rate, you can save money and hassle when you take part in our Early Bird Special available until October 31st or until 1000 members have taken the offer.  (To our $5.95 subscribers, unfortunately we are not able to offer a discounted annual billing option. Your $5.95 rate was and is the best rate possible.)

The launch of the ArtFire Commercial category also includes a new community policing mechanism to help locate, review, and take action on shops and products that do not meet the guidelines of the marketplace they are currently in.  The visible portion of this new mechanism is a “Report this Shop” link that appears at the bottom of a seller’s shop and can only be seen by other logged in ArtFire sellers. Logged out visitors and logged in buyers do not see a link at all.   The “Report this Shop” link can be used to report an entire shop as being mis-categorized.  The reporting prompt provides an area to briefly explain the report as well as spots to include supporting links if there are any.  ArtFire staff members will review reports in the  order they are received and take timely and decisive action to either leave a seller in the traditional marketplaces or transfer their shop and listings to commercial.

You can learn more about the early bird $100 pre-pay discount offer as well as the new ArtFire Commercial marketplace in this ArtFire forum announcement.

If you’d like to take the $100 offer and you’re already a member of ArtFire at the $9.95 or $11.95 rate you can authorize the payment and save on your next year as an ArtFire seller from your My Billing page.