ArtFire experienced a service interruption beginning just before 12.00PM local time (Arizona).  Service was partially restored by 1:10PM and an issue affecting images was finally resolved just after 2:00PM.

The cause of this downtime was a connectivity problem between internal servers; specifically there was an issue preventing a file server from connecting to our client servers.  Other than the connectivity problem there was/is no danger to ArtFire’s files (we utilize redundant storage and backups to safeguard our users information and data).

Our development team is exploring potential upgrades to our file server infrastructure to prevent interruptions like this from occurring in the future.

7: 30 AM PDT Currently the ArtFire site servers are not responding properly to connection attempts.  We have contacted the technical team but do not have an initial diagnosis yet.   We will update as more information becomes available.

8:03 AM PDT – Remote log in not possible.  Tech team is on the way to the colocation facility.

8:20 AM PDT –  Site Service resotored.  We are investigating a potential power loss issue as remote access to all servers, hardware and systems was disabled.   For now, tech team remains on site.

One of our user generated content servers ( the servers that store the local copy of photos and images) crashed this morning.  The secondary fail over servers which mirror that content did not properly connect with our web servers.  Currently, our tech team is manually connecting them and restarting the original servers.

8:40 – Servers are restarted but not properly connecting with the web servers still.  We are investigating.

8:53 –  Out techs are unable to connect the servers remotely.  We have someone en route to the colocation facility to work on the problem.

Historically, this is a much lower traffic day for ArtFire, our team is in the office today working on optimization and database maintenance projects that are more complex.

We hope to minimize any downtime today, but there is the potential for a few minutes of site outage from time to time during this process.

At approximately 3 PM AZ time the ArtFire database servers became unresponsive.  Our CTO attempted to log in remotely to investigate the issue but was unable to access or remotely reboot the database servers.    He traveled to our co-location facility in Tucson and has been working on site since approximately 3:30 PM to determine the cause of the issue.  As part of the repair process he is duplicating data on a failed hard drive.  Due to the drive’s technical failure, this is taking longer than normal.  When we have more information we will update this blog.  As it is Sunday and most staff is out of the office,  the blog and our forums will be the only two channels updated today.

Update 5:27 PM AZ Time – The data recovery process was completed approximately 10 minutes ago, our CTO is troubleshooting the reboot of these servers.  There is an as yet unknown technical problem preventing the full restoration of the server stack.

Update 5:37 PM AZ Time – All data is secure.  However, the product table continues to “lock” ( this is something that normally happens only during full re-indexes or DB changes we make, not usually on its own).  Our CTO will continue to troubleshoot.  However it may require migrating one of the database servers to a back-up server to resolve. You may see  the site “up” intermittently, but if the table lock is not resolved it will not be stable.

Update: 6 PM AZ Time – The site has returned to normal operation.  The “locking” issues seems to be related to the referral link tracking we use to track in bound referral sources for seller items and shops in our internal stats structure.  Specifically the size of that table was quite large, and holding that data long term, seems to have caused resource slowing and the eventual lock.  We have the data backed-up but will be flushing all referrals prior to mid 2011 in the live database to alleviate the problem.

A full post-mortem and planning to avoid future repeats of the issue will take place on Monday with the technical team.

This week at ArtFire we are excited to announce the launch of our newest top level marketplace on ArtFire, the Commercial category. The new Commercial category joins the other top level marketplaces: Handmade, Fine Art, Media, Design, Vintage, and Craft Supplies, getting its own link in the marketplace box, a landing page, and it’s own category structure.  The Commercial category creates a space on ArtFire for items that are not made or produced by the shop owner and that do not meet the definition of for Vintage or Craft Supplies.

The Early Bird Special – Pre-Pay Today and Save!

To help jump-start our newest marketplace we’re offering a limited time discount to new and existing shop owners:  A full year of ArtFire for $100; that’s a discount of over 30% off the current month-to-month rate!  While the Early Bird Special was launched in tandem with the new Commercial Marketplace,  we are offering the discount to all new and existing members that would like to save by pre-paying for the next year.  Whether you’re a subscriber at the $9.95 or $11.95 rate, you can save money and hassle when you take part in our Early Bird Special available until October 31st or until 1000 members have taken the offer.  (To our $5.95 subscribers, unfortunately we are not able to offer a discounted annual billing option. Your $5.95 rate was and is the best rate possible.)

The launch of the ArtFire Commercial category also includes a new community policing mechanism to help locate, review, and take action on shops and products that do not meet the guidelines of the marketplace they are currently in.  The visible portion of this new mechanism is a “Report this Shop” link that appears at the bottom of a seller’s shop and can only be seen by other logged in ArtFire sellers. Logged out visitors and logged in buyers do not see a link at all.   The “Report this Shop” link can be used to report an entire shop as being mis-categorized.  The reporting prompt provides an area to briefly explain the report as well as spots to include supporting links if there are any.  ArtFire staff members will review reports in the  order they are received and take timely and decisive action to either leave a seller in the traditional marketplaces or transfer their shop and listings to commercial.

You can learn more about the early bird $100 pre-pay discount offer as well as the new ArtFire Commercial marketplace in this ArtFire forum announcement.

If you’d like to take the $100 offer and you’re already a member of ArtFire at the $9.95 or $11.95 rate you can authorize the payment and save on your next year as an ArtFire seller from your My Billing page.

We are currently in an opt-in only testing phase with the new ArtFire cart and checkout.  If you’d like to opt in to try the new cart and checkout as a shopper please visit this post in the ArtFire forums with the links you’ll need to turn the test on and off. Remember that you’ll only be changing the cart and checkout for yourself as a shopper and it will not affect how other shoppers who haven’t opted into the test see your studio, cart, and checkout items.  If you’d like to see how a shopper might experience your studio you can log out while still remaining in the test group to cart your own items and see how things will look.

We’re looking forward to launching this new cart and checkout to the entire community soon!   Let us know what you think in the comments and give the new cart and checkout at try!

In less than a year the @ArtFire Twitter account has grown by over 10,000 followers. Why are so many people following @ArtFire? This amount of growth has been the result of a lot of work, along with the support of our followers. With @ArtFire now on a steady growth curve, we are turning our sights to growing the @ArtFireVintage Twitter account.

If you haven’t read Twitter, You’re Doing it Wrong, it includes some of the fundamental strategies to gaining followers, and keeping them. Over the last week or so we have been gathering sources of vintage-related content, researching other vintage-focused Twitter accounts, and reaching out to “tweeps” (followers on Twitter) who love vintage. Over in the vintage forum many ArtFire members had great suggestions for content, and we can always use more! Quality content is the key to attracting followers. For every 8-10 tweets on vintage fashion trends, or DIY projects repurposing vintage items, there will be one tweet highlighting ArtFire or vintage items on ArtFire. This hopefully encourages people to come for the content, but stay to check out the great vintage items and community on ArtFire.

Do you regularly visit any vintage blogs, websites or forums? We are looking for more sources for vintage news, fashion, photography, trivia, DIY projects and even sub-genres of vintage like steampunk and upcycling. Here are some examples of what’s been  popular in the last week:

Sequential Crush – A blog devoted to preserving the memory of romance comic books of the 60s and 70s.

Tutorial for Refreshing Vintage Wood – Tutorial from A Brick House, a blog following a couple “renovating a mid century cement brick house in the middle of retirement hell.”

Vintage Headband – How to make a vintage-style headband from Craft.

Couture Allure Vintage Fashion Blog – Vintage swimsuit inspiration from the 1970s.

Over the next few months we will be actively growing @ArtFireVintage, so if you have any vintage  content sources we should check out, let us know in the comments below! You can also add the hashtag #ArtFireVintage to any tweets you put out that you think would interest a wider vintage audience.

9:10 AM PDT 7/6/2011  – The servers are currently live, but our firewall is unable to properly route traffic to them.  Our hardware team is working on the issue and we will update as we know more.

9:17 AM PDT 7/6/2011 -  Firewall operating again, but is currently seeing our primary web servers as down.  Those servers appear to be caught in a disk check during reboot.  The hardware team is working to switch web code service to back up servers.

9:23 AM PDT 7/6/2011 – Firewall pointed to back up web code servers, however the latest code version is not live still*.  We are currently pushing our latest code build to those servers to update.

9:33 AM PDT 7/6/2011 – Servers are down again, there may be an intermittent hardware problem.  We have dispatched the team to the co-location facility to visually inspect systems and run diagnostics on site.

9:45 AM PDT 7/6/2011 – Code update is complete to secondary servers.  Hardware team is on site at co-location facility.  Reports backup generators are running, may have been a power issue.  Waiting for confirmation as to cause.

9:55 AM PDT 7/6/2011 – Back up servers will not properly mount the User Generated Content server** ( this is the server that handles image uploading and processing).  Without that server listing could not take place.  This means that the fastest way back to a live site is to complete the maintenance ( disk check)  on the primary web servers.

10:05 AM PDT 7/6/2011 – Disk Check is 80% complete.  We are not willing to interrupt that as it may be repairing errors on hard drives that could affect data if cut short.  As soon as the disk check is complete, we will attempt to relaunch.

10:25 AM PDT 7/6/2011- Disk Check is not completing.  Rebooting server fails without completed disk check.  Tech team is working to connect User Generated Content server to the back up web code servers.  Will update when complete.

10:40 AM PDT 7/6/2011 – UGC server appears to not mount due to operating system differences on back up servers.  This means we have to pull updates to the Operating System.  Currently, we are hitting 404 errors on repositories for OS updates. ***

10:52 AM PDT 7/6/2011 – Team is split into parallel tasks to try to return the site to service.  One portion is continuing to pursue getting the UGC server to mount to the existing back up servers.  Another part of the team is building a new code sever from scratch with a fresh OS and code build.

11:04 AM PDT  7/6/2011 – Partially restored service, still waiting on some image pointing and myartfire access.

11:42 AM PDT 7/6/2011 – Back down for a few minutes to restart PHP ( this is needed to restore full service).

11:53 AM PDT 7/6/2011 – Normal site function restored, reviewing site for any lingering conflicts.

Running tally of process improvement key take-aways from event.

* Review protocol for maintaining updated code base on back up web code servers

** Investigate issues with UGC server not mounting to back-up web code servers

*** Review schedule for pulling OS updates on back-up servers when not in use

A Growing Site has Different Needs

As we have grown in size and a great number more products have been listed in the ArtFire marketplace, our relevancy search algorithm has been slowly becoming outdated.   Originally, when we launched the site we had fewer items and wanted to build a search that would return anything that was remotely relevant.  Obviously, shoppers who got no results would be less likely to continue shopping and we wanted to keep shoppers shopping.

However, with 1.5 million active items, the chances of zero results being returned have greatly diminished.  The same algorithm that served us well in the early days creates more noise than we want in a marketplace that now has plenty of relevant listings for shopper queries.   That less relevant items have become noise which could now frustrate potential shoppers.

The new algorithm uses a different combination of matching parameters to better match shoppers’ queries while allowing  some room for difference between how a seller describes an item and how the shopper searches.  However, the query and the item now have to show a much more significant relevancy match in order for the item to be returned in the results.  This reduces the number of  non-relevant items returned that matched perhaps one partial word, but did not have a significant change at being what the shopper wanted.

Handling Spammers

Over time and with trial and error,  sellers learn how to best  create listings to take advantage factors in the internal search. Some go so far as to  attempt to game the search system.   Recently, we have seen some aggressive tag-stuffing in an attempt to get more search exposure.   As a result, our new change devalues tags even further and has some additional anti-gaming penalties to combat this practice.   We are still working on developing a fair process for duplicate content penalties.  Unfortunately, we have not been able to devise a system that will reduce duplicate listings, without impacting sellers who make a series of very similar products.

Making ArtFire SEO match Google SEO

We strive to align our internal search with Google best practices so that our sellers, in writing good item titles and descriptions for our internal search, are also simultaneously creating the best possible content for organic Google searches.  This means titles, descriptions and categories are the main drivers for the new search algorithm. These are the same drivers that  when coupled with our automated ArtFire SEO program are the best practices for getting in front of shoppers searching major search engines.

Of course, we still have marketplace-specific factors that apply in our search.  Where some other marketplaces apply a heavy recency weight to drive re-listing fees, we use our no cost check-in button to create a reward for active custodial sellers and give less exposure to sellers who are not monitoring their shop frequently.  The search penalty process starts at 7 days and gets progressively worse beginning at 14 days from the last time the member checked in.

As a reward to our Pro members for supporting the site, we also apply a +1 to Pro accounts.  This really serves as a tie-breaker ensuring that if both an item from a Basic studio and a Pro studio have an exact relevancy score match, the Pro studio is returned first.

Best Practices for Sellers

Just being a Pro seller won’t overcome poor titles, descriptions and incorrect categories though.  A search engine is only as good as the content it searches.  If item pages are not listed correctly, they won’t be returned to shoppers nearly as often.

Here are some tips  to maximize your item exposure for the new search:

1) Do keyword research for your titles via a site like The Google Keywords Tool

2) Use keywords and phrases that best fit your product and are frequently searched as your title

3) Include secondary keyword search phrases in your descriptions

4) Properly categorize your items

5) Use the first 10 tags for those same keywords, plus your studio name and city or location

6) Check- in at least once every 7 days by pushing the check-in button on your MyArtFire panel.

7) Don’t “stuff” your tags with hundreds or thousands of words