There’s a thought-provoking article for you ecommerce folks out there over at Ecommerce Times called “Stop Experimenting on Your Customers.” The purpose of the article is to argue against A/B field testing on your web site and to instead intercept customers while they’re on your web site and ask if they will participate in a customer experience test.
The argument the author of the article makes is that A/B field testing will reveal what test variable is better than the other, but not why. Other downsides to A/B field testing that the author provides is that sales can decline from a poor-performing page and that testing without customers’ knowledge can damage your trust for your brand.
With this proposed participatory research method, you, the web site operator, can find out why one test variable works better than another. Customers that know they are participating in a test will readily tell you what their goals for the visit are and what they are experiencing.
Frankly, my opinion is that both methods have their place, but I much prefer traditional A/B field testing. For starters, revenue is not sacrificed because in a well-designed test, the test sample (the B version) is a small percentage of site traffic. Also, in a well-designed test, there is only one variable on the page (the A version is the control), so you can actually get at why one design is preferred to another.
Personally, I’ve been involved in many A/B field tests over my career, and I’ve seen great results and learnings. I’ve used them for e-mail marketing communications, site homepages and landing pages. In fact, Google makes it very easy to set up A/B tests on your Google Adwords.
I’ve also used a testing technique similar to participatory customer experience testing. We contracted Usability Sciences to deploy their WebIQ product, where a customer is asked a host of questions when entering the site and then when they complete certain tasks on the site. Only about 10% or less of people asked to participate actually participated, and the results were not statistically significant due to the low participation rate.
We all know that offers and promotions change so quickly in today’s ecommerce world that a method that produces quick results is needed. As for me, I’ll stick to the traditional A/B field tests for ecommerce sites. In many cases, the only cost of the test is the time it takes to develop a new page and analyze the results.
share this post on your social networks: