Finding a sweet spot for crowdsourcing: uTest outsources software testing

By Andy Oram
December 2, 2008 | Comments: 2

The promise of peer production seems to reach everywhere, but harnessing it is quite a trick. To start with, it's not easy to make it worthwhile for both the contributors and the beneficiaries of those contributions to join up. Achieving that, you then have to:

  • Manage relationships (often over long distances in different times zones)
  • Establish clear expectations
  • Provide feedback and reputation systems to keep people honest
  • Compensate the participants

And oh yes--making a little money along the way yourself would be nice in order to enable the whole venture, and could be the hardest trick of all.

uTest seems to have corraled all the necessary elements. Their business model can be described quite simply: uTest signs up freelance testers to work for participating software development firms, who in turn pay each tester for each bug found. (Some customers are on a subscription model, like traditional Software as a Service, but uTest still pays testers on a per-bug basis.)

And it's working: uTest opened its virtual doors as a private beta site in April 2008, formally launched in August 2008, and just got its second round of funding. They currently have over 11,700 testers from 148 countries and more than fifty customers, most of whom run test cycles every two or three weeks.

Testing is a well-understood activity, and testers at uTest follow standard practices. The model does offer a few intriguing enhancements: for instance, if you have access to several thousand testers, you can test an application such as a virtual trade show much more realistically than if you depend on scripts and virtual machines. But the interesting elements in uTest are the communication and reward systems, and that's what this article describes.

While some of the ideas in this blog may benefit people working on volunteer activities, it is directed mostly to companies trying to actually make living through organizing communities.

I met last week with CEO Doron Reuveni, and learned several interesting lessons from him that may help other organizations trying to apply crowdsourcing in their own domains. After running through the model, I'll briefly compare it to that used by another crowdsourcing firm in the software field, TopCoder.

Communication and virtual cooperation
Some interactions in uTest are highly structured and admit little ambiguity. For instance, a customer divides up the test cases and provides test scripts to testers. This is a good way for both new customers and new testers to start. uTest provides each new customer with a project manager to help them get the most out of the service.

But interactions quickly get more sophisticated. Customers start to loosen up the scripts or allow experienced testers to design the scripts. Although uTest concentrates on testing executable programs, it will offer testing on higher-level activities such as product design in Q1 of 2009.

Meanwhile, the testers form a community around each product, often coming back (with the blessing of the customer) on each test cycle and coordinating their efforts for better testing. Although rewards are individual and each tester strives to find the most bugs, incentives lie entirely on the side of cooperating rather than competing. These teams are always geographically distributed, with people from many different cultures in different time zones working on a common goal.

At an even a higher level, testers participate in forums to improve skills and study how to use uTest's system better. They also become uTest's best recruiters, bringing in their colleagues as new testers.

Communication is fundamentally text-based, through an internal messaging system that allows both interactive chat and the exchange of more structured communications. But testers, as we shall see, are pushing the envelope of these tools, and uTest plans to add more multimedia over time.

Innovation and peer production
A young company, uTest is always expanding its platform. Along the way it learns a lot from both its customers and its testers. Reuveni stressed to me, "we don't want to be the dominant partner in these relationships."

Because communication is so important, many testers find third-party tools such as slide-show and video editors that help show customers exactly where a system goes wrong. uTest tries to enhance its platform to allow testers to integrate such tools. Testers also want to share their uTest profiles in other environments, such as social networks.

Eventually, uTest will probably make parts of its platform open source so that testers and customers can enhance them and add their own features.

Granularity and choice
uTest offers a huge range of testing experience, with testers ranging from novice to expert and knowing a variety of languages and platforms. For each test cycle, a customer specifies how many testers it wants, at what skill levels. Customers can expand and contract their test bases just as cloud computing allows a customer to expand and contract the number of servers that run at any moment.

As a customer gets to know testers, it can request a specific tester for a specific function. Fundamentally, uTest is a market that dissolves and re-forms every week. If a customer doesn't like a tester, or vice versa, they just don't work together again.

uTest's flexible contractual system also makes it easy for testers to use it as a part-time secondary income source. In a typical test cycle, the customer's development team wraps up a development cycle on Friday, testing starts Saturday morning and runs all weekend, and the customers get a complete report of bugs found when they open their office on Monday morning. The timing is convenient for both sides.

Reuveni says that many testers participate not so much for the money as the chance to study new systems, work in domains outside of the ones they work in during the week, and build up reputations they can parlay into other opportunities.

Reputation and honesty
A customer pays only when a bug is found. To push back on Reuveni as skeptically as I could, I asked what would prevent a customer from denying the existence of a bug, but then quietly fixing it anyway.

The answer is simple: testers know when they're being cheated, and they share their opinions with each other. Any customer that rejected an obvious bug would quickly get a bad reputation and would have a hard time attracting testers for future test releases.

In actual operation, uTest finds that the opposite effect appears: customers and testers fall over themselves to make each other happy. Customers approached uTest, for instance, to tell them of a phenomenon uTest hadn't taken into account: the customer may decide that a report is not an actual bug, but is still worthwhile information and deserves compensation. In response to this request, uTest created its "valuable feedback" program which rewards a tester above and beyond any bugs found.

uTest does do some monitoring of its own. Anomalies in the customer/tester relationship (such as an unusually high number of rejected bug reports) trigger an investigation.

uTest's certification program also allows testers to exploit new opportunities and reach higher levels of responsibility. These certifications might even serve as a model for the whole testing community, whose current certification programs are poorly standardized.

What does all this add up to? A system rich with benefits for all concerned, that allows participation at whatever level each party finds comfortable, and that fosters cooperation, learning, and growth.

As I mentioned, it may be interesting to compare the uTest model to another unusual and successful company in the area of software outsourcing: TopCoder. Although TopCoder performs design and coding while uTest performs testing, both depend on a worldwide network of individual participants. Both have also formed supportive online communities.

Here are some aspects where they overlap and diverge, based on a conversation I recently had with one of TopCoder's leading project managers, Sean Campion, and with Jim McKeown, their Director of Communications:

Cooperation versus competition
Both firms find that participants are motivated to cooperate even though the setting involves an aspect of competition. This aspect is much stronger in TopCoder--at first glance it seems definitional--because the money awarded for each project goes only to the top two competitors. Yet TopCoder participants answer each other's questions about customer requirements on the forum for each project, and spend time in general forums and "practice rooms" to help novices learn the system. These are seen as good for everybody.
uTest offers very fine-grained rewards, which lowers barriers to entry and making it easy for novices to ramp up. TopCoder softens its winner-take-all model by offering a pool of money in "digital runs" to another tier consisting of the top five contributors to each project. Bug races (small, fast competitions to fix a single bug in a standard component) allow even new arrivals to pick up a little cash. So both uTest and TopCoder make it easy to be a casual contributor; you can be effective while making it a part-time avocation.

But for TopCoder, the chance to learn new skills and knowledge domains may be the strongest motivation to participate. Every submission that meets minimum standards gets reviewed by experts. In this community of competitors (to coin a somewhat oxymoronic but vividly realistic phrase) mere participation also builds skills. And participants can point to their TopCoder scores in job applications and other external activities.

Indeed, given the basic rules of the competition--that the applicant must complete the entire project, including tests and documentation, in order to compete--TopCoder probably could not be a success unless it offered such non-monetary rewards.

To avoid confusion, I should state that uTest does not run competitions like TopCoder: each tester on uTest gets a task and is given full responsibility for carrying it out.

Evaluation and quality control

uTest's quality control is up to the customer, who determines what deserves a reward. Testers implicitly rate customers by voting with their feet, and TopCoder participants do the same. In TopCoder, a project that is poorly defined or doesn't offer sufficient reward simply doesn't attract competitors.

Once a competition starts, however, TopCoder imposes rigorous quality controls. These start with the requirements specification (the responsibility of the customer, but checked by multiple TopCoder quality managers such as Campion); continue with stress, accuracy, and failure tests (developed by expert reviewers); and extend to unit testing (the responsibility of the individual competitors). The customer is provided with UI prototypes and can also check requirements and tests at any point during design and development.

Many of these companies' ideas will work in other peer production environments, and many seem to be unique to the companies' chosen fields. But anyone who wants to try peer production in his own industry can get some tips by studying how they found their sweet spots.

You might also be interested in:


Love the article ... crowd sourcing rules and is a extremely viable solution to many testing situations.
"The true undiscovered frontier of crowdsourcing will likely be the mobile platform."

... enter "Crowd Sourced Mobile Application Testing" Over 1,600 handsets on over 170 carriers in 75+ countries. We're especially handy for mobile developer networks ... google for our O2 Litmus press release.

Check out our site:

Thanks for readin'!

Stephen King

Really nice article.... doing research on software testing... This is really useful and informatic... I got good source about software testing in this site too. Any way thank u for this good article.

News Topics

Recommended for You

Got a Question?