Open Comparability: against anti-benchmarking EULAs

By Rick Jelliffe
November 16, 2008 | Comments: 4

It is time that legislators, regulators and procurement officials put an end to end-user license agreements (EULA) that prevent publication of comparative benchmarks.

In economic theory, a perfect market has perfect information. Information about product capabilities and comparisons is fundamental to a functioning market. But the size and complexity of modern software products makes it impossible for each purchaser, whether individual or organization, to thoroughly benchmark each product privately. Indeed, the ISO standard for Software Product Quality (IS9126) is very clear that there is an important species of quality ("quality-in-use") that is determined by deployment: the proof of the pudding.

The only way for the market to be well-informed is to be able to read and criticize each others' benchmarks and post-mortems.

But there are many common EULA's that forbid. This blog was prompted by a comment (on Kiril Grouchnikov's blog item on Sun's continued mismanagement of Java Trust is hard to build and easy to destroy) on a Sun EULA for JavaFX

177 (f) You may not publish or provide the results of any benchmark or
178 comparison tests run on Software to any third party without the prior
179 written consent of Sun.

Microsoft may be the leading perpetrator of this kind of EULA.

In the current drive for more openness, this should be a no-brainer. This kind of EULA should be banned as being fundamentally anti-competitive. And procurement officials should make its the presence of such provisions a disqualification for public tendering or licensing deals: a major demerit at the very least.

Obviously the argument against benchmarking in public is that the benchmark may get it wrong, be superceded by events, or be used by opponents to mislead opinion. Big deal, that is the market! Caveat emptor. The supposed case of the compiler benchmarks that were subverted when a vendor built in a special generator that recognized when the benchmark was being run is no reason against benchmarks. The market needs information. It needs to know coverage and performance against standards and benchmarks: it needs to know weak and strong points. The public is never served by snow jobs.

Public servants working in the area of competition policy should review whether the current laws or regulations do make these EULA provisions invalid in their jurisdiction, and, if they don't, enhance the laws or regulations appropriately.

Lawyers working creating EULAs should stop including them and pro-actively start removing them: the writing is on the wall in favour of increased openness—by 2009 it is unarguably a requirement of modern business with public entities and just a cost of doing business—, so why make trouble for your clients and serve their customers ill?

[Update: Article title changed from "Boycott and ban products with anti-benchmarking EULAs" since I wanted to focus on the EULAs.]

You might also be interested in:


Ban? Who do you see banning and in what context?

When I hear "ban" my First Amendment hackles get raised.

Andy: Not being an American, I had to look up what your First Amendment was. It is to do with protecting various freedoms, such as the freedom of speech.

So I don't see your point. EULA's that require someone to give up their freedom of speech prevent the free information that the market needs. If a product is crappy, we should be able to report it: of course, responsibly. If you value freedom of speech, then say no to freedom-curtailing EULAs!

If you think the lawyers are having fun now, just remove the EULA's and watch the suits move in. EULA's are a necessary evil that prevent skunk works projects from taking over the benchmark game.

The problem isn't with the content of the EULAs, but with the fact that the courts allow them to be treated as contracts even when they're available to the user only after the sale. In any area of business except software, allowing the seller to disclose terms after the sale would be absurd. When there's no contract in the normal sense of the word, it's no wonder the sellers think they can impose any capricious terms they want.

News Topics

Recommended for You

Got a Question?