It is time that legislators, regulators and procurement officials put an end to end-user license agreements (EULA) that prevent publication of comparative benchmarks.
In economic theory, a perfect market has perfect information. Information about product capabilities and comparisons is fundamental to a functioning market. But the size and complexity of modern software products makes it impossible for each purchaser, whether individual or organization, to thoroughly benchmark each product privately. Indeed, the ISO standard for Software Product Quality (IS9126) is very clear that there is an important species of quality ("quality-in-use") that is determined by deployment: the proof of the pudding.
The only way for the market to be well-informed is to be able to read and criticize each others' benchmarks and post-mortems.
But there are many common EULA's that forbid. This blog was prompted by a comment (on Kiril Grouchnikov's blog item on Sun's continued mismanagement of Java Trust is hard to build and easy to destroy) on a Sun EULA for JavaFX
177 (f) You may not publish or provide the results of any benchmark or
178 comparison tests run on Software to any third party without the prior
179 written consent of Sun.
Microsoft may be the leading perpetrator of this kind of EULA.
In the current drive for more openness, this should be a no-brainer. This kind of EULA should be banned as being fundamentally anti-competitive. And procurement officials should make its the presence of such provisions a disqualification for public tendering or licensing deals: a major demerit at the very least.
Obviously the argument against benchmarking in public is that the benchmark may get it wrong, be superceded by events, or be used by opponents to mislead opinion. Big deal, that is the market! Caveat emptor. The supposed case of the compiler benchmarks that were subverted when a vendor built in a special generator that recognized when the benchmark was being run is no reason against benchmarks. The market needs information. It needs to know coverage and performance against standards and benchmarks: it needs to know weak and strong points. The public is never served by snow jobs.
Public servants working in the area of competition policy should review whether the current laws or regulations do make these EULA provisions invalid in their jurisdiction, and, if they don't, enhance the laws or regulations appropriately.
Lawyers working creating EULAs should stop including them and pro-actively start removing them: the writing is on the wall in favour of increased openness—by 2009 it is unarguably a requirement of modern business with public entities and just a cost of doing business—, so why make trouble for your clients and serve their customers ill?
[Update: Article title changed from "Boycott and ban products with anti-benchmarking EULAs" since I wanted to focus on the EULAs.]