If you plan to pay for software, you often can’t find published comparative evaluations that have serious benchmarks on performance, accuracy, and so on. Why is that? The answer is a censorship regime that you’ve probably never heard of.
The DeWitt Clause is a common end-user license agreement provision for proprietary software that prevents anyone (such as researchers and scientists) from publishing information about their products (like benchmarks) that name the software unless its supplier approves it. Real benchmarks typically show the weaknesses of systems, so unsurprisingly, suppliers often don’t approve the publication of thorough research. In fact, many researchers won’t even start this kind of analysis, because they know they won’t be able to publish the results.
The clause was originally created to squelch database research being performed by Dr. David DeWitt. Larry Ellison, co-founder of Oracle, was displeased with a benchmark study done by David DeWitt in 1982, which showed that Oracle's system had poor performance. Instead of simply fixing the problems, Oracle (at Larry Ellison's behest) established the original DeWitt clause to censor any benchmark results that the vendor had not sanctioned. It had the intended effect; Oracle has a long reputation of having a huge legal department, so most people avoid publishing benchmarks about software products with a DeWitt clause. Over time these DeWitt clauses have become widespread in the software industry, making it nearly impossible to get unbiased published benchmarks about a variety of software products.
“The Devil’s in the DeWitt Clause” by Brian Moran (SQL Server Pro, Apr 2, 2003) discusses some of the arguments that vendors give to justify the DeWitt clause. As he summarizes later in “DeWitt Clauses: Serving the Wrong Master”, they are:
In the end, though, Moran states that “I don’t believe that DeWitt clauses serve customers’ best interest”. Indeed, I will go further and say that they directly opppose society’s interests. If vendors continue to be able to legally censor information, then society has no way to get or debate the information. The problem isn’t deception; we already have laws against deception. It’s true that evaluations can be misunderstood, but the solution there is to let other people try to repeat the findings and publish their own results. And let’s be honest: the DeWitt clause has nothing to do with preventing publication of misleading or grossly inaccurate results. The clause is named after efforts to squelch Dr. DeWitt’s research, and no one argues that Dr. DeWitt was clueless or deceptive. Indeed, Dr. David DeWitt is the winner of many awards specifically in the area of databases (the area he was benchmarking), including a pair of very prestigious awards: the ACM Software System Award and the IEEE Emanuel R. Piore Award. These DeWitt clauses are purely government-enabled censorship - and they are government-enabled censorship, because companies are using government courts to go after people who dare to publish truth.
You can learn much more from the 2016 report “Vendor Truth Serum” by Dr. Gregory Klass and Dr. Eric Burger (Georgetown University, Software and Security Research Center, 22 September 2016). They show that this inability to publish analysis of software is significantly hampering the security of our software. To make software secure, you really need to use tools. Which tools? Well, that’s a problem. They note that, “With many tools to choose from, a real issue is deciding which tools a given developer needs to use to ensure satisfactory test coverage over their software artifact. What is needed is a way for developers to know which tools provide what coverage, so they can make informed choices and accomplish satisfactory testing in minimal time at minimal expense. Unfortunately, there are common industry contractual practices which inhibits making such knowledge generally available.”
I can even point to a specific example of this problem. US NIST's Static Analysis Tool Exposition (SATE) produced SATE V report (NIST SP 800-326) in October 2018; it lists the names of the tools evaluated, but when you want to read how well the tools performed in different categories, all you learn about is "Tool A", "Tool B", "Tool C", and so on. In short, this report is useless for developers who want to select a good tool, and it's also ineffective for encouraging competition, because there's no useful information on how well specific tools
A serious problem is that these clauses have a corrosive effect within the areas they are used. Once one supplier adds a DeWitt clause, the others can feel that they are at a disadvantage without one. After all, when a supplier uses a Dewitt clause, then their product cannot be rigorously critiqued in public using repeatable benchmarks, and the other suppliers can be critiqued. I am actually sympathetic to the suppliers who add DeWitt clauses because their competitors do so... but that kind of decision should never be necessary.
Open source software (OSS) licenses don’t have DeWitt clauses, but again, this can mislead potential customers. The problem is that this means that the OSS can be legally critiqued, but not their proprietary competitors. There is one exception: Published benchmarks that name a product with DeWitt clauses can often be published if there are no negative statements, because then the supplier is likely to approve it. That is meaningless, because real in-depth analysis is rather unlikely to report nothing but good results. Suppliers using DeWitt clauses can even pay people to create carefully biased research that makes their products look good compared to all products without DeWitt clauses, as this is legal, and no one will be allowed to publish anything that could refute this biased research. Unsurprisingly, this situation can mislead people into thinking that any products without DeWitt clauses (including OSS) are worse, even when those products are far better. After all, only the products without DeWitt clauses (like OSS) can be legally criticized. In short, because of DeWitt clauses it's not legal to publish the truth. This perverse result only happens because DeWitt clauses have been allowed to persist.
There is a bogus legal theory that people can just choose to not get involved with these contracts and still publish truthful results. This legal theory ignores reality. If you’re going to publish a benchmark of a particular important product, you cannot avoid those contracts. The supplier of a product is typically the only legal source for that product. Also, because suppliers are pressured to add a DeWitt clause once their competitors use one, it often becomes impractical to compare common products across an entire field.
There is a US case that you might think would prevent this. That is the US case Bose Corp. v. Consumers Union of United States, Inc., 466 U.S. 485 (1984), which was a "product disparagement case ultimately decided by the Supreme Court of the United States. The Court held, on a 6-3 vote, that 'actual malice' was necessary in product disparagement cases raising First Amendment issues, as set out by the case of New York Times Co. v. Sullivan (1963)." That case does not help; DeWitt clauses are based on contract law and copyright licensing requirements, not on disparagement laws. Also, the Bose equipment was a physical device, and no one had to agree to a contract just to be allowed to use it. Software is different. Simply using software requires a license, and so software suppliers can add all sorts of clauses that are dangerous to society because nothing stops those suppliers.
There is a precedence for making such clauses illegal. In December 2016 U.S. president Obama signed the Consumer Review Fairness Act of 2016 (H.R. 5111), which had earlier passed both houses of Congress unanimously. The bill countered a dangerous trend: “businesses inserting clauses into their form contracts that attempt to limit their customers’ ability to criticize products and services online.” As “Vendor Truth Serum” notes, this and similar statutes aim "at consumer contracts only. They do not, or would not, prohibit the enforcement of a DeWitt clause. They do, however, suggest a model for what legislative action on DeWitt clauses might look like." I would like the statute to be interpreted broadly enough to also strike down DeWitt clauses, but my current guess is that this law would not prohibit enforcement of a DeWitt clause in many or all cases.
Please don’t tell me that DeWitt clauses are acceptable because “government should stay out contracts”. That’s rediculous; DeWitt clauses unnecessarily bring the government into situations where it doesn’t belong to enforce these unconscionable clauses. The government should not be in the business of censoring free speech and preventing free markets, and that’s what these clauses do. The current situation inhibits free markets, because it’s not possible for customers to learn from others about the products they are considering. Contract law presumes that all parties freely entered into an agreement, but current law forbids any alternatives, so there is no way to freely enter into an agreement.
I believe these DeWitt clauses should be struck down on first amendment grounds. The first amendment is really only about limiting what the government can prevent, but in this case contract law is tricking the government into implementing censorship on the behalf of suppliers. Free speech is about much more than the first amendment anyway. But that argument seems unlikely to succeed, even though I think it should. It could be argued that such clauses are unconscionable; if so, I would love to see that enshrined in case law. But while I would love to either result, I think these options are unlikely.
So instead, I’d like to see a law, similar to the Consumer Review Fairness Act, that makes the various forms of the DeWitt clause illegal. If you produce good software, you should not be afraid to have it benchmarked. If you produce terrible software, you should be working to make it better, not working to muzzle the truth. I believe society is best served by letting people independently evaluate products and publish their results; it’s time to let that happen.
Feel free to see my home page at https://dwheeler.com. You may also want to look at my paper Why OSS/FS? Look at the Numbers! and my book on how to develop secure programs. DeWitt Clause, or Can You Benchmark %DATABASE% and Get Away With It (Cube blog, updated 2022-06-02) also discussed the DeWitt clause.
(C) Copyright 2017- David A. Wheeler. Released under Creative Commons Attribution-ShareAlike version 3.0 or later (CC-BY-SA-3.0+).