This article originally appeared in Food in Canada and is republished with the permission of the publisher.

Our regulatory system is widely misunderstood by academics and the public alike. Last month we saw that the primary problem of regulatory burden is not the volume of regulations, but bad enforcement and the lack of responsiveness to changing science or to a changing social, political or economic environment. This month we look at another myth.

Myth number 2: Quantitative Risk Benefit Analysis has significantly improved regulation making.

In the past two decades economists have grown Cost Benefit Analysis (CBA) into a whole new industry. Canada formally adopted the requirement for CBA for major regulations in 1978 through the introduction of our Regulatory Impact Analysis Statement (RIAS) system. From the outset this was as much about clarity and transparency as it was about CBA because prior to the RIAS process there was no requirement for general pre-publication or for explanation or consultation. CBA has become increasingly more rigorous, with departments now hiring economists to submit detailed RIAS documents with all major regulatory packages.

While Canadian officials are more modest in their assessment of the role of CBA, stressing that it is only a guide to better decision making, economists in many other jurisdictions display no such caution. Cass Sunstein, the American academic who was Obama’s regulatory guru from 2009 to 2012, describes CBA as nothing less than extraordinary in its ability to fundamentally improve regulation making. He states categorically that he and his staff of 45 produced the “stunning achievement” (his words) of saving $150 billion simply by forcing agencies and departments to apply CBA more rigorously in their regulation making.

There are many problems with the process of CBA and with its many claims for savings. The most obvious is that while economists seem to have no problem monetizing risks and benefits by simply making a number of assumptions, in the real world many of these calculations lack a common sense credibility. Sunstein asserts as common knowledge, for example, that the statistical value of a life is $9 million though he admits that this number can vary by 1,000 per cent; presumably such a variation is merely a technical detail.

The problem of measuring risks and benefits is especially difficult in science-based regulation because so much science is uncertain. Indeed, the use of the term “risk” is misleading as it disguises the uncertainty underlying the policy options. Surely managing risk must at a minimum involve some notion of probability of harm, yet much of the time when I was a regulator we were just doing our best to manage deep uncertainty. For example, I remember well when risk assessors provided me with a number of complex mathematical equations with Greek letters while admitting that none of them were actually helpful in choosing regulatory options on how to respond to the threat of mad cow disease.

Almost all nutrition science is fraught with uncertainty and changing assumptions. The RIAS to support making it a crime to label a food as “low carb” was premised on the health benefit of significantly reducing fat in our diet, an assumption now proving to be based on unreliable science. How do you measure the cost and benefit of lowering the level of salt in our diet when the nutrition experts are so divided on the science?

In my experience the CBA was used primarily to justify a decision that was the right thing to do for other reasons. The quantitative CBA provided good cover but it often undermined timely response, a cost of RIAS that deserves more attention.

In many respects, CBA is part of that bigger fallacy that you can only manage what you can measure, ignoring that in the public sector you usually can’t count what really counts. It encourages managing what you can measure, not what is most important.

Good regulation making requires clear thinking and analysis weighing political, economic, environmental and social factors that can rarely be measured with enough certainty to be useful for decision making. In spite of what its practitioners claim, in my experience quantitative CBA has done little to improve the regulatory craft.