Introduction: Evidence-Based Policy-Making
Last year, Cardus partnered with economists Morley Gunderson and Ting-ting Zhang to produce a report titled “Up, Up, and Away: The Impact of Restrictive Tendering on Municipal Contracting in Ontario.” Our paper compared the difference in bidding prices on construction projects in municipalities affected by an obscure piece of Ontario labour law that inadvertently gives a small number of contractors a virtual monopoly on public infrastructure projects, with those that operate according to the legally mandated fair, competitive, and transparent bidding guidelines.
The findings of that paper suggested that municipalities under restrictive bidding regimes experienced increases of roughly 100 percent in the gaps between the winning bid and a variety of other measured bids (next highest, mean, and high). We noted that municipalities operating under the constraints of these restrictions were likely experiencing significant upward pressure on construction prices.
We noted the strength of our results, and that they suggest that restricting tendering has negative effects on the policy goals of municipal bidding, but we also acknowledged the limitations of our methodology and our data.
“Up, Up, and Away” used a variety of null hypothesis significance testing (NHST), which, in a nutshell, is a statistical method “used to decide between two inverse claims: either ‘an effect’ that posits a relationship between, say, a treatment and an outcome (typically the favoured hypothesis) or ‘no effect’ (defined as the null hypothesis).”1
In this paper, we follow the advice of statisticians Blakely B. McShane, David Gal, Andrew Gelman, Christian Robert, and Jennifer Tacket, who call for the use of P values (which we used in “Up, Up, and Away”) to be “considered as just one among many pieces of evidence”2 assisting policy-makers in their decisions. Their advice is primarily aimed at academic journals, but applies just as much to policy-makers when making policy decisions based on empirical work. These scholars suggest that a set of accompanying considerations that should act as types of evidentiary counsellors to those making decisions, including “prior knowledge . . . real world costs and benefits, and other factors.”3
Cardus has examined various factors related to restrictive tendering from a variety of angles in the set of papers that form the Cardus Construction Competitiveness Monitor. The rationale behind this paper stems from our intent to provide further observations, using data that will show the effects of closed tendering on a specific local market.