"One of the most influential studies justifying austerity policies in recent years has come in for an extensive new critique. And one of the big problems found? An Excel coding error.
The paper in question is Carmen Reinhart and Kenneth Rogoff’s famous 2010 study ’Growth in a Time of Debt,’ which found that economic growth severely suffers when a country’s debt level reaches 90 percent of GDP. That 90 percent figure has been widely cited in the past few years as a key reason why countries must trim their deficits.
But a new critique (pdf) by Thomas Herndon, Michael Ash and Robert Pollin finds that this result may need serious revision. For one, the economists argue that Reinhart and Rogoff excluded three episodes of high-debt, high-growth nations — Canada, New Zealand, and Australia in the late 1940s. Second, they argue, Reinhart and Rogoff made some debatable assumptions about weighting different episodes.
Now, those are debatable methodological critiques. But there’s also a third problem, as Mike Konczal details at length here, Reinhart and Rogoff appear to have made a troubling error with one of their Excel spreadsheet formulas. By typing AVERAGE(L30:L44) at one point instead of AVERAGE(L30:L49), they left out Denmark, Canada, Belgium, Austria, and Australia:
This error wasn’t caught earlier because Reinhart and Rogoff hadn’t released their underlying data to the public. They only shared their spreadsheet with the Herndon, Ash and Pollin after the latter three tried to replicate the initial results and failed."
Mark Thoma puts this in perspective, How Much Unemployment Was Caused by Reinhart and Rogoff's Arithmetic Mistake? "What she asserted is that pain now -- austerity -- can avoid even more pain down the road in the form of lower economic growth. Here's the problem. She is right that austerity causes pain in the short-run. But according to a review of her work with Rogoff discussed below, the lower growth from debt levels above 90 percent that austerity is supposed to avoid turns out, it appears, to be largely the result of errors in the research. In fact, there is no substantial growth penalty from high debt levels, and hence not much gain from short-run austerity."
Paul Krugman sums up the issues with the data, Holy Coding Error, Batman.
"According to the review paper, R-R mysteriously excluded data on some high-debt countries with decent growth immediately after World War II, which would have greatly weakened their result; they used an eccentric weighting scheme in which a single year of bad growth in one high-debt country counts as much as multiple years of good growth in another high-debt country; and they dropped a whole bunch of additional data through a simple coding error."
Jared Bernstein adds a nice graph putting the issue in context:
Krugman comments on Reinhart-Rogoff's response.
2 comments:
As a common user of Excel for modeling I am both sympathetic and horrified at this simple mistake. You should check your results in proportion to the importance of their use. I have also done calculations by hand or in a different newly coded spreadsheet to check results to avoid just this thing.
Maybe these researchers didn't know their results were going to be used to ruin the world. I suspect that the unusual result was more interesting so there was less motivation to check it. I think the opposite, the more surprising or unusual a results the more evidence (and confidence in that evidence) you need to back it up.
I think you're being far too kind to them. I don't think they viewed their results as unusual. Remember too, the excel mistake was just one of three pointed out. The others were deliberate choices in their model.
Post a Comment