Replicability Vs Reanalysis in Macroeconomics April 17, 2013Posted by larry in economics, Logic & Theory of Theory Testing, Statistics.
A recent furor has arisen about an analysis by Reinhart and Rogoff and a critique by Hernden et al. The discussion centers around data analysis and whether a piece of data analysis constitutes a reanalysis or is a replication of the original study. My response is to this issue. The relevant papers are (The first two are not links):
Reinhaart&Rogoff-GrowthInTimeOfDebt-AmericanEconRev-May2010 [arises out of their book];
http://images.politico.com/global/2013/04/17/reinhart_rogoff_response_to_herndon_ash_and_pollen_april_17.html [Reinhart and Rogoff provide the wrong year for this response, dating it as 2012 when it is obviously 17 April 2013.].
There is a logical problem with replicability in macroeconomics. Let me contrast for this purpose economics with ecology. Ecological studies can be divided into roughly two types, experimental and field studies. With experimental studies in most disciplines, with the exception of advanced physics which I will come to later, it is clear what must be done in order to replicate the study. Experimental ecological studies generally satisfy these requirements.
But with field studies, it is not always so clear. Say you are studying a particular insect with respect to a set of attributes in a particular environment. To replicate this study, you need to find another environment that is as similar to the original environment as possible and at the same time of year with the temperature being similar, &c. You can use the same environment the following year for replicability purposes, but this has its own down sides. Strictly speaking, you will not be able to truly replicate the original study, but you may be able to get as close as the theories that are being tested need you to be. And that is what counts.
In experimental physics, you will not generally find true replications. This is because they don’t usually need them. What is sometimes referred to as a replication is actually an attempt to improve on the original experiment, usually in terms of some measure of precision. Exceptions occur when the results appear to be “off the wall”, as was felt to be the case by the physics community at large re the original cold fusion experiments.
What you have in both empirical ecological and physical studies are samples of the relevant populations with all the statistical analytical implications that that brings with it. In economics, however, you are not always presented with data samples. Sometimes you are presented with what I shall refer to as the entire data universe. For instance, say you want to know what banks have in reserve. Instead of a sample of banks qua their reserves, you may be presented with the reserves (many of which may be estimates) of every single bank. This is not a sample but the entire population of bank reserves. This procedure precludes certain standard statistical techniques being used. But as is usually the case in economics, since no statistics are employed, this difference generally makes no difference.
As for the critique of the Reinhart and Rogoff study, I think what we have here is not a true replication, but a reanalysis of more or less the same data. A reanalysis would be undertaken if you thought that either the data were poor or badly organized or that there was an error made in the original analysis. A replication, as opposed to a reanalysis, would involve gathering “new” data of the same kind and subjecting this “new” data to the same or an improved version of the original analysis.
It could be argued, and has been by some, that replications are impossible in disciplines like economics as too many factors change from one temporal interval to another, thereby making it impossible to replicate the same or similar conditions. But since replications (and reanalyses) are tests of relevant theories, if the theories in question are any good, they should be able to “tell” the researcher whether such a replication is possible or fruitful or not. On the other hand, there should exist, in principle, no theoretical obstacles to a reanalysis of the original data set. All such tests take place within a given theoretical context, including those where the theories under test are virtually completely contradictory. This latter circumstance renders the test environment more difficult to specify but not thereby impossible.
With respect to the presentation of data, whether of the entire population or of a sample thereof, it used to be relatively common practice, for instance in the thirties, to include with the data an error estimate, often plus or minus some percentage. This no longer seems to be the case. Statistics has moved on from that period, but confidence intervals or their equivalent were common then and are now. Yet they do not seem to find their way into a good number of economic analyses, whether presented in tabular or in graphical form. And this is as true of the Reinhart and Rogoff studies as many others, although it is less applicable to the figures in the Hernden et al. study. Additional statistical issues related to the nature of the data themselves are more relevant in this case.