Mayer

ETHICS IN ECONOMICS

Honesty and Integrity in Academic Economics
Thomas Mayer

There may not be much outright plagiarism or cheating in economics research, argues this economist, but there are grounds to conclude that a bias exists in statistical research. Does this amount to dishonesty? The author tackles the issue.

T

is that people try to maximize their welfare. Since dishonesty can enhance the prospect of publication, the prime success indicator in academia, economists should ask themselves whether academic economists sometimes maximize utility by being dishonest in doing or presenting their research.
HE CORE ASSUMPTION OF MAINSTREAM ECONOMICS

I
Plagiarism, in its flagrant form of copying someone else’s work, seems rare; I can recall reading about only three confirmed cases of it in the almost sixty years I have spent as an economist. Both the risk of exposure and feelings of conscience provide plausible explanations for this scarcity. Soft plagiarism in the sense of making unacknowledged
THOMAS MAYER is professor emeritus of economics at University of California–Davis. A more detailed working-paper version of this article is available at www.econ.ucdavis.edu.
Challenge, vol. 52, no. 4, July/August 2009, pp. 16–24. © 2009 M.E. Sharpe, Inc. All rights reserved. ISSN 0577–5132 / 2009 $9.50 + 0.00. DOI: 10.2753/0577–5132520402

16

Challenge/July–August 2009

Honesty and Integrity in Academic Economics use of someone else’s ideas is probably much more common, both because it is considerably less likely to be detected (and the punishment, if the act is indeed detected, is apt to be much less severe) and because it is less heinous, so one’s conscience will protest less. Another temptation is to produce a publishable paper through outright cheating by making up data out of thin air or by lying about the results of analyzing legitimate data. The risk of being caught is not all that great. While many journals require authors to make their data available, this requirement is frequently ignored. And if subsequent to publication someone asks for the data, a cheater could always claim to have lost them, or if someone does obtain the data and is unable to duplicate the author’s results, this author could claim that his misreporting of the results is just due to carelessness. When working with small samples, a keypunch “error” or the arbitrary omission of a few outliers can work wonders on one’s results. No data on such cheating are available, but my guess is that it is rare, or at least unusual. If it were common, one would hear more complaints about the inability to reproduce a previous author’s results. Moreover, my general impression of my colleagues is that they would not stoop to such a practice.

II
However, when looking at the situation from another angle, there is evidence that suggests dishonesty, or at least a certain lack of integrity, on the part of academic economists: that is, the results their papers reach are uncannily consistent with the results their previous papers reached. If economists let the facts speak freely so that the chips fall where they may, shouldn’t the probability that the empirical results of a paper support hypothesis A be no greater for an economist whose previous papers supported A than for one whose previous papers supported the rival hypothesis B? Not necessarily. One possible explanation is that the results are driven by the econometric methods or types of data used, and that an author tends to use the same methods and data in all her work

Challenge/July–August 2009 17

Mayer on a certain theory. But even if that is so, this does not necessarily mean that economists are unbiased. The reason economist X uses one econometric method and economist Y another could be that X’s methods tend to give results that she favors and Y’s method the result he favors. Another possibility is that all of X’s empirical results are based on a certain premise that Y rejects, such as the slow adjustment of wages and prices to changes in demand, or even on a difference in world views—for example, the acceptance of what Thomas Sowell has called the “tragic vision.” That may well account for some of the consistency between an economist’s new and previous results, but it seems unlikely that it can account for most of it. This conclusion suggests that bias does play a significant role in disagreements among economists. Such a bias could be ideological or simply a reluctance to admit that one’s previous results are wrong, or, more generally, the result of what psychologists call “confirmation bias.” It could also be the result of loyalty to a particular school of thought, a loyalty probably forged in graduate school, or just an attitude of “my mind is made up, don’t bother me with the facts.” Does such a bias imply dishonesty or at least a lack of sufficient intellectual integrity? It depends on the definition of dishonesty and integrity. If these terms are defined broadly, it does, since economists claim at least implicitly that their results are unbiased.

III
How can one explain this coexistence between apparently honest economists and such bias? One explanation is that some of our accepted procedures allow economists so much leeway in the methods they choose that they can produce the results they want without feeling dishonest. Let us look at such procedures. One of them is to confuse statistical and economic significance, so that finding that x affects y at the 5 percent significance level seems like an important contribution, even when the size of the effect is trivial. How common is this? In The Cult of Statistical Significance, Stephen Ziliak and Deirdre McCloskey (2008) argue that it is very common.

18

Challenge/July–August 2009

Honesty and Integrity in Academic Economics Whether it is or not can be, and has been, debated, but even if it occurs only infrequently, that is too much. A second questionable procedure is the upside-down use of significance tests—that is, to imply that if you have shown that a certain claim, e.g., that x affects y, is not confirmed at the 5 percent significance level, you have thereby shown that x does not affect y, thus waving away the possibility that what is at fault is not the hypothesis but the small size of your sample. Perhaps we should think of significance tests as being just as much of the adequacy of sample size as of the adequacy of the hypothesis. Actually, it is a joint test of both. If a hypothesis is not confirmed at the 5 percent level, all we can conclude is that either the hypothesis is false or the sample is too small (or both). Hence, to say something about the hypothesis, we have to make a more or less subjective judgment about the adequacy of the sample. Unfortunately, that brings back the subjectivism that significance tests were supposed to avoid. However, when applied consistently, as seems common, and not only when it supports the author’s results, the use of upside-down significance tests does not on average bias the results toward those that the researcher wants to come up with because there should be approximately as many cases where it militates against the researchers’ hypothesis as cases in which it supports his hypothesis, and therefore cannot account for the frequency with which the results of new studies cohere with those of the author’s previous studies. A third, and probably the most important, procedure that generates bias is data mining (alias “fishing,” “data snooping,” and “peeking”). What this means is that if my initial regressions do not confirm my hypothesis, I can just make some changes, such as switching the regression equation from natural numbers to logs, changing the definitions of a variable—e.g., using M1 instead of M2—or adding or subtracting additional variables until I get the results I want. In a classic example, Edward Leamer has shown how in this way the estimated effect of capital punishment on the homicide rate can be made to vary from strongly negative to positive. It is tempting to conclude that this sort of fishing for desired re-

Challenge/July–August 2009 19

Mayer sults is dishonest and should be eliminated. But the problem lies in the words “this sort.” There is a type of data mining, or perhaps one should say a motive for data mining, that is not only legitimate but often necessary. This is because the theoretical terms of economic theory cannot be translated unequivocally into measurable variables. For example, does the theoretical term “money” correspond to M1, M2, or M3? Similarly, when a theory tells us that something occurs with a lag, should we lag the regressor one, four, or ten periods? How can we find out what to do? The only answer is to try various definitions, lags, etc., and then to select the ones that give the best fit. Sometimes there is a straightforward way to avoid the permissiveness of standard data-mining practice. If you have a large enough sample, break it into two parts, use one part to formulate the appropriate form of the hypothesis, and test that on the other part. But in macroeconomics we usually do not have sufficient data for that. And in microeconomics, where survey data with large samples are frequently available, this procedure, though feasible, is for some reason rarely used. That is hard to justify. If it is not possible to use separate samples for specif ying the hypothesis and for testing it, the next best thing is to let the reader know about all the variants that you have fitted and the results thus obtained, so that she can decide how much credence to give to the results. We economists, or at least macroeconomists, tell the world about the importance of transparency, but we do not practice it sufficiently.

IV
It is not surprising that the mores of economists provide such loopholes. Without them many if not most economic research projects would fail. One might then expect an economist at a research university to publish only perhaps one or two papers per decade. Try explaining that to a dean. I am not denying that our journals use tough criteria—I have had too many papers rejected to doubt that. But their criteria are ones that require extensive and difficult work, such

20

Challenge/July–August 2009

Honesty and Integrity in Academic Economics as the application of the latest econometric techniques and advanced math; yet with sufficient effort, that work can be done. Just what determines which errors are permitted, and which not, is hard to say, but certain conventional errors become established, perhaps in some cases by the historical accident that an outstanding economist made this particular error, or by referees’ believing that this particular error is unavoidable or not even noticing it. Suppose a paper with an erroneous procedure—e.g., confusing statistical with economic significance—does get into the literature. This tends to make another economist less leery of also using this procedure; after all, if others think that it is all right, perhaps it is, and even if you still believe that it is not, the protests of your conscience are likely to be less vehement if you can point to others who have done the same thing. Moreover, the more a certain questionable procedure is used, the less likely it is that a referee, who perhaps used it himself, will object to your using it, or that the reader will form a bad opinion of you. As Alexander Pope put it a long time ago:
Vice is a monster of such frightful mien As to be hated needs but to be seen Yet seen too oft, familiar with the face We first endure, then pity, then embrace.

Thus, as time proceeds, additional dubious procedures will come to be seen as acceptable. However, this does not mean that the standards of economics will inevitably decline, because as readers come across a certain mistake more often, there is a greater chance that someone will catch it, and it also becomes a more valuable target for a critical paper. There is therefore also a tendency for invalid procedures to be eliminated. For example, we are now much more careful, in trying to predict the effect of government policies, to take into account their effect on the expectation of firms and households, and the subsequent effect of these changes in expectations on the behavior of these entities, and much more cautious in attributing causation to mere correlation. Moreover, in recent years, economists have become more aware of the

Challenge/July–August 2009 21

Mayer danger of data mining, and many more papers report on fragility tests (that is, on tests to see whether the results still hold when one makes reasonable changes in the definitions of variables, sample periods, or the variables included, a sort of reverse data mining), although still to a lesser extent than seems appropriate. I suspect that recently, in the race between errors becoming acceptable and errors being exposed, the latter has won out. Economics is improving, albeit at a slow pace, even when measured not by the elegance of our techniques but by what we know about how the economy operates. But that does not mean that new errors do not from time to time gain a foothold.

V
Does the use of readily available loopholes imply that economists are dishonest? In one sense it does not. The typical economist is not consciously cheating and is using procedures that are sanctified by common practice and that she thinks are valid. But in another sense, we economists collectively are cheating, because we accept practices that we would admit are questionable if we were forced to confront this issue outright. Once one takes account of group-think, there is no contradiction between individuals behaving honestly and yet the combination of these individuals being less than honest and forthright.

VI
Is this problem worse in economics than in other fields? Physicists seem more willing than economists to admit error. That is hardly surprising, since their hypotheses can be confirmed or disconfirmed much more definitively than can economic hypotheses. Economics should be appraised by comparison not with the top of the pyramid but with less elevated fields, such as psychology. Ziliak and McCloskey’s (2008) discussion of the way psychologists use significance tests, as well as some discussion within psychology,

22

Challenge/July–August 2009

Honesty and Integrity in Academic Economics suggests that psychologists are just as ready to misuse significance tests as are economists. In biology Peter Lawrence (2003) has complained about the ignoring or hiding of results that do not fit the author’s hypothesis. And biometric and medical research has generated news reports about scandalous conflicts of interest, as well as complaints about an overemphasis on statistical significance, and data mining. As for philosophers, those lovers of knowledge and wisdom, an outstanding philosopher, Robert Nozick reports:
One form of philosophical activity feels like pushing and shoving things to fit into some fixed perimeter of specified shape. All those things are lying out there, and they must be fit in. You push and shove the material into the rigid area getting it into the boundary on one side, and it bulges out [i]n another. . . . So you push and shove and clip off corners from the things so that they’ll fit and you press in until finally almost everything sits unstably more or less in there; what doesn’t gets heaved far away so that it won’t be noticed. (Of course, it’s not all that crude. There is also the coaxing and cajoling. And the body English.) Quickly, you find an angle from which it looks like an exact fit and take a snapshot. . . . All that remains is to publish the photograph as a representation of exactly how things are, and to note how nothing fits properly into any other shape. (1974: xiii; emphasis in original)

VII
That certain fields are no better in this respect than economics does not imply that we economists should not try to improve the integrity of economics. But how? An obvious answer is to foster in our day-to-day activities a climate of greater humility, to admit that our methods are imperfect and our results less compelling than they seem. This would make us more willing to challenge prevailing practices. It should also make us less prone to group-think. Turning to more specific remedies, journals in recent years have cut back sharply on the critical comments on the papers they have published, and the resulting reduction in the probability that an error will be caught is undesirable. It would also help if journals would enforce more strictly the requirement that authors make their data

Challenge/July–August 2009 23

Mayer and computer programs available, and if referees would ask authors to do more tests for fragility, including tests using alternative computer programs. More checking of published results would also help to make outright fraud more risky, though its main benefit would probably be more to reduce carelessness than outright dishonesty. Econometrics courses could require students to check a published paper, and their results should be published. This effort could perhaps be underwritten by National Science Foundation grants. Data mining could be disciplined to some extent by requiring authors who use time series data to publish after, say, three years a brief note discussing whether their results still hold when one adds three more years’ data. Similarly, authors who use data that are subsequently revised, such as GDP data, could be asked to report whether the fit of their regressions improves—or deteriorates—when the revised data are used.

For Further Reading
Lawrence, Peter. 2003. “The Mismeasurement of Science.” Current Biology 17, August 7. Nozick, Robert. 1974. Anarchy, State and Utopia. New York: Basic Books. Ziliak, Stephen, and Deirdre McCloskey. 2008. The Cult of Statistical Significance. Ann Arbor: University of Michigan Press.

To order reprints, call 1-800-352-2210; outside the United States, call 717-632-3535.

24

Challenge/July–August 2009