Haves and Have Nots

I was talking to a friend about trends in applied microeconomics the other day. Right now, the "identification fetish", to quote Chris Blattman, is kind of in a steep wane. I said I'm seeing various other things taking the obsessions with two-stage least squares. I'm seeing regression discontinuity (Journal of Econometrics devoted an entire issue to it in early 2008) and random peer group assignments. Both of them are cousins to instrumental variables, because both are focused on finding some artificial, hopefully random, treatment assignment, from which we can estimate some kind of localized treatment effect by comparing the treated and the non-treated. It helps us get around problems in observational data like unobserved selection or simply omitted variable bias.

But, if you think about that Angus Deaton piece I linked to the other day, as well as Heckman's many public poops on this stuff, does it not seem like top people in the profession have begun to seriously think all of this stuff is stupid? I mean, when you've got Levitt on his blog defending himself from criticisms that he's "ruined" economics, it means Levitt's hearing so much of this, so often, that he's a bit stressed about it.

I was refereeing a paper for a top field journal the other month. It had an instrumental variable section that was meant to really be the main point of the paper. The author found, what he argued was, a random increase in treatments, from which he could estimate the average effect at around the localized treatment, across the states that had this experience. The paper had problems, but I voted for publication. The other two referees voted against it. The paper was turned down. I came away from it feeling like the tide was turning. Ten years ago, that would've been published, and not just in a field journal either. In the JPE or the AER. Levitt got away with far less than what is routinely rejected by top journals all the time.

So where to go? I told my friend I am noting a lot of manually collected data. For instance, Manisha Shah and several others went and collected data from prostitutes in South America on individual transactions. This data has gone into a JPE publication, and more recently, in the AEA May Papers & Proceedings (not the typical AER, but I think most of us would take a May AEA any day). The Stinebrickner's have been using Berea College's rule of random roommate assignments, followed up with their own manual collection of survey data from the students, to roll out a buttload of papers in AER, Journal of Public Economics, and other places. So maybe we should already be thinking this, too. Unlike 2SLS or other approaches, you aren't passive to the data collection. You can even collect the data to coincide with a natural experiment if you still want to utilize such a thing - Levitt and his coauthor do this in their paper on prostitution, using July 4th as a date to collect data on prostitutes, which is believed to increase demand for prostitution services. You can also just get back to real scientific work when you're collecting data. I mean, if your junior and an applied microeconomist, you can't tell me you've never really been disappointed by the obsession with identification, can you? How many times have I felt like such a sham when I come home and tell my wife or a colleague I have this incredible research idea where I'll use some age cut-off to identify the effect of Medicare on health outcomes (Card already did this by the way), or use Katrina to estimate the effect of local schools on student outcomes. We've all heard this - we're starting to care more about instruments than questions. I understand the reason, too, and unlike most, I don't think it's even such a bad thing. There's value to finding causal relationships, particularly if you're wanting to think about policy. If you cannot deal with that, then you can't really hope to have some policy relevance. But in the end, if it's all we do, I think we'll be really disappointed by who we become as social scientists. Not as a group; I mean just individually. I don't want to look back and think that all I did with my career was come up with some clever instruments. Seriously.

So my friend found the trend towards manual data collection troubling, interestingly. He said that if this continues (which he also has seen), then it'll only further separate the haves from the have-nots. The costs for a Harvard researcher to going to Uganda by having an RA spend some indefinite period of time there is low compared to me or most economists. I'd love to do that, but I can't. He's right. If you're in development work, you're going to be at a disadvantage. I guess part of the appeal of instrumental variables was, in other words, the pure egalitarian nature of them. Everyone had the NLS dataset on mature male workers. But, Card noticed a "College in the county" variable in this publicly available dataset, and so used it to instrument for college attendance, in order to estimate the returns to schooling. Sure, Card was at Berkeley, but the point is, had you or me been smart enough, we could've found the same thing. And that generation build their entire career off of that stuff - Levitt, Angrist, Card, Krueger, Hoxby. A lot of their famous works were IV works - breakthrough papers that dealt with some lingering identification problem in the empirical literature. A decade later, nearly all of those breakthrough papers clearly have problems. Hoxby's paper on streams turns out is kind of sensitive to what you actually call a river or a stream. Levitt's had numerous coding problems that once you corrected, went away (both in his abortion-crime paper, and in his earlier gubernatorial cycle and police hirings paper). Card's Miami boatlift paper, and even that college in the county paper, both are disputed constantly. Borjas at one point calls Card crazy for what he wrote in that Miami boatlift paper! I think Angrist's work on the Vietnam draft is the lone one left standing, but even it gets regularly attacked. And of course, the Angrist one that looked at variation in birthdates to look at high school completion was what led to the whole "weak instruments" literature. And so on.

It also seems like editors have a much different incentive structure than I once believed. I used to think it just went like this. If the paper is good, it gets accepted, otherwise it's rejected. There's other things going on in their minds that I do not understand, to be honest. Some of it has to do with wanting to minimize a later paper that overturns that paper, which means this weak instrument stuff is really important. It's not just about trying to the test statistics on your instrument up to 12, either. That's not enough, either. So where do you turn? I really do not know. Like I said, I've been seeing manually collected data becoming really common, and right now, that's where I'm turning. I've had research assistants collect data for the last four months, and they're still not done. I bought proprietary data that cost my department $16,000. I'm right now putting a survey together to hit the field in a month. I'm moving away from things like NLS and CPS and towards things which are untouched, in other words. I'm still sticking to my research interests, but I'm moving away from snazzy approaches. I just think, based on listening to people at the top schools and widely esteemed, that this identification fad is done. And you either immediately start to transition, or you're going to get run over, and come up for tenure in six years with nothing but dozens of rejection slips from journals.