By all the evidence we have to date, I think we have to conclude that most social psychology findings are false. That’s what the data is telling us. A failure to replicate is not decisive evidence that a finding is false, but in most cases the replication attempts are higher quality studies than the originals, with larger and often more representative samples. There is no reason to grant greater epistemic standing to an original study over a replication attempt, and there are many reasons to do the opposite.
A recent paper (PDF) by Rohrer, Pashler, and Harris illustrates why we might be overvaluing original research. They attempted to replicate the findings reported by Caruso, Vohs, Baxter, and Waytz (2013). Caruso et al. claimed that momentarily exposing people to background images of money on a computer screen before an experiment made them endorse free market systems and “social inequality” more than people who were not exposed to the money image.
Rohrer, Pashler, and Harris re-ran four of the five studies from the Caruso paper, using much larger samples. They even ran one of the studies three times. They found none of the effects reported by Caruso et al. Participants exposed to the money pictures didn’t endorse free markets any more or less than participants who never saw the money.
In discussing their resounding failure to replicate with the Caruso team, they discovered a shocking fact: the Caruso team had conducted nine studies, not five, but they only reported the five that confirmed their desired hypotheses. In their paper, they failed to disclose the other four studies that didn’t go their way. They also failed to disclose two outcome variables that were meant to test their hypotheses, but which showed no effects. So out of eleven tests, five supported their hypotheses and six did not — they only reported the five. Readers would have no idea that there were six other tests that showed no results. (For details, see the Rohrer et al. paper.)
That kind of malpractice needs to be purged from the field immediately if we want to be taken seriously as a science. We absolutely cannot do that. Failure to disclose null results undermines the validity of the statistical methods we use to report significant results. (For more on how this works, see Daniel Lakens’ work.) If this malpractice is widespread in the field, there would be little reason for anyone to believe social psychology findings, irrespective of any other issues — it would be irrational and mathematically incompetent to believe published findings in that case.
Another factor that could explain why most findings don’t replicate is that we’re using invalid samples — most commonly, college kids from one university in one country (and only those taking Intro to Psychology, adding additional skew.) This was never a justifiable practice, and it’s strange that it’s persisted so long. College kids from a single campus are artificially homogenous on many variables that could interact with the variables researchers are studying. This creates a synthetic low-variability and low-noise context for our analyses, making us more likely to find an “effect” that isn’t real in humans as such. We can’t credibly make claims about human nature from such
unrepresentative and skewed samples. This is obvious to everyone outside of social psychology, and it has a serious impact on our credibility as a field, so I hope we can move forward quickly here. Research based entirely on college kids should be valued much less than debunking false findings — in that case, I agree wholeheartedly with the Union professor.
Not exactly new, this stuff, but striking. There have been discussions of how a sample size that's too small is already worthless for stasticial and scientific purposes.
After all ...
The question of whether deceiving test subjects is acceptable for the sake of verifying a hypothesis is its own philosophical and ethical question.
At least when a televangelist quotes a Bible verse out of context you can go look that up. If social scientists don't disclose null test results and use deception to get results they present as a finding then there's a sense in which, these days, the snake oil preacher may be llying to you but at least you've got a better chance of guessing that's the game.
Work in the social sciences is certainly valuable. Not arguing against the socials ciences at all. But when enough questions are raised about the statistical viability, the ethical norms, and the lack of replicatable results it might be fair to ask whether social science can be taken in diretions that don't land so readily into these failures of both ethics and science.
Jonathan Haidt has said and written plenty about how one of the big problems in the social sciences in the United States is how WEIRD it is (western, educated, industrialized, rich and democratic, which can in turn be simplified into rich white egalitarian liberals with a few variations). Even if the sample size at a college, often one college, is high, by its very nature asking a bunch of kids who can go to college what they think about stuff is going to be something other than a representative sample of anyone who hasn't been to college or even made it through high school. There could be an elite echo chamber effect going on in all that.
Now why could this matter? Well, let's just give a local example. Anyone remember how people theorized that the appeal of Mark Driscoll was his being a macho Calvinist? He didn't always have that thing going because in the earliest years of Mars Hill he wasn't even really all that Calvinist. The pitch he had, if you go back and look at how he promoted himself to Mother Jones, was appealing to generational estrangement. Dads failed us by not being there so we gotta band together and follow Jesus. Driscoll was targeting young men who he felt needed ambition and purpose in their lives. This was not necessarily explicitly about red state politics. Driscoll was making an appeal to give guys a purpose and that purpose was going to be, as it turns out, Mars Hill. As Roy Baumeister has put it, men are distinctive for being interchangeable and disposable within a cultural apparatus. Driscoll's spiel was to make a case that if "you", young man, join our movement, you will not be disposable but will play a vital role in a movement to make a better future.
I ought to know, I bought it. :(
One of the big failures of progressives and secularists when discussing Driscoll has been an inability to take his ideas seriously. There are some obvious reasons for that but they're unfortunate dismissals. Why? Because if you want to reduce the likelihood of another Driscoll or a re:surgence of the one we already know about you have to do something besides address what you assume Mark Driscoll's appeal is. If you assume it was only an appeal to right wing bigotry you'd be wrong. Bigotries abound all over the political spectrum. Driscoll worked because he had a populist appeal and made a generational appeal. He targeted a demographic that was generationally estranged, felt a need to belong, and wanted to contribute positively to the future. That's every generation ever. The great danger is telling ourselves we wouldn't fall for this. Any given generation will be likely to self-identiy as fixing the world's problems. Those of us who were 20-somethings decades ago were sure it was gonna be us. Where others failed we would succeed. Go Mars Hill.
Well, wait, where is Mars Hill now?
If there's a crisis of procedural ethics and replicability in the social sciences then it may be that the people least qualified to talk down to and about religious conservatives in the United States could be vocational social scientists. It shouldn't be that way if it is that way, but it's a possibility that may be worth considering.
People who are part of movements like Mars Hill may avoid talking to journalists and academics not "just" because they are evangelicals with conservative conceptions of Christian dogma (but not necessarily in politics), it may also be because they had doubts about the honesty and ethics of journalists and academics who said one thing in conversation with them and then published another for the periodical.
Why this particular soapbox here? Because if you boil down Mark Driscoll's aspirations and interest they boil down to a social gospel. He wants to formulate a message that will inspire guys to soldier through being husbands and fathers and to see that as a worthy aspiration. Instead of presuming to know why he's embracing what would seem to be red state values, consider that he's said a few times that his interest is to keep young guys from resorting to violence. His social gospel is crude, but it amounts to this--if young guys are happily married young and getting laid all the time and have children to feed that drastically reduces the likelihood that they will engage in anti-social or destructive behavior.
If all you see is the heteronormative parts you'll miss out on the part where it's possible to conceive of ways of solving the problem of the social identity of working class males who might be tempted to violence in other ways. If you can figure out what's at the heart of his social gospel message then you have a better shot at coming up with a more compelling alternative solution to the problems Driscoll thinks he's trying to fix. Take that nebulous "father wound". He's been selling that line for more than a decade and he's shown he's' not competent to solve that problem on his end. Roy Baumeister has written that the conundrum of individual male identity is that the individual male is disposable. There are also no rites of passage that are clear-cut indicating the transition to manhood, as some who were at Mars Hill have put it.
Instead of seeing the critique of radical individualism as a sign of reactionary politics (which it can be, but it could lead to other things), look at how the appeal is to social identity. Mars Hill gave people roles and if people in the academy are too ensconced in what Haidt called WEIRD ethics then it will be inconceivable for these sorts of scholars to understand the appeal that Mars Hill had because in their college-attending and teaching based perspective they have their social identity formulated by being in academics, not as part of a church. But historically and globally which form of social identity is more likely to fit ordinary people, being an academic or being part of a church?
I've stonewalled people over the years and in a select few cases I've talked with people freely. Those people were the ones who told me up front exactly where they landed on religion and politics. It was not necessary to be a religious liberal or conservative. It wasn't even necessary to be religious. It wasn't necessary to be a political liberal or conservative as such. What was necessary was to state where you land and what your interest in the subject was and to demonstrate a willingness to do meticulous, fact-checked research about what actually happened. I'm less interested in taking sides left and right on doctrine or policy than in talking with the people who are stubbornly committed to documenting the facts as they happened regardless of whether or not they went in a direction any of us may have wanted to see.
Jose Duarte's rants strike a chord here because theyh get at a concern I sometimes have, that we can't be sure that academics who might opt to write about Mars Hill haven't already made up their mind what they want to say before they even start the research. Let the study of the subject change how you think about it. I certainly don't think about Mars Hill the same way now that I did ten years ago. Ten years ago I was fairly happy to be there! Ten years ago there was a Mars Hill.
It can be very easy for this or that team to talk about the "narrative" as if it's something the other team does. There's a right narrative about Mars Hill and there's a left narrative about Mars Hill but fr anyone who was actually AT Mars Hill there was a lot more political diversity than would get reported by The Stranger or the Seattle Times. There was more theological diversity than might have seemed to outsiders.
The most dangerous mistake you can make about a movement like Mars Hill is the one we made while we were in it, looking out at everyone else we think we're different from and saying "Yeah, well, we won't make those mistakes."