Friday, December 18, 2015

an intermittent crisis in how much science there is in the findings of social science, thinking back on some coverage shortfalls
By all the evidence we have to date, I think we have to conclude that most social psychology findings are false. That’s what the data is telling us. A failure to replicate is not decisive evidence that a finding is false, but in most cases the replication attempts are higher quality studies than the originals, with larger and often more representative samples. There is no reason to grant greater epistemic standing to an original study over a replication attempt, and there are many reasons to do the opposite.
A recent paper (PDF) by Rohrer, Pashler, and Harris illustrates why we might be overvaluing original research. They attempted to replicate the findings reported by Caruso, Vohs, Baxter, and Waytz (2013). Caruso et al. claimed that momentarily exposing people to background images of money on a computer screen before an experiment made them endorse free market systems and “social inequality” more than people who were not exposed to the money image.
Rohrer, Pashler, and Harris re-ran four of the five studies from the Caruso paper, using much larger samples. They even ran one of the studies three times. They found none of the effects reported by Caruso et al. Participants exposed to the money pictures didn’t endorse free markets any more or less than participants who never saw the money.
In discussing their resounding failure to replicate with the Caruso team, they discovered a shocking fact: the Caruso team had conducted nine studies, not five, but they only reported the five that confirmed their desired hypotheses. In their paper, they failed to disclose the other four studies that didn’t go their way. They also failed to disclose two outcome variables that were meant to test their hypotheses, but which showed no effects. So out of eleven tests, five supported their hypotheses and six did not — they only reported the five. Readers would have no idea that there were six other tests that showed no results. (For details, see the Rohrer et al. paper.)
That kind of malpractice needs to be purged from the field immediately if we want to be taken seriously as a science. We absolutely cannot do that. Failure to disclose null results undermines the validity of the statistical methods we use to report significant results. (For more on how this works, see Daniel Lakens’ work.) If this malpractice is widespread in the field, there would be little reason for anyone to believe social psychology findings, irrespective of any other issues — it would be irrational and mathematically incompetent to believe published findings in that case.
Another factor that could explain why most findings don’t replicate is that we’re using invalid samples — most commonly, college kids from one university in one country (and only those taking Intro to Psychology, adding additional skew.) This was never a justifiable practice, and it’s strange that it’s persisted so long. College kids from a single campus are artificially homogenous on many variables that could interact with the variables researchers are studying. This creates a synthetic low-variability and low-noise context for our analyses, making us more likely to find an “effect” that isn’t real in humans as such. We can’t credibly make claims about human nature from such
unrepresentative and skewed samples. This is obvious to everyone outside of social psychology, and it has a serious impact on our credibility as a field, so I hope we can move forward quickly here. Research based entirely on college kids should be valued much less than debunking false findings — in that case, I agree wholeheartedly with the Union professor.

Not exactly new, this stuff, but striking.  There have been discussions of how a sample size that's too small is already worthless for stasticial and scientific purposes. 

After all ...

The question of whether deceiving test subjects is acceptable for the sake of verifying a hypothesis is its own philosophical and ethical question.

At least when a televangelist quotes a Bible verse out of context you can go look that up.  If social scientists don't disclose null test results and use deception to get results they present as a finding then there's a sense in which, these days, the snake oil preacher may be llying to you but at least you've got a better chance of guessing that's the game. 

Work in the social sciences is certainly valuable.  Not arguing against the socials ciences at all. But when enough questions are raised about the statistical viability, the ethical norms, and the lack of replicatable results it might be fair to ask whether social science can be taken in diretions that don't land so readily into these failures of both ethics and science.

Jonathan Haidt has said and written plenty about how one of the big problems in the social sciences in the United States is how WEIRD it is (western, educated, industrialized, rich and democratic, which can in turn be simplified into rich white egalitarian liberals with a few variations).  Even if the sample size at a college, often one college, is high, by its very nature asking a bunch of kids who can go to college what they think about stuff is going to be something other than a representative sample of anyone who hasn't been to college or even made it through high school.  There could be an elite echo chamber effect going on in all that.

Now why could this matter?  Well, let's just give a local example.  Anyone remember how people theorized that the appeal of Mark Driscoll was his being a macho Calvinist?  He didn't always have that thing going because in the earliest years of Mars Hill he wasn't even really all that Calvinist.  The pitch he had, if you go back and look at how he promoted himself to Mother Jones, was appealing to generational estrangement.  Dads failed us by not being there so we gotta band together and follow Jesus.  Driscoll was targeting young men who he felt needed ambition and purpose in their lives. This was not necessarily explicitly about red state politics.  Driscoll was making an appeal to give guys a purpose and that purpose was going to be, as it turns out, Mars Hill.  As Roy Baumeister has put it, men are distinctive for being interchangeable and disposable within a cultural apparatus.  Driscoll's spiel was to make a case that if "you", young man, join our movement, you will not be disposable but will play a vital role in a movement to make a better future.

I ought to know, I bought it.  :( 

One of the big failures of progressives and secularists when discussing Driscoll has been an inability to take his ideas seriously.  There are some obvious reasons for that but they're unfortunate dismissals.  Why?  Because if you want to reduce the likelihood of another Driscoll or a re:surgence of the one we already know about you have to do something besides address what you assume Mark Driscoll's appeal is.  If you assume it was only an appeal to right wing bigotry you'd be wrong.  Bigotries abound all over the political spectrum.  Driscoll worked because he had a populist appeal and made a generational appeal.  He targeted a demographic that was generationally estranged, felt a need to belong, and wanted to contribute positively to the future.  That's every generation ever.  The great danger is telling ourselves we wouldn't fall for this.  Any given generation will be likely to self-identiy as fixing the world's problems.  Those of us who were 20-somethings decades ago were sure it was gonna be us.  Where others failed we would succeed.  Go Mars Hill.

Well, wait, where is Mars Hill now? 

Right, exactly. 

If there's a crisis of procedural ethics and replicability in the social sciences then it may be that the people least qualified to talk down to and about religious conservatives in the United States could be vocational social scientists. It shouldn't be that way if it is that way, but it's a possibility that may be worth considering. 

People who are part of movements like Mars Hill may avoid talking to journalists and academics not "just" because they are evangelicals with conservative conceptions of Christian dogma (but not necessarily in politics), it may also be because they had doubts about the honesty and ethics of journalists and academics who said one thing in conversation with them and then published another for the periodical. 

Why this particular soapbox here?  Because if you boil down Mark Driscoll's aspirations and interest they boil down to a social gospel. He wants to formulate a message that will inspire guys to soldier through being husbands and fathers and to see that as a worthy aspiration.  Instead of presuming to know why he's embracing what would seem to be red state values, consider that he's said a few times that his interest is to keep young guys from resorting to violence.  His social gospel is crude, but it amounts to this--if young guys are happily married young and getting laid all the time and have children to feed that drastically reduces the likelihood that they will engage in anti-social or destructive behavior.

If all you see is the heteronormative parts you'll miss out on the part where it's possible to conceive of ways of solving the problem of the social identity of working class males who might be tempted to violence in other ways.  If you can figure out what's at the heart of his social gospel message then you have a better shot at coming up with a more compelling alternative solution to the problems Driscoll thinks he's trying to fix.  Take that nebulous "father wound". He's been selling that line for more than a decade and he's shown he's' not competent to solve that problem on his end.  Roy Baumeister has written that the conundrum of individual male identity is that the individual male is disposable.  There are also no rites of passage that are clear-cut indicating the transition to manhood, as some who were at Mars Hill have put it. 

Instead of seeing the critique of radical individualism as a sign of reactionary politics (which it can be, but it could lead to other things), look at how the appeal is to social identity. Mars Hill gave people roles and if people in the academy are too ensconced in what Haidt called WEIRD ethics then it will be inconceivable for these sorts of scholars to understand the appeal that Mars Hill had because in their college-attending and teaching based perspective they have their social identity formulated by being in academics, not as part of a church.  But historically and globally which form of social identity is more likely to fit ordinary people, being an academic or being part of a church?

I've stonewalled people over the years and in a select few cases I've talked with people freely.  Those people were the ones who told me up front exactly where they landed on religion and politics.  It was not necessary to be a religious liberal or conservative.  It wasn't even necessary to be religious.  It wasn't necessary to be a political liberal or conservative as such.  What was necessary was to state where you land and what your interest in the subject was and to demonstrate a willingness to do meticulous, fact-checked research about what actually happened.  I'm less interested in taking sides left and right on doctrine or policy than in talking with the people who are stubbornly committed to documenting the facts as they happened regardless of whether or not they went in a direction any of us may have wanted to see. 

Jose Duarte's rants strike a chord here because theyh get at a concern I sometimes have, that we can't be sure that academics who might opt to write about Mars Hill haven't already made up their mind what they want to say before they even start the research.  Let the study of the subject change how you think about it.  I certainly don't think about Mars Hill the same way now that I did ten years ago.  Ten years ago I was fairly happy to be there!  Ten years ago there was a Mars Hill. 

It can be very easy for this or that team to talk about the "narrative" as if it's something the other team does.  There's a right narrative about Mars Hill and there's a left narrative about Mars Hill but fr anyone who was actually AT Mars Hill there was a lot more political diversity than would get reported by The Stranger or the Seattle Times.  There was more theological diversity than might have seemed to outsiders. 

The most dangerous mistake you can make about a movement like Mars Hill is the one we made while we were in it, looking out at everyone else we think we're different from and saying "Yeah, well, we won't make those mistakes."


chris e said...

What if there wasn't a crisis? Or at least, there wasn't a crisis in the way in which those living through their 20s/30s would have defined it?

Wenatchee the Hatchet said...

A useful point, because I've been incrementally moving toward an idea that what many people in any given generation describe as a "crisis" is really just the "normal" in every generation.

It's easy for a Zimbardo to complain that young men lack direction and play video games and look at porn but the trouble is that if people think through what would "solve" that problem the solution will be some kind of social identity based on a subculture. To put it in more polemical terms, I've seen people on the left and right lament that young men lack purpose but nobody wants to propose the solution is cult formation or that cult formation is the base line human activity for us as social creatures. It's easy to say "LOL don't drink the Kool-Aid" as if the default reaction of humans was to not drink the proverbial Kool-Aid. Recognizing that may, in fact, be what our default mode is as social creatures could better equip us to figure out what kinds of social systems we do or don't want youngsters joining.

It can sure feel like a crisis of identity to be a 20 something not sure who you are but in many societies there was not necessarily an identity crisis for the average person because their social role was defined for them. We're given a kind of cultural promise that you can be anything that you want to be but there's not much by way of cultural scripts for what you do when what you choose to be doesn't pan out or turns out to be impossible.

chris e said...

There are a particular set of economic contingencies and trends that aren't 'hopeful' unless we have some kind of mid/long term reform, however when people are talking about a crisis, they are typically not referring to this, and instead putting the blame for the fact that the 1950s don't still exist squarely on the shoulder of the current generation (and then ignoring that the success of such times was built on exclusion and occurred due to a huge one off intervention).

People tend to conflate their present success with a particular set of historical contingencies. They also ignore past ills - going back to those much maligned social science studies, one fairly well founded data point is that levels of violence have decreased since the 1970s - but the present generation isn't going to get credit for that.

Obviously one can point to obvious trends like exposure to porn as troubling, however it's not clear to me exactly what problems attempts to create a 'social movement' actually solve - or indeed that the magnitude of problem is such that a social movement is needed. Are video games really that much more of an issue than male bonding over going bowling or getting drunk?

Plenty of people find their 20s hard to live through, but then people tend not to be very good at evaluating their own particular circumstances.

Wenatchee the Hatchet said...

the survivor's bias, yep.

Seen the proposal that crime has gone down because abortion is legal and the people who would have otherwise birthed today's criminals had the liberty to abort them? There's actually been an explanation of why crime has gone down that depends on the data point that today's younger generations are smaller thanks to the abortions selected by previous generations. So, yeah, you might be right that current generations are not going to get credit for being less violent than their forbears. :)

The huge one-off intervention is easy to forget but I think the combination of the New Deal and WW2 created an economic bubble that has been deflating but social conservatives want to get back to the social millieu of that bubble while progressives want to expand the impact of that bubble to everyone possible. It seems as if conservatives want to go back to a past that is imaginary and progressives want to push us into a future that is imaginary but that neither set of ideologues is necessarily contending with the real world.

I'm starting to feel like the problem is that Americans, whether left or right, have absolutely no interest in dealing with reality ... which was kind of a not so subtle plot point in an episode of South Park this season.

chris e said...

I think it is now conceded that the abortion explanation is quite unlikely, as international comparisons show that the level of violence went down everywhere, even controlling for abortion.

The point is that in the 70s there was a moral panic around violence (specifically gang related). Perhaps, if one is selling morality, it's becomes inevitable that one paints every social change (crisis or not) as a crisis of moral fibre.

Wenatchee the Hatchet said...

at the risk of cycling back to the title, the crisis that's been interesting to read being described is in the failure of the science of social science, in terms of the failure to establish replicatable result for previously given conclusions and questions about whether the means for obtaining data and results involves more deception in the screening process than may be justifiable for the results. Other forms of sociological crisis may be secondary or even tertiary to whether or not any of the data can be proven to be legitimate the first time around, but as some authors have been noting, retesting a hypothesis or even debunking previously published research is not even remotely sexy from a fundraising or professional standpoint. Nobody's going to feel that good about proving an earlier hypothesis was null because that's not seeming to forward knowledge. But to that I can't help but be reminded of an axiom my friends who are actual scientists have shared over the years, "ninety percent of science is being totally wrong and the other ten percent is the part where you figure out why you were wrong to start with". It's in that last ten percent htat scientific breakthroughs can occur.