It's hard not to think of Daniel Kahneman's Thinking Fast and Slow in which he explained why many a social scientific finding is essentially worthless because of sample size. If the sample size is too small it's not representative of a population and if a study can't be replicated multiple times it's impossible to take the finding as being all that reliable.
There's been a slow-cooker crisis related to the credibility of social scientific findings over the last decade or so.
THERE'S NOTHING NEW ABOUT scientists who fudge, or even fabricate, their results. Whole books have been written about whether Gregor Mendel tweaked the measurements of his plants to make them better fit with his theory. When attempting to fit the irregular polygon of Nature into the square hole of Theory, all researchers face a strong temptation to lop off the messy corners. Imagine that you’re going along, accumulating data points that fall into a beautiful line across the graph, and all of a sudden some dog stands there like a dummy, refusing to salivate. You ring the bell again, louder—nothing. Is he mentally defective? Is he deaf? What will you trust: the theory you have spent years developing, or the dog? (This is not to cast aspersions on one of the great pioneers of experimental psychology. As far as we know, Pavlov’s dogs really did do what he said they did.)
Then again, maybe there is no dog at all. For scientists in a real hurry to establish themselves, the quickest way to go from arresting hypothesis to eye-catching publication is to skip the research altogether and just make up results.
OUTRIGHT FAKERY IS CLEARLY more common in psychology and other sciences than we’d like to believe. But it may not be the biggest threat to their credibility. As the journalist Michael Kinsley once said of wrongdoing in Washington, so too in the lab: “The scandal is what’s legal.” The kind of manipulation that went into the “When I’m Sixty-Four” paper, for instance, is “nearly universally common,” Simonsohn says. It is called “p-hacking,” or, more colorfully, “torturing the data until it confesses.”
Sample size is a touchy topic in psychology, because undergraduate subjects often expect to be compensated, and researchers must pay them from grants that are overseen by the tight-fisted guardians of research funding. But because larger sample sizes increase the predictive power of results, Simmons now tries for at least 50 subjects in his own research. “The brutal truth is that reality is indifferent to your difficulty in finding enough subjects,” he says. “It’s like astronomy: To study things that are small and distant in the sky you need a huge telescope. If you only have access to a few subjects, you need to study bigger effects, and maybe that wouldn’t be such a bad thing.”
Of course it's no small thing when the samples turn out to be bunk. Torturing the data until it confesses may be the more common and worrisome risk than outright fraud, but Slate had a lengthy feature dealing with fraud earlier this year.
That gay marriage study that faked data, for those who remember that, that study was used as a basis to distribute money for campaign projects.
There are people who doubt scientific research because they have doubts about the research in particular, even if it's tempting to propose that those sorts of people doubt the efficacy of science in general. It's not hard to come across people who take to social media to condemn pharmaceutical companies in favor of this or that naturopathic approach. On the other hand, to tweak a line attributed to C. S. Lewis about the sciences, it may be there's cause even within the realm of science to worry that, within certain limits, people get the kind of scientific findings foundations are willing to pay for.