The Crisis of Unreplicable Research: Bayes, Evolution, & Game Theory


There is a crisis in the not-so-hard sciences:

“For the past few years, there has been a growing crisis in social science, and in biology, medicine, and other statistics-dependent fields, that many claimed research findings are fragile, are unreliable, cannot be replicated, and do not generalize outside of the lab to real-world settings (Ioannidis, 2005). Arguably the crisis is most pronounced within psychology (see Pashler & Wagenmakers, 2012).”

Andrew Gelman has some ideas on how to fix that in this paper.

Why does this problem exist? Because (says Gelman), the effects in the stats-dependent fields are intrinsically variable.

I think this view makes a lot of sense. Gelman (paraphrase) is most interested in how this crisis in the practice of science is connected to statistical problems in the modelling of interaction — and his solution is to adopt a more Bayesian approach to statistics.

I agree.

I’d like to add that the more general crisis of social science includes this one, but also includes a total lack of any serious theoretical (read: mathematical) underpinnings. This makes the search for “laws” that one finds in both physics and in biology generally, nearly impossible.

There should exist a Theoretical Social Science that is the equivalent of what exists in the hard sciences — theoretical physics means math; theoretical biology means math — theoretical social science should be a unified field of mathematics that is integrated into the fabric of social science.

Until that happens, the findings in the field will never carry the weight of the harder sciences — and for good reason.

Even without devolving into math, deeper theoretical underpinnings make the understanding of a variable effect more understandable.

EXAMPLE: Take the case of the species-concept in biology as viewed by taxonomy verses the more nuanced view given by evolutionary theory.

Traditional taxonomy was notoriously Platonic, in that it posited “universal forms”, called species, that each live in categories, organized hierarchically — these forms are given as eternal and unchanging.

The taxonomy approach presumed there was a “form” called a wolf, and that each member of the members of this wolf “family” were all imperfect expressions of its canonical form: THE wolf.

This kind of thinking is useful up to a point — the earth is NOT a sphere, but it’s useful to imagine it as an imperfect example of the canonical form, THE sphere — but just like how one needs differential geometry and topology (among other things) to deal with more complex (non-trivial) models of the earth — without more complex theories, like evolution, we can’t break free from the obvious falseness of the species-theory.

Historically, this lack of understanding of the variance in a the vary notion of species was holding our science back horrendously. Thankfully, Darwin changed all that!

In evolution, it is FALSE to say that there is a canonical wolf-form. Instead, there is a collection of individual animals that are still able to breed with one another and produce fertile offspring — that collection we call “wolves”.

In other words, the species-concept, as enhanced by evolutionary theory, accounts for the fact that each individual species is an effect (of an evolutionary process) that is also perpetually in flux and changing. There isn’t a concrete notion of the wolf-species that has any meaning outside of rough analogy (like the earth-sphere, problem) — and it is certainly not constant and unchanging!

Many of the problems in social science are remarkably similar in their base character to the problems found in evolutionary biology — duh, humans are animals!

Therefore my own feelings (I’m hardly alone, here) about how we can avoid this wide problem of less-than-awesome research come down to an adoption of (Objective/Falsificationist?) Bayesian approaches to statistics on the one side; and on the mathematical modelling side, adopt the best of what we have learned from theoretical biology and the modelling of non-human species and drive forward with the goal of producing a true Theoretical Social Science.

Now go lift something heavy,
Nick Horton

In Defence Of The Third Person

Don’t you hate it when people talk about themselves in the third person? What if I always talked like this: “Nick is drinking a cup of coffee. Nick is typing on his keyboard right now. Nick needs even more coffee…”

Well, maybe that is precisely how I should talk:

Participants were told, upon entering the lab, that they faced a nerve-wracking task: to impress a member of the opposite sex, in one study, or to give a speech. To up the ante, they also knew their performance would be videotaped and later analyzed. But right before they began, they were told to prepare themselves for the task ahead. Some participants were assigned to do so by speaking to themselves in the first person; the rest were instructed to address themselves using their own first name, as well as non-first-person pronouns like she, he, or you.

Though we don’t tend to look kindly upon those who speak of themselves in the third person, the practice is not without its benefits. According to reviewers, who were blind to participants’ condition, those who’d avoided I and me in their pep talks actually appeared less nervous, and did a better job on the task at hand. Speaking to ourselves as though we are someone else, it seems, lets us distance ourselves from an overwhelmingly stressful experience. Even your English teacher didn’t see that coming.

This finding isn’t particularly surprising, considering the total lack of mindfulness we humans have about ourselves. I’d like to see this redone with people who are engaged regularly in specific mindfulness training, like meditation, and compare the results. My prediction would be that the results would nearly vanish.

In other words, I doubt the effect can honestly be attributed to one speaking in the third person, per se, but to anything that gets you out of your own head, and allows you to be mindful of the situation you are in. Speaking in the third person is cute and all, but it isn’t a transferable skill. Meditation builds skills that ARE transferable. “Nick is feeling mindful about this.”

Now go lift something heavy,
Nick Horton

The God Delusion: Richard Dawkins Film

I found this nice film that Richard Dawkins did to accompany his wonderful book, The God Delusion — which was one of the few truly insightful books about how harmful the appeal to “faith” over reason is to our species. (Too many of the recent books in this line sound like pale rehashes of his arguments, the only exception coming to mind being Hitchens’ God is Not Great.)

The most frightening part of the movie takes Dawkins to Israel… hotbed of hatred and Religion-based violence. Worth the watch.

Sam Harris vs Israel: Where’s The Beef?


“Much of Sam’s argument would hold water if the Israelis had been in earnest about peace, and in earnest in supporting moderate Palestinian forces on the West Bank, and in earnest about taking Obama’s proposals seriously this past decade. But they haven’t been. Settlements are much more important to them than peace. And the settlements are motivated by exactly the kind of theocratic zeal that Sam normally opposes.”

— Andrew Sullivan

Anti-Religion crusader Sam Harris has come under fire for his refusal to come out against Israel’s actions of late. Andrew Sullivan has a nice discussion on why that might be.

First, Sam Harris IS attacking Israel’s positions, the problem (says Sullivan) is that he isn’t doing a very good job of it. By sticking to the abstract — for instance, Harris strongly believes that Israel shouldn’t even exist as a state in the first place — he, unwittingly, undermines his own arguments by failing to take history and pragmatism into account.

I’d have to agree with this.

The 2-state solution is the only option at this point in history, and Israel has been vehemently opposed to it in any meaningful way.

As I’ve said many times before, Hamas are a bunch of psychopaths. But, that doesn’t take away from the reality that religious zealots in Israel are standing in the way of a rational solution and have FULL SUPPORT of the Israeli government. That kind of melding of church & state is the antithesis of what good democracy and humanitarian government is about.

The Enlightenment weeps …

To paraphrase Richard Dawkins, Religion turns grownups into children. In a situation like this, we need people who can finally start acting like adults.

To quote Christopher Hitchens:

“If this was only a territorial or national fight over land, it would have been solved by now.”

More from Sullivan:

“It’s not just Palestinian terrorism and Muslim anti-Semitism that makes a one-state solution moot; it is embedded in the very meaning of Zionism. If Israel requires a Jewish majority to survive as a Jewish state, a one-state solution is anathema to it. And if all Israel wanted to do was have its tech sector thrive within (roughly) the 1967 borders, and embrace serious, US-backed security arrangements vis-a-vis Jordan, I’d be backing it to the hilt.

Instead, as Palestinian terrorism from the West Bank has declined drastically – the Israelis have intensified their theft of Palestinian land. Those settlements deeply hurt, rather than help, Israel’s security – because they alienate most of her allies, exacerbate bitterness and suspicion, and make the possibility of a two-state solution moot. You could secure the West Bank by military outposts if you wanted. But Israel is committed to engineering the demography of the place by settlements of religious fanatics of the sort Sam would usually excoriate. Netanyahu, we now know, would rather release hundreds of prisoners convicted of murdering Jews than remove a single brick from the West Bank settlements. It’s really not about security at all. It’s about race and religion in their ugliest zero-sum manifestations. Just because it isn’t as bad as Hamas doesn’t excuse it.”

The discussion continues here