Lars Jörgen Pålsson Syll is a Swedish economist who is a Professor of Social Studies and Associate professor of Economic History at Malmö University College.
What counts as evidence? I suspect we tend to overweight some kinds of evidence, and underweight others.
Yeh’s paper is a lovely illustration of a general problem with randomized control trials – that they tell us how a treatment worked under particular circumstances, but are silent about its effects in other circumstances. They can lack external validity. Yeh shows that parachutes are useless for someone jumping from a plane when it is on the ground. But this tells us nothing about their value when the plane is in the air – which is an important omission.
We should place this problem with RCTs alongside two other Big Facts in the social sciences. One is the replicability crisis … The other (related) is the fetishization of statistical significance despite the fact that, as Deirdre McCloskey has said, it “has little to do with a defensible notion of scientific inference, error analysis, or rational decision making” and “is neither necessary nor sufficient for proving discovery of a scientific or commercially relevant result.”
If we take all this together, it suggests that a lot of conventional evidence isn’t as compelling as it seems. Which suggests that maybe the converse is true.