Sep 15, 2020

Agreed, the amount of superstitious in many fields is mind boggling..

Even then in many fields like comp. sci. the role of RCT's are pretty fuzzy (and many results like OO, FP are more superstitious than scientific). So I do consider much of C.S. to be more akin to philosophy than science. Still in say Physics the concept of RCT's don't apply directly, but the effect is the same underlying hypothesis -> experiment -> re-hypothesize -> experiment cycle. That cycle requires predictions of novel events which fulfills the same role, otherwise it's just data fitting.

One "technology" that could strongly improve our ability to do proper science without RCT's or even improve RCT or hypothesis formulation would be properly integrating causation into statistics (particularly via Bayesian methods). See the work of Judea Pearl [1] for a digest of current work on causal maths. Then scientific fields can clearly define the degree of effect or causation for various results.

Even in medicine/biology RCT's don't always confirm a theory as you can have overlapping effects where the tested variable does work but not for reasons in a given hypothesis. For example see Statins & cholesterol which work to a degree but largely because cholesterol is a necessary but not only ingredient to plaque formation and heart disease. RCT's proved early results of Statins, but new more powerful ones failed to materialize benefits. It turns out it's much more complicated (one random google result plucked for context [2]).

I imagine those cases in medicine as similar to "microbenchmarking" in C.S. where it's easy to see that small scale microbenchmarks are true but fail to improve the overall system performance. Sometimes they make the overall system worse. Medicine in particular doesn't do enough real science. RCT's are only performed at a "microbenchmark" level but rarely performed for the overall system, partly due to cost and difficulty. Hence why I think something like causal maths could really help society formally and explicitly estimate causal properties from groups of RCT's.

Sorry for the semi-rant, but it's been bouncing around my head for a few weeks now!

1: 2: