Get your science fix here: research, quackery, activism and all the rest
-
FredM
- Clardic Fug
- Posts: 160
- Joined: Mon Nov 11, 2019 4:36 pm
- Location: Norwich
-
Contact:
Post
by FredM » Fri Nov 15, 2019 10:04 am
Detailed stats paper critique of MBI
here. My italics.
Conclusions
“ We have given a precise description of “magnitude-based inference” for the problem of comparing two means and discussed its interpretation in detail. “Magnitude-based inference” begins with the computation of a confidence interval (that has the usual frequentist interpretation in terms of repeated samples). We show that the calculations can be interpreted either as P values for particular tests or as approximate Bayesian calculations, which lead to a type of test. In the former case, this means that 1) the “magnitude-based inference” calculations are not derived directly from the confidence interval but from P values for particular tests and 2) “magnitude-based inference” is less conservative than standard inference because it changes the null hypothesis and uses one-sided instead of two-sided P values.
The inflated level of the test means that it should not be used. Finally, the sample size calculations should not be used. Rather than use “magnitude-based inference,” a better solution is to be realistic about the limitations of the data and use either confidence intervals or a fully Bayesian analysis.”
-
spinybear
- Fleury White
- Posts: 9
- Joined: Thu Nov 14, 2019 4:00 pm
Post
by spinybear » Fri Nov 15, 2019 10:30 am
FiveThirtyEight has
a good readable explanation.
Small sample sizes have their place. I regularly work with small sample sizes for qualitative research, but there's all kinds of wrong with such small numbers for statistical inference. There are quant methods that can deal well with relatively low sample sizes (though all way higher than "just 11"!), but these would only be methodologically sound on really straightforward questions with clearly defined boundaries.
-
greyspoke
- Fuzzable
- Posts: 252
- Joined: Mon Nov 11, 2019 4:36 pm
Post
by greyspoke » Fri Nov 15, 2019 5:31 pm
It isn't clear how widespread it is in sports science generally.
In competitive sport, there is a big difference between "may improve performance or may have no effect" and "may either improve or reduce performance". I'd take a long shot on the former, but would be wary of the latter.