Do likelihood ratios and Bayesian model comparison provide superior & sufficient alternatives to null-hypothesis testing?

Question

In response to a growing body of statisticians and researchers that criticize the utility of null-hypothesis testing (NHT) for science as a cumulative endeavour, the American Psychological Association Task Force on Statistical Inference avoided an outright ban on NHT, but instead suggested that researchers report effect sizes in addition to p-values derived from NHT.

However, effect sizes are not easily accumulated across studies. Meta-analytic approaches can accumulate distributions of effect sizes, but effect sizes are typically computed as a ratio between raw effect magnitude and unexplained "noise" in the data of a given experiment, meaning that the distribution of effect sizes is affected not only by the variability in the raw magnitude of the effect across studies, but also variability in the manifestation of noise across studies.

In contrast, an alternative measure of effect strength, likelihood ratios, permit both intuitive interpretation on a study-by-study basis, and can be easily aggregated across studies for meta-analysis. Within each study, the likelihood represents the weight of evidence for a model containing a given effect relative to a model that does not contain the effect, and could typically be reported as, for example, "Computation of a likelihood ratio for the effect of X revealed 8 times more evidence for the effect than for its respective null". Furthermore, the likelihood ratio also permits intuitive representation of the strength of null findings insofar as likelihood ratios below 1 represent scenarios where the null is favoured and taking the reciprocal of this value represents the weight of evidence for the null over the effect. Notably, the likelihood ratio is represented mathematically as the ratio of unexplained variances of the two models, which differ only in the variance explained by the effect and thus is not a huge conceptual departure from an effect size. On the other hand, computation of a meta-analytic likelihood ratio, representing the weight of evidence for an effect across studies, is simply a matter of taking the product of likelihood ratios across studies.

Thus, I argue that for science seeking to establish the degree of gross evidence in favour of a effect/model, likelihood ratios are the way to go.

There are more nuanced cases where models are differentiable only in the specific size of an effect, in which case some sort of representation of the interval over which we believe the data are consistent with effect parameter values might be preferred. Indeed, the APA task force also recommends reporting confidence intervals, which can be used to this end, but I suspect that this is also an ill-considered approach.

Confidence intervals are lamentably often misinterpreted (by students and researchers alike). I also fear that their ability for use in NHT (by assessment of inclusion of zero within the CI) will only serve to further delay the extinction of NHT as an inferential practice.

Instead, when theories are differentiable only by the size of effects, I suggest that Bayesian approach would be more appropriate, where the prior distribution of each effect is defined by each model separately, and the resulting posterior distributions are compared.

Does this approach, replacing p-values, effect sizes and confidence intervals with likelihood ratios and, if necessary, Bayesian model comparison, seem sufficient? Does it miss out on some necessary inferential feature that the here-maligned alternatives provide?

Answer 1

The main advantages of a Bayesian approach, at least to me as a researcher in Psychology are:

1) lets you accumulate evidence in favor of the null

2) circumvents the theoretical and practical problems of sequential testing

3) is not vulnerable to reject a null just because of a huge N (see previous point)

4) is better suited when working with small effects (with large effects both Frequentist and Bayesian methods tend to agree pretty much all the time)

5) allows one to do hierarchical modeling in a feasible way. For instance, introducing item and participant effects in some model classes like Multinomial Processing Tree models would need to be done in a Bayesian framework otherwise computing time would be insanely long.

6) gets you "real" confidence intervals

7) You require 3 things: the likelihood, the priors, and probability of the data. the first you get from your data, the second you make up, and the third you don't need at all given proportionality. Ok, maybe I exaggerate a little ;-)

Overall, one can invert your question: Does this all mean that classical frequentist stats are not sufficient? I think that saying "no" is too harsh a verdict. Most problems can be somewhat avoided if one goes beyond p-values and looks at stuff like effect sizes, the possibility of item effects, and consistently replicate findings (too many one-experiment papers get published!).

But not everything is that easy with Bayes. Take for instance model selection with non-nested models. In these cases, the priors are extremely important as they greatly affect results, and sometimes you dont have that much knowledge on most of the models you wanna work with in order to get your priors right. Also, takes reaaaally long....

I leave two references for anybody that might be interested in diving into Bayes.

"A Course in Bayesian Graphical Modeling for Cognitive Science" by Lee and Wagenmakers

"Bayesian Modeling Using WinBUGS" by Ntzoufras