13
$\begingroup$

In response to a growing body of statisticians and researchers that criticize the utility of null-hypothesis testing (NHT) for science as a cumulative endeavour, the American Psychological Association Task Force on Statistical Inference avoided an outright ban on NHT, but instead suggested that researchers report effect sizes in addition to p-values derived from NHT.

However, effect sizes are not easily accumulated across studies. Meta-analytic approaches can accumulate distributions of effect sizes, but effect sizes are typically computed as a ratio between raw effect magnitude and unexplained "noise" in the data of a given experiment, meaning that the distribution of effect sizes is affected not only by the variability in the raw magnitude of the effect across studies, but also variability in the manifestation of noise across studies.

In contrast, an alternative measure of effect strength, likelihood ratios, permit both intuitive interpretation on a study-by-study basis, and can be easily aggregated across studies for meta-analysis. Within each study, the likelihood represents the weight of evidence for a model containing a given effect relative to a model that does not contain the effect, and could typically be reported as, for example, "Computation of a likelihood ratio for the effect of X revealed 8 times more evidence for the effect than for its respective null". Furthermore, the likelihood ratio also permits intuitive representation of the strength of null findings insofar as likelihood ratios below 1 represent scenarios where the null is favoured and taking the reciprocal of this value represents the weight of evidence for the null over the effect. Notably, the likelihood ratio is represented mathematically as the ratio of unexplained variances of the two models, which differ only in the variance explained by the effect and thus is not a huge conceptual departure from an effect size. On the other hand, computation of a meta-analytic likelihood ratio, representing the weight of evidence for an effect across studies, is simply a matter of taking the product of likelihood ratios across studies.

Thus, I argue that for science seeking to establish the degree of gross evidence in favour of a effect/model, likelihood ratios are the way to go.

There are more nuanced cases where models are differentiable only in the specific size of an effect, in which case some sort of representation of the interval over which we believe the data are consistent with effect parameter values might be preferred. Indeed, the APA task force also recommends reporting confidence intervals, which can be used to this end, but I suspect that this is also an ill-considered approach.

Confidence intervals are lamentably often misinterpreted (by students and researchers alike). I also fear that their ability for use in NHT (by assessment of inclusion of zero within the CI) will only serve to further delay the extinction of NHT as an inferential practice.

Instead, when theories are differentiable only by the size of effects, I suggest that Bayesian approach would be more appropriate, where the prior distribution of each effect is defined by each model separately, and the resulting posterior distributions are compared.

Does this approach, replacing p-values, effect sizes and confidence intervals with likelihood ratios and, if necessary, Bayesian model comparison, seem sufficient? Does it miss out on some necessary inferential feature that the here-maligned alternatives provide?

$\endgroup$
  • $\begingroup$ Could be a more focused question? Perhaps one about likelihood oriented approaches to a specific inference problem? $\endgroup$ – conjugateprior Jan 17 '11 at 17:17
  • 2
    $\begingroup$ But while we're here: On exposition: have you mixed up measures of effect size, usually identified with a parameter, for measures of comparative evidence for a complete model? LRs only look like candidates for the latter. Also, if you want Likelihood functions alone or in combination to tell you everything the data are trying to telling you about a model, then you're basically a Bayesian. Because that's the Likelihood Principle. (Come on in, the water's lovely :-) $\endgroup$ – conjugateprior Jan 17 '11 at 17:24
  • $\begingroup$ Your title and your concluding para seem to disagree on whether you're suggesting using confidence intervals or replacing them. $\endgroup$ – onestop Jan 17 '11 at 17:26
  • $\begingroup$ @onestop: indeed, I just realized I forgot to change the title; I changed my mind regarding confidence intervals while writing the question. I have edited the title now. Apologies for the confusion. $\endgroup$ – Mike Lawrence Jan 17 '11 at 17:43
  • $\begingroup$ @Conjugate Prior: Completely agree with your first two sentences. But you can accept the likelihood principle without being a Bayesian if you don't like the idea of priors and base inference on likelihoods alone - see books by Edwards books.google.com/books?id=2a_XZ-gvct4C and Royall books.google.com/books?id=oysWLTFaI_gC. Although someone (and i wish i remember who and where) once likened this to breaking eggs but not eating the omelette. $\endgroup$ – onestop Jan 17 '11 at 22:30
3
$\begingroup$

The main advantages of a Bayesian approach, at least to me as a researcher in Psychology are:

1) lets you accumulate evidence in favor of the null

2) circumvents the theoretical and practical problems of sequential testing

3) is not vulnerable to reject a null just because of a huge N (see previous point)

4) is better suited when working with small effects (with large effects both Frequentist and Bayesian methods tend to agree pretty much all the time)

5) allows one to do hierarchical modeling in a feasible way. For instance, introducing item and participant effects in some model classes like Multinomial Processing Tree models would need to be done in a Bayesian framework otherwise computing time would be insanely long.

6) gets you "real" confidence intervals

7) You require 3 things: the likelihood, the priors, and probability of the data. the first you get from your data, the second you make up, and the third you don't need at all given proportionality. Ok, maybe I exaggerate a little ;-)

Overall, one can invert your question: Does this all mean that classical frequentist stats are not sufficient? I think that saying "no" is too harsh a verdict. Most problems can be somewhat avoided if one goes beyond p-values and looks at stuff like effect sizes, the possibility of item effects, and consistently replicate findings (too many one-experiment papers get published!).

But not everything is that easy with Bayes. Take for instance model selection with non-nested models. In these cases, the priors are extremely important as they greatly affect results, and sometimes you dont have that much knowledge on most of the models you wanna work with in order to get your priors right. Also, takes reaaaally long....

I leave two references for anybody that might be interested in diving into Bayes.

"A Course in Bayesian Graphical Modeling for Cognitive Science" by Lee and Wagenmakers

"Bayesian Modeling Using WinBUGS" by Ntzoufras

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.