4
$\begingroup$

I have a linear mixed model and am interested in the significance of the fixed effects.

I run run 2 identical models, except the second eliminates data points that are outliers (i.e., a sensitivity test).

Assuming the use of p-values to determine statistical significance, what is the best method to determine whether a change in p-values (from significant to non-significant) is due the reduction in data points?

  • Power analysis?
  • Effect size?

In which case, what is the method for computing the above with linear mixed models in R?

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ A better alternative would probably be to re-analyse using robust methods instead of removing outliers. $\endgroup$
    – mkt
    Commented Jan 29 at 15:25
  • 2
    $\begingroup$ Following up on @mkt's comment, check out cran.r-project.org/web/packages/robustlmm/index.html $\endgroup$
    – Erik Ruzek
    Commented Jan 29 at 15:51

2 Answers 2

Reset to default
6
$\begingroup$

This is not possible in principle, because once you have collected your data, there is a one-to-one relationship between the p value and power. A p value greater than 0.05 tells you that your study was not sufficiently powered to detect the observed effect size as statistically significant. This is simply a restatement of the "post hoc power" fallacy. See Figure 1 in Hoenig & Heisey, 2001, and actually the entire paper.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Ok; then what is the optimal alternative to post-hoc power analysis to interpret these results? $\endgroup$
    – SilvaC
    Commented Jan 29 at 14:39
  • 5
    $\begingroup$ Quite honestly, there is no good statistical way of dealing with insignificant results, or your specific situation. You could discuss that these "outliers" (a term many statisticians do not much like) seem to be driving your significance, but little more. Also, see Gelman & Stern (2006), who warn against over-interpreting results like yours. It may well be useful to dig into what distinguishes "outliers" from other data points. $\endgroup$
    – Stephan Kolassa
    Commented Jan 29 at 14:44
2
$\begingroup$

It is important understand why certain data points are labelled as outliers. Are they true anomalies, or do they represent a valid, albeit extreme, variation within the data? Investigating the nature of these outliers can provide insights into the data's structure and the phenomena being studied.

While p-values indicate whether the test statistic you obtained is likely if the null hypothesis is true, they do not convey the magnitude of the effect. Calculating and reporting effect sizes can offer a more nuanced understanding of your results. In R, packages like lme4 for linear mixed models can be used to compute effect sizes. Consider Cohen's d or other relevant effect size measures appropriate for your data.

Examining confidence intervals around your estimates can be more informative than solely relying on p-values. Confidence intervals provide a range of plausible values for the effect size and can offer insights into the precision of your estimates. Wide intervals might suggest more data is needed for precise estimates.

Comparing models with and without outliers using information criteria like AIC (Akaike Information Criterion) or BIC (Bayesian Information Criterion) can provide insights into model fit and complexity. A significant change in these criteria might suggest the outliers have a substantial impact on the model.

Beyond just removing outliers, conducting a sensitivity analysis where you systematically vary your inclusion criteria or model specifications can reveal how robust your findings are to different assumptions and data manipulations.

Bayesian methods can offer a different perspective, especially in understanding the probability of a hypothesis given the data. This approach might provide a more intuitive interpretation of your results than frequentist p-values.

In the absence of clear statistical guidance, a qualitative discussion about the nature and impact of the outliers, the context of the study, and potential theoretical implications can be valuable. This can include speculating about why outliers might be influencing the results and what this means for your field of study.

Complex models and situations like yours often benefit from collaboration with a statistician who can provide tailored advice based on the specifics of your data and research questions.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ p values do not "determine whether an effect exists". They only determine whether the test statistic you got is likely if the null is true. $\endgroup$
    – Peter Flom
    Commented Jan 29 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.