Should I remove a significant covariate from my mixed-effects model because it's underpowered?

Question

I tried figuring out the answer by reading the comments in this thread but I am still confused. Should I remove a covariate from analysis when it comes as significant but power simulations finds that it is underpowered?

Using lme4, I created a mixed-effects model with several fixed effects like so:

m1 <- lmer(outcome ~ x1 + x2 + x3 + x4 + (1 | participant), data = data)

Each of the fixed variables is a factor with two levels 'yes/no', indicating 'presence/absence'. I find that for all variables the estimated value is significant, thanks to lmertest.

Next with simr, I run 200 simulations using Kenward-Roger approximation to find if based on observed effect each of fixed variables is sufficiently powered (above 80%). x1, x2, x3 come around ~95%, while x4 is ~50%.

My understanding was that, if a variable is underpowered, it is more likely to find a false positive. However, after speaking to someone, they suggested that since I achieved significance, the power analysis does not matter. They cited the linked question as a source.

I'm unsure what conclusion to draw from my power analysis. Should I remove x4?

Answer 1

You should not base your decision to remove the variable on your post hoc power analysis.

First, there is an inherent problem with post hoc power analysis, you can read more about this here:

• Gelman, A. (2019). Don't calculate post-hoc power using observed estimate of effect size. Ann. Surg, 269, e9-e10. (http://www.stat.columbia.edu/~gelman/research/published/power_surgery_3.pdf)

Apart from that, your statement "My understanding was that, if a variable is underpowered, it is more likely to find a false positive" is wrong! Underpowered tests are more likely to run into a false negative. That is, assuming your post-hoc power was a reasonable estimate of the actual power of your study, you found an effect despite and not due to low(er) power. So the power should not be of concern.

On a more general note, statistical significance is not a good yardstick to judge which variables to select for your model. This should be guided by your goal (e.g., statistical inference, causal inference, prediction?) and an evaluation of metrics and assumptions that judges how well your model achieves this goal. Read more about exactly these topics in variable selection, e.g., here:

• Heinze, G., Wallisch, C., & Dunkler, D. (2018). Variable selection–a review and recommendations for the practicing statistician. Biometrical Journal, 60(3), 431-449.

• Heinze, G., & Dunkler, D. (2017). Five myths about variable selection. Transplant International, 30(1), 6-10. (https://onlinelibrary.wiley.com/doi/epdf/10.1111/tri.12895)