2
$\begingroup$

I am analyzing a data set to identify a useful predictive model. I used a model selection approach (Burnham & Anderson, 2002) referring to AIC to select the most useful model for prediction. However, one of the included predictor variables did not reach a statistical significance (P < 0.5).

I am aware that a model selection using an information criterion and the significance testing are two different things. However, I am struggling to justify that it is okay to include non-significant predictor variables in a model.

I would like to know if there are any references (preferably, peer-reviewed research articles) that I can read into and cite for such a case where a certain coefficient did not reach the statistical significance but was included in the model (with the smallest AIC value among all candidates) when taking a model selection approach.

Details of the background

I aim to identify the best statistical model including predictor variables that best predicts learners' behavior (while avoiding overfitting). With this model, I would like to predict learners' performance (on a certain task; for example, GPA) by considering the characteristics of situations based on the included predictor variables.

My general model comparison approach is as follows:

model1 <- lm (y~x1+x2+x3, data = dat)
model2 <- lm (y~x1+x3, data = dat)
model3 <- lm (y~x1+x2+x4, data = dat)
AIC(model1, model2, model3)

So, let's say the model 3 was selected as the best model as indicated by the smallest AIC value. However, the included x4 is (p = .10) so not reaching the 'significant' level at p < .5.

How should I interpret this x4? x4 if useful for prediction as AIC suggested but not " statistically significant"? I was asked to discuss my interpretation of this by referring to articles.

Lastly, by 'useful predictive model,' I mean a statistical model that includes a combination of predictor variables that are useful to make predictions of the learner performance (here GPA). Although I am assuming that the prediction may provide a quite rough estimation on that given the limited number of predictors), it would be cool if people can estimate a specific student's GPA scores based on identified predictors. Additionally, I would like to highlight the influence of each predictor on GPA in general too. (one might say these are two different aims, but I also feel weird to run both hypothesis-testing analysis and model selection at the same time in the same paper, as they, in general, produce quite similar results). Because of our practical reason, I cannot use a cross-validation approach to select predictors.

Thank you very much for your help.

added:

I am aware of the discussion in CV: Why applying model selection using AIC gives me non-significant p-values for the variables and Why p-values are not significant even though AIC values improved a lot in model selection using GAM mix modelling and beta regression). But, I am still struggling to find references that I can cite. It does not have to be a statistics paper (although it's preferable); other research papers discussing the inclusion of non-significant predictors in the optimal model when conducting a model selection approach would also be of great help!

I would appreciate it if I could hear any specific references that I can cite for my research papers. Please and thank you.

Improve this question
$\endgroup$
7
  • $\begingroup$ Are you looking for your own edification, or as justification to a reviewer? $\endgroup$
    – Demetri Pananos
    Commented Jun 21, 2020 at 1:08
  • 2
    $\begingroup$ You have to define what "useful predictive model" means to you precisely or clarify what you want to do with this model. Also you need to clarify if you are adjusting the p-values for the uncertainty associated with your model selection or are they just the regular ones from the min-AIC selected model? $\endgroup$
    – dimitriy
    Commented Jun 21, 2020 at 1:10
  • $\begingroup$ Thank you for asking questions to clarify my intention! I am looking for references as justification for a reviewer. And, the regular ones from the min-AIC selected model; I am comparing several statistical models by referring to AIC to identify the model with smallest AIC. I hope this helps. $\endgroup$
    – user8460166
    Commented Jun 21, 2020 at 12:09
  • 2
    $\begingroup$ Perhaps these will be helpful: Hyndman & Kostenko "Forecasting without significance tests?" (2008) and Hyndman "Why I don't like statistical tests". $\endgroup$
    – Richard Hardy
    Commented Jun 21, 2020 at 12:48
  • $\begingroup$ Thank you very much, @RichardHardy! Forecasting without significance tests?" seems to be the kind of articles that I have been looking for! Would you mind posting that as one answer to my question? I would also like to continue to keep this question open so as to learn many more references that I and other people can read and cite. :) $\endgroup$
    – user8460166
    Commented Jun 21, 2020 at 14:27

1 Answer 1

Reset to default
2
$\begingroup$

Perhaps these will be helpful:

  1. Hyndman & Kostenko "Forecasting without significance tests?" (2008)
  2. Hyndman "Why I don't like statistical tests".
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.