I am analyzing a data set to identify a useful predictive model. I used a model selection approach (Burnham & Anderson, 2002) referring to AIC to select the most useful model for prediction. However, one of the included predictor variables did not reach a statistical significance (P < 0.5).
I am aware that a model selection using an information criterion and the significance testing are two different things. However, I am struggling to justify that it is okay to include non-significant predictor variables in a model.
I would like to know if there are any references (preferably, peer-reviewed research articles) that I can read into and cite for such a case where a certain coefficient did not reach the statistical significance but was included in the model (with the smallest AIC value among all candidates) when taking a model selection approach.
Details of the background
I aim to identify the best statistical model including predictor variables that best predicts learners' behavior (while avoiding overfitting). With this model, I would like to predict learners' performance (on a certain task; for example, GPA) by considering the characteristics of situations based on the included predictor variables.
My general model comparison approach is as follows:
model1 <- lm (y~x1+x2+x3, data = dat)
model2 <- lm (y~x1+x3, data = dat)
model3 <- lm (y~x1+x2+x4, data = dat)
AIC(model1, model2, model3)
So, let's say the model 3 was selected as
the best model as indicated by the smallest AIC value. However, the included
x4 is (p = .10) so not reaching the 'significant' level at p < .5.
How should I interpret this
x4 if useful for prediction as AIC suggested but not
" statistically significant"? I was asked to discuss my interpretation of this by referring to articles.
Lastly, by 'useful predictive model,' I mean a statistical model that includes a combination of predictor variables that are useful to make predictions of the learner performance (here GPA). Although I am assuming that the prediction may provide a quite rough estimation on that given the limited number of predictors), it would be cool if people can estimate a specific student's GPA scores based on identified predictors. Additionally, I would like to highlight the influence of each predictor on GPA in general too. (one might say these are two different aims, but I also feel weird to run both hypothesis-testing analysis and model selection at the same time in the same paper, as they, in general, produce quite similar results). Because of our practical reason, I cannot use a cross-validation approach to select predictors.
Thank you very much for your help.
I am aware of the discussion in CV: Why applying model selection using AIC gives me non-significant p-values for the variables and Why p-values are not significant even though AIC values improved a lot in model selection using GAM mix modelling and beta regression). But, I am still struggling to find references that I can cite. It does not have to be a statistics paper (although it's preferable); other research papers discussing the inclusion of non-significant predictors in the optimal model when conducting a model selection approach would also be of great help!
I would appreciate it if I could hear any specific references that I can cite for my research papers. Please and thank you.