1
$\begingroup$

I am using GLM for modeling the binary data. For the predictive variables I am using different number of Zernike Basis functions. To test how many Zernike basis functions I need, I am using AICc/BIC to choose the best model. For example, I use k={1,2,3,...,20} basis functions and AICc chooses the model which has k=5 basis functions, say X1,X2,X3,X4,X5. However, using GLM shows me that not all these 5 basis functions are significant, i.e., the p-value>0.05. For example the coefficients of X2 and X4 are not significant. What should I do now? In the end when I want to find one model for my data should I use the model which only has X1,X3,X5? or Should I not care about the p-value of the coefficients and include X1,X2,X3,X4,X5?

$\endgroup$
2
  • $\begingroup$ I am not familiar with Zernike Basis functions, but... Are the different numbers of the Zernike Basis functions your different explanatory variables (i.e. this is a multiple regression)? Or is the number of Zernike Basis functions your single explanatory variable, and the numbers are just the levels (values) of this single variable? If they are different levels, then it makes sense that not all levels are significantly different from the baseline (but the variable itself should be kept in the model, as suggested by AIC). $\endgroup$ – Tilen Feb 15 '17 at 22:05
  • $\begingroup$ Each Zernike Basis function can be considered as one explanatory variable. $\endgroup$ – Mina Feb 16 '17 at 19:18
5
$\begingroup$

The AIC, the BIC and the $p$-values all address different questions. For feature selection (variable selection, model selection), only the former two are relevant. See e.g. Rob J. Hyndman's blog posts "Statistical tests for variable selection" and "Facts and fallacies of the AIC".

AIC is best suited for forecasting purposes as it targets maximization of the likelihood of a new observation from the same data generating process (under normality of errors, maximization of likelihood is equivalent to minimization of square loss). BIC is a consistent model selector and as such may be useful in finding which model among a given set is the true model (if the true model is in the set). So depending on your goal (prediction or identification of the data generating process), AIC or BIC may be relevant for you.

In this particular application I suspect that AIC is more relevant as the models you are considering probably are not thought to contain any true model among them.

Furthermore, model averaging as suggested by @Björn might be a good (better) alternative solution.

$\endgroup$
5
  • $\begingroup$ Thanks for your answer. Could you please tell me when I should use the p-value then? $\endgroup$ – Mina Feb 15 '17 at 20:56
  • $\begingroup$ $p$-values are used for hypotheses testing. However, model selection is a different task. The 5% significance level typically used in hypothesis tests is not a relevant level when forecasting; AIC-based selection kicks out variables at around 15% level (so you may notice that your variables are significant at 15% level but not necessarily 5% level). You may check older posts on model selection to find a more detailed explanation. $\endgroup$ – Richard Hardy Feb 16 '17 at 6:13
  • $\begingroup$ For the downvoter: I would appreciate some explanation what appears to be the problem. $\endgroup$ – Richard Hardy Feb 16 '17 at 6:13
  • $\begingroup$ @RichardHardy I saw the problem in the unquestioning acceptance of the OP's thinking that he actually should select a single model, if he is going to do prediction. Very rarely is that a sensible thing to do. $\endgroup$ – Björn Feb 16 '17 at 7:12
  • $\begingroup$ @Björn, that's fine, I agree model averaging can be better (edited the post to include this). +1 to you. $\endgroup$ – Richard Hardy Feb 16 '17 at 7:26
2
$\begingroup$

For a prediction situation AIC is more relevant than BIC or p-values. However, to pick a single model on the basis of any of these and then to analyze the data as if no model selection had been done, is likely inappropriate (unless one candidate model is so vastly better than the others as to remove all model uncertainty). Model averaging is usually more appropriate, if the goal is prediction.

$\endgroup$
7
  • $\begingroup$ I actually don't want to do prediction. My goal is mostly explanation of my data. In that case I guess BIC is better, though I cannot claim that the model is definitely among the models that I am using. In other words, because I am modeling the neural activities, the chance that the model is among those models that I have chosen is almost zero. $\endgroup$ – Mina Feb 16 '17 at 19:26
  • $\begingroup$ Also one more thing. When I am plotting the AIC or BIC as a function of different models the changes in the value of both of them are not much. They are in the range of 600-650 and its in the favor of less complicated model. But since I'm using the surrogate data I know that the more complicated model gives me the correct one. Then how can I explain that with the higher AIC and BIC? $\endgroup$ – Mina Feb 16 '17 at 19:27
  • $\begingroup$ @Mina, BIC and AIC have their nice properties asymptotically. A particular finite sample might not be large enough for BIC to pick the true model. $\endgroup$ – Richard Hardy Feb 16 '17 at 19:54
  • $\begingroup$ @Richard, I have more than 1 million samples. Wouldn't that be enough? $\endgroup$ – Mina Feb 16 '17 at 20:14
  • $\begingroup$ @Mina, Oh, then it must be something else... $\endgroup$ – Richard Hardy Feb 16 '17 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.