I have a doubt related to using AIC for model selection - in which case it may not recommend the true best predictive model (based on my understanding). I understand AIC has 2 terms - goodness of fit (which can be obtained my finding error on model training dataset) and complexity term (2*no. of parameters in model). I discuss the case below:

I have 2 models - 1st model is non-parametric model that interpolates each data point and overfits, hence no. of parameters (K) is same as no. of instances in train set (n i.e. sample size, lets say 500). SSE on train set (goodness of fit) is very good, say 1e-4 (as it overfits the trained dataset). Its calculated AIC value (using formula, n*ln(SSE/n)+2K) would be -6712. The second model is a parametric model (2nd order polynomial regressive model) with 6 parameters. Its goodness of fit is not as good as non-parametric model with SSE being 1e-1. Its calculated AIC would be -4246.

Based on delta(AIC), we would select model 1, but we know that the 1st model overfits the data and hence would not generalize well on new data.

So, how do we use AIC in such cases when a model overfits data but the complexity term does not penalize it well enough to reject it among others. Does this case imply we can not use AIC for differentiating between parametric and non-parametric models?

I have a doubt related to using AIC for model selection - in which case it may not recommend the true best predictive model (based on my understanding). I understand AIC has 2 terms - goodness of fit (which can be obtained my finding error on model training dataset) and complexity term (2*no. of parameters in model). I discuss the case below:

I have 2 models - 1st model is non-parametric model that interpolates each data point and overfits, hence no. of parameters (K) is same as sample size (lets say 500). SSE on train set (goodness of fit) is very good, say 1e-4 (as it overfits the trained dataset). Its calculated AIC value (using formula, n*ln(SSE/n)+2K) would be -6712. The second model is a parametric model (2nd order polynomial regressive model) with 6 parameters. Its goodness of fit is not as good as non-parametric model with SSE being 1e-1. Its calculated AIC would be -4246.

Based on delta(AIC), we would select model 1, but we know that the 1st model overfits the data and hence would not generalize well on new data.

So, how do we use AIC in such cases when a model overfits data but the complexity term does not penalize it well enough to reject it among others. Does this case imply we can not use AIC for differentiating between parametric and non-parametric models?

I have a doubt related to using AIC for model selection - in which case it may not recommend the true best predictive model (based on my understanding). I understand AIC has 2 terms - goodness of fit (which can be obtained my finding error on model training dataset) and complexity term (2*no. of parameters in model). I discuss the case below:

I have 2 models - 1st model is non-parametric model that interpolates each data point and overfits, hence no. of parameters (K) is same as no. of instances in train set (n i.e. sample size, lets say 500). SSE on trainset (goodness of fit) is very good, say 1e-4 (as it overfits the trained dataset). Its calculated AIC value (using formula, n*ln(SSE/n)+2K) would be -6712. The 2nd model is a parametric model (2nd order polynomial regressive model) with 6 parameters. Its gof is not as good as non-parametric model with SSE being 1e-1. Its calculated AIC would be -4246.

Based on delta(AIC), we would select model 1, but we know that the 1st model overfits the data and hence would not generalize well on new data.

So, how do we use AIC in such cases when a model overfits data but the complexity term does not penalize it well enough to reject it among others. Does this case imply we can not use AIC for differentiating between parametric and non-parametric models? Thanks

I have a doubt related to using AIC for model selection - in which case it may not recommend the true best predictive model (based on my understanding). I understand AIC has 2 terms - goodness of fit (which can be obtained my finding error on model training dataset) and complexity term (2*no. of parameters in model). I discuss the case below:

I have 2 models - 1st model is non-parametric model that interpolates each data point and overfits, hence no. of parameters (K) is same as no. of instances in train set (n i.e. sample size, lets say 500). SSE on train set (goodness of fit) is very good, say 1e-4 (as it overfits the trained dataset). Its calculated AIC value (using formula, n*ln(SSE/n)+2K) would be -6712. The second model is a parametric model (2nd order polynomial regressive model) with 6 parameters. Its goodness of fit is not as good as non-parametric model with SSE being 1e-1. Its calculated AIC would be -4246.

Based on delta(AIC), we would select model 1, but we know that the 1st model overfits the data and hence would not generalize well on new data.

So, how do we use AIC in such cases when a model overfits data but the complexity term does not penalize it well enough to reject it among others. Does this case imply we can not use AIC for differentiating between parametric and non-parametric models?