1
$\begingroup$

I still have a doubt related to using AIC to compare parametric models with non parametric models. In my previous question AIC can recommend an overfitting model?, I received some comments/answers that seemed to clear the air, but I strongly feel AIC could not be used for comparing parametric model (eg. 2nd order polynomial) with a non parametric model (eg. gaussian process regression or GPR). Atleast for an example, lets keep the comparison between a 'polynomial' parametric model & 'GPR' non parametric model; without generalizing to any parametric & non parametric model for clarity. Some of my questions why I feel so (comparison between polynomial regression and GPR using AIC is not justified) are as follows:

  1. GPR does not have a predefined model form unlike 2nd order polynomial model. So, what are the parameters in GPR model that should be counted as parameters for AIC - are they the noise and parameters of the covariance function? Do not these parameters differ in their utlities? - meaning coefficients of 2nd order polynomial define the model form completely, while coefficients of covariance function and noise in GPR define the covariance functions that help in obtaining the posterior probability.
  2. Since GPR models the noise its goodness of fit will be far superior to 2nd order polynomial (if the data generating process is not a 2nd order polynomial function obviously). So, does it not make GPR always favorable than polynomial regression (as gof of GPR becomes much lesser than polynomial!).

I feel this comparison using AIC is something analogous to comparing apples to oranges. Maybe theoretically we can compare, but I dont think it makes sense. (Also, kindly let me know if my understanding is correct - AIC can be used to compare models whose parameters are determined by MLE estimates (or equivalent like LS techniques), so parameters of covariance function of GPR are determined by MLE estimate, I hope that this satisfies the condition of applying AIC on GPR models)

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ What is AOC for nonparametric models? How do you define it? $\endgroup$
    – Aksakal
    Commented Jun 9, 2020 at 17:17
  • $\begingroup$ @Aksakal: If we are talking about GPR I think we can treat $k$ as the number of effective hyper-parameters of the model. We already have the likelihood anyway. Granted it is a leap of faith involved where we assume the number of hyper-parameters equates the number of estimated parameters in a standard parametric model. $\endgroup$
    – usεr11852
    Commented Jun 9, 2020 at 18:02
  • $\begingroup$ there's your problem: hyperparameters cannot be treated like parameters, because they are not estimated the same. they're "tuned" or whatever other word you want to use for the process, but estimated wouldn't be the right one in spirit and letter $\endgroup$
    – Aksakal
    Commented Jun 9, 2020 at 18:27
  • $\begingroup$ @Aksakal: Agreed, "it is a leap of faith". I have seen it done. I do not fully understand what you mean by "they are not estimated the same"; they are estimated exactly the same way in my mind. We stuck them in optimiser and get out the optimal values in terms of log-likelihood. $\endgroup$
    – usεr11852
    Commented Jun 9, 2020 at 22:32
  • $\begingroup$ What do you mean by hyperparameters? Number of layers in NN is one example, the regularization constant is another. If you think they are optimized the same way as model parameters then we have a bigger issue $\endgroup$
    – Aksakal
    Commented Jun 9, 2020 at 23:44

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.