1
$\begingroup$

I've tried searching for answers on this site, but I've not found a clear answer. I have a dataset with around 9471 observations and 10 attributes. I wish to fit a Ridge Regression model, and figure out what the optimal Alpha I should finally end up using in the model. I could try a range of values from 0 to 10.

What metric should I use when I compare models of different alpha values? How do I know which is better, i.e which model has a more optimal alpha? How should I go about finding this optimal alpha?

$\endgroup$
  • 7
    $\begingroup$ Keyword: cross validation. $\endgroup$ – Richard Hardy Feb 23 '18 at 18:37
  • $\begingroup$ Which metric is used in the cross-val method, to compare the models? $\endgroup$ – ricksanchez Feb 23 '18 at 18:59
  • 4
    $\begingroup$ Use the same metric that you use to evaluate the final model. $\endgroup$ – Richard Hardy Feb 23 '18 at 19:35
  • $\begingroup$ Yeah so if I'm not wrong, we use R2 to find how good the model is. But in the case of Ridge, as we increase the Alpha value, we're essentially reducing the overfitting of the model. Therefore with each increase of alpha, the model fit will be lesser, and the R2 value will keep decreasing. So wouldn't it be pointless then to use R2 value or something like Sum of Squared Errors? $\endgroup$ – ricksanchez Feb 24 '18 at 0:27
  • 2
    $\begingroup$ In cross validation you evaluate out-of-sample predictive performance on a test fold, so you do not use $R^2$. $\endgroup$ – Richard Hardy Feb 24 '18 at 13:16
1
$\begingroup$

As Richard mentions, you can use cross validation. Another option which does not require that you do a K-fold cross-validation is generalized cross validation. See e.g., the smooth.spline function in R or the mgcv package in R (and the book by Simon Wood). Particularly, see the H argument in the gam function in the mgcv package.

$\endgroup$
0
$\begingroup$

This presentation gives a good overview of the strategies you can use to tune the ridge penalization factor. Information criteria such as AIC or BIC can also be used to tune regularisation, and they each asymptotically approximate a particular form of cross validation:

  • AIC approximately minimizes the prediction error and is asymptotically equivalent to leave-1-out cross-validation (LOOCV) (Stone 1977); LOOCV in turn is approximated by generalized cross validation (GCV), but LOOCV should always be better than GCV. AIC is not consistent though, which means that even with a very large amount of data ($n$ going to infinity) and if the true model is among the candidate models, the probability of selecting the true model based on the AIC criterion would not approach 1.
  • BIC is an approximation to the integrated marginal likelihood $P(D|M,A) (D=Data, M=model, A=assumptions)$, which under a flat prior is equivalent to seeking the model that maximizes $P(M|D,A)$. Its advantage is that it is consistent, which means that with a very large amount of data ($n$ going to infinity) and if the true model is among the candidate models, the probability of selecting the true model based on the BIC criterion would approach 1. This would come at a slight cost to prediction performance though if $n$ were small. BIC is also equivalent to leave-k-out cross-validation (LKOCV) where $k=n[1−1/(log(n)−1)]$, with $n=$ sample size (Shao 1997). There is many different versions of the BIC though which come down to making different approximations of the marginal likelihood or assuming different priors. E.g. instead of using a prior uniform of all possible models as in the original BIC, EBIC uses a prior uniform of models of fixed size (Chen & Chen 2008) whereas BICq uses a Bernouilli distribution specifying the prior probability for each parameter to be included.

Note that the LOOCV error can also be calculated analytically from the residuals and the diagonal of the hat matrix, without having to actually carry out any cross validation. This would always be an alternative to the AIC as an asymptotic approximation of the LOOCV error.

Instead of using ridge regression with a fixed penalization of your coefficients it would be better to use iterated adaptive ridge regression though, as the latter approximates L0 penalized regression (aka best subset), where the log likelihood of a GLM model is penalized based on a multiple of the number of nonzero coefficients in the model - see Frommlet & Noel 2016. This has the advantage that you can then directly optimize the AIC or BIC objective directly, by choosing the appropriate regularization level $lambda$, as a $lambda = 2$ then optimizes AIC and $lambda=log(n)$ would optimize BIC. This is what is done in the l0ara R package. To me this make more sense than first optimizing your coefficients under one objective (e.g. ridge), only to then tune the regularization level of that model based on some other criterion (e.g. minimizing cross validation prediction error, AIC or BIC). The other advantage of L0-penalized regression over ridge regression or LASSO regression is that it gives you unbiased estimates, so you can get rid of the bias-variance tradeoff that plagues most penalized regression approaches. Plus like ridge it also works for high-dimensional problems with $p>n$.

References

Stone M. (1977) An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. Journal of the Royal Statistical Society Series B. 39, 44–7.

Shao J. (1997) An asymptotic theory for linear model selection. Statistica Sinica 7, 221-242.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.