Tweeted twitter.com/StackStats/status/959064077636112385
4 added 20 characters in body
source | link

Consider elastic net regression with glmnet-like parametrization of the loss function$$\mathcal L = \frac{1}{2n}\big\lVert y - \beta_0-X\beta\big\rVert^2 + \lambda\big(\alpha\lVert \beta\rVert_1 + (1-\alpha) \lVert \beta\rVert^2_2/2\big).$$ I have a data set with $n\ll p$ (44 and 3000 respectively) and I am using repeated 11-fold cross-validation to select the optimal regularization parameters $\alpha$ and $\lambda$. Normally I would use squared error as the performance metric on the test set, e.g. this R-squared-like metric: $$L_\text{test} = 1-\frac{\lVert y_\text{test} - \hat\beta_0 - X_\text{test}\hat\beta\rVert^2}{\lVert y_\text{test} - \hat\beta_0\rVert^2},$$ but this time I also tried using correlation metric (note that for the un-regularized OLS regression minimizing the squared error loss is equivalent to maximizing the correlation): $$L_\text{test}=\operatorname{corr}(y_\text{test}, X_\text{test}\hat\beta).$$

It's clear that these two performance metrics are not exactly equivalent, but weirdly, they disagree rather strongly:

Elastic net cross-validation

Note in particular what happens at small alphas, e.g. $\alpha=.2$ (green line): maximum test-set correlation is achieved when test-set $R^2$ drops quite substantially compared to its maximum. In general, for any given $\alpha$, correlation seems to be maximized at larger $\lambda$ than squared error.

Why does it happen and how to deal with it? Which criterion should be preferred? Has anybody encountered this effect?

Consider elastic net regression with glmnet-like parametrization $$\mathcal L = \frac{1}{2n}\big\lVert y - \beta_0-X\beta\big\rVert^2 + \lambda\big(\alpha\lVert \beta\rVert_1 + (1-\alpha) \lVert \beta\rVert^2_2/2\big).$$ I have a data set with $n\ll p$ (44 and 3000 respectively) and I am using repeated 11-fold cross-validation to select the optimal regularization parameters $\alpha$ and $\lambda$. Normally I would use squared error as the performance metric on the test set, e.g. this R-squared-like metric: $$L_\text{test} = 1-\frac{\lVert y_\text{test} - \hat\beta_0 - X_\text{test}\hat\beta\rVert^2}{\lVert y_\text{test} - \hat\beta_0\rVert^2},$$ but this time I also tried using correlation metric (note that for the un-regularized OLS regression minimizing the squared error loss is equivalent to maximizing the correlation): $$L_\text{test}=\operatorname{corr}(y_\text{test}, X_\text{test}\hat\beta).$$

It's clear that these two performance metrics are not exactly equivalent, but weirdly, they disagree rather strongly:

Elastic net cross-validation

Note in particular what happens at small alphas, e.g. $\alpha=.2$ (green line): maximum test-set correlation is achieved when test-set $R^2$ drops quite substantially compared to its maximum. In general, for any given $\alpha$, correlation seems to be maximized at larger $\lambda$ than squared error.

Why does it happen and how to deal with it? Which criterion should be preferred? Has anybody encountered this effect?

Consider elastic net regression with glmnet-like parametrization of the loss function$$\mathcal L = \frac{1}{2n}\big\lVert y - \beta_0-X\beta\big\rVert^2 + \lambda\big(\alpha\lVert \beta\rVert_1 + (1-\alpha) \lVert \beta\rVert^2_2/2\big).$$ I have a data set with $n\ll p$ (44 and 3000 respectively) and I am using repeated 11-fold cross-validation to select the optimal regularization parameters $\alpha$ and $\lambda$. Normally I would use squared error as the performance metric on the test set, e.g. this R-squared-like metric: $$L_\text{test} = 1-\frac{\lVert y_\text{test} - \hat\beta_0 - X_\text{test}\hat\beta\rVert^2}{\lVert y_\text{test} - \hat\beta_0\rVert^2},$$ but this time I also tried using correlation metric (note that for the un-regularized OLS regression minimizing the squared error loss is equivalent to maximizing the correlation): $$L_\text{test}=\operatorname{corr}(y_\text{test}, X_\text{test}\hat\beta).$$

It's clear that these two performance metrics are not exactly equivalent, but weirdly, they disagree rather strongly:

Elastic net cross-validation

Note in particular what happens at small alphas, e.g. $\alpha=.2$ (green line): maximum test-set correlation is achieved when test-set $R^2$ drops quite substantially compared to its maximum. In general, for any given $\alpha$, correlation seems to be maximized at larger $\lambda$ than squared error.

Why does it happen and how to deal with it? Which criterion should be preferred? Has anybody encountered this effect?

3 added 56 characters in body
source | link

Consider elastic net regression with glmnet-like parametrization $$\mathcal L = \frac{1}{2n}\big\lVert y - \beta_0-X\beta\big\rVert^2 + \lambda\big(\alpha\lVert \beta\rVert_1 + (1-\alpha) \lVert \beta\rVert^2_2/2\big).$$ I have a data set with $n\ll p$ (44 and 3000 respectively) and I am using repeated 11-fold cross-validation to select the optimal regularization parameters $\alpha$ and $\lambda$. Normally I would use squared error as the performance metric on the test set, e.g. this R-squared-like metric: $$L_\text{test} = 1-\frac{\lVert y_\text{test} - \hat\beta_0 - X_\text{test}\hat\beta\rVert^2}{\lVert y_\text{test} - \hat\beta_0\rVert^2},$$ but this time I also tried using correlation metric (note that for the un-regularized OLS regression minimizing the squared error loss is equivalent to maximizing the correlation): $$L_\text{test}=\operatorname{corr}(y_\text{test}, X_\text{test}\hat\beta).$$

It's clear that these two performance metrics are not exactly equivalent, but weirdly, they disagree rather strongly:

Elastic net cross-validation

Note in particular what happens at small alphas, e.g. $\alpha=.2$ (green line): maximum test-set correlation is achieved when test-set $R^2$ drops quite substantially compared to its maximum. In general, for any given $\alpha$, correlation seems to be maximized at larger $\lambda$ than squared error.

Why does it happen and how to interpretdeal with it? Which criterion should be preferred? Has anybody encountered this effect?

Consider elastic net regression with glmnet-like parametrization $$\mathcal L = \frac{1}{2n}\big\lVert y - \beta_0-X\beta\big\rVert^2 + \lambda\big(\alpha\lVert \beta\rVert_1 + (1-\alpha) \lVert \beta\rVert^2_2/2\big).$$ I have a data set with $n\ll p$ (44 and 3000 respectively) and I am using repeated 11-fold cross-validation to select the optimal regularization parameters $\alpha$ and $\lambda$. Normally I would use squared error as the performance metric on the test set, e.g. this R-squared-like metric: $$L_\text{test} = 1-\frac{\lVert y_\text{test} - \hat\beta_0 - X_\text{test}\hat\beta\rVert^2}{\lVert y_\text{test} - \hat\beta_0\rVert^2},$$ but this time I also tried using correlation metric (note that for the un-regularized OLS regression minimizing the squared error loss is equivalent to maximizing the correlation): $$L_\text{test}=\operatorname{corr}(y_\text{test}, X_\text{test}\hat\beta).$$

It's clear that these two performance metrics are not exactly equivalent, but weirdly, they disagree rather strongly:

Elastic net cross-validation

Note in particular what happens at $\alpha=.2$ (green line): maximum test-set correlation is achieved when test-set $R^2$ drops quite substantially compared to its maximum. In general, for any given $\alpha$, correlation seems to be maximized at larger $\lambda$ than squared error.

Why does it happen and how to interpret it? Has anybody encountered this effect?

Consider elastic net regression with glmnet-like parametrization $$\mathcal L = \frac{1}{2n}\big\lVert y - \beta_0-X\beta\big\rVert^2 + \lambda\big(\alpha\lVert \beta\rVert_1 + (1-\alpha) \lVert \beta\rVert^2_2/2\big).$$ I have a data set with $n\ll p$ (44 and 3000 respectively) and I am using repeated 11-fold cross-validation to select the optimal regularization parameters $\alpha$ and $\lambda$. Normally I would use squared error as the performance metric on the test set, e.g. this R-squared-like metric: $$L_\text{test} = 1-\frac{\lVert y_\text{test} - \hat\beta_0 - X_\text{test}\hat\beta\rVert^2}{\lVert y_\text{test} - \hat\beta_0\rVert^2},$$ but this time I also tried using correlation metric (note that for the un-regularized OLS regression minimizing the squared error loss is equivalent to maximizing the correlation): $$L_\text{test}=\operatorname{corr}(y_\text{test}, X_\text{test}\hat\beta).$$

It's clear that these two performance metrics are not exactly equivalent, but weirdly, they disagree rather strongly:

Elastic net cross-validation

Note in particular what happens at small alphas, e.g. $\alpha=.2$ (green line): maximum test-set correlation is achieved when test-set $R^2$ drops quite substantially compared to its maximum. In general, for any given $\alpha$, correlation seems to be maximized at larger $\lambda$ than squared error.

Why does it happen and how to deal with it? Which criterion should be preferred? Has anybody encountered this effect?

2 edited tags
| link
1
source | link