Tweeted twitter.com/StackStats/status/959064077636112385 occurred Feb 1 '18 at 14:01 4 added 20 characters in body edited Feb 1 '18 at 13:54 amoeba 64.9k1616 gold badges218218 silver badges273273 bronze badges Consider elastic net regression with glmnet-like parametrization of the loss function$$\mathcal L = \frac{1}{2n}\big\lVert y - \beta_0-X\beta\big\rVert^2 + \lambda\big(\alpha\lVert \beta\rVert_1 + (1-\alpha) \lVert \beta\rVert^2_2/2\big).$$ I have a data set with $$n\ll p$$ (44 and 3000 respectively) and I am using repeated 11-fold cross-validation to select the optimal regularization parameters $$\alpha$$ and $$\lambda$$. Normally I would use squared error as the performance metric on the test set, e.g. this R-squared-like metric: $$L_\text{test} = 1-\frac{\lVert y_\text{test} - \hat\beta_0 - X_\text{test}\hat\beta\rVert^2}{\lVert y_\text{test} - \hat\beta_0\rVert^2},$$ but this time I also tried using correlation metric (note that for the un-regularized OLS regression minimizing the squared error loss is equivalent to maximizing the correlation): $$L_\text{test}=\operatorname{corr}(y_\text{test}, X_\text{test}\hat\beta).$$ It's clear that these two performance metrics are not exactly equivalent, but weirdly, they disagree rather strongly: Note in particular what happens at small alphas, e.g. $$\alpha=.2$$ (green line): maximum test-set correlation is achieved when test-set $$R^2$$ drops quite substantially compared to its maximum. In general, for any given $$\alpha$$, correlation seems to be maximized at larger $$\lambda$$ than squared error. Why does it happen and how to deal with it? Which criterion should be preferred? Has anybody encountered this effect? Consider elastic net regression with glmnet-like parametrization $$\mathcal L = \frac{1}{2n}\big\lVert y - \beta_0-X\beta\big\rVert^2 + \lambda\big(\alpha\lVert \beta\rVert_1 + (1-\alpha) \lVert \beta\rVert^2_2/2\big).$$ I have a data set with $$n\ll p$$ (44 and 3000 respectively) and I am using repeated 11-fold cross-validation to select the optimal regularization parameters $$\alpha$$ and $$\lambda$$. Normally I would use squared error as the performance metric on the test set, e.g. this R-squared-like metric: $$L_\text{test} = 1-\frac{\lVert y_\text{test} - \hat\beta_0 - X_\text{test}\hat\beta\rVert^2}{\lVert y_\text{test} - \hat\beta_0\rVert^2},$$ but this time I also tried using correlation metric (note that for the un-regularized OLS regression minimizing the squared error loss is equivalent to maximizing the correlation): $$L_\text{test}=\operatorname{corr}(y_\text{test}, X_\text{test}\hat\beta).$$ It's clear that these two performance metrics are not exactly equivalent, but weirdly, they disagree rather strongly: Note in particular what happens at small alphas, e.g. $$\alpha=.2$$ (green line): maximum test-set correlation is achieved when test-set $$R^2$$ drops quite substantially compared to its maximum. In general, for any given $$\alpha$$, correlation seems to be maximized at larger $$\lambda$$ than squared error. Why does it happen and how to deal with it? Which criterion should be preferred? Has anybody encountered this effect? Consider elastic net regression with glmnet-like parametrization of the loss function$$\mathcal L = \frac{1}{2n}\big\lVert y - \beta_0-X\beta\big\rVert^2 + \lambda\big(\alpha\lVert \beta\rVert_1 + (1-\alpha) \lVert \beta\rVert^2_2/2\big).$$ I have a data set with $$n\ll p$$ (44 and 3000 respectively) and I am using repeated 11-fold cross-validation to select the optimal regularization parameters $$\alpha$$ and $$\lambda$$. Normally I would use squared error as the performance metric on the test set, e.g. this R-squared-like metric: $$L_\text{test} = 1-\frac{\lVert y_\text{test} - \hat\beta_0 - X_\text{test}\hat\beta\rVert^2}{\lVert y_\text{test} - \hat\beta_0\rVert^2},$$ but this time I also tried using correlation metric (note that for the un-regularized OLS regression minimizing the squared error loss is equivalent to maximizing the correlation): $$L_\text{test}=\operatorname{corr}(y_\text{test}, X_\text{test}\hat\beta).$$ It's clear that these two performance metrics are not exactly equivalent, but weirdly, they disagree rather strongly: Note in particular what happens at small alphas, e.g. $$\alpha=.2$$ (green line): maximum test-set correlation is achieved when test-set $$R^2$$ drops quite substantially compared to its maximum. In general, for any given $$\alpha$$, correlation seems to be maximized at larger $$\lambda$$ than squared error. Why does it happen and how to deal with it? Which criterion should be preferred? Has anybody encountered this effect? 3 added 56 characters in body edited Feb 1 '18 at 12:06 amoeba 64.9k1616 gold badges218218 silver badges273273 bronze badges Consider elastic net regression with glmnet-like parametrization $$\mathcal L = \frac{1}{2n}\big\lVert y - \beta_0-X\beta\big\rVert^2 + \lambda\big(\alpha\lVert \beta\rVert_1 + (1-\alpha) \lVert \beta\rVert^2_2/2\big).$$ I have a data set with $$n\ll p$$ (44 and 3000 respectively) and I am using repeated 11-fold cross-validation to select the optimal regularization parameters $$\alpha$$ and $$\lambda$$. Normally I would use squared error as the performance metric on the test set, e.g. this R-squared-like metric: $$L_\text{test} = 1-\frac{\lVert y_\text{test} - \hat\beta_0 - X_\text{test}\hat\beta\rVert^2}{\lVert y_\text{test} - \hat\beta_0\rVert^2},$$ but this time I also tried using correlation metric (note that for the un-regularized OLS regression minimizing the squared error loss is equivalent to maximizing the correlation): $$L_\text{test}=\operatorname{corr}(y_\text{test}, X_\text{test}\hat\beta).$$ It's clear that these two performance metrics are not exactly equivalent, but weirdly, they disagree rather strongly: Note in particular what happens at small alphas, e.g. $$\alpha=.2$$ (green line): maximum test-set correlation is achieved when test-set $$R^2$$ drops quite substantially compared to its maximum. In general, for any given $$\alpha$$, correlation seems to be maximized at larger $$\lambda$$ than squared error. Why does it happen and how to interpretdeal with it? Which criterion should be preferred? Has anybody encountered this effect? Consider elastic net regression with glmnet-like parametrization $$\mathcal L = \frac{1}{2n}\big\lVert y - \beta_0-X\beta\big\rVert^2 + \lambda\big(\alpha\lVert \beta\rVert_1 + (1-\alpha) \lVert \beta\rVert^2_2/2\big).$$ I have a data set with $$n\ll p$$ (44 and 3000 respectively) and I am using repeated 11-fold cross-validation to select the optimal regularization parameters $$\alpha$$ and $$\lambda$$. Normally I would use squared error as the performance metric on the test set, e.g. this R-squared-like metric: $$L_\text{test} = 1-\frac{\lVert y_\text{test} - \hat\beta_0 - X_\text{test}\hat\beta\rVert^2}{\lVert y_\text{test} - \hat\beta_0\rVert^2},$$ but this time I also tried using correlation metric (note that for the un-regularized OLS regression minimizing the squared error loss is equivalent to maximizing the correlation): $$L_\text{test}=\operatorname{corr}(y_\text{test}, X_\text{test}\hat\beta).$$ It's clear that these two performance metrics are not exactly equivalent, but weirdly, they disagree rather strongly: Note in particular what happens at $$\alpha=.2$$ (green line): maximum test-set correlation is achieved when test-set $$R^2$$ drops quite substantially compared to its maximum. In general, for any given $$\alpha$$, correlation seems to be maximized at larger $$\lambda$$ than squared error. Why does it happen and how to interpret it? Has anybody encountered this effect? Consider elastic net regression with glmnet-like parametrization $$\mathcal L = \frac{1}{2n}\big\lVert y - \beta_0-X\beta\big\rVert^2 + \lambda\big(\alpha\lVert \beta\rVert_1 + (1-\alpha) \lVert \beta\rVert^2_2/2\big).$$ I have a data set with $$n\ll p$$ (44 and 3000 respectively) and I am using repeated 11-fold cross-validation to select the optimal regularization parameters $$\alpha$$ and $$\lambda$$. Normally I would use squared error as the performance metric on the test set, e.g. this R-squared-like metric: $$L_\text{test} = 1-\frac{\lVert y_\text{test} - \hat\beta_0 - X_\text{test}\hat\beta\rVert^2}{\lVert y_\text{test} - \hat\beta_0\rVert^2},$$ but this time I also tried using correlation metric (note that for the un-regularized OLS regression minimizing the squared error loss is equivalent to maximizing the correlation): $$L_\text{test}=\operatorname{corr}(y_\text{test}, X_\text{test}\hat\beta).$$ It's clear that these two performance metrics are not exactly equivalent, but weirdly, they disagree rather strongly: Note in particular what happens at small alphas, e.g. $$\alpha=.2$$ (green line): maximum test-set correlation is achieved when test-set $$R^2$$ drops quite substantially compared to its maximum. In general, for any given $$\alpha$$, correlation seems to be maximized at larger $$\lambda$$ than squared error. Why does it happen and how to deal with it? Which criterion should be preferred? Has anybody encountered this effect? 2 edited tags | link edited Jan 31 '18 at 19:29 amoeba 64.9k1616 gold badges218218 silver badges273273 bronze badges 1 asked Jan 31 '18 at 16:15 amoeba 64.9k1616 gold badges218218 silver badges273273 bronze badges