In elastic net regularisation, will dividing the OLS term the number of observations cause misleading results when cross-validating?

Question

Two formulations of the elastic net regression function

Consider sklearn's implementation of elastic net regularisation (Wikipedia link). From the docs, it works by minimising the following objective function:

1 / (2 * n_samples) * ||y - Xw||^2_2 [OLS part]
+ alpha * l1_ratio * ||w||_1 + 0.5 * alpha * (1 - l1_ratio) * ||w||^2_2 [penalty part]

Comparing this to the objective function from the Wikipedia article:

$\hat{\beta} \equiv \underset{\beta}{\operatorname{argmin}} (\| y-X \beta \|^2 + \lambda_2 \|\beta\|^2 + \lambda_1 \|\beta\|_1)$

They are very similar: alpha * l1_ratio equates to $\lambda_1$, 0.5 * alpha * (1 - l1_ratio) equates to $\lambda_2$. But there's a difference in the 'OLS part' of the objective function - in the sklearn implementation, we divide by 2 * n_samples.

Differences between the cross-validated models

Now, this does not necessarily matter, because the regularisation hyperparameters are free - so we can get the same results as the Wikipedia formula by multiplying the objective function by 2 * n_samples. But I'd argue this leads to misleading results when cross-validating.

Suppose we have a dataset composed of 1000 observations. We want to do a Ridge regression (so we ignore the L1 penalty term by setting l1_ratio or $\lambda_1$ to 0) of one column y on the rest of the columns X. First, in order to find the best hyperparameter $\tilde \lambda$ to use, we do a 5-fold cross-validation using this dataset (so each fold has 800 training observations and 200 test observations).

For this we use the sklearn method, and find that $\tilde \lambda = 1$. Out of interest, we try again using the Wikipedia method, and find that (as Sycorax suggested) the best hyperparameter ($\lambda^{*}$) under this method is just a rescaled version of $\tilde \lambda$. In fact, $\lambda^{*} = 2n \tilde \lambda = 1600$. In each cross-validation training set, the models under the different methods are the same when trained with the respective optimal hyperparameters.

Now, let's see what happens when we try to train these models (with the associated hyperparameters) on the whole dataset ($n = 1000$). We can write down the respective objective functions:

sklearn:

$\hat{\beta} \equiv \underset{\beta}{\operatorname{argmin}} (\| y-X \beta \|^2/2n + \tilde \lambda \|\beta\|^2)$

$\hat{\beta} \equiv \underset{\beta}{\operatorname{argmin}} (\| y-X \beta \|^2/2000 + \|\beta\|^2)$

Wikipedia:

$\hat{\beta} \equiv \underset{\beta}{\operatorname{argmin}} (\| y-X \beta \|^2 + \lambda^{*} \|\beta\|^2)$

$\hat{\beta} \equiv \underset{\beta}{\operatorname{argmin}} (\| y-X \beta \|^2 + 1600 \|\beta\|^2)$

Clearly the values of $\hat{\beta}$ that we obtain from these models will be different - in particular, those from the sklearn model will be more strongly regularised.

The sklearn implementation is less suited for cross-validation problems

Therefore the sklearn and Wikipedia methods result in different models, when selecting the optimal hyperparameter based on cross-validation on a subsample of the dataset. Which one makes more sense?

I'd argue that it's the Wikipedia version. Ceteris paribus the more data you have, the lower variance your predictions, and so the less you want to regularise. The Wikipedia approach - in which the OLS part of the objective function is allowed to scale with the number of samples, rather than being an average - means that this happens automatically as the size of your dataset increases, without having to change the regularisation parameter. This means that it is better suited to cases where you select the optimal parameter on a subset of data, as with cross-validation.

Answer 1

It's correct that when the sample size is fixed, there is not a difference between the two statements of the optimization problem.

Your demonstration in the revised question makes it clear that the choice of objective function implies that $\lambda$ selected from one sample size is not "portable" to a different sample size.

The question then becomes "What can we do about this?" On the one hand, cross-validation is the preferred way to estimate the optimal $\lambda$. On the other hand, the two alternative statements of the optimization task will yield two alternative choices of "optimal" $\lambda$. You could try to re-estimate $\lambda$ on the whole data set, but this begs the question because using the whole data set implies that out-of-sample data is not available to assess the quality of fit.