# Tuning penalty strength in scikit-learn logistic regression

From scikit-learn's user guide, the loss function for logistic regression is expressed in this generalized form:

$$\min_{w,c}\frac{1-\rho}{2}w^{T}w+\rho\lVert w\rVert_{1}+C\sum_{i=1}^{n}\log\left(\exp\left(-y_{i}\left(x_{i}^{T}w+c\right)\right)+1\right).$$

This is all fine if you are working with a static dataset. What I don't get is, once you have tuned your C using some cross-validation procedure, and then you go out and collect more data, you might have to proportionally adjust the optimal C or even re-tune C altogether.

In the extreme case, assume iid distribution of all samples, if we flood the original dataset with 100x more data, and we repeat our CV procedure, the new optimal C will surely look very different from the original one. This seems very unecessary to me. If we have a stable structure of the model, whether we fit the model on the original small sample or the flooded big sample, we should have similar betas and C's.

Instead, why don't we express penalty strength in terms of mean per-sample loss:

$$\min_{w,c}\frac{1-\rho}{2}w^{T}w+\rho\lVert w\rVert_{1}+\frac{c}{n}\sum_{i=1}^{n}\log\left(\exp\left(-y_{i}\left(x_{i}^{T}w+c\right)\right)+1\right).$$

That way I feel more comfortable leaving the optimal c alone and only re-fit my betas when more data comes in.

Please share if you've encountered some discussion on this point.

However, different samples have different sampling variations (noise) so you may want to retune $$C$$ for different training samples, even if they are of the same size (as there is no telling that the value for the first sample of data was not an "outlier" in the distribution of optimal values).