From scikit-learn's user guide, the loss function for logistic regression is expressed in this generalized form:

$$ \min_{w,c}\frac{1-\rho}{2}w^{T}w+\rho\lVert w\rVert_{1}+C\sum_{i=1}^{n}\log\left(\exp\left(-y_{i}\left(x_{i}^{T}w+c\right)\right)+1\right). $$

This is all fine if you are working with a static dataset. What I don't get is, once you have tuned your C using some cross-validation procedure, and then you go out and collect more data, you might have to proportionally adjust the optimal C or even re-tune C altogether.

In the extreme case, assume iid distribution of all samples, if we flood the original dataset with 100x more data, and we repeat our CV procedure, the new optimal C will surely look very different from the original one. This seems very unecessary to me. If we have a stable structure of the model, whether we fit the model on the original small sample or the flooded big sample, we should have similar betas and C's.

Instead, why don't we express penalty strength in terms of mean per-sample loss:

$$ \min_{w,c}\frac{1-\rho}{2}w^{T}w+\rho\lVert w\rVert_{1}+\frac{c}{n}\sum_{i=1}^{n}\log\left(\exp\left(-y_{i}\left(x_{i}^{T}w+c\right)\right)+1\right). $$

That way I feel more comfortable leaving the optimal c alone and only re-fit my betas when more data comes in.

Please share if you've encountered some discussion on this point.


This is indeed a reasonable approach from a machine learning perspective, and I did something similar in my Weighted Least-Squares Support Vector Machine implementation (see this paper) so that the range of hyper-parameter values that you need to search is more compact and the optimal value less dependent on the number of training samples.

I suspect the reason it is not as commonly seen in more statistically based models (rather than models from a more machine learning oriented source) is that Bayesian model selection schemes would require the overall loss, rather than the per-pattern loss, as might AIC or BIC.

However, different samples have different sampling variations (noise) so you may want to retune $C$ for different training samples, even if they are of the same size (as there is no telling that the value for the first sample of data was not an "outlier" in the distribution of optimal values).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.