1
$\begingroup$

Last week my team and I discovered a strange phenomenon with the coefficients of a logistic regression (LR). As we included more samples from a static dataset, the magnitude of the coefficients of the LR kept on growing, until a certain size. After that, the magnitudes decrease rapidly.

We thought this had something to do a with a peculiarity in our dataset, with the amount of regularisation or with the visualisation itself. But after a while we replicated the phenomenon with randomly initialised data (see below).

What causes this strange behaviour of LR coefficients? Is it possible to determine the transition point (it to occur at a stable dataset size)?

Note: we use very little regularisation (C=1000).

Reproducible example

Preparation, specify number of rows N and dimension D

from sklearn.linear_model import LogisticRegression
import pandas as pd

N = 500
D = 100
sigma = 1

Generate some random data. X is an NxD array of N(0, 1) distributed data. Y is a Nx1 array of ground truth labels. model specifies the relation between X (+ noise) and Y, it has coefficients between -10 and 10.

X = pd.np.random.randn(N, D)
model = pd.np.random.randint(-10, 10, (D, 1))
noise = pd.np.random.randn(N, D) * sigma
Y = ((X + noise).dot(model) > 0).squeeze()

The next code fits a logistic regression model to subsets of the whole data. The size of the subsets increases from 10 data points to all 500 data points. For each subset the coefficients of the logistic regression are saved.

def lr_coefficients_for(sample_size):
    lr = LogisticRegression(penalty='l2', C=1000).fit(X[:sample_size,], Y[:sample_size])
    return lr.coef_

coefficients = pd.np.concatenate(list(map(lr_coefficients_for, sample_sizes)))

The mean absolute coefficient for each of the subsets is computed and plotted.

sample_sizes = range(10, N+1, 5)
pdf_coefs = pd.Series(pd.np.abs(coefficients).mean(axis=1), index=sample_sizes, name='Mean absolute coefficient')
pdf_coefs.plot(figsize=(8, 6), title='Mean absolute logistic regression coefficient vs. sample size')

The result

The image shows how the magnitude of the coefficients increase when the subset has 250 or less data points. When the number of data points gets larger than 250, suddenly the magnitude of coefficients decreases.

enter image description here

$\endgroup$
  • 1
    $\begingroup$ For those not intimately acquainted with the details of the functions you are using, could you please describe your algorithm? $\endgroup$ – whuber Aug 10 '18 at 14:53
  • $\begingroup$ @whuber, added some explanation to the code $\endgroup$ – Pieter Aug 10 '18 at 17:55
  • 2
    $\begingroup$ Top of head thought: It could be that with small subsets of the data you had complete separation, and since you're applying L2 regularization you were still getting finite (but increasingly large) coefficient estimates. Then after a certain point when you've added enough data, the data are no longer completely separated and the coefficients come back down to a sensible range. $\endgroup$ – Jake Westfall Aug 10 '18 at 18:52
  • $\begingroup$ Thank you. However, the code doesn't appear to do what you describe. What is this "l2 penalty" and precisely how is it determined? That little detail alone could account for what's happening. Also, you don't appear to be creating "subsets" of the data: you are selecting features. There's a huge difference! $\endgroup$ – whuber Aug 10 '18 at 19:32
  • $\begingroup$ I am subsetting the first index of a numpy array, i.e. the rows. $\endgroup$ – Pieter Aug 11 '18 at 8:26
1
$\begingroup$

My sense is that this represents what you might expect with overfitting of a model, perhaps with some near-perfect separation added in.

A useful rule of thumb to avoid overfitting in logistic regression is to have 15 to 20 events per predictor. "Events" in this context represents the number of cases in the sample with the lower prevalence of the 2 classes. With 10 predictors in your model and as I understand it a 50/50 split of the 2 classes, then you would want 300 - 400 cases total to avoid overfitting. This is in the range where the sum of the absolute values of coefficients in your example has stabilized at a low value.

My guess is that something related to perfect separation or the Hauck-Donner phenomenon accounts for the peak at intermediate sample sizes. At very low sample sizes you do not have enough cases to find any substantial relations of classes to predictors. But at intermediate sample sizes you might pick up combinations of predictors that happen to do very well on the particular sample at hand but do not generalize well to new samples; thus the large-magnitude apparent coefficients fade away as more cases are considered.

$\endgroup$
  • $\begingroup$ The labels are indeed distributed by a 50/50 split. The number of features however is 100 and not 10, would this suggest that I need 3000 samples? $\endgroup$ – Pieter Aug 11 '18 at 8:32
  • 1
    $\begingroup$ @Pieter that’s the rule of thumb for completely unpenalized regression. Penalization can be thought of as reducing the number of effective predictors so that you can work with fewer cases. Don’t know how to gauge the amount of penalization introduced by your C=1000 parameter setting, but that might explain why things stabilized at lower sample sizes than might be expected absent penalization. $\endgroup$ – EdM Aug 11 '18 at 12:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.