Last week my team and I discovered a strange phenomenon with the coefficients of a logistic regression (LR). As we included more samples from a static dataset, the magnitude of the coefficients of the LR kept on growing, until a certain size. After that, the magnitudes decrease rapidly.
We thought this had something to do a with a peculiarity in our dataset, with the amount of regularisation or with the visualisation itself. But after a while we replicated the phenomenon with randomly initialised data (see below).
What causes this strange behaviour of LR coefficients? Is it possible to determine the transition point (it to occur at a stable dataset size)?
Note: we use very little regularisation (
Preparation, specify number of rows
N and dimension
from sklearn.linear_model import LogisticRegression import pandas as pd N = 500 D = 100 sigma = 1
Generate some random data.
X is an
NxD array of N(0, 1) distributed data.
Y is a
Nx1 array of ground truth labels.
model specifies the relation between X (+ noise) and Y, it has coefficients between -10 and 10.
X = pd.np.random.randn(N, D) model = pd.np.random.randint(-10, 10, (D, 1)) noise = pd.np.random.randn(N, D) * sigma Y = ((X + noise).dot(model) > 0).squeeze()
The next code fits a logistic regression model to subsets of the whole data. The size of the subsets increases from 10 data points to all 500 data points. For each subset the coefficients of the logistic regression are saved.
def lr_coefficients_for(sample_size): lr = LogisticRegression(penalty='l2', C=1000).fit(X[:sample_size,], Y[:sample_size]) return lr.coef_ coefficients = pd.np.concatenate(list(map(lr_coefficients_for, sample_sizes)))
The mean absolute coefficient for each of the subsets is computed and plotted.
sample_sizes = range(10, N+1, 5) pdf_coefs = pd.Series(pd.np.abs(coefficients).mean(axis=1), index=sample_sizes, name='Mean absolute coefficient') pdf_coefs.plot(figsize=(8, 6), title='Mean absolute logistic regression coefficient vs. sample size')
The image shows how the magnitude of the coefficients increase when the subset has 250 or less data points. When the number of data points gets larger than 250, suddenly the magnitude of coefficients decreases.