What explains a sudden change in the magnitude of logistic regression coefficients when increasing the sample size

Last week my team and I discovered a strange phenomenon with the coefficients of a logistic regression (LR). As we included more samples from a static dataset, the magnitude of the coefficients of the LR kept on growing, until a certain size. After that, the magnitudes decrease rapidly.

We thought this had something to do a with a peculiarity in our dataset, with the amount of regularisation or with the visualisation itself. But after a while we replicated the phenomenon with randomly initialised data (see below).

What causes this strange behaviour of LR coefficients? Is it possible to determine the transition point (it to occur at a stable dataset size)?

Note: we use very little regularisation (C=1000).

Reproducible example

Preparation, specify number of rows N and dimension D

from sklearn.linear_model import LogisticRegression
import pandas as pd

N = 500
D = 100
sigma = 1

Generate some random data. X is an NxD array of N(0, 1) distributed data. Y is a Nx1 array of ground truth labels. model specifies the relation between X (+ noise) and Y, it has coefficients between -10 and 10.

X = pd.np.random.randn(N, D)
model = pd.np.random.randint(-10, 10, (D, 1))
noise = pd.np.random.randn(N, D) * sigma
Y = ((X + noise).dot(model) > 0).squeeze()

The next code fits a logistic regression model to subsets of the whole data. The size of the subsets increases from 10 data points to all 500 data points. For each subset the coefficients of the logistic regression are saved.

def lr_coefficients_for(sample_size):
lr = LogisticRegression(penalty='l2', C=1000).fit(X[:sample_size,], Y[:sample_size])
return lr.coef_

coefficients = pd.np.concatenate(list(map(lr_coefficients_for, sample_sizes)))

The mean absolute coefficient for each of the subsets is computed and plotted.

sample_sizes = range(10, N+1, 5)
pdf_coefs = pd.Series(pd.np.abs(coefficients).mean(axis=1), index=sample_sizes, name='Mean absolute coefficient')
pdf_coefs.plot(figsize=(8, 6), title='Mean absolute logistic regression coefficient vs. sample size')

The result

The image shows how the magnitude of the coefficients increase when the subset has 250 or less data points. When the number of data points gets larger than 250, suddenly the magnitude of coefficients decreases. • For those not intimately acquainted with the details of the functions you are using, could you please describe your algorithm? – whuber Aug 10 '18 at 14:53
• @whuber, added some explanation to the code – Pieter Aug 10 '18 at 17:55
• Top of head thought: It could be that with small subsets of the data you had complete separation, and since you're applying L2 regularization you were still getting finite (but increasingly large) coefficient estimates. Then after a certain point when you've added enough data, the data are no longer completely separated and the coefficients come back down to a sensible range. – Jake Westfall Aug 10 '18 at 18:52
• Thank you. However, the code doesn't appear to do what you describe. What is this "l2 penalty" and precisely how is it determined? That little detail alone could account for what's happening. Also, you don't appear to be creating "subsets" of the data: you are selecting features. There's a huge difference! – whuber Aug 10 '18 at 19:32
• I am subsetting the first index of a numpy array, i.e. the rows. – Pieter Aug 11 '18 at 8:26