The following Python function creates synthetic binary labeled data that is supposed to perfectly follow the logistic regression model:

def generate_data(params, N, epsilon_std=.2):
    X = pd.DataFrame()
    X_p = pd.DataFrame()
    for ix in params.index:
        if ix == 'intercept':
            X[ix] = np.ones(N)
            X[ix] = np.random.rand(N)
        X_p[ix] = X[ix] * params[ix]

    X_p['__epsilon__'] = np.random.randn(N) * epsilon_std
    X_p['logit'] = X_p.sum(axis=1)
    X_p['label'] = X_p.logit > 0.0
    X['label'] = X_p['label']
    return X

The function above receives a vector of parameters (interecept and coefficients), generates random numbers, adds some noise, multiplies the parameters and the generated numbers to calculate the theoretical logit value and then assigns a label (positive if the logit is positive and negative, otherwise).

Now, generate the data, and use sklearn logistic regression model to calculate the parameters.

X = generate_data(params, 5000, epsilon_std=.1)
lr = LogisticRegression()
lr.fit(X[params.index], X.label)
result = lr.coef_.ravel()

The problem is that, although the resulting model is accurate, the calculated parameters don't even resemble the theoretical ones.

The following code generates a lot of models and draws parameter histogram. The vertical lines represent parameter values that I expect to see:

params = pd.Series({'intercept': 0,
                   'x': -1.,
                   'y': 1.})
params = params
fit_values = []
for i in range(100):
    X = generate_data(params, 5000, epsilon_std=.5)
    lr = LogisticRegression()
    lr.fit(X[params.index], X.label)
    result = lr.coef_.ravel()
    result = result 
fit_values = pd.DataFrame(fit_values, columns=params.index)
fig, axes = plt.subplots(1, len(params), figsize=(9, 2))
axes = axes.ravel()
for p, ax in zip(params.index, 
    ax.hist(fit_values[p], alpha=.3)

parameter distribution

I tried the same process using statsmodels. The result was the same. What am I missing?

  • $\begingroup$ X_p['label'] = X_p.logit > .5 doesn't seem to be assigning a label based on the sign of the logit: it is comparing it to $1/2$, not $0$! Regardless, this is not the underlying model for logistic regression. These two errors likely conspire to produce the strange results you report. $\endgroup$
    – whuber
    Commented Jun 20, 2015 at 19:56
  • $\begingroup$ you are right, but even then, the parameters (not the intercept) look scaled. $\endgroup$
    – David D
    Commented Jun 20, 2015 at 20:18
  • $\begingroup$ Please add a reproducible example for people to work with. This seems to be more of a statistical / conceptual question than a coding question, & is hence on topic here, but you haven't set the random seed, or provided specified parameter values, or displayed the erroneous results. $\endgroup$ Commented Jun 20, 2015 at 20:36
  • 1
    $\begingroup$ The data generating process that you are using is a Probit and not a Logit. Logit and Probit parameters are not directly comparable. $\endgroup$
    – Josef
    Commented Jun 20, 2015 at 21:14
  • $\begingroup$ @Josef: do you want to post your comment(s) as an answer? Better to have a short answer than no answer at all. Anyone who has a better answer can post it. $\endgroup$ Commented Sep 18, 2019 at 10:34


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.