# Why can't I reconstruct parameters of a synthetic data set?

The following Python function creates synthetic binary labeled data that is supposed to perfectly follow the logistic regression model:

def generate_data(params, N, epsilon_std=.2):
X = pd.DataFrame()
X_p = pd.DataFrame()
for ix in params.index:
if ix == 'intercept':
X[ix] = np.ones(N)
else:
X[ix] = np.random.rand(N)
X_p[ix] = X[ix] * params[ix]

X_p['__epsilon__'] = np.random.randn(N) * epsilon_std
X_p['logit'] = X_p.sum(axis=1)
X_p['label'] = X_p.logit > 0.0
X['label'] = X_p['label']
return X


The function above receives a vector of parameters (interecept and coefficients), generates random numbers, adds some noise, multiplies the parameters and the generated numbers to calculate the theoretical logit value and then assigns a label (positive if the logit is positive and negative, otherwise).

Now, generate the data, and use sklearn logistic regression model to calculate the parameters.

X = generate_data(params, 5000, epsilon_std=.1)
lr = LogisticRegression()
lr.fit(X[params.index], X.label)
result = lr.coef_.ravel()


The problem is that, although the resulting model is accurate, the calculated parameters don't even resemble the theoretical ones.

The following code generates a lot of models and draws parameter histogram. The vertical lines represent parameter values that I expect to see:

params = pd.Series({'intercept': 0,
'x': -1.,
'y': 1.})
params = params
fit_values = []
for i in range(100):
X = generate_data(params, 5000, epsilon_std=.5)
lr = LogisticRegression()
lr.fit(X[params.index], X.label)
result = lr.coef_.ravel()
result = result
fit_values.append(result)
fit_values = pd.DataFrame(fit_values, columns=params.index)
fig, axes = plt.subplots(1, len(params), figsize=(9, 2))
axes = axes.ravel()
for p, ax in zip(params.index,
axes):
ax.hist(fit_values[p], alpha=.3)
ax.axvline(params[p])
ax.set_title(p)


I tried the same process using statsmodels. The result was the same. What am I missing?

• X_p['label'] = X_p.logit > .5 doesn't seem to be assigning a label based on the sign of the logit: it is comparing it to $1/2$, not $0$! Regardless, this is not the underlying model for logistic regression. These two errors likely conspire to produce the strange results you report.
– whuber
Commented Jun 20, 2015 at 19:56
• you are right, but even then, the parameters (not the intercept) look scaled. Commented Jun 20, 2015 at 20:18
• Please add a reproducible example for people to work with. This seems to be more of a statistical / conceptual question than a coding question, & is hence on topic here, but you haven't set the random seed, or provided specified parameter values, or displayed the erroneous results. Commented Jun 20, 2015 at 20:36
• The data generating process that you are using is a Probit and not a Logit. Logit and Probit parameters are not directly comparable. Commented Jun 20, 2015 at 21:14
• @Josef: do you want to post your comment(s) as an answer? Better to have a short answer than no answer at all. Anyone who has a better answer can post it. Commented Sep 18, 2019 at 10:34