# Logistic regression: Understanding convergence towards coefficients from synthetic model

In preparation for working on real-world datasets, I am exploring classifiers on syntethically generated data. First I generate random variables $$X_1 ... X_8$$ that represent observables with physical meaning, see first image below for bivariate distributions ($$X_1$$ is the top row and so on).

In a next step, based on an ad-hoc model how these observables/their interaction affect the observed target class I map them to another set of random variables $$Y_1, .. ,Y_6$$. (Shown in the first six rows in the second image.) These variables consitute the basis for the generative logistic model that can have two outcomes $$A\in \{0,1\}$$.

I assume a relations

\begin{align} Y_1 &= f(X_1,X_2)\\ Y_2 &= Y_1*g(X_3)\\ Y_3 &= Y_1*h(X_4,X_5)\\ Y_4 &= Y_1*i(X_6)\\ Y_5 &= j(X_7)\\ Y_6 &= Y_1*k(X_8) \end{align}

where $$f(\cdot),g(\cdot),h(\cdot),i(\cdot),j(\cdot),k(\cdot)$$ are functions.

In order to establish a class-membership for each observation I set $$\mu_i(A=1)=\sigma(w_M^Ty_i-w_0)$$, where $$\sigma$$ is the sigmoid function, $$y_i$$ is the vector of random variables $$Y_i$$ for observation $$i$$,$$w_M$$ is a fixed vector specifying the impact of each component and $$w_0$$ a bias. In the figure below $$PHI$$ denotes the sum inside the sigmoid (third last row). Finally, I draw from a binomial distribution with probability $$\mu_A$$ (second last row) to obtain a class membership $$A_i$$ for each sample(last row). (Resulting in 109 samples with $$A=1$$ and 891 samples with $$A=0$$.)

In a first step I wanted to see if I can retrieve the coefficients $$w$$ (incl. $$w_0$$) from a direct fit of a logistic model to the (in reality hidden) random vectors $$y_i$$. The table below shows the coefficients as set in the synthetic model and the ones found by fitting with sklearn.LogisticRegression(solver='lbfgs', C=very_large) on a train split (n=1000, train=0.65).

To illustrate the problem I show representative examples of the fitted coefficients, I find similar outcome for other splits. It can be seen that the fitted coefficients approximate the synthetic/model ones (except $$w_1$$ see question below).

|              | w0/intercept_ | w1        | w2       | w3       | w4       | w5        | w6       |
|--------------|---------------|-----------|----------|----------|----------|-----------|----------|
| synth. coef_ | -3.0          | 0.01      | 0.002    | 0.02     | 0.005    | -3.0      | 0.05     |
| fitted       | -2.898314     | -0.009813 | 0.001818 | 0.027478 | 0.003471 | -2.611516 | 0.147478 |


Question: Can you help me understand why (around 70% over different splits) the coefficient $$w_0$$ shows the wrong signum?