In preparation for working on real-world datasets, I am exploring classifiers on syntethically generated data. First I generate random variables $X_1 ... X_8$ that represent observables with physical meaning, see first image below for bivariate distributions ($X_1$ is the top row and so on).

enter image description here

In a next step, based on an ad-hoc model how these observables/their interaction affect the observed target class I map them to another set of random variables $Y_1, .. ,Y_6$. (Shown in the first six rows in the second image.) These variables consitute the basis for the generative logistic model that can have two outcomes $A\in \{0,1\}$.

I assume a relations

\begin{align} Y_1 &= f(X_1,X_2)\\ Y_2 &= Y_1*g(X_3)\\ Y_3 &= Y_1*h(X_4,X_5)\\ Y_4 &= Y_1*i(X_6)\\ Y_5 &= j(X_7)\\ Y_6 &= Y_1*k(X_8) \end{align}

where $f(\cdot),g(\cdot),h(\cdot),i(\cdot),j(\cdot),k(\cdot)$ are functions.

In order to establish a class-membership for each observation I set $\mu_i(A=1)=\sigma(w_M^Ty_i-w_0)$, where $\sigma$ is the sigmoid function, $y_i$ is the vector of random variables $Y_i$ for observation $i$,$w_M$ is a fixed vector specifying the impact of each component and $w_0$ a bias. In the figure below $PHI$ denotes the sum inside the sigmoid (third last row). Finally, I draw from a binomial distribution with probability $\mu_A$ (second last row) to obtain a class membership $A_i$ for each sample(last row). (Resulting in 109 samples with $A=1$ and 891 samples with $A=0$.)

enter image description here

In a first step I wanted to see if I can retrieve the coefficients $w$ (incl. $w_0$) from a direct fit of a logistic model to the (in reality hidden) random vectors $y_i$. The table below shows the coefficients as set in the synthetic model and the ones found by fitting with sklearn.LogisticRegression(solver='lbfgs', C=very_large) on a train split (n=1000, train=0.65).

To illustrate the problem I show representative examples of the fitted coefficients, I find similar outcome for other splits. It can be seen that the fitted coefficients approximate the synthetic/model ones (except $w_1$ see question below).

|              | w0/intercept_ | w1        | w2       | w3       | w4       | w5        | w6       |
| synth. coef_ | -3.0          | 0.01      | 0.002    | 0.02     | 0.005    | -3.0      | 0.05     |
| fitted       | -2.898314     | -0.009813 | 0.001818 | 0.027478 | 0.003471 | -2.611516 | 0.147478 |

Question: Can you help me understand why (around 70% over different splits) the coefficient $w_0$ shows the wrong signum?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.