0
$\begingroup$

In preparation for working on real-world datasets, I am exploring classifiers on syntethically generated data. First I generate random variables $X_1 ... X_8$ that represent observables with physical meaning, see first image below for bivariate distributions ($X_1$ is the top row and so on).

enter image description here

In a next step, based on an ad-hoc model how these observables/their interaction affect the observed target class I map them to another set of random variables $Y_1, .. ,Y_6$. (Shown in the first six rows in the second image.) These variables consitute the basis for the generative logistic model that can have two outcomes $A\in \{0,1\}$.

I assume a relations

\begin{align} Y_1 &= f(X_1,X_2)\\ Y_2 &= Y_1*g(X_3)\\ Y_3 &= Y_1*h(X_4,X_5)\\ Y_4 &= Y_1*i(X_6)\\ Y_5 &= j(X_7)\\ Y_6 &= Y_1*k(X_8) \end{align}

where $f(\cdot),g(\cdot),h(\cdot),i(\cdot),j(\cdot),k(\cdot)$ are functions.

In order to establish a class-membership for each observation I set $\mu_i(A=1)=\sigma(w_M^Ty_i-w_0)$, where $\sigma$ is the sigmoid function, $y_i$ is the vector of random variables $Y_i$ for observation $i$,$w_M$ is a fixed vector specifying the impact of each component and $w_0$ a bias. In the figure below $PHI$ denotes the sum inside the sigmoid (third last row). Finally, I draw from a binomial distribution with probability $\mu_A$ (second last row) to obtain a class membership $A_i$ for each sample(last row). (Resulting in 109 samples with $A=1$ and 891 samples with $A=0$.)

enter image description here

In a first step I wanted to see if I can retrieve the coefficients $w$ (incl. $w_0$) from a direct fit of a logistic model to the (in reality hidden) random vectors $y_i$. The table below shows the coefficients as set in the synthetic model and the ones found by fitting with sklearn.LogisticRegression(solver='lbfgs', C=very_large) on a train split (n=1000, train=0.65).

To illustrate the problem I show representative examples of the fitted coefficients, I find similar outcome for other splits. It can be seen that the fitted coefficients approximate the synthetic/model ones (except $w_1$ see question below).

|              | w0/intercept_ | w1        | w2       | w3       | w4       | w5        | w6       |
|--------------|---------------|-----------|----------|----------|----------|-----------|----------|
| synth. coef_ | -3.0          | 0.01      | 0.002    | 0.02     | 0.005    | -3.0      | 0.05     |
| fitted       | -2.898314     | -0.009813 | 0.001818 | 0.027478 | 0.003471 | -2.611516 | 0.147478 |

Question: Can you help me understand why (around 70% over different splits) the coefficient $w_0$ shows the wrong signum?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.