How to train a Bayesian network with Bernoulli switch variable?

Question

I model my problem as a simple V-structured Bayesian network. There is an $outcome$ variable, the binary $switch$ variable, and some environment features $X$. All the variables are observed during training but $outcome$ and $switch$ are not known during inference. I am interested in estimating the marginal $P(outcome \mid X)$. From the domain knowledge,

$$P(outcome \mid switch = 0, X) = 0,$$

so, the model can be decomposed as

$$P(outcome \mid X) = P(switch = 1 \mid X) \cdot P(outcome \mid switch = 1, X).$$

Fitting $P(switch = 1 \mid X)$ is straightforward: I just use logistic regression. But how to fit $P(outcome \mid switch = 1, X)$?

My current approach is to filter the training data to retain only examples for which $[switch=1]$ as dictated by the form of this probability. However, it does not seem sensible when it comes to inference. There will be some $[switch=0]\ $ examples that have non-zero $P(switch = 1 \mid X)$ prediction, which can be amplified by $P(outcome \mid switch = 1, X)$ as we did not train that model on similar examples.

Should I instead train $P(outcome \mid switch = 1, X)$ on the $[switch=0]$ examples as well, just weighting them proportionally to the $P(switch = 1 \mid X)$ prediction? Or should I reformulate my model completely?

Will appreciate any references.

Answer 1

You know that your Data may be distributed in an absolute different way than you latent space. You are sampling your latent space given/from your data.

The Bernoulli Dstribution for your latent space can either only be obtained through backprop. or through a parameterization given your Latent distribution´like a reverse generative model.

Though you have:

--data distro-- --Bernoulli distro--

X1   -     ...   -    Y1
X2   -     ...   
X3   -     :W:   -    Y2
X4   -     ...
X5   -     ...   -    Y3

A generative model would learn a Gassian/Beta/Gamma/Poisson data noise with Bernoulli prior, so for the encoding case, you need a Bernoulli data noise with any data prior.

The parameters can be obtained with the KL-Divergence or EM... Be sure not to learn too much values. E.g. $diag( \sigma_.^2)$ and the bias $\mu_x$ for simple space invariance. But you could even use just a recurrent / convolutional layer for the spacial invariance.

Note that usually your $f(x,W)$ shall already depict your mean value of Y! (And $f^{-1}(W^{-1},y)$ mathematically you data mean... if I didn't overcomplicate something.)

Bayesian it would be:

$P(X|Y,\Theta) = \frac{P(Y|X,\Theta)P(x)}{P(Y)}$

which you can approximate with $\frac{P(Y|X,\Theta)P(X)}{\sum_{x' \in ROI(X)}P(Y|x',\Theta)P(x')}$ where $ROI(X)$ would be the region of interest from X given the $M'$ highest marginal probabilities over your Bernoulli distribution.

There are papers about binary to binary generative models which you should read considering the math you are going to implement!