Consider a binary stochastic variable $D$ with possible states on and off, denoted by $D^1$ and $D^0$ . The state of $D$ is informed by $n$ independent binary stochastic variables $x_0$ through $x_n$, also with possible states on and off.

When $x_i$ is on it sends probability $f_i$ to $D$, which is the probability that $D$ is on given $x_i$:

$$P(D^1 | x_i) = f_i$$ $$P(D^0 | x_i) = 1 - f_i$$

When multiple $x$ are on simultaneously then $D$ receives multiple probabilities; if we consider these as independent their product gives their joint probability, which we define as being the final probability for state $D^1$:

$$ P(D^1) = \prod x_i f_i $$

(x is defined as having value 0 or 1 in states off and on respectively)

The models parameters are the $f_i$, hence we can train the model by following the gradient at each $f_i$:

$$ \partial_{f_i} P(D^1) = \partial_{f_i} \prod_i x_i f_i $$ $$ \partial_{f_i} P(D^1) = \left(\prod_{j \neq i} x_j f_j \right) x_i $$



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.