3
$\begingroup$

I would like to check if I correctly derived my the expression for $p(\mathbf{s}|\mathbf{d})$. Here's the question:

Consider a model of diseases and symptoms. $s_i\in\{0,1\}$ is a binary random variable indicating whether the patient is showing the $i$-th symptom and $d_j\in \{0,1\}$ is a binary random variable indicating whether the patient has $j$-th disease. A model for this is given by $$p(\mathbf{s,d})=\frac{1}{Z}\exp(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}+\mathbf{b}^T\mathbf{d})$$

where $Z$ is the normalisation constant and $\mathbf{W},\mathbf{a},\mathbf{b}$ are the parameters of the model.

I would like to draw the Markov Network for this model. Assume there are $n$ total symptoms and $m$ total diseases.

enter image description here

Question 1: Would this Markov Network be correct? Since it's a Markov Network, I don't have arrows, otherwise if it was a belief network, here's how I would draw it:

enter image description here

Question 2 Would this belief network be correct?

So now, I have the question:

Derive the expression for $p(\mathbf{s}|\mathbf{d})$. Is this distribution factorised?

Here's my solution:

$$p(\mathbf{s}|\mathbf{d}) = \frac{p(\mathbf{s},\mathbf{d})}{p(\mathbf{d})} = \frac{\exp\left(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}+\mathbf{b}^T\mathbf{d}\right)}{\sum_{\mathbf{s}}\exp\left(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}+\mathbf{b}^T\mathbf{d}\right)} = \frac{\exp\left(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}\right)}{\sum_{\mathbf{s}}\exp\left(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}\right)} \\ \propto \exp\left(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}\right) = \exp(\mathbf{s}^T\mathbf{W}d+\mathbf{s}^T\mathbf{a}) = \exp(\mathbf{s}^T(\mathbf{W}\mathbf{d}+\mathbf{a})) = \\ \exp\left(\sum_i^n\mathcal{1}_{\{s_i=1\}}\sum_j^m W_{ij}d_j+a_i\right) \\ \prod^n_i \exp(\sum_j^mW_{ij}d_j+a_i) = \prod_i^np(s_i|\mathbf{d}) $$

The distribution is factoried and $p(s_i|\mathbf{d}) = \sigma(a_i+\sum_jW_{ij}d_j)$ where $\sigma(x) = e^x/(1+e^x)$

Is my derivation correct?

$\endgroup$

1 Answer 1

1
+50
$\begingroup$

$\newcommand{\sv}{\mathbf{s}} \newcommand{\dv}{\mathbf{d}} \newcommand{\Wv}{\mathbf{W}} \newcommand{\av}{\mathbf{a}} \newcommand{\bv}{\mathbf{b}} \newcommand{\pperp}{\perp\kern-5pt\perp} \newcommand{\npperp}{\not\perp\kern-8pt\perp} $ Your first question:
The probability $p(\mathbf s, \mathbf d)$ can be rewritten as follows: $$ \begin{align} p(\sv, \dv) &= \frac{1}{Z}\exp\left(\sv^T\Wv \dv + \av^T\sv + \bv^T\dv\right)\\ &= \frac{1}{Z}\prod_{ik}\exp(s_iW_{ik}d_k)\prod_r\exp(a_rs_r)\prod_{j}\exp(b_jd_j). \end{align} $$ Thus, you obtain exactly the factorization depicted by your Markov network (MN). There is an edge between $s_i$ and $d_k$ if and only if $W_{ik}\ne 0$.

Your second question:
We know now that your MN is describing conditional independence relation (CI) in a generic pdf of the given type, i.e. if the MN has a connection between two nodes, this would in the generic case yield dependence between the corresponding random variables. In particular, we deduce from the MN that in general: $$ \left(d_i \npperp d_k\right). $$ But in your belief network (BN), all the $s_r$ are colliders, thus the BN would require: $$ \left(d_i \pperp d_k\right), \quad i\ne k. $$ Therefore, your BN is not suitable in the sense that it is not an I-map for the generic pdf of the given family.

Your third question:
The fact that $p(\sv|\dv)$ factors in $\sv$ follows from the MN. Conditioning on $\dv$ completely separates this bipartite graph, so all the $d_i$ are independent, i.e. the pdf factors. Your derivation in formulas appears quite right, although I am a bit confused about the term $1_{s_i=1}$ and that the variable $\sv$ disappeared from the second to last expression. So I would suggest the following, slightly changed version of your derivation:
Following you, I will consider only factors containing $s_i$ and combine all factors containing $d_i$ into the overall constant factor: $$ \begin{align} p(\sv|\dv) &= \frac{p(\sv, \dv)}{p(\dv)}\\ &\propto \exp\left( \sv^T\Wv\dv + \av^T\sv\right)\\ &= \exp\left(\sv^T(\Wv\dv + \av) \right)\\ &= \prod_i \exp(s_i(\Wv\dv + \av)_i)\\ &= \prod_i f_i(s_i). \end{align} $$ This confirms that $p(\sv|\dv)$ does indeed factor.

$\endgroup$
1
  • $\begingroup$ Thanks a lot for the answer. I agree I wrote confusing/unnecessary notation, but our answers are the same in the end. Actually the fact that I know $p(\mathbf{s}|\mathbf{d})=\prod_ip(s_i|\mathbf{d})$ where $p(s_i|\mathbf{d}) = \sigma(a_i+\sum_jW_{ij}d_j)$ where $\sigma(x) = e^x/(1+e^x)$ is because there's an identical problem in BRML (exercise 4.4) and I wanted to double check their answer. Also if you don't mind taking a look at this Q, it should take very quick to answer imo $\endgroup$
    – user
    Aug 8, 2022 at 20:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.