# Expression for $p(\mathbf{s}|\mathbf{d})$ and the respective Markov Network

I would like to check if I correctly derived my the expression for $$p(\mathbf{s}|\mathbf{d})$$. Here's the question:

Consider a model of diseases and symptoms. $$s_i\in\{0,1\}$$ is a binary random variable indicating whether the patient is showing the $$i$$-th symptom and $$d_j\in \{0,1\}$$ is a binary random variable indicating whether the patient has $$j$$-th disease. A model for this is given by $$p(\mathbf{s,d})=\frac{1}{Z}\exp(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}+\mathbf{b}^T\mathbf{d})$$

where $$Z$$ is the normalisation constant and $$\mathbf{W},\mathbf{a},\mathbf{b}$$ are the parameters of the model.

I would like to draw the Markov Network for this model. Assume there are $$n$$ total symptoms and $$m$$ total diseases. Question 1: Would this Markov Network be correct? Since it's a Markov Network, I don't have arrows, otherwise if it was a belief network, here's how I would draw it: Question 2 Would this belief network be correct?

So now, I have the question:

Derive the expression for $$p(\mathbf{s}|\mathbf{d})$$. Is this distribution factorised?

Here's my solution:

$$p(\mathbf{s}|\mathbf{d}) = \frac{p(\mathbf{s},\mathbf{d})}{p(\mathbf{d})} = \frac{\exp\left(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}+\mathbf{b}^T\mathbf{d}\right)}{\sum_{\mathbf{s}}\exp\left(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}+\mathbf{b}^T\mathbf{d}\right)} = \frac{\exp\left(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}\right)}{\sum_{\mathbf{s}}\exp\left(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}\right)} \\ \propto \exp\left(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}\right) = \exp(\mathbf{s}^T\mathbf{W}d+\mathbf{s}^T\mathbf{a}) = \exp(\mathbf{s}^T(\mathbf{W}\mathbf{d}+\mathbf{a})) = \\ \exp\left(\sum_i^n\mathcal{1}_{\{s_i=1\}}\sum_j^m W_{ij}d_j+a_i\right) \\ \prod^n_i \exp(\sum_j^mW_{ij}d_j+a_i) = \prod_i^np(s_i|\mathbf{d})$$

The distribution is factoried and $$p(s_i|\mathbf{d}) = \sigma(a_i+\sum_jW_{ij}d_j)$$ where $$\sigma(x) = e^x/(1+e^x)$$

Is my derivation correct?

$$\newcommand{\sv}{\mathbf{s}} \newcommand{\dv}{\mathbf{d}} \newcommand{\Wv}{\mathbf{W}} \newcommand{\av}{\mathbf{a}} \newcommand{\bv}{\mathbf{b}} \newcommand{\pperp}{\perp\kern-5pt\perp} \newcommand{\npperp}{\not\perp\kern-8pt\perp}$$ Your first question:
The probability $$p(\mathbf s, \mathbf d)$$ can be rewritten as follows: \begin{align} p(\sv, \dv) &= \frac{1}{Z}\exp\left(\sv^T\Wv \dv + \av^T\sv + \bv^T\dv\right)\\ &= \frac{1}{Z}\prod_{ik}\exp(s_iW_{ik}d_k)\prod_r\exp(a_rs_r)\prod_{j}\exp(b_jd_j). \end{align} Thus, you obtain exactly the factorization depicted by your Markov network (MN). There is an edge between $$s_i$$ and $$d_k$$ if and only if $$W_{ik}\ne 0$$.

We know now that your MN is describing conditional independence relation (CI) in a generic pdf of the given type, i.e. if the MN has a connection between two nodes, this would in the generic case yield dependence between the corresponding random variables. In particular, we deduce from the MN that in general: $$\left(d_i \npperp d_k\right).$$ But in your belief network (BN), all the $$s_r$$ are colliders, thus the BN would require: $$\left(d_i \pperp d_k\right), \quad i\ne k.$$ Therefore, your BN is not suitable in the sense that it is not an I-map for the generic pdf of the given family.
The fact that $$p(\sv|\dv)$$ factors in $$\sv$$ follows from the MN. Conditioning on $$\dv$$ completely separates this bipartite graph, so all the $$d_i$$ are independent, i.e. the pdf factors. Your derivation in formulas appears quite right, although I am a bit confused about the term $$1_{s_i=1}$$ and that the variable $$\sv$$ disappeared from the second to last expression. So I would suggest the following, slightly changed version of your derivation:
Following you, I will consider only factors containing $$s_i$$ and combine all factors containing $$d_i$$ into the overall constant factor: \begin{align} p(\sv|\dv) &= \frac{p(\sv, \dv)}{p(\dv)}\\ &\propto \exp\left( \sv^T\Wv\dv + \av^T\sv\right)\\ &= \exp\left(\sv^T(\Wv\dv + \av) \right)\\ &= \prod_i \exp(s_i(\Wv\dv + \av)_i)\\ &= \prod_i f_i(s_i). \end{align} This confirms that $$p(\sv|\dv)$$ does indeed factor.
• Thanks a lot for the answer. I agree I wrote confusing/unnecessary notation, but our answers are the same in the end. Actually the fact that I know $p(\mathbf{s}|\mathbf{d})=\prod_ip(s_i|\mathbf{d})$ where $p(s_i|\mathbf{d}) = \sigma(a_i+\sum_jW_{ij}d_j)$ where $\sigma(x) = e^x/(1+e^x)$ is because there's an identical problem in BRML (exercise 4.4) and I wanted to double check their answer. Also if you don't mind taking a look at this Q, it should take very quick to answer imo