# Deriving the expression for $p(\mathcal{K})$ where $\mathcal{K} = \{(\mathbf{s}^k,\mathbf{d}^k), k = 1,..., K\}$

This is a follow up from this question.

Consider a model of diseases and symptoms. $$s_i\in\{0,1\}$$ is a binary random variable indicating whether the patient is showing the $$i$$-th symptom and $$d_j\in \{0,1\}$$ is a binary random variable indicating whether the patient has $$j$$-th disease. A model for this is given by $$p(\mathbf{s,d})=\frac{1}{Z}\exp(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}+\mathbf{b}^T\mathbf{d})$$

where $$Z$$ is the normalisation constant and $$\mathbf{W},\mathbf{a},\mathbf{b}$$ are the parameters of the model.

In my previous question it was derived that $$p(\mathbf{s}|\mathbf{d})$$:

$$p(\mathbf{s}|\mathbf{d})=\prod_i^np(s_i|\mathbf{d})$$ where $$p(s_i|\mathbf{d}) = \sigma(a_i+\sum_jW_{ij}d_j)$$ where $$\sigma(x) = e^x/(1+e^x)$$ ---- this follows from the fact that $$s_i\in \{0,1\}$$.

Now, in this question I'd like to add a bit of a spin:

Consider there to be $$K$$ patients, each with a patient record $$(\mathbf{s}^k,\mathbf{d}^k)$$. Suppose that you want to learn the parameters of the model by Maximum Likelihood. I'd like to derive the expression for the log-likelihood $$L$$, assuming that the patients are i.i.d.

I tried deriving it:

Let $$\mathcal{K} = \{(\mathbf{s}^k,\mathbf{d}^k), k = 1,..., K\}$$. Then $$P(\mathcal{K})=\prod_kp(\mathbf{s}^k|\mathbf{d}^k)$$

and so $$L = \log p(\mathcal{K}) = \sum^K_{k=1}\log \left(\prod_i^np(s_i^k|\mathbf{d}^k)\right) = \sum^K_{k=1}\sum^n_{i=1}\log \left(p(s_i^k|\mathbf{d}^k)\right) \\ = \sum^K_{k=1}\sum^n_{i=1} \log\left(\frac{\exp\left(a_i+\sum_jW_{ij}d_j\right)}{1+\exp\left(a_i+\sum_jW_{ij}d_j\right)}\right)\\ = \sum^K_{k=1}\sum^n_{i=1}\left[ \left(a_i + \sum_jW_{ij}d_j\right)-\log\left(1+\exp\left(a_i+\sum_jW_{ij}d_j\right)\right)\right]$$

It this derivation correct?

The log-likelihood L is defined as $$L(\Wv, \av, \bv) = \log \prod_{k=1}^K p(\sv^k, \dv^K\;|\;\Wv, \av, \bv)$$. Thus: \begin{align} L(\Wv, \av, \bv) &= \log \prod_{k=1}^K \frac{1}{Z(\Wv, \av, \bv)}\exp\left((\sv^k)^T\Wv\dv^k + \av^T\sv^k + \bv^T\dv^k\right)\\ &= -K\log Z(\Wv, \av, \bv) + \sum_{k=1}^K \left((\sv^k)^T\Wv\dv^k + \av^T\sv^k + \bv^T\dv^k\right). \end{align}