This is a follow up from this question.

Consider a model of diseases and symptoms. $s_i\in\{0,1\}$ is a binary random variable indicating whether the patient is showing the $i$-th symptom and $d_j\in \{0,1\}$ is a binary random variable indicating whether the patient has $j$-th disease. A model for this is given by $$p(\mathbf{s,d})=\frac{1}{Z}\exp(\mathbf{s}^T\mathbf{W}\mathbf{d}+\mathbf{a}^T\mathbf{s}+\mathbf{b}^T\mathbf{d})$$

where $Z$ is the normalisation constant and $\mathbf{W},\mathbf{a},\mathbf{b}$ are the parameters of the model.

In my previous question it was derived that $p(\mathbf{s}|\mathbf{d})$:

$p(\mathbf{s}|\mathbf{d})=\prod_i^np(s_i|\mathbf{d})$ where $p(s_i|\mathbf{d}) = \sigma(a_i+\sum_jW_{ij}d_j)$ where $\sigma(x) = e^x/(1+e^x)$ ---- this follows from the fact that $s_i\in \{0,1\}$.

Now, in this question I'd like to add a bit of a spin:

Consider there to be $K$ patients, each with a patient record $(\mathbf{s}^k,\mathbf{d}^k)$. Suppose that you want to learn the parameters of the model by Maximum Likelihood. I'd like to derive the expression for the log-likelihood $L$, assuming that the patients are i.i.d.

I tried deriving it:

Let $\mathcal{K} = \{(\mathbf{s}^k,\mathbf{d}^k), k = 1,..., K\}$. Then $$P(\mathcal{K})=\prod_kp(\mathbf{s}^k|\mathbf{d}^k)$$

and so $$L = \log p(\mathcal{K}) = \sum^K_{k=1}\log \left(\prod_i^np(s_i^k|\mathbf{d}^k)\right) = \sum^K_{k=1}\sum^n_{i=1}\log \left(p(s_i^k|\mathbf{d}^k)\right) \\ = \sum^K_{k=1}\sum^n_{i=1} \log\left(\frac{\exp\left(a_i+\sum_jW_{ij}d_j\right)}{1+\exp\left(a_i+\sum_jW_{ij}d_j\right)}\right)\\ = \sum^K_{k=1}\sum^n_{i=1}\left[ \left(a_i + \sum_jW_{ij}d_j\right)-\log\left(1+\exp\left(a_i+\sum_jW_{ij}d_j\right)\right)\right]$$

It this derivation correct?


1 Answer 1


$ \newcommand{\sv}{\mathbf{s}} \newcommand{\dv}{\mathbf{d}} \newcommand{\av}{\mathbf{a}} \newcommand{\bv}{\mathbf{b}} \newcommand{\Wv}{\mathbf{W}} $

The log-likelihood L is defined as $L(\Wv, \av, \bv) = \log \prod_{k=1}^K p(\sv^k, \dv^K\;|\;\Wv, \av, \bv)$. Thus: $$ \begin{align} L(\Wv, \av, \bv) &= \log \prod_{k=1}^K \frac{1}{Z(\Wv, \av, \bv)}\exp\left((\sv^k)^T\Wv\dv^k + \av^T\sv^k + \bv^T\dv^k\right)\\ &= -K\log Z(\Wv, \av, \bv) + \sum_{k=1}^K \left((\sv^k)^T\Wv\dv^k + \av^T\sv^k + \bv^T\dv^k\right). \end{align} $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.