On p110 of Murphy's machine learning book, he derives the discriminant function for the diagonal LDA model by simplifying the full linear discriminant analysis equation (4.33):

enter image description here

It seems that he's discarded the normalization constants to get the expression for $\delta_c {\bf{x}}$. Why?

I haven't come across the notation $\delta_c {\bf{x}}$ before - what does this mean?


He's used the formula for conditional probability. Recall the conditional probability rule: $$p(A,B|C)=p(A|B,C)p(B|C)$$ This basically asserts that the joint equals the conditional times the normalizing ''constant''. The normalizing constant is given by the rule of marginalization: $$p(B|C)=\sum_a p(A=a,B|C)$$

(4.33) is basically saying: $$p(y=c|x,\theta)=\dfrac{p(x,y=c|\theta)}{\sum_cp(x,y=c|\theta)}=\dfrac{p(x,y=c|\theta)}{\color{red}{p(x|\theta)}}$$

The denominator in (4.33), which is the normalizing constant, is $p(x|\theta)$. $$\delta_c(x) =\log \underbrace{p(x,y=c|\theta)}_{\text{joint}}=\log [\underbrace{p(y=c|x,\theta)}_{(4.33)} \underbrace{p(x|\theta)}_{\text{(constant)}}]$$

$\delta_c(x)$ appears to be a fancy notation for the following: for each class $c$, determine $\delta_c(x)$ for a fixed $x$. This tells you to what extent $x$ belongs to each class, $c$.

  • $\begingroup$ thanks. What about the $|2 \pi \Sigma_c |$ factors? $\endgroup$
    – Tarrare
    Feb 5 '16 at 8:58
  • $\begingroup$ In the formulation, all $\Sigma_c=\Sigma$. So the determinant factors cancel out. $\endgroup$ Feb 23 '16 at 5:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.