What Is the Loss (Objective) Function for Linear Discriminant Analysis (LDA)?

As many algorithms can be viewed as optimization problems through the Loss function, I was wondering if such a loss function existed for LDA (linear classification). And if yes, what would it be ?

I already know these:

• For SVM: $L_{Hinge}(y,x,w)=max(0,1-yw^tx),y\in\{-1,1\}$
• For Logistic Regression: $L_{Log}(y,x,w) =log(1+e^{-yw^tx}), ,y\in \{-1,1\}$,
• For Perceptron: $L_{Perceptron}(y,x,w)=max(0,-yw^tx),y\in \{-1,1\}$

where $x$ stand for the feature, $y$ the label and $w$ the parameter of the hyperplane we have to find.

Edit: Thanks a lot for your help but i don't know if i am clear enough. I am looking for the loss for 1 instance as if i were about to implement a stochastig gradient descent. Perhaps i miss something among your answers...

• the log likelihood – Haitao Du May 25 '18 at 18:10
• @hxd1011 negative log likelihood :-) – Łukasz Grad May 25 '18 at 18:46
• @ŁukaszGrad, Could you write the explicit form? – Royi May 25 '18 at 19:08
• – Royi May 25 '18 at 19:30
• For binary $y$, LDA is equivalent to linear regression of $y$ on $X$. So the loss function is simply squared error. CC @Royi. – amoeba says Reinstate Monica May 25 '18 at 19:59

1 Answer

LDA is also called Fisher’s linear discriminant. I refer you to page 186 of book “Pattern recognition and machine learning” by Christopher Bishop. The objective function that you are looking for is called Fisher’s criterion J(w) and is formulated in page 188 of the book. The Fisher criterion is defined to be the ratio of the between-class variance to the within-class variance.

• I'm not sure this is the loss function. This is the optimization objective function. I think @laurent is looking for the loss of wrong label like in his examples. Of course they are closely related and probably the objective function can be derived from the loss function. – Royi May 25 '18 at 19:38