As many algorithms can be viewed as optimization problems through the Loss function, I was wondering if such a loss function existed for LDA (linear classification). And if yes, what would it be ?

I already know these:

  • For SVM: $L_{Hinge}(y,x,w)=max(0,1-yw^tx),y\in\{-1,1\}$
  • For Logistic Regression: $L_{Log}(y,x,w) =log(1+e^{-yw^tx}), ,y\in \{-1,1\}$,
  • For Perceptron: $L_{Perceptron}(y,x,w)=max(0,-yw^tx),y\in \{-1,1\}$

where $x$ stand for the feature, $y$ the label and $w$ the parameter of the hyperplane we have to find.

Edit: Thanks a lot for your help but i don't know if i am clear enough. I am looking for the loss for 1 instance as if i were about to implement a stochastig gradient descent. Perhaps i miss something among your answers...


LDA is also called Fisher’s linear discriminant. I refer you to page 186 of book “Pattern recognition and machine learning” by Christopher Bishop. The objective function that you are looking for is called Fisher’s criterion J(w) and is formulated in page 188 of the book. The Fisher criterion is defined to be the ratio of the between-class variance to the within-class variance.

  • $\begingroup$ I'm not sure this is the loss function. This is the optimization objective function. I think @laurent is looking for the loss of wrong label like in his examples. Of course they are closely related and probably the objective function can be derived from the loss function. $\endgroup$
    – Royi
    May 25 '18 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.