LDA cutoff (decision boundary) value

I have run a linear discriminant analysis for the simple 2 categorical group case using the MASS package lda() function in R. With priors fixed at 0.5 and unequal n for the response variable of each group, the output basically provides the group means and the LD1 (first linear discriminant coefficient) value. There is no automatic output of the cutoff (decision boundary) value estimated that is later used to classify new values of the response variable into the different groups. I have tried various unsuccessful approaches to extract this value. It is obvious that in the simple 2 group case the value will be close to the mean of the 2 group means and that the LD1 value is involved (perhaps grand mean * LD1?). I am probably missing (misunderstanding?) the obvious and would appreciate being educated in this matter. Thanks. Regards,BJ

-
Please explain what is "LD1 value" in MASS output –  ttnphns Sep 23 '11 at 21:35
LD1 is the first linear discriminant coefficient –  BJessop Sep 24 '11 at 0:14
If the OP could explain precisely what LD1 is, there would probably not be a question. As I see it, you are not supposed to use the components of an lda-object from an lda-call directly, but rather use the plot or the predict methods implemented for these objects. That does not mean that it is not interesting to figure out what the components are, and I have tried to give some explanation and further references in my answer. –  NRH Sep 24 '11 at 7:54

The LD-vectors, which are in the scaling entry of the list returned by ldain R, are not so easily comprehended. The help page says that scaling is

      a matrix which transforms observations to discriminant
functions, normalized so that within groups covariance matrix
is spherical.


which is not super informative. Inner products with these vectors give the coefficients for a projection to a $K-1$ dimensional space (with $K$ groups, that is, a one-dimensional space if there are two groups). Everything is centered using the total mean of the predictors before projection. In this projection, classification happens to the group with the nearest mean, as measured by the usual euclidean distance, if the prior probabilities are equal. Perhaps the best thing to do to understand precisely how the computation of the predictions work is to read the R-code in MASS:::predict.lda. The MASS book by Ripley and Venables may also be useful, but I find that Chapter 3 in Ripley's other book Pattern Recognition and Neural Networks is more complete. Here you will find the complete explanation behind the computations implemented in the R function lda, and a more detailed description of what the projection actually is.

For the two group case things simplify, the LD1 vector is then proportional to $\hat{\Sigma}^{-1}(\hat{\mu}_2 - \hat{\mu}_1)$, and the linear discriminator is given by taking inner product with LD1. In the case of equal prior probabilities on the two groups you don't need to know the constant of proportionality and the cutoff is $\frac{1}{2}(\hat{\mu}_2 + \hat{\mu}_1)^T\text{LD1}$. With unequal priors it seems to me that you need to know the constant of proportionality to compute the cutoff in this formulation. In the alternative, and this is essentially what MASS:::predict.lda does, you can for a given $x$ compute the inner products $c = x^T \text{LD1}$ and $c_i = \hat{\mu}_i^T \text{LD1}$ for $i=1,2$ and then $$d_i = \frac{1}{2}(c - c_i)^2 - \log \pi_i.$$ Finally, you classify to the group with the smallest $d_i$.

-
Thanks for the references. In the simple 2 categorical group case that I am concerned with I had hoped that the lda function in R would allow easy extraction of the discriminant function threshold value that is calculated and used in the classification process applied to new data. Apparently it is not directly available. I have done the discrimination and classification steps on my data. I need the decision boundary value (in this simple case, a single value) for another purpose and general interest. In this simple case, if lda() does not provide it, how can I calculate it by hand? –  BJessop Sep 24 '11 at 13:47
@BJessop, I have tried to provide a more detailed explanation for the two group case. –  NRH Sep 26 '11 at 10:14

In the 2-group situation, the cut-off value of the discriminant function scores is simply the mean of the means of the scores for the groups (those means are also called "function's values at group centroids"). Say, if for group 1 the mean score is -.742 and for group 2 it is .576, then the cut-value for classifying is their average, -.083. This is true when prior probabilities are set to .5 and when discriminant function scores used aren't normalized in some way).

Note that my answer is "general": I'm not R-user and don't know how it is all in MASS.

-
Thanks for confirming part of my understanding. The MASS lda function uses normalized scores. In this simplest of cases, is there a hand-calculation way to adjust the groups grand mean (weighted by different n) for the within-groups covariance matrix? I will look for such a formula. –  BJessop Sep 24 '11 at 13:58
The above classification rule very arbitrarily assumes a cost/utility/loss function that may not be applicable. Surely the fact that the loss is assumed to be constant over subjects is suspect. You can get posterior probabilities from LDA but logistic regression is more robust as it does not assume a distribution for the Xs whereas LDA assumes multivariate normality. –  Frank Harrell Sep 25 '11 at 13:25