Why prediction of a predicted variable from a discriminant analysis is imperfect

Question

I am puzzled by something I found using Linear Discriminant Analysis. Here is the problem - I first ran the Discriminant analysis using 20 or so independent variables to predict 5 segments. Among the outputs, I asked for the Predicted Segments, which are the same as the original segments for around 80% of the cases. Then I ran again the Discriminant Analysis with the same independent variables, but now trying to predict the Predicted Segments. I was expecting I would get 100% of correct classification rate, but that did not happen and I am not sure why. It seems to me that if the Discriminant Analysis cannot predict with 100% accuracy it own predicted segments then somehow it is not a optimum procedure since a rule exist that will get 100% accuracy. I am missing something?

Note - This situation seems to be similar to that in Linear Regression Analysis. If you fit the model $y = a + bX + \text{error}$ and use the estimated equation with the same data you will get $\hat{y}$ [$= \hat{a} + \hat{b}X$]. Now if you estimate the model $\hat{y} = \hat{a} + \hat{b}X + \text{error}$, you will find the same $\hat{a}$ and $\hat{b}$ as before, no error, and R2 = 100% (perfect fit). I though this would also happen with Linear Discriminant Analysis, but it does not.

Note 2 - I run this test with Discriminant Analysis in SPSS.

Answer 1

This is quite normal in case of machine learning -- it does not need to be optimal, it must be general.

Answer 2

I am having troubling following your reasoning, but here are some things you should consider.

Generally, the harder you fit a model to your training data, the worse the model will perform on independent validation data sets. By over-fitting the model to the training set, you risk capturing predictor-response relationships that are particular to the training set you are using. These relationships are likely due to random chance. When building a model for classification, you want to only capture the predictor-response relationships that are common to all training sets. This is requires careful selection of the right size model (big enough to capture the true predictor-response relationship, small enough to not to overfit to your particular training set.)

Also, the fact that a linear regression gives an R^2 of 1 doesn't mean much. For example, I can generate a 101 X 100 matrix of N(0,1) observations, take the first column to be the "response", and the other 100 columns to be "predictors." This will give me an R^2 value of 1, even though the "response" and "predictors" are independent (assuming the rows/columns are linearly independent, which they will be with probability 1 if they are all N(0,1) observations.) So in your n=5 observation, p=20 predictor case, you can choose any 5 predictors and get a perfect fit. R^2 is generally a pretty poor model assessment metric.

Also, unless you are certain the conditional distribution of the predictors is multivariate normal and that the predictors have a common covariance matrix, LDA may not be the best choice here. There are several better nonparametric/semiparametric methods available.

Maybe you can clarify your post a little bit to get a better response.