Suppose, I have samples from APPLE, MANGO, and ORANGE --- these 3 classes. The goal is to do multiclass classification.
Now, let's say, I have calculated 4 features from all of the 3 classes.
By rules and practice, we have already known that while considering features from different classes, we need to make sure there is shall not be correlated features into the final dataset, to avoid the overfitting problem while doing the classification task (or we may say to avoid the multicollinearity problems).
By keeping those thing into mind, if I have considered the following approach will it make any sense ?
Step 1: I am going to consider 4 features from all the 3 classes.
Step 2: Now I check the feature correlations for all the classes and keep the correlated features. Suppose, I have calculated the correlation coefficient from 3 sets of features, i.e., APPLE SET (APPLEF1, APPLEF2,APPPLEF3, and APPLEF4), MANGO SET (MANGOF1, MANGOF2,MANGOF3, and MANGOF4) and ORANGE SET (ORANGEF1, ORANGEF2,ORANGEF3, and ORANGEF4) to keep the correlated features.
Suppose, after calculating the correlation coefficient, I have a combined set of feature pool where I have kept the highly correlated features determined by the correlation coefficient. e.g., new combined set of features (APPLEF1, APPLEF2,MANGOF1, MANGOF2,ORANGEF1, ORANGEF2). So, here are total 6 features and they are highly correlated.
Step 3: Now, I will apply LDA on to those features to maximize the distances. That is how, I am dealing with the correlated features. Actually the idea is to increase the feature separability as they are highly correlated features.
Step 4: Finally I consider a classifier (Multiclass SVM), and apply, and obtained good results as my dataset is not too large.
So in short, here I have kept the highly correlated features (which I shall not do while doing multiclass classification), but then I have applied LDA to increase the distance among the feature space (by treating them in a supervised way). Thus by considering the correlated features, I am able to make a Multiclass classifier.
The final question is, IS THIS APPROACH REALLY SENSIBLE? I AM CURIOUS TO KNOW.
BECAUSE as I know before applying LDA, if you remove the correlated features that will make sense to remove the problem of multicollinearity. Some suggest to apply PCA before LDA.