For example, we have a dataset in which the samples contain 400 features. In this case, if we try to perform classification, we get very low accuracy because our learning model will become very complex. This automatically results in overfitting.
In this case, dimensionality reduction is inevitable. Generally I prefer feature selection because I do not want to lose the track of original features. One of the most successful feature selection method is Pearson Correlation Coefficient analysis. It investigates the relationship between the features and labels. The most related ones are generally the most effective ones for classification.
My question occurs in this part. Someone told me that Pearson Correlation coefficient is time consuming because you have to compute correlation matrix. If you have many features, its computation consumes too much time. Instead, you can use "mean investigation method"
I asked him what "mean investigation method" is. He explained it like this:
"You have to split your dataset into the classes. Then, you have to compute mean of all values that each feature took. This computation must be performed separately for each class. After this process, you obtain n mean values, which is equal to the number of features. For example, you have 3 features, and you compute the mean values like this:"
sample 1(class 1) sample 2(class 0)
feature 1 -> 0.57 0.82
feature 2 -> 0.94 2.33
feature 3 -> 0.12 4.55
"Feature 3 is the best one, since it is the feature that separates distribution of two classes from each other perfectly."
I thought this method, and it made very sense, since it prevents class overlapping. It enables us to choose the feature that makes our class distributions most distant. We can sort distance between mean values of each class over all features, and we can choose the largest n features whose mean values are farthest.
What do you think about this dimensionality reduction method ?
