I have a classification dataset with 148 input (independent) features, most of which are expected beforehand to be irrelevant. So, at the moment, I am using feature selection methods to discard the irrelevant features before removing outliers.

From one perspective, this seems to be a good idea since an outlier in one feature space may not be so in a different (reduced) feature space. On the other hand, however, the presence of the outliers in the dataset, on which the feature selection methods are applied, may negatively influence the selected features.

The feature selection methods I am using are information gain, gain ratio, symmetrical uncertainty, fisher score and gini index.

The question is: what is the "most" appropriate order of applying these two dataset preprocessing steps?

  • $\begingroup$ Are you using the response to discard the irrelevant features before removing outliers? $\endgroup$ – James Sep 10 '15 at 18:41
  • $\begingroup$ Yes I am. The input to the outlier removal algorithm is a dataset with less than 60 independent features. $\endgroup$ – PatternRecognition Sep 10 '15 at 19:10

Outliers can affect your decision about whether the feature is irrelevant, so you have to remove them first. More importantly, even if outliers are not a problem, you are going to overfit your model by pre-selecting the relevant features. What you can do is to reduce the dimensionality of feature space by using a method that does not need the response, eg, PCA.

  • 3
    $\begingroup$ How about the following paragraph in [this article][1]: "Four important properties of high - dimensional data... (4) Almost every point is an outlier". [1]: goo.gl/O0bXTX $\endgroup$ – PatternRecognition Sep 11 '15 at 13:29
  • 2
    $\begingroup$ Let me give a simple example to illustrate the concern raised in the question. In a 2-d space, the point (3,50) may seems to be an outlier relative to the points (2,1) and (4,1). However, removing the y axis will make this outlier point a perfectly normal one. $\endgroup$ – PatternRecognition Sep 11 '15 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.