Let's consider I have a data set of student details. Age would be a typical feature in such a data set. Just because there are typically fewer people aged above 40 in such a data set, which is expected given it involves student records, should they be eliminated or handled differently ? The fact that older students behave differently and therefore might have an impact on the classification itself cannot be ignored.
My question is:
If I eliminate the outliers or handle them differently, am I not causing information loss. Would it be worth homogenizing the data set to students in the typical age group of 20-30 or 20-25 in order to run a classification model? If I were to do this, I wouldn't know if "Age" is impacting the classification. In fact, could rather remove "Age" as a feature in this case as they won't impact the classification? Also, I wonder if this is similar to the class imbalance problem where one class (valid transactions) has more representation than the other(fraud transactions) because of it's inherent distribution. Likewise, older students are naturally fewer in the data set.
Should I or should I not handle outliers similar to the "Age" example. If so, how ?
Option 1: Can I bin the Age as 20-25, 25-30 etc. ? But that would be arbitrary ? Also, in this case do I keep both the original Age and the new binned feature ?
Option 2: Add another feature as outlier/non-outlier based on Age < 40 and Age > 40. Threshold 40 again is arbitrary. It has now become a binary variable.
Option 3: This post recommends creating an augmented class label which is akin to removal of outliers.
Option 4: Try Random Forest as they are robust in handling outliers ? I don't want to limit to trying RF. I want to try logistic regression, SVM etc.
Option 5: Remove records of outliers from the data set.
Option 6: Do nothing, and use data set as given as the outliers carry meaning.
Option 7: It depends on number of outliers ? The number of values and not the value itself decided if the feature is an Outlier? Is there any rule of thumb such as, if out of 100, I have say 10 values for the feature "Age" > 40, it needs to be removed but, having 20 values although very far away from typical mean / median would mean that it is not an outlier ?
I am very confused. Please advice.
Edit: This POST suggests discretization / binning should be avoided EVEN IF the variable is skewed. That said, the extreme age values are valid values that fall in the tail. I don't have a skewed distribution.