Handling Features with Outliers in Classification Let's consider I have a data set of student details. Age would be a typical feature in such a data set. Just because there are typically fewer people aged above 40 in such a data set, which is expected given it involves student records, should they be eliminated or handled differently ? The fact that older students behave differently and therefore might have an impact on the classification itself cannot be ignored.
My question is:
If I eliminate the outliers or handle them differently, am I not causing information loss.
Would it be worth homogenizing the data set to students in the typical age group of 20-30 or 20-25 in order to run a classification model? If I were to do this, I wouldn't know if "Age" is impacting the classification. In fact, could rather remove "Age" as a feature in this case as they won't impact the classification? Also, I wonder if this is similar to the class imbalance problem where one class (valid transactions) has more representation than the other(fraud transactions) because of it's inherent distribution. Likewise, older students are naturally fewer in the data set.
Should I or should I not handle outliers similar to the "Age" example. If so, how ?
Option 1:
Can I bin the Age as 20-25, 25-30 etc. ? But that would  be arbitrary ? Also, in this case do I keep both the original Age and the new binned feature ?
Option 2:
Add another feature as outlier/non-outlier based on Age < 40 and Age > 40. Threshold 40 again is arbitrary. It has now become a binary variable.
Option 3:
This post recommends creating an augmented class label which is akin to removal of outliers.
Option 4:
Try Random Forest as they are robust in handling outliers ? I don't want to limit to trying RF. I want to try logistic regression, SVM etc.
Option 5:
Remove records of outliers from the data set.
Option 6:
Do nothing, and use data set as given as the outliers carry meaning.
Option 7:
It depends on number of outliers ? The number of values and not the value itself decided if the feature is an Outlier? Is there any rule of thumb such as, if out of 100, I have say 10 values for the feature "Age" > 40, it needs to be removed but, having 20 values although very far away from typical mean / median would mean that it is not an outlier ?
I am very confused. Please advice.
Edit: This POST suggests discretization / binning should be avoided EVEN IF the variable is skewed. That said, the extreme age values are valid values that fall in the tail. I don't have a skewed distribution.
 A: Another solution is to use robust classification. For instance, you can look at RANSAC techniques (which is kind of a downsampling technique) or techniques from scikit-learn-extra. For instance this example (which implement something similar to logistic regression), this type of technique will not eliminate the outliers, it will take them into account but in a very downweighted way so as not to perturb the prediction too much.
A: I'd probably first try the things with the least impact on the data, and see what you get - use the data as is where possible, and check your results. If you find students in some age bracket are being typically mis-classified, you can handle the outliers using one or more of the options you suggested. It's good practice to have a training, validation, and test set of data if you're going to be playing with multiple model types and pre-processing options (note this is three sets, rather than the typical two). Refining your models gets done on the training and validation sets, and then once you've got an approach you're happy with you can confirm it on the test set.
You may also want to look into over/undersampling as techniques for dealing with class imbalance.
Discretising age (option 1 or 2) would make sense - it will occur within RF anyway. It is an arbitrary choice if you do it yourself, but it's also common for good reason - it associates some data points more strongly, in this case forcing (say) all ages over 40 to be treated in the same way, giving you more data points for that class. You wouldn't use both the continuous feature and the discretised feature together if you do that, though you could use the continuous feature in one model and the discretised feature in another. This paper concludes that discretising continuous variables should be avoided, but it is acceptable when the distribution is skewed - which in your case it is.
A: Firstly, define the question you want answered.
Are you examining the school population? How likely someone is to apply for a dorm?
A properly define question will help you know what records (students) to look at.
Secondly, and in more general terms outliers (Age  = 40) are different than anomalies (Age =302). Although in statistics we generally use the word outliers to mean both.
Anomalies are always removed.
Outliers are valid data points and removal depends on the question being asked.
Options 1,2 and 3
Can be done but check against baseline and/or use feature selection or importance to see if they had any impact.
Option 4
Some ml methods handle outliers better than others
Options 5,6,7
Try each but check against a baseline
