3
$\begingroup$

Let's consider I have a data set of student details. Age would be a typical feature in such a data set. Just because there are typically fewer people aged above 40 in such a data set, which is expected given it involves student records, should they be eliminated or handled differently ? The fact that older students behave differently and therefore might have an impact on the classification itself cannot be ignored.

My question is:

If I eliminate the outliers or handle them differently, am I not causing information loss. Would it be worth homogenizing the data set to students in the typical age group of 20-30 or 20-25 in order to run a classification model? If I were to do this, I wouldn't know if "Age" is impacting the classification. In fact, could rather remove "Age" as a feature in this case as they won't impact the classification? Also, I wonder if this is similar to the class imbalance problem where one class (valid transactions) has more representation than the other(fraud transactions) because of it's inherent distribution. Likewise, older students are naturally fewer in the data set.

Should I or should I not handle outliers similar to the "Age" example. If so, how ?

Option 1: Can I bin the Age as 20-25, 25-30 etc. ? But that would be arbitrary ? Also, in this case do I keep both the original Age and the new binned feature ?

Option 2: Add another feature as outlier/non-outlier based on Age < 40 and Age > 40. Threshold 40 again is arbitrary. It has now become a binary variable.

Option 3: This post recommends creating an augmented class label which is akin to removal of outliers.

Option 4: Try Random Forest as they are robust in handling outliers ? I don't want to limit to trying RF. I want to try logistic regression, SVM etc.

Option 5: Remove records of outliers from the data set.

Option 6: Do nothing, and use data set as given as the outliers carry meaning.

Option 7: It depends on number of outliers ? The number of values and not the value itself decided if the feature is an Outlier? Is there any rule of thumb such as, if out of 100, I have say 10 values for the feature "Age" > 40, it needs to be removed but, having 20 values although very far away from typical mean / median would mean that it is not an outlier ?

I am very confused. Please advice.

Edit: This POST suggests discretization / binning should be avoided EVEN IF the variable is skewed. That said, the extreme age values are valid values that fall in the tail. I don't have a skewed distribution.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
$\begingroup$

Another solution is to use robust classification. For instance, you can look at RANSAC techniques (which is kind of a downsampling technique) or techniques from scikit-learn-extra. For instance this example (which implement something similar to logistic regression), this type of technique will not eliminate the outliers, it will take them into account but in a very downweighted way so as not to perturb the prediction too much.

Improve this answer
$\endgroup$
0
$\begingroup$

I'd probably first try the things with the least impact on the data, and see what you get - use the data as is where possible, and check your results. If you find students in some age bracket are being typically mis-classified, you can handle the outliers using one or more of the options you suggested. It's good practice to have a training, validation, and test set of data if you're going to be playing with multiple model types and pre-processing options (note this is three sets, rather than the typical two). Refining your models gets done on the training and validation sets, and then once you've got an approach you're happy with you can confirm it on the test set.

You may also want to look into over/undersampling as techniques for dealing with class imbalance.

Discretising age (option 1 or 2) would make sense - it will occur within RF anyway. It is an arbitrary choice if you do it yourself, but it's also common for good reason - it associates some data points more strongly, in this case forcing (say) all ages over 40 to be treated in the same way, giving you more data points for that class. You wouldn't use both the continuous feature and the discretised feature together if you do that, though you could use the continuous feature in one model and the discretised feature in another. This paper concludes that discretising continuous variables should be avoided, but it is acceptable when the distribution is skewed - which in your case it is.

Improve this answer
$\endgroup$
10
  • $\begingroup$ There are lots of posts that discourages binning such as: this and this and this. Also, I am wondering if the problem of having few outliers is similar to class imbalance and therefore treated similarly ? $\endgroup$
    – learner
    Commented Sep 4, 2020 at 9:23
  • 2
    $\begingroup$ I by and large agree with this answer except as a first attempt I wouldn't bin. I wouldn't actually apply any special treatment for the outliers. There are enough methods that can handle outliers particularly if they behave in a systematical way that a flexible classification method can pick up. Some methods will be affected but if they are, you'll find out using (maybe double) cross-validation. Obviously one can do in depth analysis of results and see whether even the best method you find is affected, but I wouldn't expect that if you try out enough (like support vector, random forests etc.). $\endgroup$
    – Christian Hennig
    Commented Sep 4, 2020 at 9:28
  • 1
    $\begingroup$ Ah, my bad, yes it does split automatically - but the principle is still the same in that binning is happening. I agree with @Lewian and would not bin to start with; sorry if that wasn't clear. As to whether to treat outliers as class imbalance, I think it depend a bit on the context. For age, I would. For other sorts of measurements, maybe not - it would depend on whether the outlier was likely to be indicative of some rare class, or measurement error $\endgroup$
    – Elenchus
    Commented Sep 4, 2020 at 9:56
  • 1
    $\begingroup$ I'd be using precision and recall as performance measures here, especially to start with. You want to know if it can discriminate correctly between the classes $\endgroup$
    – Elenchus
    Commented Sep 4, 2020 at 10:23
  • 1
    $\begingroup$ Thank You @Elenchus I will look into he paper you have suggested. This post is a question I had asked and the answer still discourages binning even if the variable is skewed. That said, is the "Age" skewed ? I think they are valid but extreme values that fall on the right tail and not skewed. $\endgroup$
    – learner
    Commented Sep 4, 2020 at 10:34
0
$\begingroup$

Firstly, define the question you want answered.

Are you examining the school population? How likely someone is to apply for a dorm?

A properly define question will help you know what records (students) to look at.

Secondly, and in more general terms outliers (Age = 40) are different than anomalies (Age =302). Although in statistics we generally use the word outliers to mean both.

Anomalies are always removed.
Outliers are valid data points and removal depends on the question being asked.

Options 1,2 and 3
Can be done but check against baseline and/or use feature selection or importance to see if they had any impact.

Option 4
Some ml methods handle outliers better than others

Options 5,6,7
Try each but check against a baseline

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.