Outliers detection : mismatch between columns

Question

Most of the methods I found to detect the outliers in a dataset deal with the values by column (one by one). For each column it detects the values that are too far from the median or the mean.

Is there a way to detect the incoherences by relation between features ? An example will explain it better. I have a set of houses :

df = pd.DataFrame({"Nbr of rooms":[2,2,3,8],"Size":[330,2000,440,3200]})

   Nbr of rooms  Size
0             2   330
1             2  2000
2             3   440
3             8  3200

With the methods by columns the last rows will be delete because 8 rooms is far from the mean of number of rooms. But I think the outlier should be the house number 1 (row 2), because if the number of rooms is two the size is probably not 2000 square feet.

I thought about comparing the coefficient between each column (col1/col2) and delete the outlier :

df['coeff'] = df.iloc[:,1] / df.iloc[:,0]

   Nbr of rooms  Size        coeff
0             2   330   165.000000
1             2  2000  1000.000000
2             3   440   146.666667
3             8  3200   400.000000

There is two limits to this one

• it only compares two features
• it suggests that the relation between two features is always linear

Is there a smarter way to do that ?

Answer 1

One possibly sensible approach to take would be to regress the number of rooms onto floor space (or vice versa) and then examine the residuals (i.e. the difference between the prediction of the model and the actual value). Where a residual was particularly large (you could perhaps standardise them) this would suggest that there was some sort of incoherence between the two features (e.g., an unusually large house with a small number of rooms).

You could do this with linear regression, although this would still be making an assumption that the relationship between rooms and floor space was linear. Other types of regression (e.g. non-linear ones) may better represent the true relationship.

A regression residual based approach may not be as flexible if there were multiple features involved though.

Answer 2

If you want a method which applies for more than two variables you could consider using principal components and then plotting the data in the space of the first two components which will show you points which fall a long way from the others (perhaps). You could also look at the last few components which may reveal points which are responsible for other sots of anomaly. Not sure how you do this in Python, search for biplots which may help you.

Answer 3

Adding to @Ian_Fin's answer: I would suggest taking a look at the Mahalanobis distance and using $\chi^2$ test in order to extract probabilistic thresholds. Just pay attention to $D_{m}$ as it holds many priors.

Answer 4

How about doing so clustering (k-means, ...) which works on more than 2 features? Then small clusters might be outliers. The though part here is to get the distance-measure right, for example by calculating additional attributes like your 'coeff'.