Most of the methods I found to detect the outliers in a dataset deal with the values by column (one by one). For each column it detects the values that are too far from the median or the mean.
Is there a way to detect the incoherences by relation between features ? An example will explain it better. I have a set of houses :
df = pd.DataFrame({"Nbr of rooms":[2,2,3,8],"Size":[330,2000,440,3200]})
Nbr of rooms Size
0 2 330
1 2 2000
2 3 440
3 8 3200
With the methods by columns the last rows will be delete because 8 rooms is far from the mean of number of rooms. But I think the outlier should be the house number 1 (row 2), because if the number of rooms is two the size is probably not 2000 square feet.
I thought about comparing the coefficient between each column (col1/col2) and delete the outlier :
df['coeff'] = df.iloc[:,1] / df.iloc[:,0]
Nbr of rooms Size coeff
0 2 330 165.000000
1 2 2000 1000.000000
2 3 440 146.666667
3 8 3200 400.000000
There is two limits to this one
- it only compares two features
- it suggests that the relation between two features is always linear
Is there a smarter way to do that ?