1
$\begingroup$
----------------------------------
  X         gender location   type
----------------------------------
  33.2      f      AAA        xxx
  44.0      f      BBB        zzz
  12.8      f      AAA        yyy
  8.1       f      BBB        yyy
  65.3(*)   m      BBB        yyy
  73.1(*)   m      BBB        xxx
  ................................
----------------------------------

Consider the data above.

We assume the values in the X column with an asterisks as outliers among other X values.

(Assume we're detecting outlier among X column using the simple mean/deviation. Numbers with asterisk may not be actual outliers in this example, but you got the idea.)

But there are other columns with categorical data which are describing the values in the X column (breakdowns), so I need to identify on which level the problem exists.

After selecting an initial outlier I need to detect the exact level of the problem.

Example: if we will take only {gender, location} columns and throw the type (and aggregate the values), we will see that the outlier exists here too:

--------------------------
  gender location     X
--------------------------  
    f      AAA      46.0
    f      BBB      52.9
    m      BBB      138.4 (*)
--------------------------

so the actual problem is in the point with gender m and location BBB, because if we will throw away gender column and look at the table:

-------------------------
  location type     X
-------------------------  
    AAA    xxx     33.2
    AAA    yyy     12.8
    BBB    xxx     73.1
    BBB    yyy     73.4
    BBB    zzz     44.8
-------------------------  

we will see that there are no apparent outliers. So no matter which type is, there is actually one outlier.

Problem

There are can be much more categorical columns, so generating all combinations and performing multiple checks are time consuming. Also, there can be situations when such checks will not work (for example one of the categories in column contains values that are much more greater than the others, so these simple checks can be wrong).

May be there is a more sophisticated method to detect the exact level of the outliers?

$\endgroup$

closed as unclear what you're asking by Michael Chernick, kjetil b halvorsen, mdewey, Peter Flom Oct 6 '17 at 19:33

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ What exactly do you mean with "level"? It is clear that for gender "m", location "BBB" and type "yyy" you have an outlier, but why do you want to aggregate some categories? Also multiple correspondence analysis may help you to identify these outlying cells quickly. $\endgroup$ – Knarpie Oct 6 '17 at 11:56
  • $\begingroup$ @Knarpie for example, there can be another gender m, location BBB and type zzz with some value, which is outlier in initial data too.. If we remove the type, it will be aggregated point from these two outliers, and eventually it can be outlier in the aggregated data without type column too, so the "level" is exact combination where point appears to be an outlier. $\endgroup$ – Vsevolod Oct 6 '17 at 15:12
  • $\begingroup$ @Knarpie I've updated the example to be more clear. $\endgroup$ – Vsevolod Oct 6 '17 at 18:31