Non parametric: How to detect different groups

Question

I have data sets for different serial numbers of devices. Those data are not following a normal distribution.

I would like to know which serial numbers are behaving differently from the overall distribution.

I have made some box plots to "see" which are different, and I've tried different non-parametric tests: Mood, Kruskal-Wallis, etc.

The way I did it:

• Sample 1: All the points belonging to that SN (pointsIN in the table below)
• Sample 2: All the points not belonging to that SN (pointsIN in the table below)
• Perform test (Sample 1, Sample 2)

Unfortunately, I have every time rejected H0, so I need to conclude every time that this serial number behaves differently.

Here's an example in attachment.

Can you tell me which test I should use?

Thanks a lot.

Python code:

import scipy.stats
import pickle
import pandas
data = pickle.load(open("example_dataset.pickle", "rb"))
def kruskalVsAll(data, values, pivot, alpha=0.05, plot=True):
    ret = []
    for subject in data[pivot].unique():
        filt = data[pivot] == subject
        dataIn = data.loc[filt, values]
        dataOut = data.loc[~filt, values]
        (h, p) = scipy.stats.kruskal(dataIn, dataOut)
        ret.append(
            {
                pivot: subject,
                'allPoints': data.shape[0],
                'inPoints': dataIn.shape[0],
                'outPoints': dataOut.shape[0],
                'H': h,
                'p': p,
                'h0': p >= alpha,
            }
        )
    ret = pandas.DataFrame(ret)
    ret.set_index(pivot, inplace=True)
    if plot:
        data.reset_index(inplace=False, drop=False).pivot(index='index', columns=pivot, values=values).plot(kind='box')
    return ret
kva = kruskalVsAll(data, 'value', 'sn')
print(kva)

Answer 1

The first two steps, 'prescanning' and doing one-vs.-all, may not be entirely defensible. Why not do KW on all SNs, this way you may be able to see which ones are the most different. Alternatively, you can test for a given degree of departure from the overall population mean (but that would change with each new SN added) or analyze the ratio of outliers/datapoints above a certain threshold (eg 24) to total n(SNx), which can then be analyzed using goodness-of-fit. But the latter still boils down to getting a p-value to support a subjective decision.

Another possibility would be to do some kind of change-point analysis to see whether there is, in fact, a distinct breakpoint in the variable your call 'behavior' to justify calling it a distinct abnormality, and using that as a threshold for the approach above.