I have a long vector (~1.000.000 entries) of integers from 1 to 2500 each of which expresses the number of occurrences of some sort of event for a certain user. The data can be illustrated as:
Apparently there is a lot of nose in there so my goal is to compute two cutoff values that omit both upper- and lower- tails. My problem, however, is how to compute efficiently these two cutoffs (statistically speaking) in right-skewed distributions. Any help is appreciated.
('median', 1.0) ('mean', 3.9751942718652025) ('std', 13.888936478353791) ('min', 1) ('max', 2434)
I consider very low and very high occurrences (of some sort of events) as noise / scam / non-reliable data and thus I want to get rid of them. If I had to do this by hand I would say that anything bellow 10 or above 100 should be removed. But, apparently, this is only a personal judgement! In other words, I would like to come up with idea that computes these two numbers (one for the lower and one for the upper bound) that is based on some mathematical rules. The problem is, I don't know how to start!