0
$\begingroup$

I have a long vector (~1.000.000 entries) of integers from 1 to 2500 each of which expresses the number of occurrences of some sort of event for a certain user. The data can be illustrated as:

bar plot

Apparently there is a lot of nose in there so my goal is to compute two cutoff values that omit both upper- and lower- tails. My problem, however, is how to compute efficiently these two cutoffs (statistically speaking) in right-skewed distributions. Any help is appreciated.

EDIT 1

('median', 1.0)
('mean', 3.9751942718652025)
('std', 13.888936478353791)
('min', 1)
('max', 2434)

EDIT 2

I consider very low and very high occurrences (of some sort of events) as noise / scam / non-reliable data and thus I want to get rid of them. If I had to do this by hand I would say that anything bellow 10 or above 100 should be removed. But, apparently, this is only a personal judgement! In other words, I would like to come up with idea that computes these two numbers (one for the lower and one for the upper bound) that is based on some mathematical rules. The problem is, I don't know how to start!

Improve this question
$\endgroup$
5
  • $\begingroup$ You need to supply cutoff criteria that are objective and can be quantified. There is no universal rule. To make progress with this question then, please edit it to include information about how these cutoffs will be used in analyzing or presenting your data. $\endgroup$
    – whuber ♦
    Dec 17, 2014 at 16:02
  • $\begingroup$ @whuber no idea what sort of criteria should I provide here... that's why I decided to write to you; I just want some hints on how to define them. btw, I have updated my question in order to provide some basic statistics. $\endgroup$
    – user2295350
    Dec 17, 2014 at 16:24
  • $\begingroup$ Hint: a threshold is defined by giving a number. But what are you going to do with that number? What meaning does this process have for you? How do you want it to affect your analysis or your visualizations? If you don't tell us, we can't help you! $\endgroup$
    – whuber ♦
    Dec 17, 2014 at 16:32
  • $\begingroup$ @whuber ok, so I consider very low and very high occurrences (of some sort of events) as noise / scam / non-reliable data and thus I want to get rid of them. If I had to do this by hand I would say that anything bellow 10 or above 100 should be removed. But, apparently, this is only a personal judgement! In other words, I would like to come up with idea that computes these two numbers (one for the lower and one for the upper bound) that is based on some mathematical rules. The problem is, I don't know how to start! $\endgroup$
    – user2295350
    Dec 17, 2014 at 16:42
  • $\begingroup$ OK, that's a good start. (Please consider editing the question to include your comment.) I wonder, though, whether there's anything inherently wrong or inferior about using judgment compared to mathematics. Mathematical formulas cannot help you until they are informed by quantitative statements of your objectives. Usually that's done by stating how much it would hurt you to weed out values that are not noise and how much it would hurt to include values that are noise. There will usually then be an optimal balance between the two costs, leading to a formula for the threshold. $\endgroup$
    – whuber ♦
    Dec 17, 2014 at 16:46

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.