0
$\begingroup$

I have a dataset with right skewed data. It represent frequency of candidates vs their TTB (which is essentially the number of days)

enter image description here

Now the TTB values less than 14 or so are possible. But not with as much frequency. Some of it has been incorrectly entered into the system. There is no way of ascertaining how many of them are incorrect.

So I want to reduce the density of data starting from the lower values until about 14 or so. Is there any statistical method to do that?

I thought of (Median - 2.5*Median absolute deviation), but my median is 22 and MAD is 13. I also thought of double MAD but in that case my MAD is 0.

Can someone suggest any approach to this?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Would it be correct to paraphrase your question as saying "I know there are some bad data here, but I don't know specifically which ones or how many of them there are, so is there some formula that will tell me those quantities"? $\endgroup$
    – whuber ♦
    Commented Dec 7, 2015 at 22:45
  • 1
    $\begingroup$ What's the underlying problem you're trying to solve? (i.e. why are you trying to do this "reducing density of outliers" -- what's it for?); it may make more sense to help you tackle the original problem in the presence of potentially contaminated data. $\endgroup$
    – Glen_b
    Commented Dec 7, 2015 at 23:02

1 Answer 1

Reset to default
1
$\begingroup$

In the data as you have presented it, none of the low numbers are outliers by any reasonable definition - unless you make additional assumptions, although some may be data entry errors. However, since you say "there is no way of knowing how many of them are incorrect" there is nothing you can do.

By most definitions some of your high numbers may be outliers, but it really depends on what distribution the variable should have.

Are you sure there is no way of knowing how many are incorrect? Because, if that's right, the only solutions are either to start over and get new data or to use the data you've got. It would not be sound to just drop some cases.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.