1
$\begingroup$

I have a few questions. If you would be so kind to help.

so i am wondering, because this is pretty subjective (imo). What would you consider a "normal" percentage of outliers in your data set?

Lets say for example i have a data set of 10 million records and i perform a cluster analysis.

I am asking because i want to be sure that the cluster analysis i do actually does its' job. Would it be sensible to say that if you have an x amount of outliers or x percentage of them in your data, then my cluster analysis is not really accurate and i should either increase clusters count or change the method?

And if yes. How do you determine that?

Or at the very least examine the data again and rethink my strategy?

Regards,

Emil

Improve this question
$\endgroup$
5
  • $\begingroup$ There is no such a thing... In some cases single outlier may influence your results. Moreover, if you have more then "normal" amount of outliers you still have to deal with them somehow. $\endgroup$
    – Tim ♦
    Oct 25, 2016 at 7:18
  • $\begingroup$ @Tim Well.. My understanding is that there can always be some kind of a surge in the data. For example a really lucky day where you could sell a lot, or a nice day at the financial market that net you 5-6 times over the normal amount you usually earn. I am not concerned with the results (as in gross revenue for example or something like that). But if there is a pattern - you have low amount of sales 3 weeks of the month and then the last week you generate 95% of the sales for your target, then there is probably some pattern that i'd have to investigate further. That is what i want to do :-) $\endgroup$
    – Emil Filipov
    Oct 25, 2016 at 7:41
  • $\begingroup$ @Tim I was just wondering if there was some way to standardize outliers and everything around them. Just an idea, though. I wasn't hoping for much, i just wanted to hear what other people with way more experience then me think. :-) $\endgroup$
    – Emil Filipov
    Oct 25, 2016 at 7:44
  • $\begingroup$ But then you are not describing outliers... Outliers are datapoints that do not fit your model and can make it produce biased results. In your comment you rather seem to be talking about novelty detection. $\endgroup$
    – Tim ♦
    Oct 25, 2016 at 8:03
  • $\begingroup$ @Tim Well, now i learned something new. I should model with clean data (without outliers, i am relatively new, don't judge me :-D). I have to read a little bit more though.. I don't really notice a big difference between outliers and novelty observations.. $\endgroup$
    – Emil Filipov
    Oct 25, 2016 at 8:30

2 Answers 2

Reset to default
1
$\begingroup$

It depends on your distribution and the model behind it, and also on your definition of an outlier. If you expect a normal distribution of your data points, for example, then you can define an outlier as any point that is outside the $3 \sigma$ interval, which should encompass 99.7% of your data points. In this case, you'd expect that around 0.3% of your data points would be outliers. If it's significantly more, then you should probably look for an error in your data acquisition method. (Side note, if your number of data points is small, you should consider using Student's t-distribution instead of a normal distribution.)

So you need an expected distribution, then you need to define an interval in which you expect a certain percentage of your data based on that distribution, and then you can define anything outside that interval as an outlier.

It's important to note, however, that being an "outlier" does not really mean anything. It always depends on the context.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Well i have a data base that contains volumes of sales grouped by date,time. The idea is to identify if the sales team doesn't take its' job seriously 3/4ths of the month and starts generating sales and production requests at the end of the month making it hard on the production guys. The company though has no definition of "unusualness" so i can't really use regressions to model this. But you are right. It depends on the distribution of the data and its' overall fundamentals. I will probably write a paper on this after i finish (hopefully). Thanks for the tips, guys! $\endgroup$
    – Emil Filipov
    Oct 25, 2016 at 7:36
  • $\begingroup$ I'm not sure where the outliers come into this yet, but in order for me to understand this, you would need to explain your cluster analysis more in detail. It sounds to me like you could simply make a histogram of the generated sales/requests, look at that, and then define some sort of interval. In any case, I answered your question didn't I? $\endgroup$
    – PoorYorick
    Oct 25, 2016 at 7:59
  • $\begingroup$ I could do that (the histogram). This will be one of the things i will try actually. I think i will probably have to break down the data and analyse it a bit deeper to come to any conclusions. I will be using an R script to do the clustering which gives out the number and percentage of detected outliers. I will have to break down the code as well to understand exactly how it works so i can tweak it if needed. I am just thinking outloud in a forum, there are probably a lot of ways to do that, that i don't know. $\endgroup$
    – Emil Filipov
    Oct 25, 2016 at 8:07
  • $\begingroup$ Okay. If you think I've answered your original question sufficiently, I'd be grateful if you ticked my answer as "accepted". $\endgroup$
    – PoorYorick
    Oct 25, 2016 at 8:29
0
$\begingroup$

First of all you need a definition of outlier. Outlier just means unusual datapoint, so what people find unusual varies from dataset to dataset and even from person to person. I would even go so far as to say that it would in most cases be better to base your definition of an outlier on the proportion of outliers that definition produces, rather than the other way around.
The general problem of finding a good number of clusters has also no easy answers. Depending on your dataset and distance measure, multiple numbers of clusters might be perfectly reasonable. So to get any more detailed answers, you are going to need to show us your data and tell us what you want to use the clusters for.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.