3
$\begingroup$

I have a data set in which I am trying to find outliers. I am using python libraries to get the Z-score value using below code :

df['z_score']=stats.zscore(df[column_Name])
new_df=df.loc[df['z_score'].abs()>3]

Now the problem is that I get a good percent of my sample data which is having Z-Score > 3 or <-3. And due to which I cant drop it.

So, I checked the Z-Scores for all these columns and rows. The value of Z-Score is ranging from -17 to +20. Is it normal to get so high values of Z-Scores. And what does it shows about my data?

And in this case, how should I proceed, clearly I cant have Z-Score compared with 3. So, how do we do this in real world.

I am new to data science, I googled but did not find much help regarding this. So any leads will be appreciated.
Also, I am not able to understand this range of -5 to 10 which gets displayed at the bottom of box plot. If I look at that, it looks like the data beyond this value of -5 to 10 is my outlier.

enter image description here

$\endgroup$
  • 4
    $\begingroup$ Is $z$-score basically the mean value divided by standard deviation? In that case it is perfectly normal. But what do you want to achieve with your analysis? $\endgroup$ – Ott Toomet Jun 6 at 19:53
  • $\begingroup$ Please do say what you want to do with your data. A number of common tests make assumptions that this tendency to have extreme points (the high kurtosis I mention in my answer) might violate quite severely. $\endgroup$ – Dave Jun 6 at 20:07
  • $\begingroup$ i wanted to find out the outliers and drop them to better tune my model $\endgroup$ – Onki Jun 7 at 0:34
  • $\begingroup$ We generally discourage dropping extreme points. What do you want to model? $\endgroup$ – Dave Jun 7 at 5:51
  • 1
    $\begingroup$ The master is against outlier detection: statmodeling.stat.columbia.edu/2014/06/02/…. Plenty of people will do it, but Gelman is right that statisticians tend not to think highly of it. What would you want to do with your data after you’ve discarded the outliers, though? $\endgroup$ – Dave Jun 7 at 15:49
10
$\begingroup$

This is totally fine. It might be inconvenient, but it doesn’t meant that there’s something wrong with the data.

What it means is that your data set is more prone to extreme observations than a normal distribution with the same variance. For a norma distribution, you have about a $0.06\%$ chance of getting an observation with a z-score of magnitude greater than $3$, and it’s extraordinarily unusual to observe z-scores with magnitudes like $17$ and $20$.

So you don’t have a normal distribution.

This is related to a quantity called kurtosis, which quantifies the propensity of a distribution to have extreme values. Every normal distribution has a kurtosis of $3$. If you stick your data into R and call kurtosis in the moments package, I would expect you to get quite a bit higher value than 3. The Python implementation, since you’re into Python, is scipy.stats.kurtosis, though I think scipy subtracts 3 to give you the so-called excess kurtosis.

| cite | improve this answer | |
$\endgroup$
7
$\begingroup$

$Z$-scores with magnitude $\sim 15$ are extremely uncommon given that your data is drawn from a normal distribution. If the underlying population has heavy tails (i.e. $t$ or Cauchy distribution) or if there is heavy skew (i.e. exponential, lognormal) then it is not uncommon for a $Z$-score to be much larger than $3$.

For example, consider taking $n=10000$ draws from (i) a normal distribution, (ii) a $T$ distribution with $3$ degrees of freedom and (iii) an Exponential distribution. This figure shows the Z-scores corresponding to each distribution. Notice that the range of the Z-scores corresponding to the $t$-distribution is quite extreme, i.e. $-15$ to $15$.

enter image description here

So what does this tell you about your data? I would say that you can safely conclude that a normal distribution is not an appropriate model for your data. Based on your provided boxplots, the data is generated by a distribution with heavy tails.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.