2
$\begingroup$

Currently, I am building my analytics portfolio as part of the Google Data Analytics course. I chose the option to analyze Divvy Bike Sharing data for the year 2021. But now I'm currently stuck in the part where I need to identify outliers in the dataset. I'm focusing on the 'ride_length' column which shows the duration of each ride and I'm using two methods which are:

  1. IQR (data points that fall below 25th or above 75th percentile are outliers)
  2. 1% and 99% rule (data points that fall below 1% percentile or above 99% percentile are outliers)

Note: the ride_length column is counted in minutes

A) IQR METHOD

The first method that I use to detect outliers is the IQR proximity rule (The data points which fall below the 25th percentile or above the 75th percentile are outliers). Here's the code:

lower_bound_iqr <- quantile(df_2021_test$ride_length, 0.25)
upper_bound_iqr <- quantile(df_2021_test$ride_length, 0.75)

lower_bound_iqr
25% 
6.98 

upper_bound_iqr
75% 
20.98 

Key takeaways:

  1. ride_length that falls below 6.98 minutes is considered an outlier
  2. ride_length that falls above 20.98 minutes is considered an outlier

Then I count the percentages of outliers in the data:

outliers_iqr <- which(df_2021_test$ride_length < lower_bound_iqr | df_2021_test$ride_length > upper_bound_iqr)
(count(df_2021_test[outliers_iqr, ]) / count(df_2021_test)) * 100 

         n
1 49.89316

The result is that 49.89 % of data are considered outliers. I think this is too much data to exclude for the analysis to begin as it will reduce the accuracy of the analysis. Or am I wrong? Therefore I move to the second method

B) 1% and 99% Percentile Rule

This method state that data points that are far from the 99% percentile and less than 1% percentile are considered an outlier. Here's the code:

lower_bound <- quantile(df_2021_test$ride_length, 0.01)
upper_bound <- quantile(df_2021_test$ride_length, 0.99)

lower_bound
1% 
1.82 
upper_bound
99% 
115.63 

Key takeaways:

  1. ride_length that falls below 1.82 minutes are considered outliers
  2. ride_length that falls above 115.63 minutes (approx. 2 hours) are considered outliers

Again, I count the percentages of outliers in the data:

outliers <- which(df_2021_test$ride_length < lower_bound | df_2021_test$ride_length > upper_bound)
(count(df_2021_test[outliers, ]) / count(df_2021_test)) * 100

         n
1 1.982182

The result is that 1.98 % of data are considered outliers. I think this is fine to exclude for the analysis to begin as it will not reduce the accuracy of the analysis that much. Or am I wrong?

Here are my questions:

  1. When identifying outliers in the data, what should you choose between the two of the method above? Or is there another better method?
  2. Is my way of identifying outliers in the dataset correct? Or am I missing something?

I have detailed all of my steps to identify outliers above and again, it's not an error in the code it's just that I'm confused as to whether my method of identifying them is correct or if is there any better way or something that I miss.

$\endgroup$
1
  • 2
    $\begingroup$ It depends also on what you are going to do with the data. Will you plot the data? Then you do not need to do anything. Will you run a linear model? Maybe use robust regression. $\endgroup$ Jan 23, 2023 at 7:32

2 Answers 2

19
$\begingroup$

It should be obvious that 50% of your data will fall above the 75th or below the 25th centile, and that 2% is either above the 99th or below the 1st.

There is no data-driven answer to 'what is an outlier'. Data points might be far away from the rest of a distribution for many different reasons, and how you identify these and what you do with them should depend on why you think they occurred and what downstream analysis you are planning (ultimately what question you are asking of the data).

If you can specify these things you'll have a clearer idea of how to identify and handle outliers. Do not rely on simple data-driven rules.

$\endgroup$
2
  • 1
    $\begingroup$ Thank you for the explanation, George! I agree that we cannot simply remove outliers based on simple data-driven rules alone. I just came up with an idea to do a kind of A/B testing where I keep all the outliers and where I remove the outliers (the 1% and 99% method). Given that the data is public and we won't know the exact reason why the outliers are there, I think this project will be challenging. Anyway, thanks for shedding some light George! Appreciate it. $\endgroup$ Jan 22, 2023 at 14:01
  • $\begingroup$ Yes, a sensitivity analysis like this is usually a good idea. $\endgroup$ Jan 22, 2023 at 15:02
9
$\begingroup$

George is correct. Rarely do data-driven "rules" behind outlier detection/removal work alone. As an example using the starwars dataset in R, I have plotted the heights and masses of the Star Wars movie characters below:

#### Load Library ####
library(tidyverse)
theme_set(theme_bw())

#### Plot Obvious Outlier ####
starwars %>% 
  ggplot(aes(x=mass,
             y=height))+
  geom_point()+
  geom_smooth(method = "lm")+
  labs(x="Mass",
       y="Height",
       title = "Star Wars: Mass x Height")

You can see a very obvious outlier. This is Jabba the Hut, whose mass is several leagues above the others in this data. Here it is reasonable to assume that it greatly affects the regression plotted and doesn't model this relationship in a very accurate way.

enter image description here

We can remove it simply by filtering it with this code:

#### Remove Outlier and Plot ####
starwars %>% 
  filter(mass < 1000) %>% 
  ggplot(aes(x=mass,
             y=height))+
  geom_point()+
  geom_smooth(method = "lm")+
  labs(x="Mass",
       y="Height",
       title = "Star Wars: Mass x Height")

enter image description here

Now let's say we didn't have this very obvious outlier and we already started with this particular subset of the data. Then after, we tried to flag the outliers using an often used 1.5x x IQR method to decide what to remove:

#### Highlight Outliers in Plot ####
starwars %>% 
  filter(mass < 1000) %>% 
  mutate(outlier.height = rstatix::is_outlier(height),
         outlier.mass = rstatix::is_outlier(mass)) %>% 
  ggplot(aes(x=mass,
             y=height))+
  geom_point(aes(color=outlier.height))+
  geom_smooth(method = "lm")+
  labs(x="Mass",
       y="Height",
       title = "Star Wars: Mass x Height",
       color = "Height Outlier?")

You will notice that not only are a substantial amount of outliers now present in the data, but shaving them would fundamentally alter the real associations in the data:

enter image description here

Therefore, it is best to decide what is a meaningful inclusion/disclusion of an outlier. In the case of Jabba the Hut, even his separation from the pack has questionable outcomes (there may be others from his species who are also as large, and we may want to predict that in some way). With this being the case, outliers require more thought than a quantile rule.

$\endgroup$
3
  • 1
    $\begingroup$ Wow, you really explained it in detail and it really helps to understand with some examples! I think my condition is similar to your "Jabba the Hut" data point where the 99% centile is around 115 minutes (approx. 2 hours) and suddenly spiked in the 100% quantile having values around 56160 minutes (approx. 39 days). Personally speaking, it looks like an error in the data collection. Should I remove the data that falls above that 99% centile? Again, thank you for the explanation! $\endgroup$ Jan 22, 2023 at 14:13
  • 2
    $\begingroup$ If you think it is a legitimate error, that can be a reasonable decision to exclude. I do think that 56,160 minutes is rather extreme. Considering the implausibility of somebody biking for 39 days, it may be safe to remove. $\endgroup$ Jan 22, 2023 at 14:23
  • 3
    $\begingroup$ @AtthoriqPangestu That is sort of the point. You can remove the 56160 minutes because you believe it is a data error. Which percentile it is is fairly irrelevant. $\endgroup$
    – quarague
    Jan 23, 2023 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.