Dealing with probable outliers in the dependent variable

Question

I am trying to fit a simple regression to a data set with ~45,000 observations. The dependent variable is revenue growth, but I'm concerned some observed values are incorrectly entered data.

To elaborate: The mean growth was about 6% (standard deviation is approximately 18%), but observations range from -100% to 200%. I'm certain 200% was incorrectly entered, but there is a decent number of observations above 100% which disconcerts me. While growth in this range is plausible and has been observed, I have doubts that all of them are accurate (my skepticism is an increasing function of the number of observations in this range).

What is the best way to deal with this data, especially because there are too many observations to check manually? Should I remove all of these data points and run the model for on data which I am confident is correct? Or will this bias the model because it bounds the dependent variable below 100%? Is there something more sophisticated that can salvage these observations?

Answer 1

Aside from the technicalities of identifying outliers (Tukey, SD, hotelling t2 for multivariate data, etc) I think that the first question should be: Why do we want to identify outliers? If your primary purpose is e.g. to model basic principles for economic growth in the general body of companies, then you may want to exclude rocket companies from your data, since they obviously work by a different mechanism.

Conversely, you may want to identify potential rockets. Then you will identify a small set of outliers from your general data swarm and in a subsequent step discard those outliers that arise from simple input errors. Then you can use some clever modeling principle or deep analysis to identify causes for these observations to stand out from the others...

Your fundamental analysis question will determine the best approach for identifying outliers, but as a general principle you should make sure to check why observations are outliers. Not just ascertain that they are.

I don't know much about quantitative economics, but why not make e.g. a normality test before/after a series of transformations (e.g. log transformations) and see if observations become modelable before deciding on . aprinciple for identifying outliers. Then, as I mentioned above, if your aim is to find sort of general principles of economic growth, you may want to sort out not only obvious errors but also those observations that are governed by other principles, since they probably will not be well described by general mechanisms.

---

EDIT: Your comment suggests that you may be more interested in identifying the interesting outliers than modeling the bulk of the observations. Then why not train a classification model? You haven't specified your independent data, but I'm guessing you have some input parameters...

Then train your independent data on a 3 class problem: 1. Normal companies 2. Apparent faulty entries 3. Rockets

You may also want to add other categories, such as bombs...

That way you could investigate whether there is something systematically different between the different types of outliers. An analysis of important variables distinguishing different types of companies could then help you make an informed decision on how to treat your outliers in relation to your subsequent analyses (e.g. whether they should be included in the main model, completely discarded or analysed separately in a different model).

HTH HAND
Carl

Answer 2

One of the ways to deal with outliers is to use tukey border values to either set the data outside it to missing or to set it to the border value. For the lower part of the distribution it is defined as $Q1-1.5IQR$ and for the higher part $Q3 + 1.5 IQR$, where Q1 and Q3 are 1st and 3rd quartile values, and IQR is inter-quartile range. The values lower than the first, or higher than the second value should be seto to NA or replaced with these border values. I prefer setting the values to missing if there are not more than 10% of missing values after this transformation.

The other way to deal with the problem is to use $M\pm3SD$. You set all values above and below these values to NA.

Here are two functions in R you can use to do the work for you.

# Function identifies either tukey (tuk) or sd (sd) border values (type),
# and the magnitude of the range (level) - normal tukey range is 1.5IQR, 
# and usual range with SD is 3)

GRW <- function(x, type, level){
  x <- unlist(x)
  if (type == "tuk"){
    IQR <- quantile(x,0.75, na.rm = T) - quantile(x,0.25, na.rm=T)
    Q1 <- quantile(x, 0.25, na.rm=T) - level*IQR
    Q2 <- quantile(x, 0.75, na.rm=T) + level*IQR
    names(Q1) <- NULL
    names(Q2) <- NULL
    return (c(Q1 = Q1, Q2 = Q2))
  }

  if (type == "sd"){
    SD_l <- mean(x, na.rm=T) - level*sd(x, na.rm=T)
    SD_h <- mean(x, na.rm=T) + level*sd(x, na.rm=T)
    names(SD_l) <- NULL
    names(SD_h) <- NULL
    return(c(SD_l = SD_l, SD_h = SD_h))
  }
}

# Function returns vector with tukey/sd (type) with manually defined 
# range (level), only on positive(pos), negative (neg) or both ends (double)
# and it either replaces the outliers with border values (replace = TRUE) 
# or sets them to NA (replace = FALSE)

REPL <- function (var, type, level, end, replace) {
  # Only positive end
  if (end == "pos") {
    if (replace == FALSE) {
      var[var>GRW(var,type,level)[2]] <- NA
      var
    }
    if (replace == TRUE){
      var[var>GRW(var,type,level)[2]] <- GRW(var,type,level)[2]
      var
    }
  }

  # Only negative end  
  if (end == "neg") {
    if (replace == FALSE) {
      var[var<GRW(var,type,level)[1]] <- NA
      var
    }
    if (replace == TRUE){
      var[var<GRW(var,type,level)[1]] <- GRW(var,type,level)[1]
      var
    }
  }

  # Double end  
  if (end == "double"){
    if (replace == FALSE) {
      var[var<GRW(var,type,level)[1]] <- NA
      var[var>GRW(var,type,level)[2]] <- NA
      var
    }
    if (replace == TRUE){
      var[var<GRW(var,type,level)[1]] <- GRW(var,type,level)[1]
      var[var>GRW(var,type,level)[2]] <- GRW(var,type,level)[2]
      var
    }
  }
  var
}

Btw. if you want to check for outliers visually, use the function boxplot(), or in ggplot geom_boxplot()

Answer 3

Robust regression methods are a general approach to dealing with outliers. You can also run the regression with different inclusion rules and report how sensitive your results are to different degrees of data inclusion.