1
$\begingroup$

In this question, I would like to ask two things:

  1. outlier detection
  2. normality test

Details are as follows:

I need to detect and remove outliers in my data. Before doing that, I want to test if my data is normally distributed or not. I have two variables X(independent) and Y(dependent) and have 951 records for both of them.

I want to know that while testing the normality, do I need consider both the variables simultaneously or both the variables but one at a time? (Somewhere, I have read that only dependent variable is considered to test the normality).

The attached figures show the results of normality test (Analyse>>Descriptive >>Explore) of dependent variable. If normality test is done only on dependent variable, then it shows that the data is highly skewed. In such a case, how can I remove the outliers?The image shows histogram of dependent variable

The significance level of Shapiro-Wilk test and Kolmogorov-Smirnov test is 0.00. Skewness has statistic of 22.909 with SE of 0.079.

Improve this question
$\endgroup$
16
  • 4
    $\begingroup$ Many people ask for normality testing, even if no normality testing is needed. Please consider your reasons and what you likely want to do is test the residuals (not the independent and the dependend) variable for normality by means of a quartile-quartile-plot. $\endgroup$
    – Bernhard
    Aug 31, 2018 at 11:19
  • 1
    $\begingroup$ As for outliers: consider where these data come from and whether it makes sense, that some data are so much higher then the rest. If so, the so called outliers may contain worthwhile information. If not, try to define the cut off from the underlying science, not from statistics. If that does not work, have you considered using robust measures like the 5%-trimmed mean instead of the mean or some form of robust regression? $\endgroup$
    – Bernhard
    Aug 31, 2018 at 11:25
  • 3
    $\begingroup$ Did you look at some of the highly voted threads under your tags? There have been hundreds -- possibly even thousands -- of questions similar to this. $\endgroup$
    – Nick Cox
    Aug 31, 2018 at 11:36
  • 1
    $\begingroup$ The variables are rainfall (independent variable) and runoff (dependent variable). As hydrologists believe that runoff measurement by human means involves error, so they tend to remove outliers (Ref: various scientific papers). That's why, I have been after removing outliers. But just after posting the question and receiving answers, I went to my data again and found that higher runoff (17mm) is caused due to higher rainfall (13mm), which I think makes sense. Cont... $\endgroup$
    – Alexia k Boston
    Aug 31, 2018 at 12:12
  • 1
    $\begingroup$ And 0 is also, I don't think will be erroneous because if all the water infiltrates and rainfall is little, runoff would be zero. And moreover, this is satellite data. So, now I myself, thinking that if removing outlier would be fair or not!! Cont... $\endgroup$
    – Alexia k Boston
    Aug 31, 2018 at 12:13

1 Answer 1

Reset to default
3
$\begingroup$

You write:

I need to detect and remove outliers in my data

Why do you need to do this? Detecting outliers is a good thing, but automatically removing them is not (and that seems like what you want to do). Since, from your question, it seems like you have some sort of regression problem, you should consider keeping the data and changing the regression method to e.g. quantile regression or robust regression.

You should also be aware that even OLS regression does not make assumptions about the distribution of the data (except that the DV is continuous or nearly so) but about the error.

You then write:

Before doing that, I want to test if my data is normally distributed or not

Again, why? But, if you want to test normality, as @brad said in his answer, graphical methods are best. I like both density plots (as Brad suggested) and quantile plot (as Nick suggested). However, the latter take a bit of experience to use well. You could also try box plots.

Then you write:

I want to know that while testing the normality, do I need consider both the variables simultaneously or both the variables but one at a time? (Somewhere, I have read that only dependent variable is considered to test the normality).

This makes me strongly suspect you are doing regression. As I noted above, neither variable needs to be normally distributed (I don't doubt that you read what you read, but it's incorrect).

Finally, you show a histogram of your DV. The histogram is not a very useful plot (as William S. Cleveland notes in his books).

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.