3
$\begingroup$

I want to fit a scatter plot with a polynomial, and find the correlation between two variables.

1) How can I define and remove outliers from data points? (in the figure the outliers on the right misled the polynomial fit, which didn't capture the linear relationship)

enter image description here

where SE is the squared error, R is the Pearson correlation coefficient, and $\rho$ is the Spearman Correlation Coefficient.

$\endgroup$
  • $\begingroup$ First I'd start by recommending thinking carefully about that datapoint and if you really should be removing it. A far out point like this could indeed be a valid observation that you should take into account during your modeling process. If that point is truly a mistake for some reason, then it's probably okay to simply remove as long as the process that led to the mistake isn't somehow related to your variables of interest. Can you tell us more about your data and why you think this point should be removed as an "outlier?" $\endgroup$ – StatsStudent May 24 '19 at 22:01
0
$\begingroup$

In the picture, you posted, outlier is on the x axis. We can remove them using IQR and example code of doing it in R can be found here

Here is an example on simulated data for your case:left subfigure is the data without outlier, the right subfigure is the data with outlier. (I am manually adding 3 data points in mtcars data.)

As you can see, those 3 data points make the regression line flat.

enter image description here

Code

par(mfrow=c(1,2))
d=mtcars[,c("wt","mpg")]
plot(d)
fit=lm(mpg~wt,d)
summary(fit)
abline(fit)

d2=rbind(d,c(40,20),c(45,20),c(50,20))
plot(d2)
fit2=lm(mpg~wt,d2)
summary(fit2)
abline(fit2)
$\endgroup$
  • 1
    $\begingroup$ Thank you for answering. What do you mean by "outlier is on the x axis" ? I think the outliers are those at around points (1.5, 14), (0.7, 12) and (1, 17). $\endgroup$ – Karl Alexius May 2 '17 at 17:36
  • $\begingroup$ no, the major problem is the 3 data points on the right side. I can give you a demo on a simulate data. if you want. $\endgroup$ – Haitao Du May 2 '17 at 17:42
  • $\begingroup$ It would be great to see the demo, thanks. Do you mean that those three points do not affect correlation or the regression? $\endgroup$ – Karl Alexius May 2 '17 at 17:47
  • 2
    $\begingroup$ Would you please post either the raw data, or a link to the raw data? $\endgroup$ – James Phillips May 2 '17 at 21:28
  • 1
    $\begingroup$ I would strongly caution against blindly applying procedures like this that simply remove suspected outliers, based on some crude approach such as excluding those that fall outside of the IQR. $\endgroup$ – StatsStudent May 24 '19 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.