1
$\begingroup$

First let me start off by saying I know the consequences that come with removing/ignoring outliers.. but for this particular case I am just looking at weekly trends in the equipment I collect data from (a little over 100 sensors). I need to ignore "leverage points" that ruin R sqd values. I need to see if any of my sensors are trending out of their normal operating limits over time using simple regressions.

rm(list=ls())
mtcars <- mtcars
myList <- lapply(mtcars, function(x) summary(lm(mtcars$wt ~ x))$r.squared)
myList

## For example, Rsqd value is .75 without outliers added to my data set 
##$mpg
##[1] 0.7528328
##Works Great! 

##Now we add some outliers 
mtcars$mpg[c(5,10,15,20)] <- 100

## without lmrob the value for mpg is myList$mpg [1] 0.001461735
##using robust regression
require(robustbase)
summary(lmrob(mtcars$wt ~ mtcars$mpg))$r.squared

##summary(lmrob(mtcars$wt ~ mtcars$mpg))$r.squared
## [1] 0.7187418
##gives me a representative r squared value for the data set and "Leverage"
##points/extreme outliers don't ruin relationships I am trying to see.

#but if I try this... 
myList <- lapply(mtcars, function(x) summary(lmrob(mtcars$wt ~ x))$r.squared)

#it doesn't work and by that I mean I receive the error
#Warning messages:
#1: In lmrob.S(x, y, control = control, mf = mf) :
# S-estimated scale == 0:  Probably exact fit; check your data
#2: In seq_len(ar) : first element used of 'length.out' argument

#Error in summary(lmrob(mtcars$wt ~ x)) : 
# error in evaluating the argument 'object' in selecting a method for     function 'summary': Error in numeric(seq_len(ar)) : invalid 'length' argument 

I have also tried just removing any data points in each column that are outside of +-3 Sigma but I wasn't able to get that working...

$\endgroup$
  • $\begingroup$ I think in this instance that removal of outliers might be a conservative strategy, but I don't see any reason to do so, when you are already using robust methods. The whole point of robust methods is to blunt the effects of outliers on the final results. Also don't see any reason to look at "r-squared" if the goal is to set tolerance limits. (I wouldn''t guess that R-squared would be meaningful with robust methods because they generally don't use squared deviations anyway.) You should describe the actual use case fully, and probably should be seeking statistical advice at CrossValidated.com. $\endgroup$ – DWin Jul 13 '15 at 16:21
  • 2
    $\begingroup$ I feel that this is more of a data modeling question than a programming question. I've voted to migrate to Cross Validated where statistical questions are on-topic. $\endgroup$ – MrFlick Jul 13 '15 at 16:23
  • $\begingroup$ I see as the true question that the last line of code (myList <- ...) "doesn't work". @Jacob can you specify what "doesn't work" means, i.e. do you get a R error or is your R squared not what you expect? This is crucial. $\endgroup$ – mts Jul 13 '15 at 16:25
  • $\begingroup$ @BondedDust I tried removing outliers because the robust lm function wasn't working... I am aware what the robust methods do. I am not trying to set control limits (they are already in place) as much as see when some of the sensors on my deposition tool (wafer fabrication) trend within my control limits so I can make adjustments or calibration fixes. If I get a decent r sqd value, I can see a relationship and then explore further. $\endgroup$ – Jacob Odom Jul 13 '15 at 16:29
  • $\begingroup$ @mts The lapply function containging the "lmrob" returns an error. Ideally it would return a list just as the "lm" function does in the first few lines. I am not sure how to avoid the error. $\endgroup$ – Jacob Odom Jul 13 '15 at 16:35
2
$\begingroup$

I persist in my belief that this is probably not a meaningful exercise, but this is a method of allowing you to avoid the error that is thrown when you try to regress wt on wt. The error has nothing to do with outliers, but rather your choice of functions, BTW, so you probably do need to amend the title.

myList <- lapply( names(mtcars)[!grepl("wt", names(mtcars))], 
                  function(x) summary(lmrob(mtcars[['wt']] ~ mtcars[[x]]))$r.squared)
myList

I thought the warning "Probably exact fit; check your data" was more helpful than the actual error message.

$\endgroup$
0
$\begingroup$

The error comes from trying to do:

lmrob(mtcars$wt ~ mtcars$wt)

You'll need to remove wt from the dataframe.

In lm it doesn't throw an error, it simply throws a warning and keeps going.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy