Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I need something clarified and that is when you have non-constant variance, estimates won't be biased but will be a problem when it comes to the S.E. formulas and efficiency. Therefore OLS estimates will be inefficient because they give equal weight to outliers. It makes sense to me that giving equal weight to these outliers will cause problems, but why would it cause problems if these data points are genuine and provide information?

share|improve this question
add comment

2 Answers 2

up vote 4 down vote accepted

In general, if you have outliers, they cause bias. Outliers are data that come from some other population than the rest of your data and/or the population that you are trying to model. As a result, your parameter estimates reflect the mixture of the population you are after and some sliver of a contaminating population.

The problem of efficiency is different. You can think of efficiency as being similar to statistical power. The idea of efficiency is that you use the information that is available to you optimally. If you have non-constant variance, different data provide differing amounts of information about the conditional mean of $Y$ at a given point in $X$. Clearly, if you give equal weight to each point, you are not using the information optimally, but if you can allocate the weights in accordance with the amount of information in each point, you can achieve greater efficiency. From a practical standpoint, greater efficiency can mean that you have more power to reject a false null, for example.

Here is a simple simulation, in R to demonstrate this idea:

set.seed(1)                        # this makes the simulation exactly reproducible
b0 = 10                            # this is the true value of the intercept
b1 = .5                            # this is the true value of the slope
n  = 10                            # this is the number of data I have at each point in X
x  = rep(c(0, 2, 4), each=n)       # these are the x data
wt = 1 / rep(c(1, 4, 16), each=n)  # these are a-priori correct weights
uw.p.vector = vector(length=10000) # these 2 vectors will hold the results of
w.p.vector  = vector(length=10000) #   the simulation

for(i in 1:10000){                        # I run this simulation 10k times
  y.x0 = rnorm(n, mean=b0,          sd=1) # here I am generating simulated data
  y.x2 = rnorm(n, mean=(b0 + 2*b1), sd=2) #  the SD at each point is different &
  y.x4 = rnorm(n, mean=(b0 + 4*b1), sd=4) #  the variances are 1, 4, & 16
  y    = c(y.x0, y.x2, y.x4)              # I put the data into a single vector
  unweighted.model = lm(y~x)              # I fit an identical model w/ the same data
  weighted.model   = lm(y~x, weights=wt)  #   w/o & then w/ the weights
  uw.p.vector[i]   = summary(unweighted.model)$coefficients[2,4]  # the p-values
  w.p.vector[i]    = summary(weighted.model)$coefficients[2,4]
}
mean(uw.p.vector<.05)  # using the unweighted regression, the power was ~39%
# [1] 0.3927
mean(w.p.vector<.05)   # w/ the weighted regression, the power was ~47%
# [1] 0.4732

Here, I contrast an unweighted regression with a regression that weights the data according to how much information they have to offer about the slope (set of conditional means). Notice that the amount of information is the reciprocal of the variance at a given point in $X$, and that I am not estimating it, for this demonstration, I am using the correct value a-priori. (How well this works when the weights are estimated depends on how good the estimates are.)

To understand heteroscedasticity more thoroughly, it may help to read a couple other answers I've provided on the topic:

share|improve this answer
    
So its best to allocate more weight on Y that demonstrate greater precision? That is being efficient? –  stat_genius Apr 22 at 3:59
    
Efficiency in stats has to do w/ using the info available in the optimal way. If some data cluster closely to their mean & some data disperse far from their mean, the locations of the former provide more info about the location of their mean than the locations of the latter do about the other mean. If you gave all data equal weight, you would not be using the info optimally. The true amount of information in each point relative to its mean scales as 1/variance(point); w/ constant var, you treat them equally, but w/ non-constant they should be weighted w/ more weight going w/ greater precision. –  gung Apr 22 at 14:11
    
Perfect. You are awesome gung! –  stat_genius Apr 22 at 15:12
    
One more thing actually. Lets suppose, I am looking at the resid vs. fitted and see non constant variance. I proceed to use a log transformation on the response and that gives me the constant variance that I want. Did this transformation increase the information in every point? I am guessing that the transformation comes at the expense of interpretation now that we have introduced a log transformation to our response. –  stat_genius Apr 22 at 15:16
    
It does change the interpretation, so a weighted LS approach may be preferable. In addition, transformations typically induce a curvilinear relationship where there was a rectilinear 1 before. Re efficiency, it changes the info about the location of the mean differently at different points in X. This can equalize the info in each data point allowing equally-weighted estimation to be optimal. –  gung Apr 22 at 15:23
show 1 more comment

It's not that there's equal weight given to outliers that's the issue (if the data are generated according to the correct model, the observations won't really be outliers).

The issue corresponding to what you're talking about is that information of different precision is given equal weight, when more weight should be placed on the information with greater precision (i.e. that lies on average closer to the true line).

But even so, that's an issue of efficient use of information. It's possible that the waste of information (and the noisier estimates that result) may not especially bother you.

There's another problem, however - one of larger concern. This problem to do with standard errors of the coefficients, and hence with inference such as hypothesis tests and confidence intervals.

If you don't properly account for the fact that the variance isn't constant, the standard errors of the coefficients are biased.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.