3
$\begingroup$

I have a question about MLE and how it relates to OLS. I know how to relate OLS and MLE when the noise is normal and homoskedastic. I can apply the same reason for heteroskedastic noise. My question is that, clearly, the noise terms are no longer identical (though still independent). So, we can apply MLE even when the distributions are not identical?

To make it more clear, if I take two samples, can I apply MLE when one sample's noise is normal, and the other's is Laplace? If not, why does MLE work for heteroskedastic noise?

Is it because, when estimating the parameter vector, we can simply take the different variances as known scaling factors* uniform variance $\sigma^2$, giving rise to the weights?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ Yes, you can apply maximum likelihood estimation (MLE) even if the distributions aren't identical. If you can write down the likelihood function, you can search for what parameters give the maximum likelihood. But you need to understand the data generating process, what drives the heteroskedasticity well enough to write down a likelihood function (that's sensibly parameterized). Eg. letting the variance of each observation $\sigma^2_i$ be a different, unknown parameter will almost certainly not be a useful approach. $\endgroup$
    – Matthew Gunn
    Commented Jan 31, 2017 at 7:53
  • $\begingroup$ Wait, but isn't that what we do in weighted least squares? Except for the unknown part (we assume each variance is known). Unless you are telling me that I can't estimate all the (unknown) parameters if they are all unknown (if that even makes sense). $\endgroup$
    – learning
    Commented Jan 31, 2017 at 9:07
  • $\begingroup$ Also, I guess from your explanation, I can apply MLE for heteroskedastic noise in my example because I have known variances and am estimating only one unknown. Is my reason for this correct (along with the final paragraph in the question)? $\endgroup$
    – learning
    Commented Jan 31, 2017 at 9:10

1 Answer 1

Reset to default
1
$\begingroup$

There seems to be multiple questions here. To use maximum likelihood estimation (mle) you need to be able to write down the likelihood function, that is, the joint density (maybe joint probability mass function) of the observations. There is no need for there to be identical distributions, you see that by using likelihood methods for regression.

But, in practice, likelihood methods are most useful when the distribution of the data can be written using a relatively small number of parameters which are common for all the observations. So, a unique variance for each observation cannot be expected to work---that would lead to more parameters than observations. But, still, there is no need to assume the same variance for all the observations. What you need is some way to describe how the variance varies, maybe as a function of the mean, maybe as a function of some known covariate.

You could even, in principle, in a regression model, assume normal errors for some of the observations and Laplace errors for others, there is no obstacle in principle. But it is difficult to think of a situation where such would be a natural way to model!

If you are using R, there are some packages that allows for separate modeling of expectation and variance, among them, dglm and gamlss. See for instance Simulate linear regression with heteroscedasticity and Is it possible to calculate variable confidence intervals, conditional on $\hat{Y}$ to address heteroscedasticity?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.