0
$\begingroup$

I saw some machine learning code assuming that variance of gaussian noise is a learnable parameter in linear regression problem. I'm wondering how is this solved theoretically?

Below you see typcial MLE derviation of linear regression. Noise (epsilon) is assumed to have fixed std which is sigma squared, what if we learened it just like w? what is sigma(x) ?

enter image description here

Improve this question
$\endgroup$
5
  • $\begingroup$ Hi: if you wanted to learn the variance, one way is to give it a prior and then update the posterior each time a new observation came in. See "Dynamic Linear Models" by West and Harrison for a more in-depth discussion of this approach which is really the state space approach. $\endgroup$
    – mlofton
    Commented Aug 1, 2019 at 10:41
  • $\begingroup$ @mlofton Wouldn't that then not be MLE, maybe MAP? $\endgroup$
    – Dave
    Commented Aug 1, 2019 at 10:44
  • $\begingroup$ @Dave Exactly. I don't want to go Bayesian, I'm using MLE, I want to pick variance that varies with x, just like y varies with x. $\endgroup$
    – John Deterious
    Commented Aug 1, 2019 at 11:16
  • $\begingroup$ @Dave I think the solution is to assume y = wx + epsilon, where epsilon is normal with variance = w2 * x. I.e. I'm assuming there's linear change with x, and I want the slope. Does that make sense? solution would be easy extension. $\endgroup$
    – John Deterious
    Commented Aug 1, 2019 at 11:19
  • $\begingroup$ @Dave: I'm not sure what it would be called as far as an estimator but you're right that it's not an MLE. I'm not sure about obtaining MLE by learning since the likelihood is tricky. I'll stay out of this discussion and apologies. $\endgroup$
    – mlofton
    Commented Aug 1, 2019 at 15:10

1 Answer 1

Reset to default
1
$\begingroup$

I am pretty sure that you can find the solution within the framework of the generalized linear mixed model. For instance, in the R package nlme, you can estimate heteroscedastic variances as a function of other model variables and they do it using a MLE.

Maybe you can go in this direction for a solution. This reference looks promising: https://doi.org/10.1007/0-387-22747-4_5

(Sorry for paywall but I could not find a similarly exhaustive treatment that is open.)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.