1
$\begingroup$

The probabilistic/statistical formulation of linear regression mean of y is assumed to be linearly related to x with a Gaussian zero mean error. Then we train to learn the parameters w that maximize the likelihood function p(y|x,w). But I don't understand why we learn probability of y, when we need to predict a value of y given x? We are not calculating the probabilities of every possible values o f y and then predicting the one with maximum probability. So how the prediction works after training?

Please note: I am looking for an theoretical understanding, not some library function that do the regression.

$\endgroup$
2
$\begingroup$

You use the phrase "maximize the likelihood function". The likelihood function comes from the probability assumed for y given x and w. If you assume a different probability function then you have a different likelihood to maximize.

Basically, the y values do not fall perfectly on a straight line, plane, hyperplane, etc. so we need to account for the differences somehow. One approach is to assume that the y values come from a distribution with mean (or other parameter) determined by x and w, then maximize the equivalent likelihood. If we use the Least Squares criteria, that is equivalent to assuming the y is normal and maximizing the likelihood for the normal. On the other hand, if we believe that the y's come from a Laplace distribution (double exponential) then that is equivalent to minimizing the sum of the absolute values of the residuals (and the line represents the median more than the mean). For other combinations of probability, likelihood, and "Best Fit" we get different answers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.