# Similarities and differences between regression and estimation

What's the similarities and differences between parametric regression analysis and estimation theory?

I notice that they are both about parameter estimation, and both require some models for estimation.

One difference is that regress requires both independent and dependent variables, while estimation only requires observed variables. Also, regression minimizes the distance between the observed values and the values predicted by the model (least square), as the estimation, like MMSE estimator, minimizes the mean square error (MSE) of the to-be-estimated parameters.

For linear model with Gaussian noise, the maximum likelihood (ML) estimator will identical with the regression in form of (weighted) least square. In other words, the estimate achieves maximum likelihood, and also minimizes the residual.

Is there any other similarity or difference between these two?

• This may be just me but I don't know what you mean by "estimation theory" - could you give a reference? As far as I'm concerned, estimation is a process of estimating values for parameters in a model. One such model might be a linear model with a Gaussian response; and the particular estimation technique commonly used is ordinarly least squares although other techniques are possible. You need an estimation theory to know which technique to use but I can't see how you can compare similarities and differences between estimation theory as such and regression models. – Peter Ellis Jul 29 '12 at 5:40
• @PeterEllis I don't know the exact definition of the estimation theory, but what I mean is something like the maximum likelihood (ML) estimation, maximum a posteriori (MAP) estimation, and minimum mean squared error (MMSE) estimation, etc. See here. I don't know if this kind of estimation overlap with regression, or one belongs to the other? – chaohuang Jul 29 '12 at 15:43

Regression analysis is a form of statistical model.

Estimation methods like maximum likelihood, method of moments, or least squares (which is the same as minimum mean squared error - minimising the total of squared residuals is the same as minimising the mean of them) are ways of estimating the values of parameters of a statistical model, given the sample of observations available to us.

Hence there are no differences or similarities as such. An estimation method is needed to fit your regression model. Hence you cannot have a regression model without an "estimation theory" of some sort.

A common method of estimating the parameters in a regression is ordinary least squares, which is also the maximum likelihood method if certain assumptions are met (equal variance, Gaussian error terms, model specified correctly).

• The least squares estimator may not minimize the MSE if the error/noise has non-zero mean. – chaohuang Sep 8 '12 at 18:35

One perspective is estimation theory provides a unified view for estimation where as mean square error can be chosen as a loss function, a metric for describing a state. A maximum likelihood estimator is similar to the most probable Bayesian estimator given a uniform prior distribution on the parameters. A central concept in estimation theory is probability distributions of parameters (e.g. theta and x's) both prior and posteriori.

• Seems to be a statement rather then an answer to the question. – Michael R. Chernick Dec 7 '17 at 18:34