What's the similarities and differences between parametric regression analysis and estimation theory?
I notice that they are both about parameter estimation, and both require some models for estimation.
One difference is that regress requires both independent and dependent variables, while estimation only requires observed variables. Also, regression minimizes the distance between the observed values and the values predicted by the model (least square), as the estimation, like MMSE estimator, minimizes the mean square error (MSE) of the to-be-estimated parameters.
For linear model with Gaussian noise, the maximum likelihood (ML) estimator will identical with the regression in form of (weighted) least square. In other words, the estimate achieves maximum likelihood, and also minimizes the residual.
Is there any other similarity or difference between these two?