In statistics books least squares method is placed among other methods for distribution parameters estimation methods (like method of moments, maximum likelihood method). However in none of the books I've read there is a single example on how to estimate distribution parameters using LSM. Only linear and non linear regression examples are given. Could anybody give an example of PDF estimation using LSM?

EDIT: The least squares error function is


Where $x_i$ are the sample elements, and $\theta$ are the estimated parameters and $h(\theta_1,\theta_2,...,\theta_k)$ is some arbitrarily chosen function. How would this function look like when estimating the parameters of let's say normal distribution? Would this be:


where $f_i$ are the empirical probability density values? But then I would need to make some estimate of $f_i$ what is a different problem itself.

  • 1
    $\begingroup$ In general it doesn't make much sense to use least squares to estimate parameters of distributions outside of estimating the population mean and if that's a parameter of some distribution, the parameter will only be well estimated via least squares (e.g. with efficiency near 1 compared to the estimate with the best available efficiency) in pretty particular circumstances. $\endgroup$ – Glen_b Nov 4 '15 at 8:24
  • 2
    $\begingroup$ The idea is, the form of the conditional distribution of the dependent variable is assumed to be known (e.g. Normal) and then the parameters of that conditional distribution (e.g mean and variance in case of Normal) are estimated. That makes the OLS estimates to be the estimates of the distribution parameters. (OLS is a more standard acronym for the least squares than LSM, I believe.) $\endgroup$ – Richard Hardy Nov 4 '15 at 9:15
  • 1
    $\begingroup$ I edited my comment (again). Maybe it's clearer now. Also, how is error function related to all this? What kind of error function do you have in mind? $\endgroup$ – Richard Hardy Nov 4 '15 at 9:19
  • 1
    $\begingroup$ OLS is always the same: it minimizes the sum of squared errors between the dependent variable and the fitted values. If it does something else, it's not OLS anymore. The first equation in your post ("the least squares error function") is used to obtain the parameters of the conditional distribution corresponding to my first comment. $\endgroup$ – Richard Hardy Nov 4 '15 at 9:39
  • 1
    $\begingroup$ Multiple linear regression is perhaps the most widely used example of $h$. $\endgroup$ – Richard Hardy Nov 4 '15 at 9:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.