I know that when estimating and using ordinary least squares, irregular terms have to be iid N(0, σ2 ). Now I want to use non-linear least squares method for estimations. Are there any specific conditions that irregular terms have to fullfill? Or do they have to be still iid N(0, σ2 )?

  • $\begingroup$ The errors don't have to be normal to fit a model via least squares, but as you move away from normality you'll start to lose efficiency, and the usual inferential calculations (such as hypothesis tests or confidence intervals) rely on it. $\endgroup$ – Glen_b Dec 16 '16 at 23:14
  • $\begingroup$ @Glen_b there may be a disconnect between math/stats communities here, but I commonly see "pseudo-normal" style error estimates for (local) nonlinear optimizations*, which to me seemed simply based on the idea of a local quadratic model, which should be positive-definite at a local minimum. (*i.e. not even necessarily least squares, though at least 2nd-order smooth one would hope! seems to be commonly built into optimization libraries as an output.) $\endgroup$ – GeoMatt22 Dec 17 '16 at 6:56
  • 1
    $\begingroup$ (continued) The connection being that a multivariate Gaussian negative log-likelihood is simply a positive-definite quadratic. The main issue I have seen with this is commonly optimization Hessian results for massively non-convex problems will be presented as "the uncertainty", when really they are just the "local sensitivity" of the objective function to the parameters. $\endgroup$ – GeoMatt22 Dec 17 '16 at 7:09
  • $\begingroup$ Does my answer resolve your question? $\endgroup$ – Mark L. Stone Dec 20 '16 at 20:38

What do you mean by "irregular terms"? It sounds like you are just talking about the distribution of errors per "equation" (data point).

If using generalized least squares https://en.wikipedia.org/wiki/Generalized_least_squares rather than ordinary least squares, the errors do not need to be iid Normal. A zero mean Multivariate Normal with general covariance matrix can be handled efficiently by generalized least squares.

Similarly, for nonlinear least squares, errors having zero mean Multivariate Normal with general covariance matrix can be handled efficiently by using generalized nonlinear least squares. Specifically, let $C$ be that covariance matrix, and let $R$ be the upper triangular Cholesky factor of $C$, i.e., $R' R = C$. Let $Y$ be the column vector of LHS values, and $f(X)$ be the column vector of RHS values (fitted function), so that $Y - f(X)$ is the column vector of residuals. Generalized nonlinear least squares can be conducted as ordinary nonlinear least squares applied to the transformed residuals $R^{-T}(Y - f(X))$. This works because the transformed residuals are iid $N(0,1)$. When this procedure is used, it is equivalent to minimizing $(Y - f(X))^TC^{-1}(Y - f(X))$.

  • $\begingroup$ I added "efficiently" in a couple of places to make explicit a point in the comment by @Glen_b . $\endgroup$ – Mark L. Stone Dec 17 '16 at 6:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.