For any given regression problem, there are literally an infinite number of possible loss functions that, when minimized, give something pretty close to an OLS or TLS solution. For example, when fitting a parabola to some data, in addition to "ordinary" solutions, we could choose an arbitrary formulation for a parabola, say:

$$F(x,y) = Ax^2 + 4ACxy + Cy^2 + Dx + Ey + 1 = 0$$

...subtract everything to the LHS (i.e. solve for 0) and minimize the absolute value or square of the LHS. In this case, we would find $[A,C,D,E]$ so that $|F(x,y)|$ is minimized. Indeed, this seems like the simple thing to do. Such a thing could be done for arbitrarily complex functions, and even for vector-valued functions by replacing $|\cdot|$ with $||\cdot||$.

Why is the "subtract-everything-to-one-side" loss not more popular in regression analysis, and what are possible pitfalls of such a method?

  • 2
    $\begingroup$ Hint: when you explicitly include the stochastic part of this model, its implicit assumptions and how it differs from any other model will immediately become evident. $\endgroup$ – whuber Oct 15 '18 at 15:22
  • 1
    $\begingroup$ Ah. if dx is the residual in x, OLS assumes that VAR(dy)=C,VAR(dx)=0 ∀x. TLS assumes that COV(dx,dy)=0,VAR(dy)=VAR(dx) ∀x. In this formulation the covariance and variance assumptions vary with x and y. This will manifest at the very least as optimal coefficients on the same data translated around the plane that do not correspond to the same surface under that same translation. $\endgroup$ – Scott Oct 16 '18 at 23:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.