Suppose we have one dimensional data with following conditions.
1. The value of this single feature lies in the closed interval [-a,a].
2. The associated target value with each data also lies in the closed interval [-a, a].
If we plot the feature value on x-axis and the associated target value on y-axis, then we will get a circle.

Assume that we fit a linear regression to the data, we can get infinite regression lines, passing through origin.
1. My question is can we derive the formula for error?
2. Will the error be same for every regression line passing through origin?

  • $\begingroup$ The linear regression solution depends on how the data are sampled. Extreme examples are provided by the datasets $((1,1),\,(-1,-1))$ and $((1,-1),\,(-1,1)),$ with correlations $1$ and $-1$ respectively. These also demonstrate you do not necessarily get "infinite regression lines." More to the point, isn't it obvious that under your assumptions, linear regression is an inappropriate procedure? $\endgroup$ – whuber Jan 18 '19 at 15:15
  • $\begingroup$ yeah i was also thinking in the same direction that's why I ask is it possible to calculate the error in the first place. $\endgroup$ – Prayalankar Jan 19 '19 at 9:54

If it forms a circle and it has infinite number of points (or large enough to be approximated as below) and points are uniformly distributed around the circle, we can conduct some analysis; also note that we have contradicting values for a given $x$, i.e. $y=\pm\sqrt{a^2-x^2}$. When we fit a line, our estimations will be of the form $y=bx$. And, MSE will be $\text{MSE}=E[(Y-bX)^2]=\int_{-a}^{a}(y-bx)^2f(x)dx$, which can be written explicitly as follows: $$\int_{-a}^{a}(\pm\sqrt{a^2-x^2}-bx)^2f(x)dx=\frac{1}{2a}\int_{-a}^{a}(a^2-x^2\pm2bx\sqrt{a^2-x^2}+b^2x^2)dx=\frac{a^2(2+b^2)}{3}$$

which is your error. The $\pm$ sign doesn't matter in the error calculation since errors are symmetric below and above the x-axis. Minimum error is achieved when $b=0$, i.e. $y=0$ line. So, not every line produces the same error.

  • 1
    $\begingroup$ The calculation is interesting--but it assumes much more than "infinite number of points:" it also assumes they are uniformly sampled around the perimeter. $\endgroup$ – whuber Jan 18 '19 at 21:46
  • 1
    $\begingroup$ Right @whuber, I was implicitly assuming it, and now stated in my answer. Thanks. $\endgroup$ – gunes Jan 19 '19 at 6:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.