3
$\begingroup$

Why does minimizing $$RSS(f) = \sum_{i=1}^N(y_i - f(x_i))^2$$ lead to infinitely many solutions?

I saw it from the book The Elements of Statistical Learning,second edition (Chapter 2 section 2.7 under the topic Difficulty of the problem. I don't really understand but trying to learn it. Also, I want to know a function that only has finite or single solution

Improve this question
$\endgroup$
9
  • 3
    $\begingroup$ They are infinitely many functions $f$ that passes through all of the points $(x_i,y_i)_{1\leq i \leq N}$ (you can draw infinitely many curves that goes through these points). Each one of this function will give $RSS=0$ and thus minimizes it. $\endgroup$
    – periwinkle
    Commented Oct 19, 2020 at 21:03
  • $\begingroup$ @winperikle Please if I should understand, some of these curves can be linear and polynomial right ? $\endgroup$
    – EA Lehn
    Commented Oct 19, 2020 at 21:29
  • $\begingroup$ Yes. But even so, there are still infinitely many polynomials that will have identical values at every one of the $x_i.$ $\endgroup$
    – whuber ♦
    Commented Oct 19, 2020 at 21:31
  • $\begingroup$ @whuber Please is it because of the squared that is why it has infinite solution or can I get an example of a function that has finite or one solution ? $\endgroup$
    – EA Lehn
    Commented Oct 19, 2020 at 21:48
  • 1
    $\begingroup$ Lagrange polynomials are an example how you can see that a polynomial is always able to pass through all points. And you can extend it such that infinitely many polynomials can pass through the points. en.m.wikipedia.org/wiki/Lagrange_polynomial $\endgroup$
    – Sextus Empiricus
    Commented Oct 19, 2020 at 22:46

2 Answers 2

Reset to default
2
$\begingroup$

When you have $n$ points on a graph, there are many curves you can draw to pass through those $n$ points, provided no two points have a same $x$-coordinate and different $y$-coordinate.

Since RSS depends on the residuals in those points (which can be made zero by a curve that passes through the points), there will be many curves you can draw to make RSS = 0. Each of these curves serve as a solution to the function. That is why we have infinite number of solutions.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ I have adapted your answer a bit. E.g. generalizing 3 points to $n$ points. Feel free to revert anything back which you don't like. $\endgroup$
    – Sextus Empiricus
    Commented Oct 20, 2020 at 8:26
  • $\begingroup$ Do we still have infinite solutions of an input value has multiple output values e.g. regression on categorical variables? That we would not be able to achieve $RSS=0$ concerns me. $\endgroup$
    – Dave
    Commented Oct 20, 2020 at 9:33
  • 2
    $\begingroup$ @Dave Yes. This answer is a little misleading because it is not necessary that all $x_i$ be distinct. Literally any function that passes through each point of the form $(x_i, \bar y_i)$ where $\bar y_i$ is the mean of the $y$ values of all data points of the form $(x_i,y)$ will minimize the sum of squared residuals, but it cannot possibly reduce the RSS to zero unless there is zero dispersion of those $y$ values. For instance, consider the dataset $(0,1),(0,-1),(1,0)$ and let $f$ be any function: the function $x\to x(1-x)f(x)$ minimizes RSS (equal to $2$) and every minimizer has this form. $\endgroup$
    – whuber ♦
    Commented Oct 20, 2020 at 17:27
0
$\begingroup$

Consider the case where all the $x_i$ values are distinct (purely for simplicity) and suppose you set $f(x_i) = y_i$ at each point $i=1,...,n$. Then you have $RSS(f) = 0$, which is minimised. Now, if $n$ is finite then there are an infinite number of other points of $f$ that you didn't specify. Since any specification of those other points leads to a valid minimiser, there are an infinite number of solutions to the minimisation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.