1
$\begingroup$

Let assume that we have n pairs of real numbers:

(x_1, y_1), (x_2, y_2), ..., (x_n, y_n)

Let as also assume that n is small (for example between 10 and 20). Although it is not essential.

Is there theoretically the best curve y = f(x) that approximate the data?

Among the many possible "candidate" curves there are some that are further from (or closer to) the data points (according to some measure of vicinity). There are also curves that go exactly through all the points. Obviously that such a vicinity cannot be used as a measure of goodness (because of the overfit). But what can be?

The problem seems to be mathematically ill posed. However, on the other hand, when we (human) look on the data points and different "candidate" curves can (sometimes) say that one curve is indeed better than another one.

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ This is indeed an ill-posed mathematical question: and, as asked, seems to be psychological in nature (concerning our human responses to looking at curves). In applications you need to consider the consequences of the fitted curve deviating from some idealized "true, underlying" curve and then you must quantify that deviation. Only then can it make sense to ask whether there is any "optimal" fit. No set of data can, by itself, determine how such deviations should be quantified. $\endgroup$
    – whuber ♦
    Sep 17, 2019 at 15:34
  • $\begingroup$ @whuber, but isn't machine learning trying to solve the ill posed problem that I have just described? $\endgroup$
    – Roman
    Sep 17, 2019 at 16:13
  • $\begingroup$ If it is, then the result of such an ML procedure is either useless or lucky: take your pick ;-). One could be less brutal (and less flippant) by recognizing that certain ML procedures happen to be well-adapted to certain kinds of problems. The pitfalls then lie in attempting to apply those procedures to other kinds of problems, where they might work not as well (or spectacularly poorly). To avoid those pitfalls, it behooves the user to understand exactly what the implicit problem is and how the procedure is solving it. Your abstract problem, though, has no "theoretically best" solution. $\endgroup$
    – whuber ♦
    Sep 17, 2019 at 17:36

2 Answers 2

Reset to default
1
$\begingroup$

Usually a "best" fit is defined by some function of proximity to the data, like ordinary least squares linear fits.

Otherwise, "best" is defined by some function of proximity to new data drawn from the same population. Overfitting is when your data matches the sample very closely but the new data from the same population very poorly.

In either case, you're going to want a mathematical function to describe this "best" fit you are looking for, otherwise it seems like you're just asking for a function that broadly passes the "eyeball test" from many individuals. But people are unreliable judges of good fits, functions are at least consistent.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yes, I am asking for a function. What should it be? Least squares would be a bad choice because it will lead to overfit. I guess we need to take into account the smoothness of the curve. But acceptable smoothness should also be determined by data. $\endgroup$
    – Roman
    Sep 17, 2019 at 14:57
  • $\begingroup$ All of the functions used for fitting pass the "eyeball test", at least for their authors. Least squares, least absolute deviation, principle component regression (diagonal distance to line of best fit), maximum likelihood.... etc. $\endgroup$
    – RegressForward
    Sep 19, 2019 at 19:07
1
$\begingroup$

The point isn't to fit the data sample. It's to try to model the underlying physical processes that led to your data sample, in a way that accomplishes some desired goal.

Trivially, the "best fit" to your data sample is a function defined only at the observed values of $x_i$, taking the corresponding observed value of $y_i$ at each $x_i$. I know that's not what you are looking for, but starting with such an extreme example can help clarify the issues involved. What you presumably want is some continuous function that could reasonably have produced your observed data sample.

As @whuber pointed out, you thus need to consider the nature of that underlying continuous function. If you have some reasonable physical model of the relationship between the $x_i$ and the $y_i$, depending perhaps on some parameter values that could be estimated from the data, then you can use that to start. Linear regression does this by assuming an underlying linear relationship; non-linear fitting is appropriate if the underlying physical reality suggests a fundamental non-linear relationship.

The phrase "machine learning" as used in one of your comments encompasses several approaches to discover potentially useful relationships from data. Note that many successful approaches that start with minimal assumptions about the functional form of the data, like boosted regression trees, do not return any simple function at all. They nevertheless can do quite well at capturing relationships within a data set. So, no, machine learning is not trying to solve an ill-posed problem: it attempts to learn relationships from a set of data (typically with many more than 10-20 data points) that can be useful for some particular purpose. Those relationships might or might not have simple functional forms of the type you seek.

If your interest is simply in a reasonably smooth curve that nicely interpolates ($x,y$) data without being too wiggly, then you could consider either loess (locally weighted regression) or splines. See this page and its links for example.

But how do you know that your data sample was actually taken from a process with a non-wiggly underlying physical process? This page shows the same 9 ($x,y$) data points fit with linear interpolation, a high-degree polynomial, and a restricted cubic spline. Although the restricted cubic spline looks nice and happens to match pretty closely the particular function from which these data points were sampled, what if the underlying function had actually been the polynomial? That's why the question is ill posed, absent specific criteria for what is meant by the "best fit."

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.