Is it possible to define an optimal fit? Let assume that we have n pairs of real numbers:
(x_1, y_1), (x_2, y_2), ..., (x_n, y_n)

Let as also assume that n is small (for example between 10 and 20). Although it is not essential.
Is there theoretically the best curve y = f(x) that approximate the data?
Among the many possible "candidate" curves there are some that are further from (or closer to) the data points (according to some measure of vicinity). There are also curves that go exactly through all the points. Obviously that such a vicinity cannot be used as a measure of goodness (because of the overfit). But what can be?
The problem seems to be mathematically ill posed. However, on the other hand, when we (human) look on the data points and different "candidate" curves can (sometimes) say that one curve is indeed better than another one.
 A: Usually a "best" fit is defined by some function of proximity to the data, like ordinary least squares linear fits.
Otherwise, "best" is defined by some function of proximity to new data drawn from the same population. Overfitting is when your data matches the sample very closely but the new data from the same population very poorly.
In either case, you're going to want a mathematical function to describe this "best" fit you are looking for, otherwise it seems like you're just asking for a function that broadly passes the "eyeball test" from many individuals. But people are unreliable judges of good fits, functions are at least consistent.
A: The point isn't to fit the data sample. It's to try to model the underlying physical processes that led to your data sample, in a way that accomplishes some desired goal.
Trivially, the "best fit" to your data sample is a function defined only at the observed values of $x_i$, taking the corresponding observed value of $y_i$ at each $x_i$. I know that's not what you are looking for, but starting with such an extreme example can help clarify the issues  involved. What you presumably want is some continuous function that could reasonably have produced your observed data sample.
As @whuber pointed out, you thus need to consider the nature of that underlying continuous function. If you have some reasonable physical model of the relationship between the $x_i$ and the $y_i$, depending perhaps on some parameter values that could be estimated from the data, then you can use that to start. Linear regression does this by assuming an underlying linear relationship; non-linear fitting is appropriate if the underlying physical reality suggests a fundamental non-linear relationship.
The phrase "machine learning" as used in one of your comments encompasses several approaches to discover potentially useful relationships from data. Note that many successful approaches that start with minimal assumptions about the functional form of the data, like boosted regression trees, do not return any simple function at all. They nevertheless can do quite well at capturing relationships within a data set. So, no, machine learning is not trying to solve an ill-posed problem: it attempts to learn relationships from a set of data (typically with many more than 10-20 data points) that can be useful for some particular purpose. Those relationships might or might not have simple functional forms of the type you seek.
If your interest is simply in a reasonably smooth curve that nicely interpolates ($x,y$) data without being too wiggly, then you could consider either loess (locally weighted regression) or splines. See this page and its links for example.
But how do you know that your data sample was actually taken from a process with a non-wiggly underlying physical process? This page shows the same 9 ($x,y$) data points fit with linear interpolation, a high-degree polynomial, and a restricted cubic spline. Although the restricted cubic spline looks nice and happens to match pretty closely the particular function from which these data points were sampled, what if the underlying function had actually been the polynomial? That's why the question is ill posed, absent specific criteria for what is meant by the "best fit."
