Does the convergence of algorithms in machine learning to train set make sense? Think of the hypothetical situation where we have a y reponse and a scalar input x, we want a function that maps x to y perfectly at least in the training set. This does not make sense, it is just imagining a situation where there are two instances with the same value of x but different responses, it is mathematically impossible to propose a function that correctly predicts these two instances. In view of this problem, why are convergence criteria used in some ML algorithms such as neural networks?
 A: Having two different $y$'s for the same value of $x$ will make the function not well defined, you can try plugging a data like this to a machine learning algorithm(say NN), if all your data is the same $x$ but all have different $y$, the algorithm will try to approximate the function but will not succeed to converge!
As for the mathematical analysis of machine learning, you can check PAC Learning
You can try with this example(or tweak it) but it will not converge with this kind of data:
from tensorflow import keras
import numpy as np
x=np.array([1 for _ in range(100000)],dtype=float)
y=np.array([x for x in range(2,2+len(x))],dtype=float)
model=keras.Sequential([keras.layers.Dense(10,input_shape=[1])])
model.compile(optimizer="SGD",loss="MSE")
model.fit(x,y,batch_size=500)

I hope this answer's your question!
A: Neural networks learn mathematical functions by minimizing the loss function between the function and the observed data. A function in mathematics is a mapping, that maps $X$ values to $Y$, $f: X \to Y$. There are one-to-one functions, where each of the elements in $X$ maps to a distinct element in $Y$, and functions that can map multiple $X$ elements to the same $Y$ values (e.g. absolute value function). There are no functions that map the same inputs $X$ to different outputs $Y$.
In most cases, we apply neural networks to learn from real-life data that is noisy, $y = f(x) + \varepsilon$, so we assume that the observed values $y$ depend on the deterministic function $f(x)$ and random noise $\varepsilon$. In such a case, it can happen that different $X$'s are paired with the same $Y$'s because of the random noise. What neural network (or any other machine learning algorithm) would do in such cases, is it will either predict something like an average between the two possible outputs or learn to pick one of them as a prediction. Another solution might be to use a probabilistic model. It will learn how probable are the outcomes and assign the same probabilities to both outcomes. To make a point prediction, you could do something like predicting the expected value (an average) or choosing between them at random, based on the predicted probabilities.
The convergence criteria used in such a case is observing that training the model further does not lead to any improvement. Obviously, this does not give you any guarantees whatsoever that a better solution is not possible, it shows only that the model that you are using, with the random initialization and hyperparameters that you used, does not seem to improve anymore. It still may be possible that using a different model, random seed, hyperparameters, or optimizer, etc could lead to better result.
TL;DR What you are asking is impossible. You cannot have perfectly fitting one-to-many mappings, they cannot be learned. There are however solutions that would yield optimal, though not exact predictions. In such a case, you train the model until it stops improving.
