1
$\begingroup$

If you have a function that you want a neural network to learn, and you have sufficient examples of data with the right coverage, and there is no noise in the data, then is it reasonable to expect to achieve a (nearly) zero error rate, i.e. 100% accuracy, with a neural network?

$\endgroup$
0

2 Answers 2

2
$\begingroup$

You should clearly separate between training error and prediction error.

Being a universal approximator, a neural network is in principle able to reproduce any given set of data exactly, i.e. the training error can always be brought to 0%. In particular, it doesn't matter whether your data contains noise or not -- it simply reproduces all given training results when feeding in the corresponding predictors. (However, this is hardly desired as it leads to overfitting and thus usually weak prediction results. To avoid this, methods such as regularization/weight decay, early stopping, Bayesian ANNs, special network topologies, etc. are often used).

On the other hand, you have prediction error. If you want to approximate an unknown function (again, let there be noise or not), in general you cannot expect to obtain 0% prediction error. This is because you make a special assumption about the underlying model (in the case of ANNs it is a stacked sum of sigmoids), which in general might behave arbitrarily badly between two training points. In reality, however, the underlying models are often smooth and ANNs will work well, but you can never be sure. (Yet, in special cases, it might work. A synthetic example: if you use an ANN to reproduce another ANN, you can get an exact match of the models).

So, the answer to your question is you can have 100% accuracy for the training error, but in general never for the prediction error.

$\endgroup$
2
  • $\begingroup$ I am curious, if the data generating process is actually distributed as a stacked sum of sigmoids and there is no noise, wouldn't prediction error be 0 sooner or later? $\endgroup$
    – CarrKnight
    Jul 4, 2014 at 13:14
  • 1
    $\begingroup$ Yes, that is exactly the statement of the last sentence in parentheses in the second-to-last block. $\endgroup$
    – davidhigh
    Jul 4, 2014 at 22:39
1
$\begingroup$

That depends on the problem and how you model it. See this paper for a detailed analysis. For inverse problems which are very common in robotics and image processing, a standard MLP won't work. Regardless of whether you have noise or not.

In inverse problems one basically attempts to infer how the observed data was generated. For example, in image processing one observes a blurred image and ignores the exact process that generated the blur. Still, one attempts to deblur the image by trying to find some blur function which might caused it. There might be several solutions for the problem.

To put it in more visual terms, consider this plot inverse problem. The plot corresponds to the function $y = x + 0.3*sin(2\pi x)$. The problem you would like to solve is: given $y$, tell me which $x$ caused that value. There is a range of values for which several possible x's are possible.

Otherwise, for well-behaved functions, you can expect you achieve arbitrary high accuracy.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.