2
$\begingroup$

I read about the universal approximation theorem and they say it's valid to learn continuous function. So I was wondering how this theorem could be extended in the case of classification tasks ? I guess that, in this case, the theorem can not directly be applied as classification is a discrete task.

I also heard about the fact that any 3 hidden layers network could learn any boolean function but I am not sure about it.

Could someone shed light on this ?

Thanks

$\endgroup$
1
  • $\begingroup$ Have you ever found an answer to this? $\endgroup$
    – ABIM
    Mar 23, 2020 at 8:48

1 Answer 1

1
$\begingroup$

Most of the time we use a continuous activation function. Therefore the output must be continuous. Yet the uniform limit of continuous functions must be continuous. So if your target is not continuous, because it takes values in $\{0,1\}$, then there is no way you can uniformly approximate it with neural networks, at least not with continuous activation function (which probably discards most of usual cases).

However the set of continuous functions over a compact $K$ is dense in the set $L^p(K,\mu)$ of integrable functions on $K$, endowed with $L^p$ convergence, for $p\in[1,+\infty)$ and any Borel measure $\mu$ finite on compacts (for instance the Lebesgue measure). For example (measurable) bounded functions are Lebesgue-integrable on any compact. Therefore you classification target with values in $\{0,1\}$ can be approached arbitrarily closely on any compact by a one hidden layer network in the sense of $L^p$-convergence.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.