I read about the universal approximation theorem and they say it's valid to learn continuous function. So I was wondering how this theorem could be extended in the case of classification tasks ? I guess that, in this case, the theorem can not directly be applied as classification is a discrete task.
I also heard about the fact that any 3 hidden layers network could learn any boolean function but I am not sure about it.
Could someone shed light on this ?
Thanks
Most of the time we use a continuous activation function. Therefore the output must be continuous. Yet the uniform limit of continuous functions must be continuous. So if your target is not continuous, because it takes values in $\{0,1\}$, then there is no way you can uniformly approximate it with neural networks, at least not with continuous activation function (which probably discards most of usual cases).
However the set of continuous functions over a compact $K$ is dense in the set $L^p(K,\mu)$ of integrable functions on $K$, endowed with $L^p$ convergence, for $p\in[1,+\infty)$ and any Borel measure $\mu$ finite on compacts (for instance the Lebesgue measure). For example (measurable) bounded functions are Lebesgue-integrable on any compact. Therefore you classification target with values in $\{0,1\}$ can be approached arbitrarily closely on any compact by a one hidden layer network in the sense of $L^p$-convergence.
