Let's assume we have a multilayer feedforward network with one hidden layer. The hidden layer must consist of at least two neurons with logistic activation function. How many hidden neurons are necessary in order to approximate a linear function $ f(x) = \beta_0 + \beta_1 x + \epsilon$ with an accuracy similar to a linear regression model?


It appears that you're referring to the the Cybenko universal approximation theorem. One of the hypotheses of this theorem is that you're approximating a function on a compact subset. Outside of this subset, Cybenko's UAT is silent, so the approximation could be arbitrarily poor for certain functions. Are you asking about the error inside of this interval, or outside?

On the other hand, linear functions are linear everywhere, so the OLS model could do well on any interval if the true function is linear, or poorly if it is not. For example, you can approximate the linear part of $\tanh$ well with a line, or any of its asymptotes, but the linear approximation is poor if you consider the entire function.

Likewise, the answer depends on the level of precision you wish to achieve. Among other terms and conditions, the Cybenko UAT says that for some desired level of precision, there exists a neural network with sigmoid activations and some finite number of hidden units that can approximate certain kinds of functions to the desired level of precision. If you want more precision, you'll need more units.

It's not possible to make a general statement relating this to OLS, because different OLS models will achieve different levels of precision, depending on the problem. If the true model is $y = \beta_0 + \beta_1 x + \epsilon$, then the expected MSE depends on the distribution of $\epsilon$. Are the $\epsilon$s independent, identical realizations from some distribution? Which distribution? Or something else?

It's not clear what you mean about the accuracy of a linear model. A linear model can be a good choice when the phenomenon is linear. Or it can be a bad choice when it's not.

Finally, Cybenko doesn't comment on the real applications of NNs, which is using a finite data set and some method of iterative parameter updates to discover a good choice of weights and biases. So even if we choose the number of hidden nodes correctly for our desired level of precision and satisfy all of the other technical conditions of the theorem, there's not reason to believe that this particular network is feasible to train with our data set using our primitive technology. It might take too long, or it might not happen because optimization is hard and the data are noisy.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.