Which function can approximated with Neural Networks using only linear activation functions?

Question

I want to find out which functions can be approximated up to arbitrary accuracy using Neural Networks with only linear activations. On this page I found out that with linear activation functions, the error in the prediction stays constant and is not depending on $x$. This means probably it has a bad accuracy, but I have not found out, if there is something like a URT for linear functions.

What I don't understand yet, how to relate the activation function to the actual model I want to predict.

My idea is that with only linear activation functions you can approximate only linear functions up to arbitrary accuracy. I would argue that the cost function:

\begin{align*} C(\mathbf{X}, \mathbf{y}, \mathbf{w})&=\frac{1}{N} \sum_{i=1}^{N}\left(y_{i}-\hat{y}\left(\mathbf{x}_{i} ; \mathbf{w}\right)\right)^{2}\\ &= \frac{1}{N} \sum_{i=1}^{N}\left(y_{i}-\mathbf{w}\cdot \mathbf{x}_i - b_i\right)^{2} \end{align*}

contains only a sum of linear functions, which cannot approximate a quadratic function for example. Maybe someone can help me writing that in a more rigorous way.

Answer 1

The composition of one or more linear functions is itself a linear function. A neural network using only linear activations can be rewritten as a linear function. What is the purpose of a neural network activation function?

Using a linear function, you'll be able to approximate functions which are single, straight lines, i.e. linear functions. Approximating a nonlinear function with a linear function will have some amount of error -- perhaps small enough to be ignored, perhaps too large to be acceptable. This judgement depends on what you need from your model.