Is this statement about linear-activation neural network in the PRML in error?

Question

I found the following statement in "Pattern Recognition and Machine Learning" (C. M. Bishop, 2016) p.229.

If the activation functions of all the hidden units in a network are taken to be linear, then for any such network we can always find an equivalent network without hidden units. This follows from the fact that the composition of successive linear transformations is itself a linear transformation. However, if the number of hidden units is smaller than either the number of input or output units, then the transformations that the network can generate are not the most general possible linear transformations from inputs to outputs because information is lost in the dimensionality reduction at the hidden units.

However, I think that there are two exceptions: D < M < K, and D > M > K, where D, M, and K are the numbers of input, hidden, and output units respectively.

For example, in the former case, when the input and hidden units are connected by identity matrix (only D of M nodes are used, and others are zero), the neural network (without activation) can represent all possible linear transformation using linear transformation between M and K.

Is my understanding correct?

Answer 1

The easier way to think of this is as matrices: a two-layer linear-activation neural network (with bias units) can be written $$ f(x) = W (V x + b) + c = (W V) x + (W b + c) .$$ Using your notations, the first layer has its weights $V$ of shape $M \times D$ (and bias $b$ of shape $M$), and the second layer has weights $W$ of shape $K \times M$ (and bias of shape $K$).

This is equivalent to a single layer, with linear activations, whose weights are $W V$ of shape $K \times D$ and bias $W b + c$.

Because $c$ can be anything, we can ignore the effect of $W b$ there, and get any bias we want for the equivalent one-layer network.

$W V$ can be any $K \times D$ matrix whose rank is at most $\min(M, K, D)$. So, if $M = K$ or $M = D$, the one-layer network can be any possible matrix. If $M$ is smaller than both $K$ and $D$, then it limits the set of possibilities. So, in your example, if $D < M < K$ then $M$ doesn't actually limit the final outputs.

I don't think Bishop's intended meaning was wrong, but he wrote it in a slightly confusing way: "if the number of hidden units is smaller than either the number of input or output units" should mean "$M < \min(K, D)$", not the perhaps more natural reading of "$M < K$ or $M < D$."