Given enough time, can a fully connected layer approximate a causal convolution layer?

Question

In the paper WaveNet: A Generative Model for Raw Audio, the authors try to capture spatial data as follows:

They do this by limiting the scope of the hidden layers to particular sections of the input, this approach improves performance as supported by experimental evidence. However, is it not possible for the same structure to arise naturally through the use of a straightforward fully connected layer, or am I overlooking something?

Is the purpose of the new layer just to artificially "accelerate training" by having humans narrow down the search space to smaller and smaller subsets, or is there something distinct about it?

Would a fully connected layer have eventually uncovered the same results if given enough time and computation resources (but the exact same dataset)?

Answer 1

Probably not.

As you allow the model more flexibility to fit the (in-sample) data, you let it use that flexibility to get a smaller value of the loss function.

The genius of conventional neural networks is not just that they limit this flexibility but that they do so in a clever way that makes sense from the standpoint of how humans see (or how we think we see).

Answer 2

It is certainly possible for a fully connected layer to stumble upon the same solution, or even a better one - but its greater expressive power also reduces the probability that the algorithm will ever find them. Simply increasing the training time, even to infinity, does not guarantee that a neural net algorithm will find a good solution, since it could get trapped in local minima. A fully connected layer without engineered constraints like those discussed in this paper could, in fact, be prone to a wider range of such local minima. On p. 2 the paper even mentions a clear case in point: "By using causal convolutions, we make sure the model cannot violate the ordering in which we model the data..." A fully connected model lacking that constraint would have a greater capacity to violate those orders, leading to just one of several classes of suboptimal solutions. Carefully crafted features of modern neural nets, such as convolutional topologies, are designed to stack the deck against certain classes of poor solutions by baking domain knowledge right into the architecture. Without them, navigating to ideal solutions becomes more improbable and dependent on sheer luck in some cases.