I've been trying to understand the wavenet paper. In order to do so, I am using this implementation that I found on github because it gets good results and it is pretty clear. But I have a question regarding the architecture that I don't seem to be able to solve.

On one side, the wavenet has this dilation scheme:

wavenet's dilation structure

It would seem from here that different layers take different amount of inputs and outputs according to the dilation.

On the other side, we have this structure for the entire wavenet's network:

wavenet's network structure

In which every hidden layer inputs and outputs the same number of values. This seems clear in the fact that the residual values are just summed at the output of the layer with the value produced by the dilated convolution and the gated activation units.

I understand both structures on its own, but I am having trouble with understanding how they coexist.

It looks like every layer inputs and outputs tensors with the same shape, is that correct?

If so, does the dilation just neglect several values (doesn't take them into account)? That would make the convolutions faster, but I don't see the point in propagating a bunch of values that you're never going to use.

And a little bonus one, is that 1x1 convolution after the gated activation units a re-sampler in order to restore the initial dimensionality?

It's interesting to see in the code that there are 2 1x1 convs after the gated activation units, one for the skip connections and another one for the residual but that these are not equal (they have different shapes). So it would seem that if the 1x1 are re-sampling the tensors, they are propagating different shapes to the next hidden layer and to the skip-connections.

  • $\begingroup$ I don't understand nearly enough to confidently answer most of your questions but I'll take a crack at dilation. My understanding is that dilations allow later convolutional layers to have a larger receptive field without expensive computation cost. Basically, the assumption is that you don't actually want all of the values propagated—however, how do you know which ones you want and which you don't? $\endgroup$ Commented Apr 17, 2018 at 11:15
  • $\begingroup$ In dilated convolutions you want every X value. In this case every other- you take one, you leave one. This makes sense for compressing sound signals since you can argue that every layer has the same 'time' receptive field and 'frequency' receptive field. In the rogue implementation I cite here, they take the last half of the samples to sum in the residual connection after the dilated convolution. That seems very odd. $\endgroup$ Commented Jul 24, 2018 at 9:18


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.