10
$\begingroup$

In the recent WaveNet paper, the authors refer to their model as having stacked layers of dilated convolutions. They also produce the following charts, explaining the difference between 'regular' convolutions and dilated convolutions.

The regular convolutions looks like Non dilated Convolutions This is a convolution with a filter size of 2 and a stride of 1, repeated for 4 layers.

They then show an architecture used by their model , which they refer to as dilated convolutions. It looks like this. WaveNet Dilated Convolutions They say that each layer has increasing dilations of (1, 2, 4, 8). But to me this looks like a regular convolution with a filter size of 2 and a stride of 2, repeated for 4 layers.

As I understand it, a dilated convolution, with a filter size of 2, stride of 1, and increasing dilations of (1, 2, 4, 8), would look like this. Actual Dilated Convolution

In the WaveNet diagram, none of the filters skip over an available input. There are no holes. In my diagram ,each filter skips over (d - 1) available inputs. This is how dilation is supposed to work no?

So my question is, which (if any) of the following propositions are correct?

  1. I don't understand dilated and/or regular convolutions.
  2. Deepmind did not actually implement a dilated convolution, but rather a strided convolution, but misused the word dilation.
  3. Deepmind did implement a dilated convolution, but did not implement the chart correctly.

I am not fluent enough in TensorFlow code to understand what their code is doing exactly, but I did post a related question on Stack Exchange, which contains the bit of code that could answer this question.

$\endgroup$
  • $\begingroup$ I found your question and answer below quite interesting. Since the WaveNet paper does not explain that equivalence of stride and dilation rate, I decided to summarize the key concepts in a blog post: theblog.github.io/post/… you might find it interesting if you're still working with autoregressive neural networks $\endgroup$ – Kilian Batzner Feb 28 '19 at 19:42
10
$\begingroup$

From wavenet's paper:

"A dilated convolution (also called a trous, or convolution with 
holes) is a convolution where the filter is applied over an area larger 
than its length by skipping input values with a certain step. It is 
equivalent to a convolution with a larger filter derived from the 
original filter by dilating it with zeros, but is significantly more 
efficient. A dilated convolution  effectively allows the network to 
operate on a coarser scale than with a normal convolution. This is 
similar to pooling or strided  convolutions, but 
here the output has the same size as the input. As a special case, 
dilated convolution with dilation 1 yields the standard convolution. 
Fig. 3 depicts dilated causal convolutions for dilations 1, 2, 4, and 
8."

The animations shows fixed stride one and dilation factor increasing on each layer. Animated Fig. 3 from Google's wavenet blog post

| cite | improve this answer | |
$\endgroup$
7
$\begingroup$

The penny just dropped on this one for me. Of those 3 propositions the correct one is 4: I did not understand the WaveNet paper.

My problem was that I was interpreting the WaveNet diagram as covering a single sample, to be run on different samples arranged in a 2D structure with 1 dimension being the sample size and the other being the batch count.

However, WaveNet is just running that whole filter over a 1D time series with a stride of 1. This obviously has a much lower memory footprint but accomplishes the same thing.

If you tried to do the same trick using a strided structure, the output dimension would be wrong.

So to summarize, doing it the strided way with a 2D sample x batch structure gives the same model, but with a much higher memory usage.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.