2
$\begingroup$

I'm reading the Wavenet paper which says:

Stacked dilated convolutions enable networks to have very large receptive fields with just a few layers, while preserving the input resolution throughtout the network as well as computational efficiency. In this paper, the dilation is doubled for every layer up to a limit and then repeated: e.g.,

1, 2, 4, ..., 512, 1, 2, 4, ..., 512, 1,2,4, ..., 512.

...Each 1, 2, 4, ..., 512 block has a receptive field size of 1024...Stacking these blocks further increases the model capacity and the receptive field size.

I'm trying to understand what the receptive field is after the Nth block of 1, 2, 4, ..., 512. After the second block, is it 1024*1024, and after the third block is it 1024^3 ? Or am I misunderstanding how the receptive field size expands?

$\endgroup$
3
$\begingroup$

I think it should be 1024*3.

After the first block, the indices of the receptive fields of the outputs should be 1-1024, 2-1025, 3-1026, etc. (assuming no padding, but receptive field size should be same with padding anyways). When you make the second block with a receptive field size of 1024, the first output of that block will "see" the outputs that had receptive field indices 1-1024, 2-1025, ... 1024-2048. So its receptive field covers 1-2048. So each block just adds 1024 to the overall receptive field size I think.

In general, I think the formula for the receptive field size s of a layer l should be:

$s_{l_0} = 1$

$s_{l_i}=s_{l_i} + (kernel size - 1) \cdot dilationfactor$

If this is correct, their kernel size seems to be 2 (to arrive at 1024 receptive field size), which is a bit surprising, I hope it is not due to some fault of my logic :)

Stacking of the blocks might be also more useful to refine outputs at a more finegrained level after having processed larger receptive fields in the previous block, rather than just maximally increasing receptive field size.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.