I have been trying to implement a wavenet. From the papers and designs I have looked at on github I have come up with the following...

for i, (last, d) in enumerate(is_last([1, 2, 4, 8, 16, 32, 64, 128, 256] * 4)):
        h = layers.Conv1D(64, 2, dilation_rate = d, padding = 'causal', activation = 'tanh', name = 'h_%d' % i)(r)
        t = layers.Conv1D(64, 2, dilation_rate = d, padding = 'causal', activation = 'sigmoid', name = 't_%d' % i)(r)
        x = h * t
        s = s + layers.Conv1D(256, 1, name = 's_%d' % i)(x)
        if not last:
            r = r + layers.Conv1D(64, 1, name = 'r_%d' % i)(x)

In this code block h and t are the dilated/gated convolutions. The s variable is my skip connection which will eventually have a relu applied to it before the post processing layers. The r variable is my residual connection which is fed into the next layer. What I don't understand is why the convolution that is added to r does not have an activation function. I know having two linear layers in a row can just be simplified to a single linear layer. Am I missing something here? What is the point of having a linear convolution?


1 Answer 1


To me it appears to that the residual 1x1 convolution for $r$ is computing 64 channel-wise linear combinations of $x$. During training that convolution will 'learn' which linear combinations of channels from $x$ are useful for each channel of $r$ when being applied as a residual connection.

Something common for residual connections is directly adding $x$ to $r$ with perhaps some learnable scalar parameter $\alpha$ such that $r = r + x \cdot \alpha$. This linear 1x1 convolution can learn to do the exact same thing, but it can also learn a more complex linear relationship.

The lack of a nonlinear activation is likely for simplicity; the purpose of residual connections is to carry information across layers which may have been 'lost', and non-linearity in the residual connection itself would introduce the possibility of information loss (think saturation of sigmoids or dying of ReLUs) and may introduce additional points of instability during training.

I don't have much experience with wavenets compared to other network architectures, so take my response with a grain of salt.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.