WaveNet Global and local conditioning

Question

WaveNet is a deep learning framework able to generate raw audio signal from a sequence like text sequence. https://arxiv.org/abs/1609.03499

It is also possible to "imitate" in a way the voice of the speaker learned during the training adding an additional variable input h. Two methods are described in the WaveNet's paper : global conditioning and local conditioning.

Global conditioning is characterised by a single latent representation h that influences the output distribution across all timesteps, e.g. a speaker embedding in a TTS model. The activation function from Eq. (2) now becomes: z = tanh ( W f,k ∗ x + V T f,k h ) σ ( W g,k ∗ x + V T g,k h )

For local conditioning we have a second timeseries h t , possibly with a lower sampling frequency than the audio signal, e.g. linguistic features in a TTS model. We first transform this time series using a transposed convolutional network (learned upsampling) that maps it to a new time series y= f ( h ) with the same resolution as the audio signal, which is then used in the activation unit as follows: z = tanh ( W f,k ∗ x + V f,k ∗ y ) σ ( W g,k ∗ x + V g,k ∗ y )

• For the Global conditioning part, I do not get what h represents exactly. Is it something similar to a one hot vector [ 0 0 1 0 0] where the 1 value match with the id associated of the speaker ?

• Regarding the Local conditioning, I do not get what ht is. How can we refers to a specific speaker with a time serie ?

Answer 1

I had the same question, and I think indeed is not explained very clearly in the paper, if you haven't seen features like that before.

With global conditioning the idea is exactly as the paper mentions it to use a speaker embedding, which is yes, the one hot vector, that you have mentioned above. The main concept here is that is has to be something that is not a time series, or does not influence the input in a time-wise fashion. It could be gender of speaker, religion of speaker, mood of speaker, etc.

The equation states that what you do then is have two inputs to your neural network, and the global one you just propagate through linear connections instead of convolutions. It is not an audio, not an image, so convolution does not really make sense, filtering one-hot vectors is not a good idea in general.

With local conditioning, now you have a time series data. This could be also categorical, but then it's two dimensional, because it is a categorical time series across time. In TTS, this would be graphemes (written text) or phones (pronunciation of the written text), so that we learn a function which will be able to syntesise speech for. The problem is that the sampling rate of such signal is not necessarily the same. Their first solution was just to "repeat/broadcast values", but they report it have not worked well, so that is why you use a transposed convolution for upsampling and learning a good upsampled representation. For processing, a 1x1 convolution is used, which is I think for matching the dimensionality of the convolution filterbank.