WaveNet is a deep learning framework able to generate raw audio signal from a sequence like text sequence. https://arxiv.org/abs/1609.03499
It is also possible to "imitate" in a way the voice of the speaker learned during the training adding an additional variable input h. Two methods are described in the WaveNet's paper : global conditioning and local conditioning.
Global conditioning is characterised by a single latent representation h that influences the output distribution across all timesteps, e.g. a speaker embedding in a TTS model. The activation function from Eq. (2) now becomes: z = tanh ( W f,k ∗ x + V T f,k h ) σ ( W g,k ∗ x + V T g,k h )
For local conditioning we have a second timeseries h t , possibly with a lower sampling frequency than the audio signal, e.g. linguistic features in a TTS model. We first transform this time series using a transposed convolutional network (learned upsampling) that maps it to a new time series y= f ( h ) with the same resolution as the audio signal, which is then used in the activation unit as follows: z = tanh ( W f,k ∗ x + V f,k ∗ y ) σ ( W g,k ∗ x + V g,k ∗ y )
For the Global conditioning part, I do not get what h represents exactly. Is it something similar to a one hot vector [ 0 0 1 0 0] where the 1 value match with the id associated of the speaker ?
Regarding the Local conditioning, I do not get what ht is. How can we refers to a specific speaker with a time serie ?