What are the senones in a Deep Neural Network?

Question

I am reading this paper: skype translator where they use CD-DNN-HMMs (Context dependent Deep neural Networks with Hidden Markov Models). I can understand the idea of the project and the architecture they've designed but I don't get what are the senones. I have been looking for a definition but I haven't found anything

—We propose a novel context-dependent (CD) model for large-vocabulary speech recognition (LVSR) that leverages recent advances in using deep belief networks for phone recognition. We describe a pre-trained deep neural network hidden Markov model (DNN-HMM) hybrid architecture that trains the DNN to produce a distribution over senones (tied triphone states) as its output

Please if you could give me an explanation about this I would really appreciate it.

EDIT:

I've found this definition in this paper:

We propose to model subphonetic events with Markov states and treat the state in phonetic hidden Markov models as our basic subphonetic unit -- senone. A word model is a concatenation of state-dependent senones and senones can be shared across different word models.

I guess they are used in the Hidden Markov Model part of the architecture in the first paper. Are they the states of the HMM? The outputs of the DNN?

Answer 1

"Senones" was named by me in 1992. See my ICASSP 1992 paper[1]. It's just a fancy name for a cluster of shared Markov states, representing similar acoustic events. It came from the contrast with IBM's fenones, where the "f" means "frame" and my "s" means "state".

The initial idea came from my 1991 Eurospeech (now called Interspeech) work, where I used top-down clustering on Markov states. You can find my CMU tech report in 1991 here: https://www.semanticscholar.org/paper/Shared-distribution-hidden-Markov-models-for-speech-Hwang-Huang/33ea989f1655636162b7e9b8e0cfe3fcce92c37d

In 1992, I decided to move it to tree clustering so that one could model unseen cd phones as well.

[1] Hwang, Mei-Yuh, and Xuedong Huang. "Subphonetic modeling with Markov states-Senone." In [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 1, pp. 33-36. IEEE, 1992. link: https://ieeexplore.ieee.org/document/225979

Answer 2

This is what I've finally understood:

In this architecture the DNN is used to transform the noise into phones.

In phonetics and linguistics, the word phone may refer to any speech sound or gesture considered as a physical event without regard to its place in the phonology of a language.

The last layer of the DNN is formed by all the possibles phones, having one output neuron per phone. The activation of these neurons is the probability of that the input noise corresponding to that phone.

The combination of these activations is the input of the Hidden Markov Model and establishes the senones of the HMM, which obtains a list of candidate texts by means of a dictionary.

The senones are the states of the HMM, in the following image, the senones would be x1 x2 and x3.

Please correct me if I've said something wrong, hope it helps!

Answer 3

In Speech Recognition, we often include some context about neighboring phones when modelling a certain phone. This means that our system not only knows phones for A, B and so on, but instead has a concept for E-then-A, O-then-B, X-then-A and so on.

These context-depended units are called senones in literature, which is of course a made-up word.

For speech recognition systems, these senones usually equal the HMM states of the acoustic model, which can be predicted by a neural network, if a DNN/HMM hybrid approach for acoustic modelling is used.

The term Senones was coined by the developers of the Janus speech recognition toolkit. It was then adopted by Dong Yu and Li Deng for their ASR book. That was in the time before NNs were used for acoustic modelling. Therefore the term is confusing.