I am not an expert in language modelling domain.
There are mainly two approaches that are being used nowadays. Denoising autoencoders and masking. Recent models like BERT use masking and outperforms other models on a number of tasks. Now, my questions are:
1) Why is masking so useful compared to stacking a bunch of denoising autoencoders?
2) Are there any disadvantages of masking that make it a worse approach compared to using a denoising autoencoder?
I also had the same question on the difference between the Denoising and Masked AEs. I am not fully aware of potential differences and/or benefits in the area of language modeling but from the general perspective each is aimed for a different purpose:
As we can see they both have to do something with the hidden representation but one in an explicit way the other in an implicit way. I am not aware of a work that uses MAE together with denoising. It might be a valid thing to do or not, but intuitively in my view, it could potentially be beneficial as each addresses two different issues.
One example to use denoising on top of MAE in the language modeling that I can think of is that we would like to model the $P(\hat{x_i}\mid x_{i-1},\dots)$ where $x_{i-1}$ is the last observed letter and $\hat{x}$ is the probability of the next letter. Now in the denoising case, we could consider that the $x$ is perturbed with a probability (changed to some other letter). This could make our (generative) language model robust to potential mistakes in our corpus and in a sense, our model not only learns the succession of letters following other letters, but also the succession of mistakes.
Why we would prefer a Generative Model over a Discriminative Model would be another topic but computational overhead and other restrictions aside, we would certainly benefit more from a Generative Model and so that is why MAE would be preferable.
I hope both of your questions were answered. If not, I can elaborate more and we can discuss it further!
