Cross layer parameter sharing in ALBERT Model I am reading the paper "ALBERT: LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS". ALBERT uses cross-layer parameter sharing to improve model performance. I don't understand how does parameter sharing helps the model? In a Google AI blog, they say that "the network often learned to perform similar operations at various layers, using different parameters of the network. This possible redundancy is eliminated in ALBERT by parameter-sharing across the layers, i.e., the same layer is applied on top of each other.". How did they find out that networks often performed similar operations at different layers?.
Any help would be appreciated.
 A: Regarding your first question: "How does parameter sharing help the model?"
The main purpose of sharing is the radical reduction of parameters, one of the main selling points was to reduce the impractical size of Bert.
The authors do mention that the parameter reduction also acts as regularisation/generalisation. It does not improve the performance of the model though ("This approach slightly diminishes the accuracy, but the more compact size is well worth the tradeoff.").
As to your second question, "How did they find out that network often performed similar operations at different layers?" - I don't know, but I can guess that they might have used some technique to inspect the network. 
Van Aken et al (2019) "present a layer-wise analysis of BERT’s hidden states" by applying "probing tasks that reveal the information stored in each representation layer". Another example for probing into Bert is presented by Tenney et al (2019). Note that these do not particularly attempt to measure existing redundancy and don't seem to make explicit claims about it.
However, there are a few papers which notice redundancy when looking at attention heads in particular and what they attend to (e.g. certain heads mostly attend to special tokens or punctuation, other heads attend to tokens that relate to syntactic relations, e.g. direct objects attend to their verbs). Clark et al (2019) find that there are groups of heads behaving quite similarly, in particular heads within the same layer. Voita et al (2019) and Michel et al (2019) show that many heads can be pruned away without hurting model performance.
I do not know if the redundancy that the Albert authors mention is the same that the above papers found. What's inconsistent is that the most redundancy was found between attention heads within one layer, as opposed to across different layers. But it is still something worth considering.
