Why BERT Boosts Performance for NLP Tasks? Is there a laymen explanation for why BERT can boost performance for NLP tasks? After reading a lot of articles, still not clear about where the performance boost comes from. Is it because of attention mechanism used in BERT?
 A: One of the reasons for popularity of deep neural networks, is that they are able to do automatic feature extraction from the data. Traditionally, when solving some data problem, you would need to gather the data, clean it, and extract some features that would help your model to learn something from the data. Those features could be different transformations of the data that extract, or highlight some important pieces of information that is encoded within the data. Deep neural networks are able to take as input almost raw data (e.g. text, or images) and learn from it directly. The popular example of this are convolutional neural networks, where you can actually see how different layers learn different level of granularity of information, with vary basic "building blocks" in early layers, and more detailed features in the deeper layers (see image below).

Similar things happen in deep neural networks for natural language processing problems. The networks also learn to encode the data into some latent feature vectors (layers of the network).
Models like BERT are often used either for feature generation, where you pass some data directly through them to generate some features (same as you could use PCA for tabular data), or transfer learning, where you re-train them on your data. In first case, the model is "frozen", in the sense that you don't train it, you use it just as a function that encodes your data. In second case, part, or all the layers of the model are unfrozen and you train them along with your model. There are also two ways of using such models, you could either use them as stand-alone feature generation utilities, or glue them to your network as early layers.
Why does it work? That's the kind of question nobody can answer. There is a lot of speculation, but basically, different network architectures, use many different clever tricks, that seem to work well for extracting information from the data. This area is dynamically evolving, what is popular now, could not be popular and state-of-art in a year or two. Moreover, it is hard to point single reason for success of such models, since they are build based on a lot of moving parts, that together lead to the success.
One thing, that we know for sure, is that the mode data you have, the better results you get and this is especially true for deep learning and for natural language processing. BERT was trained on enormous amount of data, so it "knows" a lot about human language and it surely could have lead to it's success. There are even papers showing that with huge amounts of data the model choice becomes less important, since you gain performance from the data alone.
