As the title of the question suggests, I'm interested in training a BERT-like model (and then use it to make some experiments on text-similarity).

Question: Could you share some references on the process of training a BERT model from scratch?


  1. A good reference to understand what do I mean by 'training from scratch' is something similar to what is done for the complete encoder-decoder training section in the article: 'The Annotated Transformer'.
  2. I'm particularly interested in the conceptual understanding of how a BERT-like model is trained.

Thanks in advance!

  • 1
    $\begingroup$ Perhaps you could strengthen your question by clarifying what you’d like to know that isn’t described in the original BERT paper. $\endgroup$ Aug 22 at 22:33


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy