2

You are indeed not the first one who thinks like this. The pragmatic answer would be: people tried, but the negative-log likelihood seems to work better. There are several relatively successful attempts (mostly in machine translation): Von mises-fisher loss for training sequence to sequence models with continuous outputs A Margin-based Loss with Synthetic ...


1

It sounds like you might want to have a look at the Levenshtein (edit) distance. The algorithm compares two strings and identifies how they differ by considering insertions (insert a new character), swaps (swap two characters) and deletions (delete a character), where each of these operations have a cost associated with it. You can then compare the correct ...


1

In NLP, I have seen it done with one-hot encoding: https://colab.research.google.com/github/AndreasMadsen/python-textualheatmap/blob/master/notebooks/huggingface_bert_example.ipynb But I've seen more places use embedding, then normalize the embedding to get a single score per token. This recent survey of input saliency shows better results for aggregating ...


1

why does masking requires us to sample (take only 15%) of the words? I think this guarantees that only about 1/7 of the words are masked, which is just like the window size in word2vec. That is on average in BERT we use 7 words as context to predict one word. The more words we mask the smaller the "window size" and the smaller the context. Google ...


1

To complete this question I write this answer. Is there some convention about hashing function that I'm not aware of (meaning the k-functions should be obvious for the reader), for example is there any canonical hashing function which makes the reference optional? I think what matters when we consider a hash function is how rarely the collisions would ...


Only top voted, non community-wiki answers of a minimum length are eligible