2
You are indeed not the first one who thinks like this. The pragmatic answer would be: people tried, but the negative-log likelihood seems to work better.
There are several relatively successful attempts (mostly in machine translation):
Von mises-fisher loss for training sequence to sequence models with continuous outputs
A Margin-based Loss with Synthetic ...
1
It sounds like you might want to have a look at the Levenshtein (edit) distance.
The algorithm compares two strings and identifies how they differ by considering insertions (insert a new character), swaps (swap two characters) and deletions (delete a character), where each of these operations have a cost associated with it.
You can then compare the correct ...
1
In NLP, I have seen it done with one-hot encoding: https://colab.research.google.com/github/AndreasMadsen/python-textualheatmap/blob/master/notebooks/huggingface_bert_example.ipynb
But I've seen more places use embedding, then normalize the embedding to get a single score per token. This recent survey of input saliency shows better results for aggregating ...
1
why does masking requires us to sample (take only 15%) of the words?
I think this guarantees that only about 1/7 of the words are masked, which is just like the window size in word2vec. That is on average in BERT we use 7 words as context to predict one word. The more words we mask the smaller the "window size" and the smaller the context.
Google ...
1
To complete this question I write this answer.
Is there some convention about hashing function that I'm not aware of (meaning the k-functions should be obvious for the reader), for example is there any canonical hashing function which makes the reference optional?
I think what matters when we consider a hash function is how rarely the collisions would ...
Only top voted, non community-wiki answers of a minimum length are eligible
