New answers tagged

1 vote

Is pretraining on test set texts (without labels) ok?

TLDR: seems fine, which is weird. I'll check that the experiment code is correct. This answer contains empirical experiments similar to my experiment with PCA here. We'll run a few experiments with ...
chicxulub's user avatar
  • 1,260
0 votes

using latent dirichlet allocation to reduce the number of dimensions in bag of words model?

EDIT: So after some more research, it appears to me that what they (e.g. the SAM paper) refer to as "topic proportion" is actually alluding the normalized posterior Dirichlet parameters $\...
Kevin's user avatar
  • 41
0 votes

Why are language modeling pre-training objectives considered unsupervised?

The original GPT paper seems to consider the language modeling objective semi-supervised: Our work broadly falls under the category of semi-supervised learning for natural language. https://cdn....
Warlax56's user avatar
  • 101
0 votes

Why are the embeddings of tokens multiplied by $\sqrt D$ (note not divided by square root of D) in a transformer?

As mentioned in the transformer paper 'Attention is all you need', the embedding matrix in the input and the pre-softmax linear transformation in the linear layer outside the decoder block share the ...
Ethan's user avatar
  • 1
0 votes

BERT masking scheme

In my opinion, why the chosen token is not always replaced by the special token [mask] is the following: suppose the token x, and its context variable denoted c(x); BERT model tries to learn the ...
Ethan's user avatar
  • 1
-1 votes

Popular named entity resolution software

It's possible to have quite robust answers to record linkage / entity resolution problems using AI vectorisation and a vector database. An example which evaluates the DBLP and ACM data set benchmark ...
Gavin Mendel-Gleason's user avatar

Top 50 recent answers are included