Large language models, such as GPT-3, BLOOM etc, can generate open-ended text. Say I want to prompt these models to answer a question. How can I compare the semantic similarity of the answer it provides me with a reference question?

Eg, if the prompt is, 'What should I eat for breakfast?', an example response that BLOOM will output is 'when should i eat lunch? etc.). Is there any way to get some suggestions from microsoft?'

Say the reference answer to this question is 'You should eat a full English breakfast'. How can I measure how different the BLOOM response is to the reference answer, in terms of its content?

Currently, I am encoding the text into BERT embeddings, and then computing cosine similarity between the output text and reference text. But the cosine similarity scores are quite low, because, as you can see, the BLOOM answer is not really on task. Should I be finetuning the BLOOM model somehow before trying to test semantic similarity?

The original BLOOM/GPT-3 models did quite well on SQuaD, but I don't seem to find that these models are answering questions very well without finetuning.


1 Answer 1


How can I compare the semantic similarity of the answer it provides me with a reference question?

With a text generation metric. See Evaluation of Text Generation: A Survey. Typical metrics: TF-IDF cosine, Rouge, Bleu, BertScore, Sentence-Bert, and more recently, GPTScore and G-Eval. Note that they have some serious limitations when evaluating GPT output {1,2}.


Since comparing the semantic similarity of the predicted answer with a reference question has some limitations, one may look at complementing the analysis with a human evaluation. The following papers survey criteria used for human evaluation of generated texts (thanks Mengjiao Zhang for pointing me to refs {3,4}):

From {3}:

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.