3
$\begingroup$

Let's say I have a long document (let's say 10 pages) and I have a question about the content of the document. An example could be:

Document: History of Flowers Question: What type of flowers have been considered edible in the past?

My objective is to find in this document the information that answers this question. My primary objective is not to have a sentence as an answer but to find the parts of the text that contain this information.

Up to now I've tried using semantic search but:

  • I'm not sure on what is the correct way to split the document (should they be phrases? paragraphs?)
  • questions are usually written in quite a different way from the information that answer them (in fact, that information is not know to those making the question)

As an alternative I was thinking that I could:

  1. run a QA model to find the answer
  2. take the answer and use it to build for the search

In this case, the problem are:

  • the QA model can only read part of the document ( how to tackle that? I was thinking of running it on chunks and then take only the "highest scoring" answer.
  • It gets longer and more cumbersome

So, my question(s) are:

  • does this approach in you opinion make any sense?
  • what are possible alternatives?
$\endgroup$
1
  • $\begingroup$ Questions like this about unstructured data may fit better on AI StackExchange, even though they’re on-topic here according to the current guidelines; the techniques for answering them may not resemble the techniques where this site has expertise. $\endgroup$
    – Matt F.
    Commented Dec 30, 2022 at 16:04

1 Answer 1

7
+100
$\begingroup$

In 2019, Google's researchers released a Natural Questions dataset, designed to test the exact challenge you describe. Just as you want to identify, given a 10-page document and a question, a span of text that answers the question, Natural Questions tests the ability to locate from within a relevant Wikipedia article a span of text that answers a question. Whereas the earlier SQuAD dataset was limited to searching for answering spans within paragraphs and, thus, would have required something like the "chunking" solution you propose for adaptation to longer articles, the Natural Questions dataset seems almost identical to the task you describe (the "short answer" version of the task requires exactly what you have described: identifying the precise span of text from within a longer document that will answer a given question--not just a sentence-based answer).

Moreover, some of the best researchers in the world have published their attempts to address that challenge. Google's leaderboard includes results from 54 different competitors, and their list is not exhaustive. For example, Facebook just published a paper and code for Atlas, which includes few-shot results on the Natural Questions dataset.

Thus, by exploring the various published attempts to solve the Natural Questions challenge, you should find lots of potential solutions to your nearly identical use case. Depending on your available resources and accuracy requirements, you can likely find ideas and/or code by investigating those published solutions. Best of luck.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.