I'm trying to find a way to prevent Intelligent Agents with Reading Comprehension and Question Answering abilities to answer when they are not confident enough that the y can find an answer from the documents they have

To my mind we can say we have on the one hand $P=\{p_1,...,p_i\}\neq\varnothing$ a set of paragraphs, $Q=\{q_{1i},...,q_{ij}\}\neq\varnothing$ a set of questions on these paragraphs and $R=\{r_{111},...,r_{ijk}\}$ the related possible answers. We try to find these answers of these paragraphs. Which are span of the sentences of these paragraphs when there is an answer or to show that there is no answers where it's unanswerable.

That is to say for every question $q_{ij}$ and every related paragraph, we predict an answer $\hat r_{ij}$ such that $$\forall q_{ij}\in Q , \hat r_{ij}= f(p_i,q_{ij}) = \begin{cases} r_{ijk},& \text{if } \exists k,\hat r_{ij}= r_{ijk}\in R\\ \emptyset, & \text{otherwise} \end{cases}$$

I did this problem definition on my own, feel free to criticize it.

I only thought about an unsupervised attempt until now: For each sentence-question pair we compute try two distance metrics: cosine similarity between the $q_{ij}$ and $s_l$. The answer is therefore:

$$\hat{r}_{ij}^{cos}=\arg\max_i \{1 - \frac{s_{l}.q_{ij}}{||s_{l}||.||q_{ij}||}\}$$

or we can also compute the Euclidean distance between the sentences and the question.

$$\hat{r}_{ij}^{euc}=\arg\max_i \{\sqrt{\sum_{v\in \mathcal V} s_l.q_{ij}}\}$$

I haven't chose the one or the other yet, the first one seems more accurate than the second one after a run giving answers may they exists or nor in the $R$ set.

Ideas of features

You can find all this work in this GitHub repository.

The distance feature

I was thinking about setting an idea of minimum to exceed for the distance for $\hat r_{ij}$ to accept a predicted answer rather than a null set. Yet, the distance feature doesn't bring much information.

As one can see on the cosinus similarity this doesn't bring much information about the existence of an answer.

enter image description here

As far as they clearly overlap, I can't create a threshold from this. The and the euclidian distance is even worse.

Update : match roots of question and sentences feature

But has low sucess rate.

The idea is to match the root of the question which is “appear” in this case to all the roots/sub-roots of the sentence. Since there are multiple verbs in a sentence, we can get multiple roots. If the root of the question is contained in the roots of the sentence, then there are higher chances that the question is answered by that sentence. Considering that in mind, I have created the following feature : the idea is trying to score those sentences where at least one root of the question $w_j'\in V_{q_{ij}}$ matched one other root $w_i\in V_{p_i}$ of this sentence and return the one with the highest score. The score of a sentence is the number of roots that match in the roots of the question.

That is to say given a paragraph $p_i$, for every question $q_{ij}$ with a set of roots $V_{q_{ij}}$ and every set $V_{s_{il}}$ of roots of a sentence $s_{il}$ of this paragraph, we receive the following answer $\hat y_{ij}$ such that $$\forall q_{ij}\in Q_i, \hat y_{ij}= \begin{cases} s_{il},& \text{if }\exists s_{il}\in S_i, \arg\max_l\{|V_{s_{il}}\bigcap V_{q_{ij}}|>0\} \\ \emptyset & \text{otherwise} \end{cases}$$ Where $S_i$ is the set of sentences of a paragraph $p_i$.

We get roughly 0.35 of correct answers with the match roots feature.

This Machine Learning problem has examples here with Stanford Question and Answer Dataset (SQuAD) and an attempt in this paper by Facebook.

  • $\begingroup$ Does "don't answer the question" need to be reframed as "What I still have yet to learn on this subject outweighs what I already know?" Or as something regarding learning as an objective? $\endgroup$ – Alexis Aug 10 '18 at 14:34
  • $\begingroup$ @Alexis Thank you for your comment. Yes, surely for the second idea! I think I have two things to do: on the one hand not to answer, for which I gave an attempt here, and on the other hand to learn when not to answer, when we had to for instance. $\endgroup$ – Revolucion for Monica Aug 10 '18 at 14:50
  • $\begingroup$ With some thought given to the relationship(s) between learning and answering as objectives. $\endgroup$ – Alexis Aug 10 '18 at 15:40
  • 1
    $\begingroup$ @Alexis Yes, yeah. Answering is the definitive goal, and learning might help for better answering. $\endgroup$ – Revolucion for Monica Aug 10 '18 at 18:59

In a little simpler set-up with FAQ answering system, I'm using simple approach with threshold - when alogithm returns probability / similarity below the threshold, I'm falling back into path "I'm not sure" (sligtly below threshold) or "I don't know" (significantly below threshold / below the second threshold).

| cite | improve this answer | |
  • $\begingroup$ Yes, that is the idea I provided for the unsupervised method. But I'm looking for a more intelligent one or to know the threshold. $\endgroup$ – Revolucion for Monica Aug 11 '18 at 11:22
  • $\begingroup$ The theshold depends on your 'appetite for risk' :) Try determining it experimentally. 'Unfortunately' meaning of language is very rich so when digitized to one dimension appears as continuous. But when you make tests with humans they also differ in their decisions - we do have also some weighs and thresholds in our heads... $\endgroup$ – MkL Aug 11 '18 at 11:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.