I'm trying to find a way to prevent Intelligent Agents with Reading Comprehension and Question Answering abilities to answer when they are not confident enough that they can find an answer from the documents they have

To my mind we can say we have on the one hand $P=\{p_1,...,p_i\}\neq\varnothing$ a set of paragraphs, $Q=\{q_{1i},...,q_{ij}\}\neq\varnothing$ a set of questions on these paragraphs and $R=\{r_{111},...,r_{ijk}\}$ the related possible answers. We try to find these answers of these paragraphs. Which are span of the sentences of these paragraphs when there is an answer or to show that there is no answers where it's unanswerable.

That is to say for every question $q_{ij}$ and every related paragraph, we predict an answer $\hat r_{ij}$ such that $$\forall q_{ij}\in Q , \hat r_{ij}= f(p_i,q_{ij}) = \begin{cases} r_{ijk},& \text{if } \exists k,\hat r_{ij}= r_{ijk}\in R\\ \emptyset, & \text{otherwise} \end{cases}$$

I did this problem definition on my own, feel free to criticize it.

I only thought about an unsupervised attempt until now: For each sentence-question pair we try to compute two distance metrics:

(1) Cosine similarity between the $q_{ij}$ and $s_l$. The answer is therefore:

$$\hat{r}_{ij}^{cos}=\arg\max_i \{1 - \frac{s_{l}.q_{ij}}{||s_{l}||.||q_{ij}||}\}$$


(2) The Euclidean distance between the sentences and the question, with the answer as:

$$\hat{r}_{ij}^{euc}=\arg\max_i \{\sqrt{\sum_{v\in \mathcal V} s_l.q_{ij}}\}$$

I haven't chosen one or the other yet. The first one seems more accurate than the second one after a run giving answers may they exists or not in the $R$ set.

Ideas of features

You can find all this work in this GitHub repository.

The distance feature

I was thinking about setting an idea of minimum to exceed for the distance for $\hat r_{ij}$ to accept a predicted answer rather than a null set. Yet, the distance feature doesn't bring much information.

As one can see on the cosine similarity this doesn't bring much information about the existence of an answer.

enter image description here

As far as they clearly overlap, I can't create a threshold from this. The Euclidean distance is even worse.

Update : match roots of question and sentences feature

But has low sucess rate.

The idea is to match the root of the question (which is “appear” in this case) to all the roots/sub-roots of the sentence. Since there are multiple verbs in a sentence, we can get multiple roots. If the root of the question is contained in the roots of the sentence, then there are higher chances that the question is answered by that sentence. Considering that in mind, I have created the following feature: the idea is trying to score those sentences where at least one root of the question $w_j'\in V_{q_{ij}}$ matched one other root $w_i\in V_{p_i}$ of this sentence and return the one with the highest score. The score of a sentence is the number of roots that match in the roots of the question.

That is to say, given a paragraph $p_i$, for every question $q_{ij}$ with a set of roots $V_{q_{ij}}$ and every set $V_{s_{il}}$ of roots of a sentence $s_{il}$ of this paragraph, we receive the following answer $\hat y_{ij}$ such that $$\forall q_{ij}\in Q_i, \hat y_{ij}= \begin{cases} s_{il},& \text{if }\exists s_{il}\in S_i, \arg\max_l\{|V_{s_{il}}\bigcap V_{q_{ij}}|>0\} \\ \emptyset & \text{otherwise} \end{cases}$$ Where $S_i$ is the set of sentences of a paragraph $p_i$.

We get roughly 0.35 of correct answers with the match roots feature.

This Machine Learning problem has examples here with Stanford Question and Answer Dataset (SQuAD) and an attempt in this paper by Facebook.

  • 2
    $\begingroup$ Does "don't answer the question" need to be reframed as "What I still have yet to learn on this subject outweighs what I already know?" Or as something regarding learning as an objective? $\endgroup$
    – Alexis
    Aug 10, 2018 at 14:34
  • $\begingroup$ @Alexis Thank you for your comment. Yes, surely for the second idea! I think I have two things to do: on the one hand not to answer, for which I gave an attempt here, and on the other hand to learn when not to answer, when we had to for instance. $\endgroup$ Aug 10, 2018 at 14:50
  • $\begingroup$ With some thought given to the relationship(s) between learning and answering as objectives. $\endgroup$
    – Alexis
    Aug 10, 2018 at 15:40
  • 1
    $\begingroup$ @Alexis Yes, yeah. Answering is the definitive goal, and learning might help for better answering. $\endgroup$ Aug 10, 2018 at 18:59
  • $\begingroup$ "We try to find these answers of these paragraphs. Which are span of the sentences of these paragraphs when there is an answer or to show that there is no answers where it's unanswerable." I don't comprehend this part. What is a 'span of the sentences'? Also, some interpunction seems confusing. $\endgroup$ Feb 13 at 19:00

1 Answer 1


In a little simpler set-up with FAQ answering system, I'm using simple approach with threshold - when alogithm returns probability / similarity below the threshold, I'm falling back into path "I'm not sure" (sligtly below threshold) or "I don't know" (significantly below threshold / below the second threshold).

  • $\begingroup$ Yes, that is the idea I provided for the unsupervised method. But I'm looking for a more intelligent one or to know the threshold. $\endgroup$ Aug 11, 2018 at 11:22
  • $\begingroup$ The theshold depends on your 'appetite for risk' :) Try determining it experimentally. 'Unfortunately' meaning of language is very rich so when digitized to one dimension appears as continuous. But when you make tests with humans they also differ in their decisions - we do have also some weighs and thresholds in our heads... $\endgroup$
    – MkL
    Aug 11, 2018 at 11:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.