I have a graph of the Cosine distance between the question and the sentence most similar to it when there is an answer and when there is none.

I want to establish a threshold on the abscissa axis from which I can say that an answer exists.

graph of the cosine similarity between the question and the passage most similar to it when there is an answer and when there is none, the bell of no answer is almost perfectly below the bell blue, even if it exceeds a little when the similarity score increases

Here, the no response curve is almost perfectly below the response existence curve. I'm not sure that taking absicisse from the global maximum is a good idea.

How to establish this threshold according to the similarities?

Here is what the data look like :

    answers     sent_emb                                      quest_emb                                           cosine_sim
0   yes     [[0.030376578, 0.044331014, 0.081356354, 0.062...   [[0.01491953, 0.021973763, 0.021364095, 0.0393...   [0.1401391625404358, 0.11776834726333618, 0.09...
1   yes     [[0.030376578, 0.044331014, 0.081356354, 0.062...   [[0.04444952, 0.028005758, 0.030357722, 0.0375...   [0.12254136800765991, 0.08665323257446289, 0.0...
2   yes     [[0.030376578, 0.044331014, 0.081356354, 0.062...   [[0.03949683, 0.04509903, 0.018089347, 0.07667...   [0.09432470798492432, 0.06841456890106201, 0.0...

And here is the code that created the graph :

stats["cos_exists_answer"] = train.apply(get_answers, axis = 1) 
stats["cos_result"] = train["cosine_sim"].apply(lambda x: np.min(x))
stats["euc_exists_answer"] = train.apply(get_answers, axis = 1) 
stats["euc_result"] = train["euclidean_dis"].apply(lambda x: np.min(x))

import matplotlib.pyplot as plt
import seaborn as sns

# Cosinus similarity
# Create a color if the group is "B"
my_color=np.where(stats['cos_exists_answer'] == False, 'orange', 'skyblue')

# Sort the dataframe by target
target_0 = stats.loc[stats['cos_exists_answer'] == True]
target_1 = stats.loc[stats['cos_exists_answer'] == False]
sns.distplot(target_0['cos_result'].dropna(), hist=False, rug=True, label= 'Exists')
sns.distplot(target_1['cos_result'].dropna(), hist=False, rug=True, label= 'Don\'t exists')

# euclidian similarity
my_color=np.where(stats['cos_exists_answer'] == False, 'orange', 'skyblue')

target_0 = stats.loc[stats['cos_exists_answer'] == True]
target_1 = stats.loc[stats['cos_exists_answer'] == False]
sns.distplot(target_0['euc_result'].dropna(), hist=False, rug=True, label= 'Exists')
sns.distplot(target_1['euc_result'].dropna(), hist=False, rug=True, label= 'Don\'t exists')

PS: I also have the equivalent with the Euclidian distance. Can this help?

enter image description here

Update : over-smoothness and root matching

Are the density plots are over-smoothed?

On the advice of user20160 I tried to see with an histogram if the smoothness was culprit for cosinus similarity and euclidean distance and I was wondering if it wasn't possible to make a span where it was more probable that we had an existing answer.

Here is the cosinus similarity on an histogram. Let me know if I should increase the binwidth.

enter image description here

And here is the euclidian distance:

enter image description here

Indeed, as you can see there is a span where there is more occurrences of existing answers in both methods. I know that the occurrences of question where no answers exists augment as well.

Make root of the question and of the context match

Another feature for this problem is the “Dependency Parse Tree”. The idea is to match the root of the question which is “appear” in this case to all the roots/sub-roots of the sentence. I am only at its creation. I do not know if it can help determine a threshold or even say what are the questions where there are no answers when there is no root.

  • 1
    $\begingroup$ Are your plots a density estimate of distance, given that an answer exists (or doesn't exist)? If so, given the large overlap between the densities, I don't think you'll be able to make a very good prediction about the existence of an answer based on the distance. $\endgroup$ – user20160 Aug 18 '18 at 18:24
  • $\begingroup$ @user20160 Okay, are there alternative unsupervised ways? What is the functionality that says when there is no answers? Should I share more about the data? $\endgroup$ – Revolucion for Monica Aug 19 '18 at 0:01
  • $\begingroup$ If the density estimates are accurate, it would suggest that distance to the nearest sentence doesn't carry much information about the existence of an answer, in which case a good prediction isn't possible by any method. Of course, that might not hold if the density plots are oversmoothed, for example. You could try to estimate the Bayes error rate, or take an empirical approach and try fitting some classifiers. But, you may want to consider looking for other, more informative features. $\endgroup$ – user20160 Aug 19 '18 at 2:03
  • $\begingroup$ @user20160 Okay, thank you for these ideas. I tried to figure out if they are oversmoothed with histograms. I have not yet estimated the Bayes error rate but I have thought about another informative feature. Can you let me know what you think about those updates? $\endgroup$ – Revolucion for Monica Aug 20 '18 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.