0
$\begingroup$

I'm trying to determine how to answer question from text with supervised learning. This used to work quite well when every questions had answers. Here is the head of dataset we used with the sentence embeddings and a "target" on the sentence which contains the answer.

    answers           text                             sentences                     target     sent_emb                                        quest_emb --------------------------------------     cosine_sim                                                      euclidean_dis
0   yes     in the late 1990s       ['Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/...   1   [array([0.03037658, 0.04433101, 0.08135635, .....   [[0.01491953 0.02197376 0.02136409 ... 0.01360...   [0.1401391625404358, 0.11776834726333618, 0.09...   [2.8352642, 2.4563262, 1.5417788, 2.9730926]
1   yes     singing and dancing     ['Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/...   1   [array([0.03037658, 0.04433101, 0.08135635, .....   [[0.04444952 0.02800576 0.03035772 ... 0.02242...   [0.12254136800765991, 0.08665323257446289, 0.0...   [2.396976, 1.7860672, 1.1152366, 2.3845947]
2   yes     2003                    ['Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/...   3   [array([0.03037658, 0.04433101, 0.08135635, .....   [[0.03949683 0.04509903 0.01808935 ... 0.04610...   [0.09432470798492432, 0.06841456890106201, 0.0...   [1.8688717, 1.4023948, 1.0954475, 1.7620108]

But with this new dataset some questions may have no answers :

    answers           text                             sentences                     target     sent_emb                                        quest_emb --------------------------------------     cosine_sim                                                      euclidean_dis
2075    no  action-adventure    ['The Legend of Zelda: Twilight Princess (Japa...   -1  [array([0.03695295, 0.04520793, 0.06620894, .....   [[0.03504493 0.01131248 0.03034939 ... 0.06862...   [0.097023606300354, 0.07994633913040161, 0.063...   [1.6347798344253954, 1.2541696560187723, 0.784...
2076    no  GameCube and Wii    ['The Legend of Zelda: Twilight Princess (Japa...   -1  [array([0.03695295, 0.04520793, 0.06620894, .....   [[0.02894322 0.02931837 0.02200499 ... 0.02773...   [0.14154821634292603, 0.08903771638870239, 0.0...   [2.304629177519047, 1.40055260190374, 1.185086...
2077    no  November 2006       ['The Legend of Zelda: Twilight Princess (Japa...   -1  [array([0.03695295, 0.04520793, 0.06620894, .....   [[0.03098885 0.01231992 0.01971022 ... 0.01789...   [0.14250332117080688, 0.1213921308517456, 0.10...   [2.3253533372257893, 1.8554215108623593, 1.195...

There are even a fourth of them that don't.

>>> print(len(stats.loc[stats['cos_exists_answer'] == True]))
11012
>>> print(len(stats.loc[stats['cos_exists_answer'] == False]))
3468

And so far, the accuracy rate have really dropped I wonder how I can improve this learning process.

All of the embedding data showed above, which are cosine and euclidean distance of each paragraph to the question, where transformed to features, standardized and minmax before being added to a multinational logistic regression model. You can see the full process on my GitHub.

>>>mul_lr = linear_model.LogisticRegression(multi_class='multinomial', solver='newton-cg')
>>>mul_lr.fit(train_x, train_y)
>>>print("Multinomial Logistic regression Train Accuracy : ", metrics.accuracy_score(train_y, mul_lr.predict(train_x)))
Multinomial Logistic regression Train Accuracy :  0.3087949035568926
>>>print("Multinomial Logistic regression Test Accuracy : ", metrics.accuracy_score(test_y, mul_lr.predict(test_x)))
Multinomial Logistic regression Test Accuracy :  0.2898089171974522

which is far far less than the result with questions for every answers as reveals this girl in her attempt.

Therefore do you know how to improve this supervised learning algorithm when we have new cases when there isn't answers?

I thought it was only adding a class: when not to answer. But maybe I am wrong and I should do a first simple logistic regression testing if a question has answers and a second simple logistic one when there is answers ? But I don't know how to cascade them. Or maybe I can't learn anything from distance as may reveal the cumulative distribution of distance of cosine and euclidean distance?

I have a sub-question : I used the multinomial logisitic regression because this is what the creator of the dataset did on their first attempt, which they describe in their paper, but I didn't understood why they didn't took a simple logistic regression to determine on each paragraph which one is answerable and which isn't.


If that can help to have a better idea, for the given examples, for line 0 the first sentences were :

'[\'Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress.\', "Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny\'s Child.", "Managed by her father, Mathew Knowles, the group became one of the world\'s best-selling girl groups of all time.", \'Their hiatus saw the release of Beyoncé\\\'s debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles "Crazy in Love" and "Baby Boy".\']'

The first question was : 'When did Beyonce start becoming popular?'

And for line 2075, the sentences were :

"['The Legend of Zelda: Twilight Princess (Japanese: ゼルダの伝説 トワイライトプリンセス, Hepburn: Zeruda no Densetsu: Towairaito Purinsesu?)', 'is an action-adventure game developed and published by Nintendo for the GameCube and Wii home video game consoles.', 'It is the thirteenth installment in the The Legend of Zelda series.', 'Originally planned for release on the GameCube in November 2005, Twilight Princess was delayed by Nintendo to allow its developers to refine the game, add more content, and port it to the Wii.', 'The Wii version was released alongside the console in North America in November 2006, and in Japan, Europe, and Australia the following month.', 'The GameCube version was released worldwide in December 2006.', '[b]']"

And the question : 'What category of game is Legend of Zelda: Australia Twilight?'

$\endgroup$
0
$\begingroup$

IMHO it's not a good way to add "no answer" value to target in parallel to values for specific answers - question-answer pair have some relation hidden behind words that are used in them and this relation is modeled to predict one of the answers for a given question.

A question without answer doesn't have this relation with any of the answers and therefore this scenario needs to baked in the process of classification in the way that will discover this lack of relation.

Simple approach to explain: instead of just predicting the best answer, predict probability of the best answer - 'predict_proba' - and if probability is less than some threshold - return 'no answer'. Similar thinking may apply to cosine distance - you can look for the 'nearest neighbour' but it must be close enough to be accepted as a matching answer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.