I'm trying to determine how to answer question from text with supervised learning. This used to work quite well when every questions had answers. Here is the head of dataset we used with the sentence embeddings and a "target" on the sentence which contains the answer.
answers text sentences target sent_emb quest_emb -------------------------------------- cosine_sim euclidean_dis
0 yes in the late 1990s ['Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/... 1 [array([0.03037658, 0.04433101, 0.08135635, ..... [[0.01491953 0.02197376 0.02136409 ... 0.01360... [0.1401391625404358, 0.11776834726333618, 0.09... [2.8352642, 2.4563262, 1.5417788, 2.9730926]
1 yes singing and dancing ['Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/... 1 [array([0.03037658, 0.04433101, 0.08135635, ..... [[0.04444952 0.02800576 0.03035772 ... 0.02242... [0.12254136800765991, 0.08665323257446289, 0.0... [2.396976, 1.7860672, 1.1152366, 2.3845947]
2 yes 2003 ['Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/... 3 [array([0.03037658, 0.04433101, 0.08135635, ..... [[0.03949683 0.04509903 0.01808935 ... 0.04610... [0.09432470798492432, 0.06841456890106201, 0.0... [1.8688717, 1.4023948, 1.0954475, 1.7620108]
But with this new dataset some questions may have no answers :
answers text sentences target sent_emb quest_emb -------------------------------------- cosine_sim euclidean_dis
2075 no action-adventure ['The Legend of Zelda: Twilight Princess (Japa... -1 [array([0.03695295, 0.04520793, 0.06620894, ..... [[0.03504493 0.01131248 0.03034939 ... 0.06862... [0.097023606300354, 0.07994633913040161, 0.063... [1.6347798344253954, 1.2541696560187723, 0.784...
2076 no GameCube and Wii ['The Legend of Zelda: Twilight Princess (Japa... -1 [array([0.03695295, 0.04520793, 0.06620894, ..... [[0.02894322 0.02931837 0.02200499 ... 0.02773... [0.14154821634292603, 0.08903771638870239, 0.0... [2.304629177519047, 1.40055260190374, 1.185086...
2077 no November 2006 ['The Legend of Zelda: Twilight Princess (Japa... -1 [array([0.03695295, 0.04520793, 0.06620894, ..... [[0.03098885 0.01231992 0.01971022 ... 0.01789... [0.14250332117080688, 0.1213921308517456, 0.10... [2.3253533372257893, 1.8554215108623593, 1.195...
There are even a fourth of them that don't.
>>> print(len(stats.loc[stats['cos_exists_answer'] == True]))
11012
>>> print(len(stats.loc[stats['cos_exists_answer'] == False]))
3468
And so far, the accuracy rate have really dropped I wonder how I can improve this learning process.
All of the embedding data showed above, which are cosine and euclidean distance of each paragraph to the question, where transformed to features, standardized and minmax before being added to a multinational logistic regression model. You can see the full process on my GitHub.
>>>mul_lr = linear_model.LogisticRegression(multi_class='multinomial', solver='newton-cg')
>>>mul_lr.fit(train_x, train_y)
>>>print("Multinomial Logistic regression Train Accuracy : ", metrics.accuracy_score(train_y, mul_lr.predict(train_x)))
Multinomial Logistic regression Train Accuracy : 0.3087949035568926
>>>print("Multinomial Logistic regression Test Accuracy : ", metrics.accuracy_score(test_y, mul_lr.predict(test_x)))
Multinomial Logistic regression Test Accuracy : 0.2898089171974522
which is far far less than the result with questions for every answers as reveals this girl in her attempt.
Therefore do you know how to improve this supervised learning algorithm when we have new cases when there isn't answers?
I thought it was only adding a class: when not to answer. But maybe I am wrong and I should do a first simple logistic regression testing if a question has answers and a second simple logistic one when there is answers ? But I don't know how to cascade them. Or maybe I can't learn anything from distance as may reveal the cumulative distribution of distance of cosine and euclidean distance?
I have a sub-question : I used the multinomial logisitic regression because this is what the creator of the dataset did on their first attempt, which they describe in their paper, but I didn't understood why they didn't took a simple logistic regression to determine on each paragraph which one is answerable and which isn't.
If that can help to have a better idea, for the given examples, for line 0 the first sentences were :
'[\'Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress.\', "Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny\'s Child.", "Managed by her father, Mathew Knowles, the group became one of the world\'s best-selling girl groups of all time.", \'Their hiatus saw the release of Beyoncé\\\'s debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles "Crazy in Love" and "Baby Boy".\']'
The first question was : 'When did Beyonce start becoming popular?'
And for line 2075, the sentences were :
"['The Legend of Zelda: Twilight Princess (Japanese: ゼルダの伝説 トワイライトプリンセス, Hepburn: Zeruda no Densetsu: Towairaito Purinsesu?)', 'is an action-adventure game developed and published by Nintendo for the GameCube and Wii home video game consoles.', 'It is the thirteenth installment in the The Legend of Zelda series.', 'Originally planned for release on the GameCube in November 2005, Twilight Princess was delayed by Nintendo to allow its developers to refine the game, add more content, and port it to the Wii.', 'The Wii version was released alongside the console in North America in November 2006, and in Japan, Europe, and Australia the following month.', 'The GameCube version was released worldwide in December 2006.', '[b]']"
And the question : 'What category of game is Legend of Zelda: Australia Twilight?'
