Lets say I have a set of unstructured text of student actions along with their grades. Is there a way if I want to know what words can lead to high grade or low grade?

For example, Data set

I was studying day by day, 80
I did my homework, 85
I missed alot of classes, 60
I stayed up late, 67

I expect that words with (study,homework) lead to high grade, while words with (missed,late) lead to low score

How can I achieve this?

I started with

  1. Pre processing text by removing stop words,punctuation, stemming and so on
  2. extracted ngrams features and used tf-idf as weighting function
  3. I ended up with a big data matrix for each word

I thought of using correlation to achieve what i want, but my matrix is very sparse and has alot of zeros

  • $\begingroup$ Please elaborate on what you have tried thus far $\endgroup$
    – Sid
    Feb 22 '18 at 6:53
  • $\begingroup$ @Sid I edited my question $\endgroup$
    – sara
    Feb 22 '18 at 7:03

The problem is very similar to the problem of CNN use for sentiment classification.

The details can be found in http://www.aclweb.org/anthology/D14-1181.

An example of a github implementation is :https://github.com/deepanwayx/CNN-For-Sentence-Classification-In-Keras

The 2 groups are positive/negative movie reviews; the files are in https://github.com/deepanwayx/CNN-For-Sentence-Classification-In-Keras/blob/master/data/rt-polarity.neg

You need to do some parsing to split the data into positive and negative groups. This ought to be pretty straightforward for the details you are describing.

An interesting point to note is that your results are also related to the topic of sentiment analysis. I am guessing a negative sentiment will correlate strongly with a poor score. You could probably get away directly with an out-of-the-box sentiment classifier too. Play around with this demo: http://nlp.stanford.edu:8080/sentiment/rntnDemo.html

to see how the results look. You can alternatively resort to training your model too :)

  • $\begingroup$ Thanks @Sid, what if the words i have doesn't hold any semantic its quite neutral , and plenty of words in my data is both repeated in the positive and the low, what is the best approach to take? $\endgroup$
    – sara
    Feb 28 '18 at 5:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.