Consider this

outcome <- c(1,0,0,1,1)
string <- c('I love pasta','hello world', '1+1 = 2','pasta madness', 'pizza madness')

df = df=data.frame(outcome,string)

> df
  outcome        string
1       1  I love pasta
2       0   hello world
3       0       1+1 = 2
4       1 pasta madness
5       1 pizza madness

Here I would like to use random forests to understand which words in the sentences contained in the string variable are strong predictors of the outcome variable.

Someone suggested to use the randomForest package

outcome <- c(1,0,0,1,1)
    string <- c('I love pasta','hello world', '1+1 = 2','pasta madness', 'pizza madness')
df <- data.frame(outcome=factor(outcome,levels=c(0,1)),string, stringsAsFactors=FALSE)

inp <- df %>% mutate(string=strsplit(string,split=" ")) %>% unnest(string)
mm <- model.matrix(outcome~string,inp)
rf <- randomForest(mm, inp$outcome, importance=TRUE)
imp <- importance(rf)

Problem is: my original dataset is much larger. Think of the string column as a short english sentence of 5/6 words. The full dataset has 800k observations. Here, randomForest fails because of obvious memory issues.

What would be an alternative algorithm for my classification purpose?

Many thanks!

  • 2
    $\begingroup$ As you've discovered, randomForest is notoriously poor for sparse data. Some other options include linear SVMs (inner products of binary indicators are cheap) or more advanced methods like neural networks for NLP tasks. $\endgroup$ – Sycorax says Reinstate Monica Oct 24 '16 at 19:41
  • $\begingroup$ thanks @Sycorax. Someone mentioned boosting. What do you think? $\endgroup$ – ℕʘʘḆḽḘ Oct 24 '16 at 19:48
  • 1
    $\begingroup$ I'm not an expert in boosting, but I don't see how that would help. $\endgroup$ – Sycorax says Reinstate Monica Oct 24 '16 at 20:17
  • 1
    $\begingroup$ Sounds like you're using your bag of words to try to classify tweet sentiment? $\endgroup$ – Wayne Oct 24 '16 at 20:53
  • $\begingroup$ i am not using twitter data at all, although its similar. think of the outcome as one if the sentence is popular. $\endgroup$ – ℕʘʘḆḽḘ Oct 24 '16 at 20:55

Try to convert your sparse data into an sparse matrix via one-hot encoding and them use random forest within Caret package...I'm not quire sure if random forest package can work with sparse matrix.,.. use other algorithm such as GBM or xgboost if random forest is giving your poor fit.

  • $\begingroup$ thanks! can you code that you using the example above? $\endgroup$ – ℕʘʘḆḽḘ Oct 25 '16 at 1:10
  • 1
    $\begingroup$ It is very simple. You need to install the library (Matrix): cran.r-project.org/web/packages/Matrix/index.html and make sure all your variables are numeric and factor. Then: sparse_matrix <- sparse.model.matrix(outcome~-1, data = df) ------------- this will dummy code all your sparse categorical variables. $\endgroup$ – RomRom Oct 25 '16 at 1:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.