Predicting a categorical variable with a bag-of-words matrix that has more columns than rows The dataframe data contain 3 columns: id, a comment and a category.
I want to predict the category from the words present in the comment.
library(text2vec)

it_train <- itoken(data$comment, 
              preprocessor = tolower,
             tokenizer = word_tokenizer, 
             ids = data$id, 
             progressbar = T)

vocab <- create_vocabulary(it_train)
termdoc <- create_dtm(it_train, vocab_vectorizer(vocab))

termdoc <- as.matrix(termdoc)

model1 <- glm(data$category ~ termdoc, family = binomial(link = "logit"))

log.predictions <- predict(model1, data$category, type="link")

Followed by the error:
"in predict.lm(object, newdata, se.fit, scale = 1, type = if (type ==  :
  prediction from a rank-deficient fit may be misleading"

log.predictions contain the same absolute values in all predictions (24 and -24).
The document-term-matrix contain columns corresponding to the number of unique words in all comments and rows corresponding to the number of comments. I understand this triggers an error since there are more predictors than cases to predict, but how can i avoid this problem?
Furthermore, the estimates from model1 are mostly NA's. To be exact, number of estimates is the same as number of rows in termdoc whereas I want it to be equal to the amount of variables/columns.
Can i trust my initial results or should i transform the termdoc-matrix in a way that is more suitable for modelling? In that case, how?
 A: It indeed seems like what Otto is mentioned might be the issue, but I will elaborate a bit more on it. 
So what you might have and is likely to have is a dimensionality issue, meaning that you have more predictors than data points. This will lead to your model being unable to fit as it can always make a perfect fit. This is also called the 'curse of dimensionality' and have been an issue for a long time in NLP and is still problematic is fields such as genetics and neuroscience. One way to solve this is as Otto mentions using regularizations. E.g. ridge regression or similar. I personally would suggest this video by Josh Starmer.
Alternatively, you might want to shift to a classifier which better handles the curse of dimensionality or you could reduce the number of predictors using by removing stopwords, frequent words and rare words. The reason why this is reasonable is:


*

*stopwords are hard to interpret e.g. what does it mean that "it" or "I" is a good predictor?

*frequent words or words which are present in almost all documents are unlikely to be differentiating features. Stopwords are typically very frequent words.

*in-frequent words i.e. words which only appear in 1-3 documents typically have very unstable estimated. Also, these words typically become perfect predictors for the category they are in which can lead to a model that generalizes poorly.


Lastly, an option is to upsample data, i.e. generating more data from already existing data points. For example given the documents:
doc   w1_count,   w2_count,   w3_count, category
1     1,            2,          8,         1
2     2,            1,          6,         1
3     11,           2,          1,         2
4     13,           2,          2,         2

You might imagine that w1 and w3 are good predictors and you might imagine that you can 'create' a new document of category 1 with:
w1_count = rnorm(1, 1, 1)
w2_count = rnorm(1, 1, 1)
w3_count = rnorm(1, 3, 1)
There exist methods for systematically doing this e.g. SMOTE.
