Is it wrong to run a Random Forest on high-dimensional, sparse, and unbalanced data? I am learning about random forests, and I have been testing using R. I have doubts about whether I am doing something wrong given that my data are: sparse, high-dimensional, and unbalanced.
Trying to understand the algorithm, I have been experimenting with a dataset of TF-IDF. The data frame is of dimension (1143, 2016). Each row represents a distinct document, and there are more than 2000 variables with the TF-IDF of the stems present in the corpus (the tf-idf tend to be sparse). I have a label L that says whether the document was written by a specific person. In summary, the data frame df looks like this:




L
tfidf_word1
....
tfidf_word2015




TRUE
$x_{1,1}$
....
$x_{1,2015}$


FALSE
$x_{2,1}$
....
$x_{2,2015}$


...
...
....
...


FALSE
$x_{1143,1}$
....
$x_{1143,2015}$




These data are highly unbalanced, with the share of L that are TRUE around 5%.
Therefore, I have constructed a weight variable as follows
weight <- 1
weight <- ifelse(df$L , $\frac{1}{0.05}$, $\frac{1}{0.95}$)

I have ran the following random following random forest:
library(ranger)
rf <- ranger(L~., data = df, case.weights = weight, importance = "impurity")

And these are the results:
Type:                             Classification 
Number of trees:                  500 
Sample size:                      1143 
Number of independent variables:  2016 
Mtry:                             44 
Target node size:                 1 
Variable importance mode:         impurity 
Splitrule:                        gini 
OOB prediction error:             0.00 % 

I am very worried about the OOB prediction error of 0%. Is there a formal reason to think that for data so sparse, unbalanced, or of high dimension there may be some sort of overfitting going on?
 A: Especially when the data is not abundant, it makes good sense to have a robust validation schema. As the OP correctly recognises OOB performance in RF might not be a good indicator for out-of-sample performance if they are low in number and it is quite unbalanced. The OP has correctly used case.weights so we weight the sampling of the training observations but that does not guarantee that the OOB sample won't contain in many cases only samples from the majority class (granted it makes unlikely). For that reason, I would strongly suggest employing a more formal resampling approach; either through stratified bootstrap or repeated k-fold cross-validation. That way we can scrutinise each fold separately.
The above being said it is important to note that unexpectedly high performance is often associated with data leakage, Leakage in Data Mining: Formulation, Detection, and Avoidance (2011) by Kaufman et al. is probably the best place to start. It would be thus very prudent to look at the most important features in terms of explanatory performance and ensure that they do not contain information that will be unavailable at new samples (e.g. indirectly encoding the author's archives indirectly, or some annotator writing something in the lines of "well this is fake!"). To that respect exactly because it is a  smaller sample it makes even more sense to invest in the interpretability of the features as the classifier might be learning information that is effective only in this smaller sample (e.g. encoding a spelling mistake the author does that while unique within this small dataset, is not be generally unique) but will not generalise further. To that extent, notice that TFIDF can easily lead to some very specific "corpus-features" that might have little meaning out of the corpus in question.
Combining somewhat the points above, I would also suggest checking that the positive data are truly unique and not just (approximate) duplicates to each other. That might make both the OOB performance over-optimistic as well make us learn patterns that might be feature learning unrealistic. It can happen with smaller imbalanced datasets where effectively we come across a "well-defined cluster" of rare positive instances but is rather uninformative in a more generalisable sense.
