Random Forest in R - How to perform feature extraction and reach the best Accuracy result?

Question

I'm working on a university project where I need to build a Random Forest model in R to predict if patients have depressive tendencies according to their EEG-data. I already preprocessed the data and built a general model for the forest. Currently, I'm fine-tuning it to receive the best possible prediction Accuracy. If I understand that correctly, I need to do some Feature Extraction (At this moment, I have 1584 Features) and tuning of the hyperparameters. But I'm not sure how to perform the Feature Extraction in R? Right now, I'm doing this:

library(dplyr)
library(caret)
library(randomForest)
library(eegkit)
library(rlist)
library(edfReader)
library(eegUtils)
library(e1071)
library(ggplot2)

yourdata_neu <- data.frame(df_test)
rownames(yourdata_neu) <- NULL

set.seed(42)
###############################Random Forest
for (t in 1:10) {
seed <- sample.int(10)
set.seed(seed)  
  seeds <- vector(mode = "list", length = 50)
  for(i in 1:50){
  seeds[[i]] <- sample.int(1000, 12)}

  ## For the last model:
  seeds[[50]] <- sample.int(1000, 1)

yourdata_neu$Depressiv <- as.factor(yourdata_neu$Depressiv)

inTraining <- createDataPartition(yourdata_neu$Depressiv[1:nrow(yourdata_neu)], p = 0.70, list = FALSE)
training <- yourdata_neu[inTraining,] 
testing <- yourdata_neu[-inTraining,]

train_control <- trainControl(method="cv", number=10, verboseIter = TRUE, seeds = seeds) 

model <- train(training[,1:ncol(yourdata_neu)-1],as.factor(training[,ncol(yourdata_neu)]), method = "rf", type="classification", metric= "Accuracy", maximize= TRUE, trControl = train_control, importance = TRUE) 

prediction2 <- predict(model, testing[,1:ncol(yourdata_neu)-1])

confusionMatrix(prediction2, as.factor(testing[,ncol(yourdata_neu)]),  positive = "1") 

model1 <- randomForest(training[,1:ncol(yourdata_neu)-1],as.factor(training[,ncol(yourdata_neu)]), 
type="classification", importance = TRUE, proximity = TRUE) 

prediction1 <- predict(model1, testing[,1:ncol(yourdata_neu)-1])
print(confusionMatrix(prediction1, as.factor(testing[,ncol(yourdata_neu)]),  positive = "1"))

Adding_columns <- NULL
varImp2 <- varImp(model, scale = TRUE)
Adding_columns <- t(varImp2$importance)
rownames(Adding_columns) <- paste0(rownames(Adding_columns),".", t)
Importance_Table <- rbind(Importance_Table, Adding_columns)
}

Importance_Table_Mean <- t(apply(Importance_Table, MARGIN = 2, function(x) mean(x, na.rm=TRUE)))
Importance_Table_Filter <- as.data.frame(Importance_Table_Mean)
Importance_Table_Filter <- Importance_Table_Filter[,Importance_Table_Filter< 50]
Importance_Table_Filter <- colnames(Importance_Table_Filter) 
Excluding_Channels <- names(yourdata_neu) %in% Importance_Table_Filter
yourdata_neu <- yourdata_neu[!Excluding_Channels]

1. Run the randomForest 10 times and save each time the importance of every feature in the data frame "Importance_Table"
2. Calculating the mean of the 10 trials for each feature in "Importance_Table_Mean"
3. Use the data frame "Importance_Table_Filter" to save all features with an importance value of under 50 (50 was chosen by me)
4. Get the colnames(feature names) of the data frame "Importance_Table_Filter" and save them in the variable "Excluding_Channels"
5. Dropping all channels with an importance value of under 50 from my data set (yourdata_neu) and keeping the ones with more than 50.

But I get the feeling that this isn't a good approach and very subjective. Does anybody have an idea to improve my model? My idea was to perform the Feature Extraction first and then optimizing the hyperparameters. It that common or advised to do it like this? I'm grateful for every input :)

EDIT: Providing the values inside of the variable "yourdata_neu" (It contains the df_test values): https://drive.google.com/file/d/1k02hyqU51cAy5ka1gs5Ydr5_gOM5vTew/view?usp=sharing

Answer 1

You may be interested in using the recursive feature elimination (RFE) function in the caret package:

1. Eliminate highly correlated variables

# You can use any threshold you want to deem a correlation too high. Here we use .80
nonColinearData = yourdata_neu[, -findCorrelation(cor(yourdata_neu), cutoff = .8)]

2. Use recursive feature elimination

# Set RFE control
ctrl = rfeControl(functions = rfFuncs, # "rfFuncs" are built-in to caret
                  method = "repeatedcv", repeats = 10,
                  saveDetails = TRUE)
# By using rfFuncs, caret will use a random forest to evaluate the usefulness of a feature.

# Set a sequence of feature-space sizes to search over:
sizes = seq(sqrt(ncol(nonColinearData))*.5, ncol(nonColinearData), by = 5)
# note, this will fit hundreds of forests (not trees), so it may take a while.

# Use caret's rfe function to fit RF models to these different feature spaces
rfeResults = rfe(x = select(nonColinearData, -Depressiv), y = nonColinearData$Depressiv,
             sizes = sizes,
             rfeControl = ctrl)

3. Evaluate results:

rfeResults$results

Answer 2

There are a number of feature selection techniques in random Forests. As Dij pointed out, RFE is a typical strategy used in random forests. Try the ones out below and see if that helps as well. These are just a few of the techniques available as R packages - some are easier to install than others, but give them a shot.

• AUCRF: Urrea, V., & Calle, M. (2012). AUCRF: variable selection with random forest and the area under the curve. R package version 1.1.
• EFS : Neumann, U., Genze, N., & Heider, D. (2017). EFS: an ensemble feature selection tool implemented as R-package and web-application. BioData mining, 10(1), 21.
• binomialRF: Zaim, S. R., Kenost, C., Lussier, Y. A., & Zhang, H. H. (2019). binomialRF: Scalable Feature Selection and Screening for Random Forests to Identify Biomarkers and Their Interactions. bioRxiv, 681973.
• VSURF: Genuer, Robin, Jean-Michel Poggi, and Christine Tuleau-Malot. "VSURF: an R package for variable selection using random forests." (2015).
• VarSelRF: Diaz-Uriarte, Ramón. "GeneSrF and varSelRF: a web-based tool and R package for gene selection and classification using random forest." BMC bioinformatics 8, no. 1 (2007): 328.
• Boruta: Kursa, M. B., & Rudnicki, W. R. (2010). Feature selection with the Boruta package. J Stat Softw, 36(11), 1-13.
• R2VIM: Szymczak, S., Holzinger, E., Dasgupta, A., Malley, J. D., Molloy, A. M., Mills, J. L., ... & Bailey-Wilson, J. E. (2016). r2VIM: A new variable selection method for random forests in genome-wide association studies. BioData mining, 9(1), 7
• Vita: Janitza, Silke, Ender Celik, and Anne-Laure Boulesteix. "A computationally fast variable importance test for random forests for high-dimensional data." Advances in Data Analysis and Classification 12, no. 4 (2018): 885-915.

Hope these help - good luck!

Answer 3

https://www.sciencedirect.com/science/article/pii/S0957417419303574 This paper compares RF variable selection methods.