2
$\begingroup$

I'm working on a university project where I need to build a Random Forest model in R to predict if patients have depressive tendencies according to their EEG-data. I already preprocessed the data and built a general model for the forest. Currently, I'm fine-tuning it to receive the best possible prediction Accuracy. If I understand that correctly, I need to do some Feature Extraction (At this moment, I have 1584 Features) and tuning of the hyperparameters. But I'm not sure how to perform the Feature Extraction in R? Right now, I'm doing this:

library(dplyr)
library(caret)
library(randomForest)
library(eegkit)
library(rlist)
library(edfReader)
library(eegUtils)
library(e1071)
library(ggplot2)

yourdata_neu <- data.frame(df_test)
rownames(yourdata_neu) <- NULL

set.seed(42)
###############################Random Forest
for (t in 1:10) {
seed <- sample.int(10)
set.seed(seed)  
  seeds <- vector(mode = "list", length = 50)
  for(i in 1:50){
  seeds[[i]] <- sample.int(1000, 12)}

  ## For the last model:
  seeds[[50]] <- sample.int(1000, 1)

yourdata_neu$Depressiv <- as.factor(yourdata_neu$Depressiv)

inTraining <- createDataPartition(yourdata_neu$Depressiv[1:nrow(yourdata_neu)], p = 0.70, list = FALSE)
training <- yourdata_neu[inTraining,] 
testing <- yourdata_neu[-inTraining,]

train_control <- trainControl(method="cv", number=10, verboseIter = TRUE, seeds = seeds) 

model <- train(training[,1:ncol(yourdata_neu)-1],as.factor(training[,ncol(yourdata_neu)]), method = "rf", type="classification", metric= "Accuracy", maximize= TRUE, trControl = train_control, importance = TRUE) 

prediction2 <- predict(model, testing[,1:ncol(yourdata_neu)-1])

confusionMatrix(prediction2, as.factor(testing[,ncol(yourdata_neu)]),  positive = "1") 

model1 <- randomForest(training[,1:ncol(yourdata_neu)-1],as.factor(training[,ncol(yourdata_neu)]), 
type="classification", importance = TRUE, proximity = TRUE) 

prediction1 <- predict(model1, testing[,1:ncol(yourdata_neu)-1])
print(confusionMatrix(prediction1, as.factor(testing[,ncol(yourdata_neu)]),  positive = "1"))

Adding_columns <- NULL
varImp2 <- varImp(model, scale = TRUE)
Adding_columns <- t(varImp2$importance)
rownames(Adding_columns) <- paste0(rownames(Adding_columns),".", t)
Importance_Table <- rbind(Importance_Table, Adding_columns)
}

Importance_Table_Mean <- t(apply(Importance_Table, MARGIN = 2, function(x) mean(x, na.rm=TRUE)))
Importance_Table_Filter <- as.data.frame(Importance_Table_Mean)
Importance_Table_Filter <- Importance_Table_Filter[,Importance_Table_Filter< 50]
Importance_Table_Filter <- colnames(Importance_Table_Filter) 
Excluding_Channels <- names(yourdata_neu) %in% Importance_Table_Filter
yourdata_neu <- yourdata_neu[!Excluding_Channels]

  1. Run the randomForest 10 times and save each time the importance of every feature in the data frame "Importance_Table"
  2. Calculating the mean of the 10 trials for each feature in "Importance_Table_Mean"
  3. Use the data frame "Importance_Table_Filter" to save all features with an importance value of under 50 (50 was chosen by me)
  4. Get the colnames(feature names) of the data frame "Importance_Table_Filter" and save them in the variable "Excluding_Channels"
  5. Dropping all channels with an importance value of under 50 from my data set (yourdata_neu) and keeping the ones with more than 50.

But I get the feeling that this isn't a good approach and very subjective. Does anybody have an idea to improve my model? My idea was to perform the Feature Extraction first and then optimizing the hyperparameters. It that common or advised to do it like this? I'm grateful for every input :)

EDIT: Providing the values inside of the variable "yourdata_neu" (It contains the df_test values): https://drive.google.com/file/d/1k02hyqU51cAy5ka1gs5Ydr5_gOM5vTew/view?usp=sharing

$\endgroup$
5
  • $\begingroup$ We do not have access to your data frame df_test. Can you provide it? $\endgroup$ Commented Jan 4, 2020 at 16:26
  • $\begingroup$ @kjetilbhalvorsen: Hi! Sorry, I totally forgot to provide any data sets. I edited my post :) (drive.google.com/file/d/1k02hyqU51cAy5ka1gs5Ydr5_gOM5vTew/…) $\endgroup$
    – Ruffybeo
    Commented Jan 4, 2020 at 16:59
  • 2
    $\begingroup$ boruta uses statistical tests of feature importance scores to eliminate features which are not more useful than random $\endgroup$
    – Sycorax
    Commented Jan 4, 2020 at 17:54
  • $\begingroup$ @SycoraxsaysReinstateMonica Thank you for your suggestion! I tried the boruta algorithm but it kinda eliminated all my features.....But I saw on video clips that it helped to get way better results! $\endgroup$
    – Ruffybeo
    Commented Jan 6, 2020 at 13:01
  • $\begingroup$ There is a beautiful tool that you are not yet using: Boruta. It can be combined with dplyr and data.table to make and evaluate very huge counts of columns. This means you can use data.table to generate all of the interactions, transformations, and lags, then use Boruta to determine which ones are relevant. $\endgroup$ Commented Apr 15, 2020 at 14:09

3 Answers 3

1
$\begingroup$

You may be interested in using the recursive feature elimination (RFE) function in the caret package:

1. Eliminate highly correlated variables

# You can use any threshold you want to deem a correlation too high. Here we use .80
nonColinearData = yourdata_neu[, -findCorrelation(cor(yourdata_neu), cutoff = .8)]

2. Use recursive feature elimination

# Set RFE control
ctrl = rfeControl(functions = rfFuncs, # "rfFuncs" are built-in to caret
                  method = "repeatedcv", repeats = 10,
                  saveDetails = TRUE)
# By using rfFuncs, caret will use a random forest to evaluate the usefulness of a feature.

# Set a sequence of feature-space sizes to search over:
sizes = seq(sqrt(ncol(nonColinearData))*.5, ncol(nonColinearData), by = 5)
# note, this will fit hundreds of forests (not trees), so it may take a while.

# Use caret's rfe function to fit RF models to these different feature spaces
rfeResults = rfe(x = select(nonColinearData, -Depressiv), y = nonColinearData$Depressiv,
             sizes = sizes,
             rfeControl = ctrl)

3. Evaluate results:

rfeResults$results
$\endgroup$
1
$\begingroup$

There are a number of feature selection techniques in random Forests. As Dij pointed out, RFE is a typical strategy used in random forests. Try the ones out below and see if that helps as well. These are just a few of the techniques available as R packages - some are easier to install than others, but give them a shot.

  • AUCRF: Urrea, V., & Calle, M. (2012). AUCRF: variable selection with random forest and the area under the curve. R package version 1.1.
  • EFS : Neumann, U., Genze, N., & Heider, D. (2017). EFS: an ensemble feature selection tool implemented as R-package and web-application. BioData mining, 10(1), 21.
  • binomialRF: Zaim, S. R., Kenost, C., Lussier, Y. A., & Zhang, H. H. (2019). binomialRF: Scalable Feature Selection and Screening for Random Forests to Identify Biomarkers and Their Interactions. bioRxiv, 681973.
  • VSURF: Genuer, Robin, Jean-Michel Poggi, and Christine Tuleau-Malot. "VSURF: an R package for variable selection using random forests." (2015).
  • VarSelRF: Diaz-Uriarte, Ramón. "GeneSrF and varSelRF: a web-based tool and R package for gene selection and classification using random forest." BMC bioinformatics 8, no. 1 (2007): 328.
  • Boruta: Kursa, M. B., & Rudnicki, W. R. (2010). Feature selection with the Boruta package. J Stat Softw, 36(11), 1-13.
  • R2VIM: Szymczak, S., Holzinger, E., Dasgupta, A., Malley, J. D., Molloy, A. M., Mills, J. L., ... & Bailey-Wilson, J. E. (2016). r2VIM: A new variable selection method for random forests in genome-wide association studies. BioData mining, 9(1), 7
  • Vita: Janitza, Silke, Ender Celik, and Anne-Laure Boulesteix. "A computationally fast variable importance test for random forests for high-dimensional data." Advances in Data Analysis and Classification 12, no. 4 (2018): 885-915.

Hope these help - good luck!

$\endgroup$
0
$\begingroup$

https://www.sciencedirect.com/science/article/pii/S0957417419303574 This paper compares RF variable selection methods.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.