I have moderate knowledge with machine learning. In a project I am involved, I need to detect recurring or common features in a binary classification task. The data-set is time dependent with about 9000 observations and more than 4000 features. In addition, the labels are very unbalanced (0:1 -> 20:1). I am interested in the label 1. Here is what I have tried:
- Construct training set (date <= 201705) & test set (date > 201705)
- Fill NAs in the training set when there are enough number of non-NAs. For ex. I remove the features when more than 70% of the corresponding rows for that feature is NA.
- Calculate correlation matrix for training set and remove one feature from feature pair when the correlation is higher than 0.70.
- As a result, I reduce the feature set significantly (let's say from 4500 to 700)
- I train my data-set with random forest (and also with gbm and glm(elastic net)) with many different parameters.
My first question is: Does it sound right to apply RF to such time series data? Am I missing something here? What other approaches can I use?
Secondly: In the best case I observe an AUC of 0.70 with a precision of 0.50 for the class 1. It is a financial data including market data such as equity values in a portfolio, foreign exchanges and interest rates. So considering the 0.70 AUC, could the results be considered acceptable?
Thanks in advance!