I have moderate knowledge with machine learning. In a project I am involved, I need to detect recurring or common features in a binary classification task. The data-set is time dependent with about 9000 observations and more than 4000 features. In addition, the labels are very unbalanced (0:1 -> 20:1). I am interested in the label 1. Here is what I have tried:

  • Construct training set (date <= 201705) & test set (date > 201705)
  • Fill NAs in the training set when there are enough number of non-NAs. For ex. I remove the features when more than 70% of the corresponding rows for that feature is NA.
  • Calculate correlation matrix for training set and remove one feature from feature pair when the correlation is higher than 0.70.
  • As a result, I reduce the feature set significantly (let's say from 4500 to 700)
  • I train my data-set with random forest (and also with gbm and glm(elastic net)) with many different parameters.

My first question is: Does it sound right to apply RF to such time series data? Am I missing something here? What other approaches can I use?

Secondly: In the best case I observe an AUC of 0.70 with a precision of 0.50 for the class 1. It is a financial data including market data such as equity values in a portfolio, foreign exchanges and interest rates. So considering the 0.70 AUC, could the results be considered acceptable?

Thanks in advance!

  • $\begingroup$ Q1: Do you have a clean validation concept (like cross-validation etc.)? Q2: None of the methods you have listed are taking into account the time dependent structure in the data. Is this by purpose? $\endgroup$ – Michael M Nov 14 '17 at 10:25
  • $\begingroup$ I had used traditional cv where the results were too good (As I have recently learnt it is not correct to apply. See: stackoverflow.com/questions/47274555/…). Now I am working on implementing time series cross validation. As for Q2, I have only one column related to time. In detail, in my data, I measure some values compared to a base date. That is, I created a new feature indicating how close the date to its corresponding base date. Should I include day, month, year separately? $\endgroup$ – mlee_jordan Nov 14 '17 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.