What I'm trying to do is adapting this research paper to another problem. In short: the authors split price variations of S&P500 index into four different classes. Then they train a Random Forest in order to predict open-close return on previously unseen data. Their approach is successful considering that they've eventually managed to systematically overperform the index. However, price fluctuations of that index are pretty balanced across classes as you can see from the figure below:

Price variations classes

For the asset which price variations I'm trying to predict, however, the situation is really different. I'm attaching its distribution below. Histogram asset's price variations

As you can see, my data is extremely unbalanced. So far my random forest has achieved a pretty decent AUROC, my micro-average across classes is 0.77. However, as you could easily imagine, most of the predictions made are about that immense number of variations around zero. What I want to do is finding a way to scientifically split data into a number of classes that is sufficiently sparse to assign pretty much the same number of data points to each class as the authors did in their paper. How would you suggest me to procede?

  • $\begingroup$ What is the statistical principle upon which this strategy is based? And note that concordance probability (AUROC) does not need balance. $\endgroup$ – Frank Harrell May 30 '19 at 11:43
  • $\begingroup$ @FrankHarrell, I'm not completely sure about what you exactly refer to when you say "statistical principle"... $\endgroup$ – Andrey E. Vedishchev Jun 1 '19 at 15:48
  • $\begingroup$ I mean does the strategy respect the information collected, e.g., no binning of continuous variables, and has the strategy been either validated theoretically or by simulation? I ask this because it seems ad hoc. $\endgroup$ – Frank Harrell Jun 1 '19 at 21:23
  • $\begingroup$ @FrankHarrell, the price variations - which are continuous variables - have been binned by the authors in four different categories: {1, 2, 3, 4}. Every category, in their original paper, contains pretty much the same number of occurrences. Their strategy has been validated by a backtest and, in the period taken into consideration, they actually overperformed the benchmark. However it should be pointed out that they haven't included transaction costs nor front running in their simulation. $\endgroup$ – Andrey E. Vedishchev Jul 4 '19 at 7:39
  • $\begingroup$ The classes are not valid. They are arbitrary and most importantly throw away information and ignore close calls (changes near those boundaries). The categories also fail to recognize that a change of +x% is not balanced by a change of -x% because % changes is an asymmetric measure. $\endgroup$ – Frank Harrell Jul 4 '19 at 12:47

Here is a list of most commonly used discretisation methods, more details as part of a (slightly dated) literature review is here.

  1. Chi-square based methods
  2. Entropy-based methods
  3. Wrapper-based methods
  4. Adaptive methods

Among these, entropy-based discretisation is quite popular. Below are some links that are good starting points:

  1. Light-weight, introductory blog
  2. A paper, by Clarke and Barton, on entropy-based discretisation. Although the title implies just Bayesian Networks, it can be applied to continuous variables regardless of classification algorithm.
  • $\begingroup$ I'm going to try your solutions during the next couple days. I'll definitely accept your answer if one of those methods will work. $\endgroup$ – Andrey E. Vedishchev Jun 1 '19 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.