Building a binary precise classifier to forecast financial outcomes (stock rise vs. fall) brings up some nifty complications within caret.
1. classifier selection: there are tons of classifiers available in care. https://topepo.github.io/caret/modelList.html. Is there literature on evaluating which of all these models are promising on noisy data (both numeric and discrete features) yielding good precision?
2. cross-validation: trControl provides k-fold repeated cv, but with financial data, there is the restriction that I cannot use future train data to predict past test data. How can I define in trControl to use rolling? (when selecting folds, testdata must always be later than train data)
3. evaluation function and tuning: precision (positive predictive value) is more important than accuracy as I dont care about detecting true negatives. However, high precision doesnt help, if the sensitivity is very low. Therefore, AUC might be misleading, since a model might be strong in detecting many TRUE outcomes, but imprecise, thereby inflating the area under the curve. How can I tell caret to tune for "detecting as MANY TRUE outcomes with e.g. max. 30% false positive rate"?
metric = 'roc'
will probably not be optimal.
4. stability over re-training Just fitting a model once and selecting features and cut-off once will not guarantee stability of the model - as features and conditions might change over time on financial markets. What are good approaches in caret here? (e.g. rolling strategies)
5. estimating cut-offs Due to the random nature of financial markets described in 4, potentially changing feature set, and the need for a precise model, the cut-off value of the probs resulting from my model on test data will probably change as well. Is there a good common practice to predict correct cut-offs? (e.g. fit a separate model to learn cut-offs on testdata...)