Building a binary precise classifier to forecast financial outcomes (stock rise vs. fall) brings up some nifty complications within caret.

1. classifier selection: there are tons of classifiers available in care. https://topepo.github.io/caret/modelList.html. Is there literature on evaluating which of all these models are promising on noisy data (both numeric and discrete features) yielding good precision?

2. cross-validation: trControl provides k-fold repeated cv, but with financial data, there is the restriction that I cannot use future train data to predict past test data. How can I define in trControl to use rolling? (when selecting folds, testdata must always be later than train data)

3. evaluation function and tuning: precision (positive predictive value) is more important than accuracy as I dont care about detecting true negatives. However, high precision doesnt help, if the sensitivity is very low. Therefore, AUC might be misleading, since a model might be strong in detecting many TRUE outcomes, but imprecise, thereby inflating the area under the curve. How can I tell caret to tune for "detecting as MANY TRUE outcomes with e.g. max. 30% false positive rate"?

metric = 'roc'

will probably not be optimal.

4. stability over re-training Just fitting a model once and selecting features and cut-off once will not guarantee stability of the model - as features and conditions might change over time on financial markets. What are good approaches in caret here? (e.g. rolling strategies)

5. estimating cut-offs Due to the random nature of financial markets described in 4, potentially changing feature set, and the need for a precise model, the cut-off value of the probs resulting from my model on test data will probably change as well. Is there a good common practice to predict correct cut-offs? (e.g. fit a separate model to learn cut-offs on testdata...)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.