1
$\begingroup$

i'm facing this kind of issue. I set up a good model (122 variables) with 1700 records: training accuracy and test are close to 80%. I'm trying to predict 8 days ahead, the movement of a stock index. Basically i did:

use both SVMRadial and RF in R created a partition 70/30 to split training set and test use repeatedCV (5 folds x 5 times)

at the end, i got encouraging accuracies in classification, both in training and test set. Finally i tried to validate the model, testing unseen data: i performed a direct 8-days step ahead to process current record (time t) to get the result for time t+8 (the latest Y to feed, the output of the classifier, is from 8 days in the past).Then i carried on training the model with the freshest data and jumping 8 records ahead on and on iteratively. Accuracy dramatically drops down to 50/55% making the classifier pointless. This happens both in SVMRadial and in RF: practically the model, even though always trained till the last available complete data (with the relevant y, that is +1 or -1), is unable to be accurate with new data.I also tried a recursive solution to fill the 8 missing Ys with random forest, getting poor results. Why my model is unable to generalize, providing good results just only 8 records ahead from the last training point? is the cause overfitting with cross validation? How can i manage it? thank in advance! regards

$\endgroup$

1 Answer 1

1
$\begingroup$

While not knowing what your predictors are, it works out that multi-instrument (and multi-market) models usually perform much better on out-of-bag (out-of-training) data. See the software package called VantagePoint, which uses separate prediction models (neural networks, not too far from SVM) for the 2-day lagged data, 5-day lagged data, and 10-day lagged data - and then determines which time-frame is the most dependable. Most of their models use data/indices from other markets and exchanges.

Another issue for stock price prediction is that many of the days or epochs of time you are using are redundant, so you may not need 1700 records or 122 predictors. Thus, you might use PCA on both the days and the 122 predictors to reduce the dimensions (decorrelate/denoise). Jurik Research sells a module known as DDR (decorrelator, dimension reducer) which can perform two-way PCA on time and features for financial data.

$\endgroup$
2
  • $\begingroup$ Thanks for your prompt answer. Basically i'm using technical indicators and other indices as predictors (e.g.S&P500).I'll sure try a PCA, that should be embedded in preProcess options (R) to reduce dimension and decorrelate. I have tried everything in feature engineering: make differences to kill trends, exponential smoothing, data transformation ,lagged variables, feature selections tools like Boruta, RFE , GAs. Random forest has an incredible ability to classify (as time ahead increases): it keeps this skill for the very following record , but cannot go up to the last effectively. Thanks! $\endgroup$ Commented Nov 11, 2021 at 22:04
  • 1
    $\begingroup$ I've tried a lot of neural network approaches to stock price prediction, and results were not that good either. Markets are a complex system, so you commonly have to use pattern recognition as well, involving e.g. fractals, etc. $\endgroup$
    – user318288
    Commented Nov 11, 2021 at 23:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.