2
$\begingroup$

I collected a sample of 600 observation (time series data) with 100 predictors variables in order to predict another one. I want to use some prediction models but I know that, unfortunately, overfitting problem exist; especially with many predictors. So I split the sample, in and out (300 vs 300), then, basically, I have to estimate models/parameters in sample and check the prediction quality out of sample against a benchmark; I use MSE and or related metrics. Until here I don't have great doubts.

I starting with regression and I used a variables selection rule with strategy like this: overfitting and selection model.

Then, I want to check for the presence of relevant non linearity in the links among predictors and predicted variable. The most flexible alternative seems me the Artificial Neural Network (ANN) models and I want to try with them. So starting from the same split above the idea is again to calibrate the ANN in sample and test it out of sample. Here I assume that no useless predictors exist (no variables selection). So, after standardization (semi-standardization for out-of-sample data), I use all of them. However in ANN there are several unknown hyperparameters that make the things complex in estimation terms. Firstly: number of hidden layers, number of nodes, type of activation function. To choose them exogenously is one possibility but seems me too strong assumption. It seem me that another possibility is to split again the "in sample data" in training and validation set (200 vs 100). Then, I define a "grid of models" with different hyperparameters. I calibrate any ANN model in training set and I check the prediction performance of them in validation set. The performance is measured with the same metrics used for, final, out of sample scrutiny. The word validation give the idea that the choice come from cross validation (CV). For dependent data, as usually time series are, the usual CV methods are inadequate but some ad hoc version exist (see here: Cross-validation techniques for time series data Choosing inner cross validation strategy for modeling time series data Using k-fold cross-validation for time-series model selection ). All these strategy bring us to obtain several split, even if much less than the standard ways. In my case I consider only one split: ($t_{1},…,t_{200}$) for training and ($t_{201},…,t_{300}$) for validation. Therefore it seems me that, what I made is not a true CV procedure.

Only the best ANN model, recalibrated on in sample data (training + validation = 300) and the best regression one achieve the final scrutiny based on out of sample data (300 unseen before).

  1. This general strategy make sense? Some weak points?

  2. The procedure described for ANN hyperparameters choice, among a grid of models/specifications, represent a proper method?

  3. In the out of sample scrutiny I have only one realization for generalized error of the two models. I measure the related $MSE$ and or others metrics. Compare them it's enough in order to choose the best one?

EDIT: The AJKOER answer suggest me to give you some information more. I said that my data have time series form, these are weak stationary or at least I suppose it. Otherwise usual inference from only one realization (what I have) is not possible. More precisely I try to predict/forecast a series of bond yield taken in first difference. At first glance this series is not far from bond or stock returns even if is negatively related with the former. However the procedure that I follow is quite general and I can use it also for return of any financial assets. It is well known that all of them are hard to forecast successfully.

Just for information, in my experience with ARMA models, even if some interesting results appear in ACF or PACF, the out of sample forecast performance, usually, is very poor. White noise benchmark seems me almost unbeatable. Moreover we have to note that, in pure forecast, multivariate models like VARs show usually worse performance that univariate models. For this reason I leave prediction models that consider only past own realizations and move to more general approach; more close to what predictive learning literature suggest.

Moreover I embrace the distinction between prediction and causation. Here I follow a purely predictive approach. Read here for more detail (Minimizing bias in explanatory modeling, why? (Galit Shmueli's "To Explain or to Predict") Structural models and relationship (statistical associations) Endogeneity in forecasting Paradox in model selection (AIC, BIC, to explain or to predict?) Regression: Causation vs Prediction vs Description). I want to built a pure data driven model, therefore any theoretical points are not considered; extremely view, I know but that's it. I aware that about bond return predictability, term premia, term structure, ecc, wide and relevant literature exist but I want precisely do not consider it here.

$\endgroup$
9
  • $\begingroup$ Since you requested my opinion, here are some points. Hastie et al. "Elements of Statistical Learning" do not recommend cross validation for selecting the numbers of layers and nodes, they set them exogenously (but I do not have a strong opinion on that). Your time series appear too short for complex models such as neural networks. Regarding the split into test vs. nontest data, consider doing time series cross validation (TSCV) via rolling windows instead; it should reduce the test error. Consider using expanding windows instead of rolling windows. $\endgroup$ – Richard Hardy Oct 8 '20 at 5:12
  • $\begingroup$ I would also definitely look into the existing literature on modelling and foreasting bond yields. There must be lots of papers doing that and lots of things to learn from them. When the sample size is only 600 observations and the effectively used part for training is even smaller due to splitting into training, validation and test, incorporating theory in your models may help considerably (though you said you are not interested in that). $\endgroup$ – Richard Hardy Oct 8 '20 at 5:13
  • $\begingroup$ About the TSCV, keeping the split-size 200 vs 100 I can split the data as follow also: ($t_{2},…, t_{201}$) as train and ($t_{202},…, t_{1}$) as test; then ($t_{3},…, t_{202}$) as train and ($t_{203},…, t_{2}$) as test; and so on. This is what you intend as rolling window strategy? $\endgroup$ – markowitz Oct 8 '20 at 16:58
  • $\begingroup$ A rolling window works like this: training from 1 to k, test on k+1 (a single observation); training from 2 to k+1, test on k+2 (a single observation), training from 3 to k+2, test on k+3 etc. until you run out of data. $\endgroup$ – Richard Hardy Oct 8 '20 at 19:56
  • $\begingroup$ What problem I can encounter if for each window I test not only on the k+1 observation but on all observations not used for training, and average over them? $\endgroup$ – markowitz Oct 9 '20 at 10:17
1
$\begingroup$

Here is my ideal opinion on a valid/rational course of action:

Step 1: identify the realm that you're operating in, whether it be economics, physical chemistry, …

Step 2: Based on Step 1, postulate all the applicable physical laws and generating processes that are likely significant drivers. This may require research/consultation with an economist, physical chemists,... etc.

Step 3: Build hypothetical simple models or inter-related models that are based on the identified generating process. Note: models and generating processes are based on the real-world opinion/research of experts in the particular arena.

Step 4: Populate said models/structures with generated data based on appropriate parent distributions via Monte Carlo methods. Start by selecting a very low level or noise (including associated inter-correlation noise structure, etc.).

Step 5: Investigate analytical tools you have available to develop parameter estimates assuming that you actually have roughly or precisely a correct singular model, or with a no model scenario, where a model-free approach is applicable. Repeat with multiple or shifting models as deemed appropriate in the realm of study.

This exercise will quickly reveal the power or lack thereof of analytical tools and associated methods applied to the various targetted arenas (so, you can stop wasting your time and effort in likely difficult reverse modeling situations).

Also, it may further, in failing, give real guidance on robust estimates of noise so that even naive poorly performing models may render rational error prediction bounds.

In essence, I am recommending expert base knowledge of a field to replicate data and then test applicable paths to extract the structure and assess error levels (if even possible), which are, in effect, all known with precision.

$\endgroup$
2
  • $\begingroup$ I added something above. $\endgroup$ – markowitz Oct 3 '20 at 14:08
  • $\begingroup$ My comment: "wide and relevant literature exist but I want precisely do not consider it here", perhaps, it may be better to build a model with PCA based independent constructs, where a time-series trend would have an independent but contributing voice. Remember, time-series forecasts are most wrong when a new significant factor surfaces (which, per the literature, maybe actually somewhat expected and should have been considered, at least as a scenario for likely model error). $\endgroup$ – AJKOER Oct 3 '20 at 15:14
1
$\begingroup$

I don't disagree with AJKOER's answer, although you may not be in a situation in which you have much background knowledge that can be used in this way. Anyway, directly to your questions:

  1. This sounds OK. The problem obvsiously is that 300 points isn't that much for making all these decisions by optimisation, so you may be disappointed with the out of sample results of what comes out as optimal on your "in sample". However, I don't really know a good cure. In fact the only alternative is to make stronger assumptions, and then it depends on background knowledge whether you can justify these. Knowing something in advance and using it doesn't waste degrees of freedom in comparisons, but of course if you don't actually have some good knowledge backing up decisions, that may do harm rather than help. So I see why you're suggesting it as you suggest, and I have done things in this way, too. Let's say it may pay off to think a bit about what you can just decide in advance, because 300 observations will be very few if you want to make 100 decisions based on them.

  2. Imprecise question. I'd call it "some sort of cross-validation", but then I'm not sure about your definition of "some sort", and does it matter how one can call it?

  3. You're only comparing two models there with no further tuning, so I'd say this is OK.

$\endgroup$
5
  • $\begingroup$ First of all thank you for your answer. About my question 2, I know that cross validation can be used for hyperparameters tuning also, and it is what I made. However I know also that CV techniques like leave one out or k-fold we have several split and as many estimates for any specification. Here, given training/validation (200 vs 100), I have only one split/estimation for any specification. I make so because in time series I dubious about the opportunity of not consider the order of the observations, moreover this opportunity is computationally too costly for my tools. $\endgroup$ – markowitz Oct 3 '20 at 14:05
  • $\begingroup$ About point 3. If in the final scrutiny I have more than two models (no tuning) some problem can appear ? $\endgroup$ – markowitz Oct 3 '20 at 14:08
  • $\begingroup$ Re point 3: Generally if you want to assess prediction quality in an unbiased way, you should not optimise anything but just assess only the previously selected method on the "out sample". That said, given that your number of observations is fairly low, you need to make economic use of your samples. If you just decide which one of two methods is best, chances are this won't introduce much bias, and the more you compare, the more selection bias you have (which is why you compare on the in sample and assess on the out sample in the first place). $\endgroup$ – Lewian Oct 3 '20 at 17:17
  • $\begingroup$ I modified something in order to clarify the point 2. $\endgroup$ – markowitz Oct 7 '20 at 22:01
  • $\begingroup$ Not quite sure what "represents a proper method" means either, however I believe it's an OK option to do it in this way. $\endgroup$ – Lewian Oct 8 '20 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.