Questions tagged [in-sample]
The in-sample tag has no usage guidance.
26
questions
0
votes
0
answers
26
views
Forecasting: choosing the sample split between "in-sample" and "out-of-sample" data
Goals:
Given approximately 11 years of time series data, to determine how much of this data should be reserved for in-sample and ...
- 101
1
vote
0
answers
18
views
Is it possible average an information criterion across models?
Is it possible to take the average of information criterion like the AIC?
For my model comparison, I have 24 different models. I use 4 different GARCH models each with 6 different distributions for ...
- 11
2
votes
0
answers
135
views
Why not use exact probability in 0.632 or 0.632+ method with small sample size?
The .632 estimator (and extensions like .632+) developed by Bradley Efron are founded on the following premise. Suppose we have a data set with $n$ observations, and we draw $B$ nonparametric ...
- 226
3
votes
0
answers
40
views
How to manage out of sample data in the long run?
For example, you are interested in testing an investment strategy and there is data from 1950 to 2022. So you split it into a train and test set, say 1950-2000 and 2000-2022. Then you build your model ...
- 31
1
vote
1
answer
1k
views
In-Sample and Out-of-sample forecasting accuracy
I am currently doing my college final project. I forecasted national soybeans yield and used MAPE to calculate the in-sample and out-of-sample forecasting accuracy. The MAPE results showed that the in-...
- 13
2
votes
1
answer
152
views
What is the posterior in-sample vs the posterior out-of-sample?
I'm watching this video on Bayesian modelling for the stock market by Thomas Wiecki, Thomas has a slide with two posterior distribution over the mean parameter in his stock return model. Around 18:26 ...
- 605
3
votes
0
answers
79
views
In practice, do we distinguish between "in-sample" and "training" error?
In Elements of Statistical Learning, it distinguishes between "in-sample" and "training" error (Frankly, I found the chapter on errors to be very confusing, especially with how ...
- 1,127
1
vote
1
answer
627
views
How does GARCH compute the realized daily volatility to be compared to the output of the model, to compute in-sample MSE?
How do GARCH and GJR-GARCH models (as implemented in rugarch or in EViews) calculate the in-sample MSE if they use the time series of daily returns as the input and don't use a time series of daily ...
- 89
0
votes
1
answer
19
views
In-sample evaluation with different classifiers
I've tested in-sample evaluation with different classifiers (Decision trees, Random Forests, Gaussian Naive Bayes) within sklearn and Iris datasets.
...
- 36
1
vote
0
answers
332
views
AIC based model selection, hyperparameter optimization and in-sample prediction
I'm using AIC to perform model selection along with hyperparameters optimization. The exact setup is the following:
I have two input variables (A and B), and a single target variable. All variables ...
- 11
2
votes
1
answer
2k
views
in-sample data vs out-of-sample data
I know that a train-validation-test splits the data into:
a training dataset - obviously my "in-sample" data
a validation dataset
a test data set - obviously my "out-of-sample" data
My question is: ...
- 75
0
votes
0
answers
32
views
Other than out of sample error, are there any other ways of comparing goodness of fit of two models, when the models come from different families?
I'm asking this within the context of time series, but the question would apply to any regression type learning problem.
It usually specified that using information criteria like the AIC or the BIC ...
- 11.6k
2
votes
1
answer
842
views
In which scenarios are the in-sample error and training error NOT the same?
In Elements of Statistical Learning, Chapter 7 (pages 228-229), the authors define the optimism of the training error rate as:
$$
op\equiv Err_{in}-\overline{err}
$$
With the training error $\...
- 11.6k
0
votes
0
answers
46
views
Is it valid to average across forecast errors for series of parts?
I'm trying to do inventory control and measure bias in my forecasts. I can calculate the forecast error (Demand-Forecast) for each series of parts per month in a 12 month period, but I want to know ...
- 239
1
vote
1
answer
2k
views
Which one should I use for rolling forecast, dynamic or static?
I'm doing a rolling forecast using a fitted arma-garch model, but I'm confused regarding the rolling method, my window length is 1209 obs, and I roll 100 times, and each time I reset my window to ...
- 23
2
votes
0
answers
279
views
Optimism bias - alternative references
In Hastie & al's book Elements of Statistical Learning, there are two subsections covering insample prediction errors and optimism bias (section 7, p.228-230).
Hastie & al explain that ...
- 41
0
votes
1
answer
127
views
Sample to use when doing ARMA order selection, OLS, GARCH estimation
I have a really basic question which I need clarifying on and would appreciate a quick response to this if possible.
I am forecasting conditional variance (volatility). I have in-sample from 1996-...
- 119
1
vote
0
answers
182
views
Data Partition for In sample and out of sample forecasting in neural network
I got confused with how to do data partition that reflects in sample and out of sample forecast when I do time series forecasting in neural network.
What I understand is we have to divide data into:
-...
- 11
1
vote
0
answers
149
views
Can you compare in-sample vs. out-of-sample using a MSPE?
Is it possible to compare in-sample and out-of-sample forecasts by calculating a MSPE for each? For example, my in sample period is say 12/1987-12/2015, and my out-of-sample forecasting period is 12/...
- 11
13
votes
1
answer
9k
views
What is the difference between in sample error and training error, and intuition of optimism?
In the book Elements of Statistical Learning in Chapter 7 (page 228), the training error is defined as:
$$
\overline{err} = \frac{1}{N}\sum_{i=1}^{N}{L(y_i,\hat{f}(x_i))}
$$
Whereas in-sample error ...
- 709
3
votes
1
answer
16k
views
Out of Sample and In Sample testing
I am very confused in testing regressions and know that there are many explanations available online, but I am still not getting anything it in my mind.
Suppose I have daily data for past 100 days, I ...
- 133
0
votes
1
answer
45
views
Is evaluating a holdout model against the full model a pseudo way of in-sample testing?
A model has been built using quarterly data over a 14 year period. The data is not available for each quarter hence the total number of observations is less than 4*14.
Then a second model is built ...
- 151
1
vote
0
answers
5k
views
In-Sample vs. Out-of-Sample One-Step Ahead Forecasts
I'm fairly new to forecasting but I find all of this quote fascinating and hope to learn something from all of you.
I have 500 observations and I'm tasked with the following: "compute recursive (...
2
votes
2
answers
4k
views
what does it mean when out of sample AUC is greater than in sample AUC?
I am fitting a logistic regression model on a data set with about 200,000 observation and 100 features. According to SAS output, the model converged correctly with an in-sample AUC of 0.85. However, ...
- 1,337
1
vote
1
answer
296
views
Predictive modeling techniques for in-sample rather than out-of-sample prediction?
Is it appropriate to apply predictive modeling variable selection and shrinkage techniques (for example, ridge regression or lasso) for in-sample prediction rather than out-of-sample prediction? ...
- 5,532
15
votes
1
answer
52k
views
Difference between "in-sample" and "pseudo out-of-sample" forecasts
Is there an explicit difference between in-sample forecasts and pseudo out-of-sample forecasts. Both is meant in the context of evaluating and comparing forecasting models.
- 705