Questions tagged [in-sample]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2 votes
0 answers
36 views

Why not use exact probability in 0.632 or 0.632+ method with small sample size?

The .632 estimator (and extensions like .632+) developed by Bradley Efron are founded on the following premise. Suppose we have a data set with $n$ observations, and we draw $B$ nonparametric ...
user avatar
3 votes
0 answers
21 views

How to manage out of sample data in the long run?

For example, you are interested in testing an investment strategy and there is data from 1950 to 2022. So you split it into a train and test set, say 1950-2000 and 2000-2022. Then you build your model ...
user avatar
0 votes
0 answers
15 views

Out of Sample Regression errors

I am trying to compute $\text{R}^2$ and $ {delta RMSE} $ from an Out of Sample Linear Model in R. $ e _{ N }$ is the vector of rolling OOS errors from the historical mean model $ e_{A}$ is the vector ...
user avatar
  • 1
1 vote
1 answer
408 views

In-Sample and Out-of-sample forecasting accuracy

I am currently doing my college final project. I forecasted national soybeans yield and used MAPE to calculate the in-sample and out-of-sample forecasting accuracy. The MAPE results showed that the in-...
user avatar
  • 13
2 votes
1 answer
94 views

What is the posterior in-sample vs the posterior out-of-sample?

I'm watching this video on Bayesian modelling for the stock market by Thomas Wiecki, Thomas has a slide with two posterior distribution over the mean parameter in his stock return model. Around 18:26 ...
user avatar
  • 595
3 votes
0 answers
52 views

In practice, do we distinguish between "in-sample" and "training" error?

In Elements of Statistical Learning, it distinguishes between "in-sample" and "training" error (Frankly, I found the chapter on errors to be very confusing, especially with how ...
user avatar
  • 927
1 vote
1 answer
334 views

How does GARCH compute the realized daily volatility to be compared to the output of the model, to compute in-sample MSE?

How do GARCH and GJR-GARCH models (as implemented in rugarch or in EViews) calculate the in-sample MSE if they use the time series of daily returns as the input and don't use a time series of daily ...
user avatar
  • 89
0 votes
1 answer
16 views

In-sample evaluation with different classifiers

I've tested in-sample evaluation with different classifiers (Decision trees, Random Forests, Gaussian Naive Bayes) within sklearn and Iris datasets. ...
user avatar
1 vote
0 answers
219 views

AIC based model selection, hyperparameter optimization and in-sample prediction

I'm using AIC to perform model selection along with hyperparameters optimization. The exact setup is the following: I have two input variables (A and B), and a single target variable. All variables ...
user avatar
  • 11
2 votes
1 answer
2k views

in-sample data vs out-of-sample data

I know that a train-validation-test splits the data into: a training dataset - obviously my "in-sample" data a validation dataset a test data set - obviously my "out-of-sample" data My question is: ...
user avatar
0 votes
0 answers
30 views

Other than out of sample error, are there any other ways of comparing goodness of fit of two models, when the models come from different families?

I'm asking this within the context of time series, but the question would apply to any regression type learning problem. It usually specified that using information criteria like the AIC or the BIC ...
user avatar
  • 10.8k
2 votes
1 answer
672 views

In which scenarios are the in-sample error and training error NOT the same?

In Elements of Statistical Learning, Chapter 7 (pages 228-229), the authors define the optimism of the training error rate as: $$ op\equiv Err_{in}-\overline{err} $$ With the training error $\...
user avatar
  • 10.8k
0 votes
0 answers
45 views

Is it valid to average across forecast errors for series of parts?

I'm trying to do inventory control and measure bias in my forecasts. I can calculate the forecast error (Demand-Forecast) for each series of parts per month in a 12 month period, but I want to know ...
user avatar
  • 239
1 vote
1 answer
1k views

Which one should I use for rolling forecast, dynamic or static?

I'm doing a rolling forecast using a fitted arma-garch model, but I'm confused regarding the rolling method, my window length is 1209 obs, and I roll 100 times, and each time I reset my window to ...
user avatar
  • 23
2 votes
0 answers
248 views

Optimism bias - alternative references

In Hastie & al's book Elements of Statistical Learning, there are two subsections covering insample prediction errors and optimism bias (section 7, p.228-230). Hastie & al explain that ...
user avatar
  • 41
0 votes
1 answer
117 views

Sample to use when doing ARMA order selection, OLS, GARCH estimation

I have a really basic question which I need clarifying on and would appreciate a quick response to this if possible. I am forecasting conditional variance (volatility). I have in-sample from 1996-...
user avatar
  • 109
1 vote
0 answers
177 views

Data Partition for In sample and out of sample forecasting in neural network

I got confused with how to do data partition that reflects in sample and out of sample forecast when I do time series forecasting in neural network. What I understand is we have to divide data into: -...
user avatar
1 vote
0 answers
134 views

Can you compare in-sample vs. out-of-sample using a MSPE?

Is it possible to compare in-sample and out-of-sample forecasts by calculating a MSPE for each? For example, my in sample period is say 12/1987-12/2015, and my out-of-sample forecasting period is 12/...
user avatar
  • 11
13 votes
1 answer
7k views

What is the difference between in sample error and training error, and intuition of optimism?

In the book Elements of Statistical Learning in Chapter 7 (page 228), the training error is defined as: $$ \overline{err} = \frac{1}{N}\sum_{i=1}^{N}{L(y_i,\hat{f}(x_i))} $$ Whereas in-sample error ...
user avatar
3 votes
1 answer
15k views

Out of Sample and In Sample testing

I am very confused in testing regressions and know that there are many explanations available online, but I am still not getting anything it in my mind. Suppose I have daily data for past 100 days, I ...
user avatar
  • 133
0 votes
1 answer
44 views

Is evaluating a holdout model against the full model a pseudo way of in-sample testing?

A model has been built using quarterly data over a 14 year period. The data is not available for each quarter hence the total number of observations is less than 4*14. Then a second model is built ...
user avatar
  • 151
1 vote
0 answers
5k views

In-Sample vs. Out-of-Sample One-Step Ahead Forecasts

I'm fairly new to forecasting but I find all of this quote fascinating and hope to learn something from all of you. I have 500 observations and I'm tasked with the following: "compute recursive (...
user avatar
2 votes
2 answers
3k views

what does it mean when out of sample AUC is greater than in sample AUC?

I am fitting a logistic regression model on a data set with about 200,000 observation and 100 features. According to SAS output, the model converged correctly with an in-sample AUC of 0.85. However, ...
user avatar
  • 1,267
1 vote
1 answer
285 views

Predictive modeling techniques for in-sample rather than out-of-sample prediction?

Is it appropriate to apply predictive modeling variable selection and shrinkage techniques (for example, ridge regression or lasso) for in-sample prediction rather than out-of-sample prediction? ...
user avatar
  • 4,514
14 votes
1 answer
51k views

Difference between "in-sample" and "pseudo out-of-sample" forecasts

Is there an explicit difference between in-sample forecasts and pseudo out-of-sample forecasts. Both is meant in the context of evaluating and comparing forecasting models.
user avatar
  • 675