# Questions tagged [in-sample]

The tag has no usage guidance.

25 questions
Filter by
Sorted by
Tagged with
36 views

### Why not use exact probability in 0.632 or 0.632+ method with small sample size?

The .632 estimator (and extensions like .632+) developed by Bradley Efron are founded on the following premise. Suppose we have a data set with $n$ observations, and we draw $B$ nonparametric ...
21 views

### How to manage out of sample data in the long run?

For example, you are interested in testing an investment strategy and there is data from 1950 to 2022. So you split it into a train and test set, say 1950-2000 and 2000-2022. Then you build your model ...
15 views

### Out of Sample Regression errors

I am trying to compute $\text{R}^2$ and ${delta RMSE}$ from an Out of Sample Linear Model in R. $e _{ N }$ is the vector of rolling OOS errors from the historical mean model $e_{A}$ is the vector ...
1 vote
408 views

### In-Sample and Out-of-sample forecasting accuracy

I am currently doing my college final project. I forecasted national soybeans yield and used MAPE to calculate the in-sample and out-of-sample forecasting accuracy. The MAPE results showed that the in-...
94 views

### What is the posterior in-sample vs the posterior out-of-sample?

I'm watching this video on Bayesian modelling for the stock market by Thomas Wiecki, Thomas has a slide with two posterior distribution over the mean parameter in his stock return model. Around 18:26 ...
52 views

### In practice, do we distinguish between "in-sample" and "training" error?

In Elements of Statistical Learning, it distinguishes between "in-sample" and "training" error (Frankly, I found the chapter on errors to be very confusing, especially with how ...
1 vote
334 views

### How does GARCH compute the realized daily volatility to be compared to the output of the model, to compute in-sample MSE?

How do GARCH and GJR-GARCH models (as implemented in rugarch or in EViews) calculate the in-sample MSE if they use the time series of daily returns as the input and don't use a time series of daily ...
16 views

### In-sample evaluation with different classifiers

I've tested in-sample evaluation with different classifiers (Decision trees, Random Forests, Gaussian Naive Bayes) within sklearn and Iris datasets. ...
1 vote
219 views

### AIC based model selection, hyperparameter optimization and in-sample prediction

I'm using AIC to perform model selection along with hyperparameters optimization. The exact setup is the following: I have two input variables (A and B), and a single target variable. All variables ...
2k views

### in-sample data vs out-of-sample data

I know that a train-validation-test splits the data into: a training dataset - obviously my "in-sample" data a validation dataset a test data set - obviously my "out-of-sample" data My question is: ...
30 views

### Other than out of sample error, are there any other ways of comparing goodness of fit of two models, when the models come from different families?

I'm asking this within the context of time series, but the question would apply to any regression type learning problem. It usually specified that using information criteria like the AIC or the BIC ...
672 views

### In which scenarios are the in-sample error and training error NOT the same?

In Elements of Statistical Learning, Chapter 7 (pages 228-229), the authors define the optimism of the training error rate as: $$op\equiv Err_{in}-\overline{err}$$ With the training error \$\...
45 views

### Is it valid to average across forecast errors for series of parts?

I'm trying to do inventory control and measure bias in my forecasts. I can calculate the forecast error (Demand-Forecast) for each series of parts per month in a 12 month period, but I want to know ...
1 vote
1k views

### Which one should I use for rolling forecast, dynamic or static?

I'm doing a rolling forecast using a fitted arma-garch model, but I'm confused regarding the rolling method, my window length is 1209 obs, and I roll 100 times, and each time I reset my window to ...
248 views

### Optimism bias - alternative references

In Hastie & al's book Elements of Statistical Learning, there are two subsections covering insample prediction errors and optimism bias (section 7, p.228-230). Hastie & al explain that ...
117 views

### Sample to use when doing ARMA order selection, OLS, GARCH estimation

I have a really basic question which I need clarifying on and would appreciate a quick response to this if possible. I am forecasting conditional variance (volatility). I have in-sample from 1996-...
1 vote
177 views

### Data Partition for In sample and out of sample forecasting in neural network

I got confused with how to do data partition that reflects in sample and out of sample forecast when I do time series forecasting in neural network. What I understand is we have to divide data into: -...
1 vote
134 views

### Can you compare in-sample vs. out-of-sample using a MSPE?

Is it possible to compare in-sample and out-of-sample forecasts by calculating a MSPE for each? For example, my in sample period is say 12/1987-12/2015, and my out-of-sample forecasting period is 12/...
7k views

### What is the difference between in sample error and training error, and intuition of optimism?

In the book Elements of Statistical Learning in Chapter 7 (page 228), the training error is defined as: $$\overline{err} = \frac{1}{N}\sum_{i=1}^{N}{L(y_i,\hat{f}(x_i))}$$ Whereas in-sample error ...
15k views

### Out of Sample and In Sample testing

I am very confused in testing regressions and know that there are many explanations available online, but I am still not getting anything it in my mind. Suppose I have daily data for past 100 days, I ...
44 views

### Is evaluating a holdout model against the full model a pseudo way of in-sample testing?

A model has been built using quarterly data over a 14 year period. The data is not available for each quarter hence the total number of observations is less than 4*14. Then a second model is built ...
1 vote
5k views

### In-Sample vs. Out-of-Sample One-Step Ahead Forecasts

I'm fairly new to forecasting but I find all of this quote fascinating and hope to learn something from all of you. I have 500 observations and I'm tasked with the following: "compute recursive (...
3k views

### what does it mean when out of sample AUC is greater than in sample AUC?

I am fitting a logistic regression model on a data set with about 200,000 observation and 100 features. According to SAS output, the model converged correctly with an in-sample AUC of 0.85. However, ...
1 vote