# Questions tagged [out-of-sample]

Refers to the practice of assessing model performance on some "test" or "holdout" or "out-of-sample" set of data that was not used for model building.

164 questions
Filter by
Sorted by
Tagged with
1 vote
14 views

### How to manage out of sample data in the long run?

For example, you are interested in testing an investment strategy and there is data from 1950 to 2022. So you split it into a train and test set, say 1950-2000 and 2000-2022. Then you build your model ...
1 vote
16 views

### Random Forests- Out of Bag Error Calculation

I was learning about the Out of Bag error in random forests and I did not understand a point about the error calculation. Assume we have N bootstraps and there are a number of Out-Of-Bag samples for ...
12 views

### Out of Sample Regression errors

I am trying to compute $\text{R}^2$ and ${delta RMSE}$ from an Out of Sample Linear Model in R. $e _{ N }$ is the vector of rolling OOS errors from the historical mean model $e_{A}$ is the vector ...
33 views

### Creating a holdout set just for feature engineering?

I recently encountered a feature engineering technique that I haven't seen before: Create the usual training, validation, and test sets. Create another set by splitting the train set; call this the &...
• 11.5k
52 views

### Poor out of time performance

I am working on a behavioral model which predicts the probability of default (PD) during the next 12 months for an existing customer with an outstanding loan. My dataset consists of monthly snapshots ...
• 11
6 views

### Do I need to separate unique IDs when doing out-of-time validation?

I'm doing a credit risk model and I thought that would be a good idea to validate out-of-time in multiple time periods (merging them to a big unique validation set). Each line of my dataset ...
87 views

### How to correctly diagnose overfitting using all information: training set, validation set and test set?

I understand that overfitting is typically defined/described as the relationship between training set error and test set error - that overfitting is when a model performs significantly worse out-of-...
• 1,082
1 vote
68 views

### How to be sure that out-of-sample results of one model are really better than those of another model?

I need to solve a standard, let say "vanilla", regression problem: I have a 2D array of real-valued features $X$ and a 1D array of real-valued targets $Y$. I choose a simple model to fit (...
• 473
23 views

### Difference in model result when scoring data

I am trying to predict/score on an out-of-time data applying the same model I had used to train and predict on test data. In the previous model, the recall and precision for correctly predicting 1s ...
16 views

### Holdout and the Multiple Comparisons Problem

I apologize in advance if this is an extremely naive question. I am attempting to analyze a large body of discretized event data (of the form "did this event occur on date X"). This data is ...
• 101
14 views

### Sample Period for Model Selection?

I am currently trying to perform pseudo-out-of-sample forecasting for monthly exchange rates with a 10-year rolling window. Before that, I select an autoregressive model using Box Jenkins ...
• 111
61 views

### Comparing logistic regression models from two data sets when a parameter isn't varied in one data set but is in the other

I want to determine if a logistic regression model makes good predictions for a data set not used in its fitting with a hypothesis test; I'll call it the "new" data. One could say that the ...
• 1,063
1 vote
41 views

### What should unseen data comply mathematically for reliable measure of generalisation (gap)?

The way machine learning models validated for their generalisation performance, generalisation gap is used, see Predicting the Generalization Gap in Deep Neural Networks. This is essentially standard ...
• 1,838
11 views

### Rigorous uncertainty estimation resulting from out-of-distribution samples

Let $\mathbf Y_{train} = \left(\mathbf y_1^{\text{T}};...;\mathbf y_N^{\text{T}} \right)$ with $\mathbf y_i = \begin{pmatrix} y_{i,1} \\ y_{i,2} \end{pmatrix}$, such that the first coordinate ...
1 vote
62 views

### Evaluating a multi-step forecasting model?

The literature is a bit confusing for me on this one, from what I understand, a great deal of papers evaluate multi-step forecasting models on a single forecasting horizon on the hold out set. It ...
• 123
1 vote
322 views

### In-Sample and Out-of-sample forecasting accuracy

I am currently doing my college final project. I forecasted national soybeans yield and used MAPE to calculate the in-sample and out-of-sample forecasting accuracy. The MAPE results showed that the in-...
• 13
47 views

### Do in-sample distributional assumptions (e.g. normally distributed residuals) apply also to out-of-sample predictions?

I appreciate that out-of-sample error is important (i.e. comparing how close a model's predictions are to leftout data) but are there any distributional assumptions about the out-of-sample predictions?...
• 135
37 views

### Out-of-sample MSE and MAE for volatility forecasting [duplicate]

I have been searching through the whole CrossValidated but couldn't find the answer. I want to test out-of-sample the volatility forecasts (if it means something ARCH-like ones, MSGARCH, Multifractal ...
111 views

### Pros and cons of R squared and normalized RMSE as a scale-free out-of-sample performance metric for regression?

I am mainly considering the nRMSE = RMSE/sd, but if you are more familiar with RMSE/mean or RMSE/min-max that would also be interesting. I am looking for pros/cons for the two, or different use cases ...
• 21
1 vote
32 views

### Bagged Decision trees / Random Forests: why ISLR uses validation set instead of OOB to compute out-of-sample MSE?

I am reading the book "An Introduction to Statistical Learning" available here. Chapter 8.3.3 at page 328 of the book computes a bagged decision tree (which is a random forest where we use ...
• 1,372
25 views

### State-of-the-art methods for out-of-sample-extension

I'm using a kernel based dimensionality reduction algorithms, and interested in extending out-of-sample data points for further analysis. I've been using the Nystrom method for this task, and some ...
• 729
151 views

### Optimal Machine Learning Sample size

I am new in ML area and I want to build a model to score 100,000 people. I wonder if it is abnormal to build (train/test/validation) the model on a dataset of the same size 100,000? Need to mention ...
• 1
19 views

### Harvey-Newbold test (2000) in R

I want to estimate the best nested model out-of sample and for that purpose, I will implement Harvey-Newbold test. However, I cannot find the respective test in any package. Can anyone help me? If ...
159 views

### Out of sample ARCH forecast

I have estimated a conditional mean model for a time series: $x_t = x_{t-1} + \epsilon_t$. Say I have estimated it using periods 1 to 10. I can do an out of sample conditional mean forecast by ...
• 860
1 vote
579 views

### How to calculate the fitted values of out-of-sample predictions from conditional fixed-effects Poisson model?

I am trying to calculate fitted values from xtpoisson fixed effects on out-of-sample data. I know how to calculate fitted values for in-sample predictions (using ...
78 views

### Model evaluation via k-fold-cross validation after variable selection

I am currently trying to implement some methods (OLS, GLM) and compare their predictive accuracy. For that I want to perform variable selection for each of the methods and then measure the predictive ...
• 129
157 views

### Nested CV vs CV with a holdout?

I just learned of nested cross-validation and wanted to understand how my current approach is worse/ok. Currently I would: Divide the data into a train/test set (80/20ish). Use k-fold cross-...
• 206
89 views

### What is the posterior in-sample vs the posterior out-of-sample?

I'm watching this video on Bayesian modelling for the stock market by Thomas Wiecki, Thomas has a slide with two posterior distribution over the mean parameter in his stock return model. Around 18:26 ...
• 595
52 views

### What I'm dealing with here VAR, VARX or something different?

I have to implement a time-series tool in my company, and I'm not sure what I'm dealing with, when looking at the old tool. We have one target variable (sales) and a lot of different independent ...
1 vote
28 views

### Deciding between linear and non-linear approach in a predictive model when the relationship 'looks linear'

When we have a relatively small number of samples it is easy to see on a plot what is intuitively happening when we fit a regression line; we can see how far each of the individual sample points are ...
• 873
1 vote
61 views

### An out of bag testing procedure I developed. Is it biased?

Trying to estimate the error of a prediction model trained on a combined group vs a subset of interest within that group (for example, Men) My procedure is First simulate a random subset like the ...
• 11
62 views

### Can entropy be used to minimize prediction surprises in machine learning?

Information theory deals with signal/noise identification, while one of its tools, entropy, measures the surprise in random probabilistic outcomes. Has there been any application of using entropy or ...
• 3,067
12 views

### How to pull in-sample fitted models towards out-of-sample optimum during training?

A common problem in prediction problems is getting a model fit on in-sample data to predict out-of-sample data with decent accuracy and precision. Assuming that we break a part of the in-sample data ...
• 3,067
40 views

### Forecasting with empirical copulas

I estimated the beta copula with 3 variable time series. Now I'd like to make forecasts to evaluate the out-of-sample performance of my model. I know 2 of the 3 variables and I have the dependence ...
1 vote
69 views

### How to evaluate multiple data imputation?

I am performing data imputation of multiple time-series using various ML techniques (such as multiple imputation, iterative imputation). I have a matrix of ~100,000 observations (rows) of 34 stations (...
• 117
1 vote
28 views

### Should test MSE be bootstrapped to compare fits?

Suppose you have a training and testing set. You fit two models, A and B, to the training set. They you predict on the testing set. You find (in this contrived example): Test MSE model A: 3 Test MSE ...
1 vote
43 views

### Should I standardize forecasted and actual values for Mincer Zarnowitz test?

I have some out-of-sample forecasted values of variance through some GARCH model, and now I am trying to perform a Mincer-Zarnowitz test for validity of my predictions. I first standardized both ...
• 11
1 vote
225 views

### Most likely sources of divergence between (adjusted)-R squared and out-of-sample predictive performance

I'm wondering which invalid assumptions are most likely to explain the wild discrepancies between a model's R-squared as a measure of predictive performance, and the actual out-of-sample predictive ...
2k views

### How to calculate the confidence interval with weighted data?

I've done some search for similar questions, but they're not the same as what I'm trying to get. Assume that there's a server that handles requests $r$ and returns a set of items $I_{r}$ of random ...
1 vote
2k views

### In sample splitting for time series data, do we randomly select data?

I'm having a hard to conceptually understanding how to do this. I would like to do my own sample splitting (not the method built into a package). Let's say you have 80 days of weather data. You ...
• 2,513
111 views

### Regression hypothesis testing via out-of-sample testing

Let's consider two linear models. \text{Full model}\\\mathbb{E}\big{[}Y\big{\vert} X_1,\dots,X_p, X_{p+1},\dots,X_{p+k}\big{]}=\\\beta_0 + \bigg[\beta_1X_1+\dots + \beta_pX_p\bigg] + \bigg[\beta_{p+...
• 31.3k
2k views

### Is there any reason to factor in sample weights when applying a scoring function to a test set?

It's my understanding that sample weights are used to ensure that each observation used to train a machine learning model are given a weight corresponding to its perceived importance/value to the ...
79 views

...
1 vote
112 views

### Out of sample prediction

I have a model in which I estimate the impact of price on acreage. My data is composed of 10 years. So I use these 10 years to estimate the model and get to coefficients. In next step, I want to use ...
• 11
37 views

### Computation of out-of-sample error

I have a question on how one would theoretically compute the out of sample error of a given hypothesis in a data learning problem. I've been working through Learning From Data: A Short Course (http://...
1 vote
28 views

### Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
1 vote
393 views

### Out-of-sample Rolling window forecast with ARIMA(0,0,0) with non-zero mean

I am doing a rolling window out-of-sample forecast and have fitted an ARIMA(0,0,1) model to a first difference time series. People argue that sometimes simpler models are better than more complicated ...
393 views

### Forecast accuracy rolling window

What is the best way to get a measure of how well an ARIMA model can predict a timeseries when doing an out-of-sample rolling window? I cant use MPE cause it contains zeroes. What I am looking for is ...