Questions tagged [out-of-sample]

Refers to the practice of assessing model performance on some "test" or "holdout" or "out-of-sample" set of data that was not used for model building.

Filter by
Sorted by
Tagged with
1 vote
0 answers
14 views

How to manage out of sample data in the long run?

For example, you are interested in testing an investment strategy and there is data from 1950 to 2022. So you split it into a train and test set, say 1950-2000 and 2000-2022. Then you build your model ...
user avatar
1 vote
1 answer
16 views

Random Forests- Out of Bag Error Calculation

I was learning about the Out of Bag error in random forests and I did not understand a point about the error calculation. Assume we have N bootstraps and there are a number of Out-Of-Bag samples for ...
user avatar
0 votes
0 answers
12 views

Out of Sample Regression errors

I am trying to compute $\text{R}^2$ and $ {delta RMSE} $ from an Out of Sample Linear Model in R. $ e _{ N }$ is the vector of rolling OOS errors from the historical mean model $ e_{A}$ is the vector ...
user avatar
  • 1
2 votes
1 answer
33 views

Creating a holdout set just for feature engineering?

I recently encountered a feature engineering technique that I haven't seen before: Create the usual training, validation, and test sets. Create another set by splitting the train set; call this the &...
user avatar
  • 11.5k
0 votes
0 answers
52 views

Poor out of time performance

I am working on a behavioral model which predicts the probability of default (PD) during the next 12 months for an existing customer with an outstanding loan. My dataset consists of monthly snapshots ...
user avatar
  • 11
0 votes
0 answers
6 views

Do I need to separate unique IDs when doing out-of-time validation?

I'm doing a credit risk model and I thought that would be a good idea to validate out-of-time in multiple time periods (merging them to a big unique validation set). Each line of my dataset ...
user avatar
0 votes
0 answers
87 views

How to correctly diagnose overfitting using all information: training set, validation set and test set?

I understand that overfitting is typically defined/described as the relationship between training set error and test set error - that overfitting is when a model performs significantly worse out-of-...
user avatar
1 vote
1 answer
68 views

How to be sure that out-of-sample results of one model are really better than those of another model?

I need to solve a standard, let say "vanilla", regression problem: I have a 2D array of real-valued features $X$ and a 1D array of real-valued targets $Y$. I choose a simple model to fit (...
user avatar
  • 473
0 votes
0 answers
23 views

Difference in model result when scoring data

I am trying to predict/score on an out-of-time data applying the same model I had used to train and predict on test data. In the previous model, the recall and precision for correctly predicting 1s ...
user avatar
0 votes
0 answers
16 views

Holdout and the Multiple Comparisons Problem

I apologize in advance if this is an extremely naive question. I am attempting to analyze a large body of discretized event data (of the form "did this event occur on date X"). This data is ...
user avatar
0 votes
0 answers
14 views

Sample Period for Model Selection?

I am currently trying to perform pseudo-out-of-sample forecasting for monthly exchange rates with a 10-year rolling window. Before that, I select an autoregressive model using Box Jenkins ...
user avatar
0 votes
0 answers
61 views

Comparing logistic regression models from two data sets when a parameter isn't varied in one data set but is in the other

I want to determine if a logistic regression model makes good predictions for a data set not used in its fitting with a hypothesis test; I'll call it the "new" data. One could say that the ...
user avatar
  • 1,063
1 vote
0 answers
41 views

What should unseen data comply mathematically for reliable measure of generalisation (gap)?

The way machine learning models validated for their generalisation performance, generalisation gap is used, see Predicting the Generalization Gap in Deep Neural Networks. This is essentially standard ...
user avatar
  • 1,838
0 votes
0 answers
11 views

Rigorous uncertainty estimation resulting from out-of-distribution samples

Let $\mathbf Y_{train} = \left(\mathbf y_1^{\text{T}};...;\mathbf y_N^{\text{T}} \right) $ with $\mathbf y_i = \begin{pmatrix} y_{i,1} \\ y_{i,2} \end{pmatrix}$, such that the first coordinate ...
user avatar
1 vote
1 answer
62 views

Evaluating a multi-step forecasting model?

The literature is a bit confusing for me on this one, from what I understand, a great deal of papers evaluate multi-step forecasting models on a single forecasting horizon on the hold out set. It ...
user avatar
  • 123
1 vote
1 answer
322 views

In-Sample and Out-of-sample forecasting accuracy

I am currently doing my college final project. I forecasted national soybeans yield and used MAPE to calculate the in-sample and out-of-sample forecasting accuracy. The MAPE results showed that the in-...
user avatar
  • 13
4 votes
1 answer
47 views

Do in-sample distributional assumptions (e.g. normally distributed residuals) apply also to out-of-sample predictions?

I appreciate that out-of-sample error is important (i.e. comparing how close a model's predictions are to leftout data) but are there any distributional assumptions about the out-of-sample predictions?...
user avatar
  • 135
0 votes
0 answers
37 views

Out-of-sample MSE and MAE for volatility forecasting [duplicate]

I have been searching through the whole CrossValidated but couldn't find the answer. I want to test out-of-sample the volatility forecasts (if it means something ARCH-like ones, MSGARCH, Multifractal ...
user avatar
2 votes
0 answers
111 views

Pros and cons of R squared and normalized RMSE as a scale-free out-of-sample performance metric for regression?

I am mainly considering the nRMSE = RMSE/sd, but if you are more familiar with RMSE/mean or RMSE/min-max that would also be interesting. I am looking for pros/cons for the two, or different use cases ...
user avatar
  • 21
1 vote
1 answer
32 views

Bagged Decision trees / Random Forests: why ISLR uses validation set instead of OOB to compute out-of-sample MSE?

I am reading the book "An Introduction to Statistical Learning" available here. Chapter 8.3.3 at page 328 of the book computes a bagged decision tree (which is a random forest where we use ...
user avatar
2 votes
0 answers
25 views

State-of-the-art methods for out-of-sample-extension

I'm using a kernel based dimensionality reduction algorithms, and interested in extending out-of-sample data points for further analysis. I've been using the Nystrom method for this task, and some ...
user avatar
  • 729
0 votes
2 answers
151 views

Optimal Machine Learning Sample size

I am new in ML area and I want to build a model to score 100,000 people. I wonder if it is abnormal to build (train/test/validation) the model on a dataset of the same size 100,000? Need to mention ...
user avatar
  • 1
0 votes
0 answers
19 views

Harvey-Newbold test (2000) in R

I want to estimate the best nested model out-of sample and for that purpose, I will implement Harvey-Newbold test. However, I cannot find the respective test in any package. Can anyone help me? If ...
user avatar
2 votes
1 answer
159 views

Out of sample ARCH forecast

I have estimated a conditional mean model for a time series: $ x_t = x_{t-1} + \epsilon_t$. Say I have estimated it using periods 1 to 10. I can do an out of sample conditional mean forecast by ...
user avatar
  • 860
1 vote
1 answer
579 views

How to calculate the fitted values of out-of-sample predictions from conditional fixed-effects Poisson model?

I am trying to calculate fitted values from xtpoisson fixed effects on out-of-sample data. I know how to calculate fitted values for in-sample predictions (using ...
user avatar
0 votes
0 answers
78 views

Model evaluation via k-fold-cross validation after variable selection

I am currently trying to implement some methods (OLS, GLM) and compare their predictive accuracy. For that I want to perform variable selection for each of the methods and then measure the predictive ...
user avatar
  • 129
0 votes
1 answer
157 views

Nested CV vs CV with a holdout?

I just learned of nested cross-validation and wanted to understand how my current approach is worse/ok. Currently I would: Divide the data into a train/test set (80/20ish). Use k-fold cross-...
user avatar
  • 206
2 votes
1 answer
89 views

What is the posterior in-sample vs the posterior out-of-sample?

I'm watching this video on Bayesian modelling for the stock market by Thomas Wiecki, Thomas has a slide with two posterior distribution over the mean parameter in his stock return model. Around 18:26 ...
user avatar
  • 595
0 votes
0 answers
52 views

What I'm dealing with here VAR, VARX or something different?

I have to implement a time-series tool in my company, and I'm not sure what I'm dealing with, when looking at the old tool. We have one target variable (sales) and a lot of different independent ...
user avatar
1 vote
1 answer
28 views

Deciding between linear and non-linear approach in a predictive model when the relationship 'looks linear'

When we have a relatively small number of samples it is easy to see on a plot what is intuitively happening when we fit a regression line; we can see how far each of the individual sample points are ...
user avatar
1 vote
0 answers
61 views

An out of bag testing procedure I developed. Is it biased?

Trying to estimate the error of a prediction model trained on a combined group vs a subset of interest within that group (for example, Men) My procedure is First simulate a random subset like the ...
user avatar
2 votes
1 answer
62 views

Can entropy be used to minimize prediction surprises in machine learning?

Information theory deals with signal/noise identification, while one of its tools, entropy, measures the surprise in random probabilistic outcomes. Has there been any application of using entropy or ...
user avatar
  • 3,067
0 votes
0 answers
12 views

How to pull in-sample fitted models towards out-of-sample optimum during training?

A common problem in prediction problems is getting a model fit on in-sample data to predict out-of-sample data with decent accuracy and precision. Assuming that we break a part of the in-sample data ...
user avatar
  • 3,067
3 votes
0 answers
40 views

Forecasting with empirical copulas

I estimated the beta copula with 3 variable time series. Now I'd like to make forecasts to evaluate the out-of-sample performance of my model. I know 2 of the 3 variables and I have the dependence ...
user avatar
1 vote
0 answers
69 views

How to evaluate multiple data imputation?

I am performing data imputation of multiple time-series using various ML techniques (such as multiple imputation, iterative imputation). I have a matrix of ~100,000 observations (rows) of 34 stations (...
user avatar
  • 117
1 vote
1 answer
28 views

Should test MSE be bootstrapped to compare fits?

Suppose you have a training and testing set. You fit two models, A and B, to the training set. They you predict on the testing set. You find (in this contrived example): Test MSE model A: 3 Test MSE ...
user avatar
1 vote
0 answers
43 views

Should I standardize forecasted and actual values for Mincer Zarnowitz test?

I have some out-of-sample forecasted values of variance through some GARCH model, and now I am trying to perform a Mincer-Zarnowitz test for validity of my predictions. I first standardized both ...
user avatar
  • 11
1 vote
2 answers
225 views

Most likely sources of divergence between (adjusted)-R squared and out-of-sample predictive performance

I'm wondering which invalid assumptions are most likely to explain the wild discrepancies between a model's R-squared as a measure of predictive performance, and the actual out-of-sample predictive ...
user avatar
0 votes
1 answer
2k views

How to calculate the confidence interval with weighted data?

I've done some search for similar questions, but they're not the same as what I'm trying to get. Assume that there's a server that handles requests $r$ and returns a set of items $I_{r}$ of random ...
user avatar
  • 1
1 vote
1 answer
2k views

In sample splitting for time series data, do we randomly select data?

I'm having a hard to conceptually understanding how to do this. I would like to do my own sample splitting (not the method built into a package). Let's say you have 80 days of weather data. You ...
user avatar
  • 2,513
3 votes
1 answer
111 views

Regression hypothesis testing via out-of-sample testing

Let's consider two linear models. $$\text{Full model}\\\mathbb{E}\big{[}Y\big{\vert} X_1,\dots,X_p, X_{p+1},\dots,X_{p+k}\big{]}=\\\beta_0 + \bigg[\beta_1X_1+\dots + \beta_pX_p\bigg] + \bigg[\beta_{p+...
user avatar
  • 31.3k
3 votes
2 answers
2k views

Is there any reason to factor in sample weights when applying a scoring function to a test set?

It's my understanding that sample weights are used to ensure that each observation used to train a machine learning model are given a weight corresponding to its perceived importance/value to the ...
user avatar
3 votes
1 answer
79 views

How to use a train set and a test set to check if my model is over-parameterized in R?

...
user avatar
1 vote
1 answer
112 views

Out of sample prediction

I have a model in which I estimate the impact of price on acreage. My data is composed of 10 years. So I use these 10 years to estimate the model and get to coefficients. In next step, I want to use ...
user avatar
  • 11
2 votes
1 answer
37 views

Computation of out-of-sample error

I have a question on how one would theoretically compute the out of sample error of a given hypothesis in a data learning problem. I've been working through Learning From Data: A Short Course (http://...
user avatar
1 vote
0 answers
28 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
user avatar
1 vote
1 answer
393 views

Out-of-sample Rolling window forecast with ARIMA(0,0,0) with non-zero mean

I am doing a rolling window out-of-sample forecast and have fitted an ARIMA(0,0,1) model to a first difference time series. People argue that sometimes simpler models are better than more complicated ...
user avatar
5 votes
3 answers
393 views

Forecast accuracy rolling window

What is the best way to get a measure of how well an ARIMA model can predict a timeseries when doing an out-of-sample rolling window? I cant use MPE cause it contains zeroes. What I am looking for is ...
user avatar
0 votes
0 answers
141 views

Ridge Regression worse results with more feature. Does it make sense?

PREMISE I am dealing with a regression problem with time-series data (of option prices data). In my setup, I need to use only piece-wise linear models or linear transformations of data. I took care ...
user avatar
3 votes
1 answer
895 views

Worthwhile to do k-fold cross-validation AND a holdout/test set?

I'm relatively new to machine learning, and most of my experience at this stage comes from working with an automated machine learning tool called DataRobot. In their tool, and in their documentation ...
user avatar
  • 155