Skip to main content

Questions tagged [out-of-sample]

Refers to the practice of assessing model performance on some "test" or "holdout" or "out-of-sample" set of data that was not used for model building.

Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

Test/validation set

I've been having a discussion with colleagues and wanted to seek your input. If I'm using holdout and cross-validation to build and test my models. In this process, the training set is used to tune ...
John Doe's user avatar
  • 101
0 votes
0 answers
23 views

Regression with noises in X. Should I use the unbiased estimator or the OLS estimator for forecasting?

I am working with a dataset that includes variables $Y$ and $X$. I assume that $$ Y = \beta X + \epsilon $$ satisfies all the assumptions of OLS. Based on industry knowledge, I know that theoretically ...
The One's user avatar
  • 225
0 votes
0 answers
17 views

Apply crossbasis transformations in DLNM to holdout dataset

I'm using the DLNM package in R to explore the use of distibuted-lag structures in otherwise-fairly simple linear models. Id like to compare the performance of my model to other models using a ...
G. Vece's user avatar
  • 548
3 votes
1 answer
75 views

Out-of-sample R square is NEGATIVE [closed]

The "Out-of-sample $R^2$" is defined as: $$ R^2_{OOS} = 1 - \frac{\sum_{t=\tau}^T\left(Y_t - \hat{Y}_{t\vert t-1}\right)^2}{\sum_{t=\tau}^T\left(Y_t - \hat{\mu}_{t\vert t-1}\right)^2} $$ ...
Alya's user avatar
  • 31
1 vote
1 answer
47 views

MSPE and $R^2_{OOS}$

I've been looking at a paper for a while that I find interesting. It's essentially a comparative analysis where the authors are comparing PCA/PLS to different machine learning methods. The aim is to ...
Nbs610's user avatar
  • 13
0 votes
0 answers
13 views

A surrogate function for validation error in order to perform hyper-parameter optimization?

Greed search CV or few other approaches may be computationally expensive in hyper-parameter tuning. Is it possible to come up with a surrogate model or a purposed model for a validation error in order ...
Lakshman's user avatar
0 votes
0 answers
42 views

Forecasting: choosing the sample split between "in-sample" and "out-of-sample" data

Goals: Given approximately 11 years of time series data, to determine how much of this data should be reserved for in-sample and ...
p.luck's user avatar
  • 101
0 votes
0 answers
28 views

Hold-out Set -- Smallest Acceptable Size given somewhat rare event?

I am contemplating the appropriate percent train-test split (e.g. 90%-10%, 80%-20%) Note I am asking about model evaluation, not cross validation/model building. This is for creating a hold out set to ...
purple-blade's user avatar
1 vote
0 answers
17 views

Different Estimates but same Out-of-sample predictive accuracy

I have two different models that give me different estimates on my data. The difference is not huge, but one model significantly shrinks the estimates towards zero. However, when I run leave-one-out ...
Zlo's user avatar
  • 137
2 votes
1 answer
221 views

How testing (new data points) works in graph neural network

In machine learning, data is divided into train and test splits. The machine learns weights using training data and we can utilize weights to predict test data. Similarly, we are also learning weights ...
Pragnesh Rana's user avatar
2 votes
1 answer
66 views

Include data for linear terms when predicting interactions?

As a simplified example, assume I have a model with two linear terms and an interaction between them: y ~ b0 + b1.x1 + b2.x2 + b3.x1*x2 ...
EcologyTom's user avatar
13 votes
2 answers
2k views

How to motivate the definition of $R^2$ in `sklearn.metrics.r2_score`?

TLDR: What motivates the definition of $R^2$ in the Python function sklearn.metrics.r2_score? DETAILS The Python machine learning package ...
Dave's user avatar
  • 64.9k
0 votes
0 answers
30 views

Standardization of Out-of-Time variables using lasso regression

I need to run Out-Of-Time predicts in LASSO regression. In Out-Of-Time sample, what should I use to standardize variables? The mean and SD of the Out-Of-Time sample or the mean and SD of the insample ...
Valt Yo's user avatar
  • 51
0 votes
0 answers
82 views

Statistical comparison of two probabilistic classifiers

TL;DR There are well-known tests to compare classifiers. How can we generalize to classifiers with random training steps? I am comparing two classification algorithms (A and B). I can see that ...
independentvariable's user avatar
3 votes
0 answers
40 views

How to manage out of sample data in the long run?

For example, you are interested in testing an investment strategy and there is data from 1950 to 2022. So you split it into a train and test set, say 1950-2000 and 2000-2022. Then you build your model ...
azertqwert's user avatar
2 votes
1 answer
41 views

Random Forests- Out of Bag Error Calculation

I was learning about the Out of Bag error in random forests and I did not understand a point about the error calculation. Assume we have N bootstraps and there are a number of Out-Of-Bag samples for ...
levitatmas's user avatar
3 votes
1 answer
111 views

Creating a holdout set just for feature engineering?

I recently encountered a feature engineering technique that I haven't seen before: Create the usual training, validation, and test sets. Create another set by splitting the train set; call this the &...
shadowtalker's user avatar
  • 12.8k
0 votes
0 answers
144 views

Poor out of time performance

I am working on a behavioral model which predicts the probability of default (PD) during the next 12 months for an existing customer with an outstanding loan. My dataset consists of monthly snapshots ...
Saman's user avatar
  • 11
1 vote
1 answer
99 views

How to be sure that out-of-sample results of one model are really better than those of another model?

I need to solve a standard, let say "vanilla", regression problem: I have a 2D array of real-valued features $X$ and a 1D array of real-valued targets $Y$. I choose a simple model to fit (...
Roman's user avatar
  • 612
0 votes
0 answers
136 views

Comparing logistic regression models from two data sets when a parameter isn't varied in one data set but is in the other

I want to determine if a logistic regression model makes good predictions for a data set not used in its fitting with a hypothesis test; I'll call it the "new" data. One could say that the ...
cgmil's user avatar
  • 1,373
1 vote
0 answers
44 views

What should unseen data comply mathematically for reliable measure of generalisation (gap)?

The way machine learning models validated for their generalisation performance, generalisation gap is used, see Predicting the Generalization Gap in Deep Neural Networks. This is essentially standard ...
patagonicus's user avatar
  • 2,550
1 vote
1 answer
252 views

Evaluating a multi-step forecasting model?

The literature is a bit confusing for me on this one, from what I understand, a great deal of papers evaluate multi-step forecasting models on a single forecasting horizon on the hold out set. It ...
Grinjero's user avatar
  • 123
1 vote
1 answer
2k views

In-Sample and Out-of-sample forecasting accuracy

I am currently doing my college final project. I forecasted national soybeans yield and used MAPE to calculate the in-sample and out-of-sample forecasting accuracy. The MAPE results showed that the in-...
adin's user avatar
  • 13
4 votes
1 answer
88 views

Do in-sample distributional assumptions (e.g. normally distributed residuals) apply also to out-of-sample predictions?

I appreciate that out-of-sample error is important (i.e. comparing how close a model's predictions are to leftout data) but are there any distributional assumptions about the out-of-sample predictions?...
user_15's user avatar
  • 185
0 votes
0 answers
47 views

Out-of-sample MSE and MAE for volatility forecasting [duplicate]

I have been searching through the whole CrossValidated but couldn't find the answer. I want to test out-of-sample the volatility forecasts (if it means something ARCH-like ones, MSGARCH, Multifractal ...
Selena Pepic's user avatar
2 votes
0 answers
283 views

Pros and cons of R squared and normalized RMSE as a scale-free out-of-sample performance metric for regression?

I am mainly considering the nRMSE = RMSE/sd, but if you are more familiar with RMSE/mean or RMSE/min-max that would also be interesting. I am looking for pros/cons for the two, or different use cases ...
Marcus's user avatar
  • 21
2 votes
1 answer
77 views

Bagged Decision trees / Random Forests: why ISLR uses validation set instead of OOB to compute out-of-sample MSE?

I am reading the book "An Introduction to Statistical Learning" available here. Chapter 8.3.3 at page 328 of the book computes a bagged decision tree (which is a random forest where we use ...
robertspierre's user avatar
2 votes
0 answers
62 views

State-of-the-art methods for out-of-sample-extension

I'm using a kernel based dimensionality reduction algorithms, and interested in extending out-of-sample data points for further analysis. I've been using the Nystrom method for this task, and some ...
Roy's user avatar
  • 839
0 votes
2 answers
561 views

Optimal Machine Learning Sample size

I am new in ML area and I want to build a model to score 100,000 people. I wonder if it is abnormal to build (train/test/validation) the model on a dataset of the same size 100,000? Need to mention ...
celo's user avatar
  • 1
0 votes
0 answers
33 views

Harvey-Newbold test (2000) in R

I want to estimate the best nested model out-of sample and for that purpose, I will implement Harvey-Newbold test. However, I cannot find the respective test in any package. Can anyone help me? If ...
Francisco Antunes's user avatar
2 votes
1 answer
440 views

Out of sample ARCH forecast

I have estimated a conditional mean model for a time series: $ x_t = x_{t-1} + \epsilon_t$. Say I have estimated it using periods 1 to 10. I can do an out of sample conditional mean forecast by ...
shenflow's user avatar
  • 1,119
1 vote
1 answer
1k views

How to calculate the fitted values of out-of-sample predictions from conditional fixed-effects Poisson model?

I am trying to calculate fitted values from xtpoisson fixed effects on out-of-sample data. I know how to calculate fitted values for in-sample predictions (using ...
Bryan E. Burke's user avatar
0 votes
0 answers
110 views

Model evaluation via k-fold-cross validation after variable selection

I am currently trying to implement some methods (OLS, GLM) and compare their predictive accuracy. For that I want to perform variable selection for each of the methods and then measure the predictive ...
n_arch's user avatar
  • 152
0 votes
1 answer
396 views

Nested CV vs CV with a holdout?

I just learned of nested cross-validation and wanted to understand how my current approach is worse/ok. Currently I would: Divide the data into a train/test set (80/20ish). Use k-fold cross-...
Josh's user avatar
  • 308
2 votes
1 answer
175 views

What is the posterior in-sample vs the posterior out-of-sample?

I'm watching this video on Bayesian modelling for the stock market by Thomas Wiecki, Thomas has a slide with two posterior distribution over the mean parameter in his stock return model. Around 18:26 ...
user27886's user avatar
  • 615
7 votes
3 answers
2k views

$R^2$ on out-sample data set

The conventional definition of $R^2$ is: $R^2 = 1-SSE/SST$, where SSE denotes sum of squared errors and SST is total sum of squares ($n\times variance$, n being number of sample points in train set). ...
Maaz's user avatar
  • 317
0 votes
0 answers
167 views

What I'm dealing with here VAR, VARX or something different?

I have to implement a time-series tool in my company, and I'm not sure what I'm dealing with, when looking at the old tool. We have one target variable (sales) and a lot of different independent ...
Patrick's user avatar
1 vote
1 answer
62 views

Deciding between linear and non-linear approach in a predictive model when the relationship 'looks linear'

When we have a relatively small number of samples it is easy to see on a plot what is intuitively happening when we fit a regression line; we can see how far each of the individual sample points are ...
ManUtdBloke's user avatar
1 vote
0 answers
64 views

An out of bag testing procedure I developed. Is it biased?

Trying to estimate the error of a prediction model trained on a combined group vs a subset of interest within that group (for example, Men) My procedure is First simulate a random subset like the ...
John Doe's user avatar
2 votes
1 answer
177 views

Can entropy be used to minimize prediction surprises in machine learning?

Information theory deals with signal/noise identification, while one of its tools, entropy, measures the surprise in random probabilistic outcomes. Has there been any application of using entropy or ...
develarist's user avatar
  • 4,025
0 votes
0 answers
12 views

How to pull in-sample fitted models towards out-of-sample optimum during training?

A common problem in prediction problems is getting a model fit on in-sample data to predict out-of-sample data with decent accuracy and precision. Assuming that we break a part of the in-sample data ...
develarist's user avatar
  • 4,025
3 votes
0 answers
59 views

Forecasting with empirical copulas

I estimated the beta copula with 3 variable time series. Now I'd like to make forecasts to evaluate the out-of-sample performance of my model. I know 2 of the 3 variables and I have the dependence ...
user13201583's user avatar
1 vote
0 answers
117 views

How to evaluate multiple data imputation?

I am performing data imputation of multiple time-series using various ML techniques (such as multiple imputation, iterative imputation). I have a matrix of ~100,000 observations (rows) of 34 stations (...
iditbela's user avatar
  • 127
1 vote
1 answer
114 views

Should test MSE be bootstrapped to compare fits?

Suppose you have a training and testing set. You fit two models, A and B, to the training set. They you predict on the testing set. You find (in this contrived example): Test MSE model A: 3 Test MSE ...
user avatar
1 vote
0 answers
63 views

Should I standardize forecasted and actual values for Mincer Zarnowitz test?

I have some out-of-sample forecasted values of variance through some GARCH model, and now I am trying to perform a Mincer-Zarnowitz test for validity of my predictions. I first standardized both ...
newbiee's user avatar
  • 11
2 votes
2 answers
805 views

Most likely sources of divergence between (adjusted)-R squared and out-of-sample predictive performance

I'm wondering which invalid assumptions are most likely to explain the wild discrepancies between a model's R-squared as a measure of predictive performance, and the actual out-of-sample predictive ...
Mark Verhagen's user avatar
2 votes
1 answer
6k views

How to calculate the confidence interval with weighted data?

I've done some search for similar questions, but they're not the same as what I'm trying to get. Assume that there's a server that handles requests $r$ and returns a set of items $I_{r}$ of random ...
Awdrtg's user avatar
  • 21
3 votes
1 answer
3k views

In sample splitting for time series data, do we randomly select data?

I'm having a hard to conceptually understanding how to do this. I would like to do my own sample splitting (not the method built into a package). Let's say you have 80 days of weather data. You ...
confused's user avatar
  • 3,273
3 votes
1 answer
334 views

Regression hypothesis testing via out-of-sample testing

Let's consider two linear models. $$\text{Full model}\\\mathbb{E}\big{[}Y\big{\vert} X_1,\dots,X_p, X_{p+1},\dots,X_{p+k}\big{]}=\\\beta_0 + \bigg[\beta_1X_1+\dots + \beta_pX_p\bigg] + \bigg[\beta_{p+...
Dave's user avatar
  • 64.9k
4 votes
2 answers
5k views

Is there any reason to factor in sample weights when applying a scoring function to a test set?

It's my understanding that sample weights are used to ensure that each observation used to train a machine learning model are given a weight corresponding to its perceived importance/value to the ...
pmse234's user avatar
  • 135