Questions tagged [out-of-sample]
Refers to the practice of assessing model performance on some "test" or "holdout" or "out-of-sample" set of data that was not used for model building.
168
questions
0
votes
0
answers
8
views
Hold-out Set -- Smallest Acceptable Size given somewhat rare event?
I am contemplating the appropriate percent train-test split (e.g. 90%-10%, 80%-20%)
Note I am asking about model evaluation, not cross validation/model building. This is for creating a hold out set to ...
0
votes
0
answers
13
views
Generalization bound on out of distribution data
Assume we have two sets of data $X_1$ and $X_2$ drawn from two different distributions. Are the loss of the empirical risk minimizer of $X_1$ on $X_2$: $l_{X_2}(f_{X_1})$ the same as the loss of the ...
- 363
1
vote
0
answers
15
views
Different Estimates but same Out-of-sample predictive accuracy
I have two different models that give me different estimates on my data. The difference is not huge, but one model significantly shrinks the estimates towards zero.
However, when I run leave-one-out ...
- 137
2
votes
1
answer
56
views
How testing (new data points) works in graph neural network
In machine learning, data is divided into train and test splits. The machine learns weights using training data and we can utilize weights to predict test data.
Similarly, we are also learning weights ...
0
votes
0
answers
16
views
Slight out-of-distribution generalization in machine learning
Suppose I have four datasets: $D_{i,train}$ $D_{i,test}$, $D_{j,train}$ and $D_{j,test}$. $D_{i,train}$ and $D_{i,test}$ come from the same distribution (e.g., can obtain them from $D_i$ by train-test ...
- 363
0
votes
0
answers
37
views
How to find the validation R² and RMSE of a caret random forest model with out-of-bag validation?
I trained six random forest models using the train function from the caret R package using out-of-bag valiadation (...
2
votes
1
answer
49
views
Include data for linear terms when predicting interactions?
As a simplified example, assume I have a model with two linear terms and an interaction between them:
y ~ b0 + b1.x1 + b2.x2 + b3.x1*x2
...
- 203
9
votes
2
answers
1k
views
How to motivate the definition of $R^2$ in `sklearn.metrics.r2_score`?
TLDR: What motivates the definition of $R^2$ in the Python function sklearn.metrics.r2_score?
DETAILS
The Python machine learning package ...
- 55.1k
0
votes
0
answers
30
views
Standardization of Out-of-Time variables using lasso regression
I need to run Out-Of-Time predicts in LASSO regression.
In Out-Of-Time sample, what should I use to standardize variables? The mean and SD of the Out-Of-Time sample or the mean and SD of the insample ...
- 51
0
votes
0
answers
73
views
Statistical comparison of two probabilistic classifiers
TL;DR There are well-known tests to compare classifiers. How can we generalize to classifiers with random training steps?
I am comparing two classification algorithms (A and B). I can see that ...
3
votes
0
answers
37
views
How to manage out of sample data in the long run?
For example, you are interested in testing an investment strategy and there is data from 1950 to 2022. So you split it into a train and test set, say 1950-2000 and 2000-2022. Then you build your model ...
- 31
1
vote
1
answer
33
views
Random Forests- Out of Bag Error Calculation
I was learning about the Out of Bag error in random forests and I did not understand a point about the error calculation. Assume we have N bootstraps and there are a number of Out-Of-Bag samples for ...
- 145
3
votes
1
answer
91
views
Creating a holdout set just for feature engineering?
I recently encountered a feature engineering technique that I haven't seen before:
Create the usual training, validation, and test sets.
Create another set by splitting the train set; call this the &...
- 11.8k
0
votes
0
answers
125
views
Poor out of time performance
I am working on a behavioral model which predicts the probability of default (PD) during the next 12 months for an existing customer with an outstanding loan. My dataset consists of monthly snapshots ...
- 11
1
vote
1
answer
92
views
How to be sure that out-of-sample results of one model are really better than those of another model?
I need to solve a standard, let say "vanilla", regression problem: I have a 2D array of real-valued features $X$ and a 1D array of real-valued targets $Y$.
I choose a simple model to fit (...
- 544
0
votes
0
answers
117
views
Comparing logistic regression models from two data sets when a parameter isn't varied in one data set but is in the other
I want to determine if a logistic regression model makes good predictions for a data set not used in its fitting with a hypothesis test; I'll call it the "new" data. One could say that the ...
- 1,313
1
vote
0
answers
43
views
What should unseen data comply mathematically for reliable measure of generalisation (gap)?
The way machine learning models validated for their generalisation performance, generalisation gap is used, see Predicting the Generalization Gap in Deep Neural Networks. This is essentially standard ...
- 2,456
1
vote
1
answer
192
views
Evaluating a multi-step forecasting model?
The literature is a bit confusing for me on this one, from what I understand, a great deal of papers evaluate multi-step forecasting models on a single forecasting horizon on the hold out set. It ...
- 123
1
vote
1
answer
1k
views
In-Sample and Out-of-sample forecasting accuracy
I am currently doing my college final project. I forecasted national soybeans yield and used MAPE to calculate the in-sample and out-of-sample forecasting accuracy. The MAPE results showed that the in-...
- 13
3
votes
1
answer
70
views
Do in-sample distributional assumptions (e.g. normally distributed residuals) apply also to out-of-sample predictions?
I appreciate that out-of-sample error is important (i.e. comparing how close a model's predictions are to leftout data) but are there any distributional assumptions about the out-of-sample predictions?...
- 125
0
votes
0
answers
46
views
Out-of-sample MSE and MAE for volatility forecasting [duplicate]
I have been searching through the whole CrossValidated but couldn't find the answer.
I want to test out-of-sample the volatility forecasts (if it means something ARCH-like ones, MSGARCH, Multifractal ...
- 21
2
votes
0
answers
251
views
Pros and cons of R squared and normalized RMSE as a scale-free out-of-sample performance metric for regression?
I am mainly considering the nRMSE = RMSE/sd, but if you are more familiar with RMSE/mean or RMSE/min-max that would also be interesting. I am looking for pros/cons for the two, or different use cases ...
- 21
2
votes
1
answer
60
views
Bagged Decision trees / Random Forests: why ISLR uses validation set instead of OOB to compute out-of-sample MSE?
I am reading the book "An Introduction to Statistical Learning" available here.
Chapter 8.3.3 at page 328 of the book computes a bagged decision tree (which is a random forest where we use ...
- 2,040
2
votes
0
answers
42
views
State-of-the-art methods for out-of-sample-extension
I'm using a kernel based dimensionality reduction algorithms, and interested in extending out-of-sample data points for further analysis.
I've been using the Nystrom method for this task, and some ...
- 799
0
votes
2
answers
400
views
Optimal Machine Learning Sample size
I am new in ML area and I want to build a model to score 100,000 people.
I wonder if it is abnormal to build (train/test/validation) the model on a dataset of the same size 100,000? Need to mention ...
- 1
0
votes
0
answers
27
views
Harvey-Newbold test (2000) in R
I want to estimate the best nested model out-of sample and for that purpose, I will implement Harvey-Newbold test. However, I cannot find the respective test in any package.
Can anyone help me?
If ...
2
votes
1
answer
354
views
Out of sample ARCH forecast
I have estimated a conditional mean model for a time series:
$ x_t = x_{t-1} + \epsilon_t$.
Say I have estimated it using periods 1 to 10. I can do an out of sample conditional mean forecast by ...
- 1,029
1
vote
1
answer
1k
views
How to calculate the fitted values of out-of-sample predictions from conditional fixed-effects Poisson model?
I am trying to calculate fitted values from xtpoisson fixed effects on out-of-sample data. I know how to calculate fitted values for in-sample predictions (using ...
0
votes
0
answers
105
views
Model evaluation via k-fold-cross validation after variable selection
I am currently trying to implement some methods (OLS, GLM) and compare their predictive accuracy. For that I want to perform variable selection for each of the methods and then measure the predictive ...
- 152
0
votes
1
answer
301
views
Nested CV vs CV with a holdout?
I just learned of nested cross-validation and wanted to understand how my current approach is worse/ok.
Currently I would:
Divide the data into a train/test set (80/20ish).
Use k-fold cross-...
- 298
2
votes
1
answer
145
views
What is the posterior in-sample vs the posterior out-of-sample?
I'm watching this video on Bayesian modelling for the stock market by Thomas Wiecki, Thomas has a slide with two posterior distribution over the mean parameter in his stock return model. Around 18:26 ...
- 595
7
votes
3
answers
1k
views
$R^2$ on out-sample data set
The conventional definition of $R^2$ is:
$R^2 = 1-SSE/SST$, where SSE denotes sum of squared errors and SST is total sum of squares ($n\times variance$, n being number of sample points in train set).
...
- 317
0
votes
0
answers
136
views
What I'm dealing with here VAR, VARX or something different?
I have to implement a time-series tool in my company, and I'm not sure what I'm dealing with, when looking at the old tool. We have one target variable (sales) and a lot of different independent ...
- 1
1
vote
1
answer
47
views
Deciding between linear and non-linear approach in a predictive model when the relationship 'looks linear'
When we have a relatively small number of samples it is easy to see on a plot what is intuitively happening when we fit a regression line; we can see how far each of the individual sample points are ...
- 883
1
vote
0
answers
64
views
An out of bag testing procedure I developed. Is it biased?
Trying to estimate the error of a prediction model trained on a combined group vs a subset of interest within that group (for example, Men)
My procedure is
First simulate a random subset like the ...
- 11
2
votes
1
answer
138
views
Can entropy be used to minimize prediction surprises in machine learning?
Information theory deals with signal/noise identification, while one of its tools, entropy, measures the surprise in random probabilistic outcomes. Has there been any application of using entropy or ...
- 3,481
0
votes
0
answers
12
views
How to pull in-sample fitted models towards out-of-sample optimum during training?
A common problem in prediction problems is getting a model fit on in-sample data to predict out-of-sample data with decent accuracy and precision. Assuming that we break a part of the in-sample data ...
- 3,481
3
votes
0
answers
53
views
Forecasting with empirical copulas
I estimated the beta copula with 3 variable time series. Now I'd like to make forecasts to evaluate the out-of-sample performance of my model. I know 2 of the 3 variables and I have the dependence ...
- 31
1
vote
0
answers
110
views
How to evaluate multiple data imputation?
I am performing data imputation of multiple time-series using various ML techniques (such as multiple imputation, iterative imputation). I have a matrix of ~100,000 observations (rows) of 34 stations (...
- 127
1
vote
1
answer
86
views
Should test MSE be bootstrapped to compare fits?
Suppose you have a training and testing set. You fit two models, A and B, to the training set. They you predict on the testing set. You find (in this contrived example):
Test MSE model A: 3
Test MSE ...
user271536
1
vote
0
answers
56
views
Should I standardize forecasted and actual values for Mincer Zarnowitz test?
I have some out-of-sample forecasted values of variance through some GARCH model, and now I am trying to perform a Mincer-Zarnowitz test for validity of my predictions. I first standardized both ...
- 11
2
votes
2
answers
685
views
Most likely sources of divergence between (adjusted)-R squared and out-of-sample predictive performance
I'm wondering which invalid assumptions are most likely to explain the wild discrepancies between a model's R-squared as a measure of predictive performance, and the actual out-of-sample predictive ...
- 606
2
votes
1
answer
4k
views
How to calculate the confidence interval with weighted data?
I've done some search for similar questions, but they're not the same as what I'm trying to get.
Assume that there's a server that handles requests $r$ and returns a set of items $I_{r}$ of random ...
- 21
2
votes
1
answer
3k
views
In sample splitting for time series data, do we randomly select data?
I'm having a hard to conceptually understanding how to do this. I would like to do my own sample splitting (not the method built into a package).
Let's say you have 80 days of weather data. You ...
- 3,063
3
votes
1
answer
239
views
Regression hypothesis testing via out-of-sample testing
Let's consider two linear models.
$$\text{Full model}\\\mathbb{E}\big{[}Y\big{\vert} X_1,\dots,X_p, X_{p+1},\dots,X_{p+k}\big{]}=\\\beta_0 + \bigg[\beta_1X_1+\dots + \beta_pX_p\bigg] + \bigg[\beta_{p+...
- 55.1k
4
votes
2
answers
4k
views
Is there any reason to factor in sample weights when applying a scoring function to a test set?
It's my understanding that sample weights are used to ensure that each observation used to train a machine learning model are given a weight corresponding to its perceived importance/value to the ...
- 135
3
votes
1
answer
235
views
1
vote
1
answer
176
views
Out of sample prediction
I have a model in which I estimate the impact of price on acreage. My data is composed of 10 years. So I use these 10 years to estimate the model and get to coefficients. In next step, I want to use ...
- 11
2
votes
1
answer
46
views
Computation of out-of-sample error
I have a question on how one would theoretically compute the out of sample error of a given hypothesis in a data learning problem. I've been working through Learning From Data: A Short Course (http://...
1
vote
0
answers
32
views
Why RANDOM noise images always predicted as BIRD?
Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%.
However when feeding into 5000 random noise images (Gaussian noise with the ...
- 11