Questions tagged [out-of-sample]

Refers to the practice of assessing model performance on some "test" or "holdout" or "out-of-sample" set of data that was not used for model building.

0
votes
2answers
40 views

What do we learn from a test set?

Suppose I split my data into two parts -- a training set (having 80% of my data) and a testing (20%) set. I train a model on my training set, and test it on the test set. What do we learn from ...
2
votes
1answer
498 views

Precision, Recall and area under ROC curve as sample size increases

The following is a question from an exam paper on evaluating the performance of search engines. To this day I looked in my text book and literally close to 50 web pages and I can't find one convincing ...
1
vote
1answer
52 views

Forecasting with AR(1) and pseudo out-of-sample using R

I'm trying to do Pseudo out-of-sample forecasting using R. And, I also have the following initial data (gdp) ...
1
vote
1answer
44 views

Asymmetric error measure for forecasts

I am building a model for forecasting some number of activations. My data set has a panel structure. Now, I want to come up with a forecast performance measure to assess the performance of my model ...
0
votes
1answer
29 views

Is there a systematic reason why a model trained on a subset of data does better out-of-sample than the same model trained on the full dataset?

I trained a linear regression model using 3000 data points. (OLS regression, no regularization.) Then I trained another model with the same predictors (about 25), but with a subset ($n=700$) of the ...
0
votes
1answer
810 views

Which one should I use for rolling forecast, dynamic or static?

I'm doing a rolling forecast using a fitted arma-garch model, but I'm confused regarding the rolling method, my window length is 1209 obs, and I roll 100 times, and each time I reset my window to ...
5
votes
0answers
152 views

Is there justification for using cross validation scores as model averaging weights?

Bayesian model averaging uses approximate Bayes factors. Some researchers use AIC to weight models. Is there justification for using, say, the Brier score, median absolute deviation, or other such ...
3
votes
0answers
282 views

Nested Cross Validation: Choosing between different best hyperparameters

I know this sort of question has been asked many times, and several answers have been already provided on this platform too (e.g., here, here, and here). Still, there is something about the idea ...
3
votes
0answers
87 views

Cross validation or EM for selecting strength of the prior?

Often when I'm looking at bayesian analyses, the influence of the prior is chosen via cross validation. For example, suppose $X$ and $Y$ represent some real valued data that I want perform a bayesian ...
3
votes
0answers
254 views

BIC vs. Out of sample performance

I have two statistical models. Model 1 uses a GLM approach while model 2 uses a time series approach for fitting. I want to compare these two models. Model 1 (i.e. GLM) has a better out of sample ...
3
votes
0answers
223 views

Can holdout validation be systematically biased?

I recently did some experimenting comparing some common method of internal validation. In my field, the use of a single 1:1 holdout validation is extremely common, even with very small datasets, and I ...
2
votes
0answers
29 views

Theoretical question on GBM out-of-time performance sensitivity

So this is more of a theoretical question, no dataset or code that I can share. It just came up in a discussion and I was not sure of the answer. Let's say I have 2 GBM models, model A and B, trained ...
2
votes
0answers
100 views

Is training multiple Random Forests equivalent to a repeated 3-fold cross-validation?

In the book "Elements of Statistical Learning" (Friedman, Hastie, Tibshirani) , the authors suggest that bootstrap and the resulting out-of-bag data for a random forest are equivalent to a 3-fold ...
2
votes
0answers
1k views

Kendall's tau for holdouts low and not significant - Conjoint

I have done my Conjoint Analysis (fractional factorial design) but when it comes to validating the model, it shows a Kendall's tau for Holdouts of 0.33 and not significant. But Pearson's R and Kendall'...
2
votes
0answers
73 views

Is this a valid method to control the FWER?

I have a huge number, say $M$, of hypotheses that are potentially correlated. I have a dataset $D$ of random samples from an unknown distribution and I want to do test the hypotheses for significance ...
1
vote
0answers
23 views

Out-of-sample predictive checks for Bayesian TVP models

Comparatively new to Bayesian econometrics so apologies if this is a silly question. I am running a time-varying parameter regression where the parameters are estimated as in Primiceri (2005). My ...
1
vote
0answers
206 views

Why using Out-of-fold predictions as metafeatures in stacking?

So my question is essentially the same as this one: Why do we generate out-of-fold predictions for meta-ensembling/stacking? However, I am not entirely satisfied with the answer (not detailed enough ...
1
vote
0answers
45 views

What is the relation between replica method and “reusable holdout” method?

Among many methods used to detect and avoid overfitting, I am particularly interested in those two: replica method reusable holdout My question is: what is their relation in the context of adaptive ...
1
vote
0answers
399 views

prediction using plm for out of sample data in R

I want to predict out of sample data for the same group (lets say state ) for new time window by either fixed or random method.."predict" function is not helping.Here i gave a example of a dataset ...
1
vote
0answers
138 views

Data Partition for In sample and out of sample forecasting in neural network

I got confused with how to do data partition that reflects in sample and out of sample forecast when I do time series forecasting in neural network. What I understand is we have to divide data into: -...
1
vote
0answers
592 views

How to obtain the same results of a random forest model using caret and randomForest?

I am trying to understand how does building a regression model with caret's train () function differs from randomForest(). For my excercise, I am using the iris dataset. As shown in the code below, I ...
1
vote
0answers
95 views

Can you compare in-sample vs. out-of-sample using a MSPE?

Is it possible to compare in-sample and out-of-sample forecasts by calculating a MSPE for each? For example, my in sample period is say 12/1987-12/2015, and my out-of-sample forecasting period is 12/...
1
vote
0answers
42 views

Inference/Prediction using a skewed sample of a population?

In statistics, the number one assumption for inference/prediction is to have a random sample that represents the target population. What if our sample is a skewed sample that does not truly represent ...
1
vote
0answers
112 views

Measure quality of out-of-sample extension for spectral embedding methods (Laplacian eigenmaps)

For my work, I analyse a rather new method for out-of-sample extension of spectral embedding methods (mostly Laplacian Eigenmaps). Instead of just providing a discrete embedding for every data point, ...
1
vote
0answers
2k views

What is the difference between oob (out of bag) error and (1 - accuracy) in RandomForest?

In a Random Forest, I know that the Out Of Bag Error is described as the fraction of number incorrect classifications over number of out of bag samples. Accuracy is defined as the number of correct ...
1
vote
0answers
41 views

Out Of Time variance test method and threshold for logistic regression models

I am interested in designing a test statistic for checking the out-of-time variance of a logistic regression model relative to the model development sample at an observation level. We did something ...
1
vote
0answers
152 views

How do I Forecast new Yts given new Xt's using a Dynamic Linear Model?

I am trying to forecast predict new observations of interest rates given new data using the DLM modeling framework. Essentially, my problem is this: I have a training set (a set of data i want to ...
1
vote
0answers
4k views

In-Sample vs. Out-of-Sample One-Step Ahead Forecasts

I'm fairly new to forecasting but I find all of this quote fascinating and hope to learn something from all of you. I have 500 observations and I'm tasked with the following: "compute recursive (...
1
vote
0answers
335 views

Out-of-sample vs. in-sample interpretation

I am running predictive regressions on stock returns and as expected less relations hold out-of-sample than in-sample, however in some cases I find a significant relation out-of-sample but not in-...
0
votes
0answers
11 views

What is the formula that is used to calculated the MSE with Random Forest regression in R?

I am using the package randomForest in R for panel-data on conflict intensity. The dependent variable is the conflict intensity (e.g. the number of battle deaths). Independent variables are population,...
0
votes
0answers
28 views

Bootstrapping vs. K-Fold: Is every data point in atleast one of the test set/out of bag - atleast once?

It's easy to see that in K-Fold cross-validation, that split training examples into K parts, in such a way that 1 of the K parts is considered to be the test set, and eventually as you shift which ...
0
votes
0answers
47 views

Classification accuracy in holdout similar to CV if set is randomly sampled, completely wrong otherwise

I'm building a classifier to predict a binary label on a dataset with 30 features and around 60000 samples of measurements from a car assembly process. While experimenting with some baseline models ...
0
votes
0answers
12 views

Is it appropriate to predict a trained KMeans model on holdout data that would not be included in the training set?

I have a KMeans model that is trained on features that are percentage-transformed descriptions of events. Each observation contains between 1 and 180 events. To help with meaningful comparisons, I ...
0
votes
0answers
28 views

Multivariate Out-of-Sample Evaluation

I have a question about multivariate hypothesis testing in out-of-sample evaluations. Generally, let`s asssume we want to predict three different stock returns in a 1-month ahead forecasting setting. ...
0
votes
0answers
30 views

Using model from one data set to predict results for another data set

I'm not certain how to phrase this question: I have a dataset of ~45000 execution times of two sets of data. Approximately 35000 of these execution times is ran in one environment, and the remaining ~...
0
votes
0answers
42 views

Can I duplicate every element of my dataset to get around some issues of small datasets?

Take this question with many grains of salt, because it's mostly a theoretical curiosity. I've built a multinomial classifier using GLMnet. The problem is that some of of my input variables have an ...
0
votes
0answers
348 views

SPSS: How to get error measurements (RMSE, MAE, MSE etc.) on holdout sample?

I split my data into "estimation" and "holdout" sample, using "Select Cases" (and select only the cases I want to use as the estimation sample). Then, I created a time series using Expert Modeler in ...
0
votes
0answers
39 views

Out-of-bag sampling from same distribution

The purpose of out-of-bag sampling is to test your model on an unseen data. However, if we have a very large dataset say 5 milllion observations, when the out-of-bag sampling follows the same ...
0
votes
0answers
86 views

evaluating out of sample accuracy

I estimate a linear regression model and compute the variance of residuals in both the training-set and also on an additional test set. Ideally these should not be very different. Does it make sense ...
0
votes
0answers
204 views

mape and mae from k-fold backtesting on time series

I performed a rolling window (i.e. do full sample, then next 4 observations until the last, and so on...) k-fold test for out of sample testing due to limited number of observations. From the MAPE ...
0
votes
0answers
169 views

Comparing PC and PLS regression performance on the test data depending on the number of components

I am performing PLS regression but have noticed that the R2 between predicted and measured values for the test set decreases with increasing the number of components. If I do principal components ...
0
votes
0answers
159 views

Why would feature scaling cause overfitting?

I'm fitting a logistic regression to a simple two-feature dataset. The feature values range from about 1 to 100. When I scale the features (using scikit-learn StandardScaler), the in-sample ...