Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [out-of-sample]

Refers to the practice of assessing model performance on some "test" or "holdout" or "out-of-sample" set of data that was not used for model building.

Filter by
Sorted by
Tagged with
1
vote
0answers
28 views

OOB vs CV for Random Forest

I know this question has been asked dozens of times, but I want to really clarify what is going on when finding the best forest using OOB Error versus CV with Accuracy. From my understanding, a Random ...
1
vote
1answer
37 views

Out of sample and In sample forecasting - R squared

Can anyone explain why R2 (R-squared) for out of sample forecasting is likely to be smaller than R2 for in-sample forecasting?
0
votes
0answers
16 views

Random forest “out-of-bag” ensemble

I am using the R package RandomForestSRC for random forest applications. In the manual for the main function (rfsrc) they mention a setting called ...
1
vote
1answer
885 views

Which one should I use for rolling forecast, dynamic or static?

I'm doing a rolling forecast using a fitted arma-garch model, but I'm confused regarding the rolling method, my window length is 1209 obs, and I roll 100 times, and each time I reset my window to ...
6
votes
1answer
5k views

Computing Out of Bag error in Random Forest: is it the average only over trees that didn't use each sample?

I am implementing a Random Forest classifier as a side-project, and I am a bit unclear on what the correct approach is to compute, say, the OOB estimate for the classifier error rate. My ...
1
vote
0answers
47 views

OOB error prediction in RF if case weights are used

I have a dataset for which grossing-up factors are given. I am using these factors as case weights for a random forest (R package ranger). Until now I was using the OOB prediction error for tuning, ...
8
votes
2answers
3k views

How can a smaller learning rate hurt the performance of a gbm?

I've always subscribed to the folk wisdom that decreasing the learning rate in a gbm (gradient boosted tree model) does not hurt the out of sample performance of the model. Today, I'm not so sure. I'...
2
votes
1answer
515 views

Precision, Recall and area under ROC curve as sample size increases

The following is a question from an exam paper on evaluating the performance of search engines. To this day I looked in my text book and literally close to 50 web pages and I can't find one convincing ...
2
votes
1answer
97 views

Is cross-validation better/worse than a third holdout set?

I see lots of papers that use just train and test datasets, without a third validation set, but they use cross-validation so that every data point is used for training and testing among the different ...
1
vote
3answers
72 views

Can a prediction be better with insignificant variables than with only significant variables (or none at all)?

I have two OLS models and want to do an out of sample prediction for wages on a test set. In the first model I excluded the insignificant variable. The second model has the insignificant variable. The ...
5
votes
0answers
154 views

Is there justification for using cross validation scores as model averaging weights?

Bayesian model averaging uses approximate Bayes factors. Some researchers use AIC to weight models. Is there justification for using, say, the Brier score, median absolute deviation, or other such ...
2
votes
2answers
68 views

How to check if i have strong linear relationship between dependent variable and independent variables in linear regression (OLS)?

I want compare the out of sample prediction from an linear regression model (OLS) and a regression tree. I read that OLS outperforms regression tree if the relationship between the dependent variable ...
0
votes
0answers
19 views

What is the formula that is used to calculated the MSE with Random Forest regression in R?

I am using the package randomForest in R for panel-data on conflict intensity. The dependent variable is the conflict intensity (e.g. the number of battle deaths). Independent variables are population,...
1
vote
1answer
545 views

Out-of bag error in Random Forest

I am trying to code my own, simple version of RandomForest function in R for learning purposes. However I have a hard time understanding the concept of the out-of-bag error. Is it simply done by ...
2
votes
1answer
98 views

Can balanced accuracy be higher than accuracy?

I have classification tree where the balanced accuracy of the test set is higher than the normal accuracy. I thought balanced accuracy can only have at his maximum the same value as the accuracy not ...
1
vote
0answers
115 views

ARIMA: Produce multi-step, out-of-sample forecasts by feeding in new history without retraining the model? [closed]

I'd like to compare the results of an LSTM model to an ARIMA model. How can I create an ARIMA model in python that trains on the first 70% of data (~2700 observations), and then produces forecasts at ...
1
vote
1answer
95 views

Forecasting with AR(1) and pseudo out-of-sample using R

I'm trying to do Pseudo out-of-sample forecasting using R. And, I also have the following initial data (gdp) ...
0
votes
0answers
29 views

Bootstrapping vs. K-Fold: Is every data point in atleast one of the test set/out of bag - atleast once?

It's easy to see that in K-Fold cross-validation, that split training examples into K parts, in such a way that 1 of the K parts is considered to be the test set, and eventually as you shift which ...
0
votes
0answers
48 views

Classification accuracy in holdout similar to CV if set is randomly sampled, completely wrong otherwise

I'm building a classifier to predict a binary label on a dataset with 30 features and around 60000 samples of measurements from a car assembly process. While experimenting with some baseline models ...
3
votes
2answers
142 views

What's the real purpose of cross validation?

As for cross evaluation (CV), I have two questions to ask: 1) CV has nothing to do with parameter selection, but only model evaluation? Specifically, which model? 2) In k-fold CV, what's the final ...
1
vote
1answer
56 views

Asymmetric error measure for forecasts

I am building a model for forecasting some number of activations. My data set has a panel structure. Now, I want to come up with a forecast performance measure to assess the performance of my model ...
8
votes
4answers
17k views

What is a good oob score for random forests with sklearn, three-class classification? [duplicate]

I have learning data consisting of ~45k samples, each has 21 features. I am trying to train a random forest classifier on this data, which is labelled to 3 classes (-1, 0 and 1). The classes are more ...
1
vote
0answers
30 views

Out-of-sample predictive checks for Bayesian TVP models

Comparatively new to Bayesian econometrics so apologies if this is a silly question. I am running a time-varying parameter regression where the parameters are estimated as in Primiceri (2005). My ...
2
votes
1answer
417 views

How do I calculate AUC with leave-one-out CV

In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
4
votes
2answers
1k views

Stacking without splitting data

I learned Stacking used in Ensemble learning. In Stacking, training data is split into two sets. The first set is used for training each model (layer-1, left figure), the second one is used for ...
0
votes
0answers
12 views

Is it appropriate to predict a trained KMeans model on holdout data that would not be included in the training set?

I have a KMeans model that is trained on features that are percentage-transformed descriptions of events. Each observation contains between 1 and 180 events. To help with meaningful comparisons, I ...
0
votes
0answers
29 views

Multivariate Out-of-Sample Evaluation

I have a question about multivariate hypothesis testing in out-of-sample evaluations. Generally, let`s asssume we want to predict three different stock returns in a 1-month ahead forecasting setting. ...
1
vote
1answer
93 views

Reverse prediction in a time series

We know using models like ARIMA we can do out of sample predictions for a Time Series. i.e. we can know what would be the value v at time t. Can we do the reverse of it, and find at what t will be v a ...
0
votes
1answer
32 views

Is there a systematic reason why a model trained on a subset of data does better out-of-sample than the same model trained on the full dataset?

I trained a linear regression model using 3000 data points. (OLS regression, no regularization.) Then I trained another model with the same predictors (about 25), but with a subset ($n=700$) of the ...
2
votes
1answer
385 views

in-sample data vs out-of-sample data

I know that a train-validation-test splits the data into: a training dataset - obviously my "in-sample" data a validation dataset a test data set - obviously my "out-of-sample" data My question is: ...
0
votes
0answers
46 views

Using model from one data set to predict results for another data set

I'm not certain how to phrase this question: I have a dataset of ~45000 execution times of two sets of data. Approximately 35000 of these execution times is ran in one environment, and the remaining ~...
9
votes
2answers
293 views

A ''significant variable'' that does not improve out-of-sample predictions - how to interpret?

I have a question that I think will be quite basic to a lot of users. Im using linear regression models to (i) investigate the relationship of several explanatory variables and my response variable ...
1
vote
0answers
249 views

Why using Out-of-fold predictions as metafeatures in stacking?

So my question is essentially the same as this one: Why do we generate out-of-fold predictions for meta-ensembling/stacking? However, I am not entirely satisfied with the answer (not detailed enough ...
3
votes
1answer
2k views

Using $R^2$ to evaluate out-of-sample performance

In this paper the $R^2$ is used to evaluate out-of-sample predictions for several methods including neural networks and tree based methods (see section 3.3 Evaluation and Validation). How is the out-...
12
votes
1answer
46k views

Difference between “in-sample” and “pseudo out-of-sample” forecasts

Is there an explicit difference between in-sample forecasts and pseudo out-of-sample forecasts. Both is meant in the context of evaluating and comparing forecasting models.
0
votes
1answer
50 views

Is it ok to keep/discard rules based on the holdout set?

We have a POC project that is looking for rules that fit our data (eg "when a=1 and b=2 and c=3 then X=6" sort of thing). We split our data into 6 sets, and we use the first 5 sets as K-fold training ...
1
vote
0answers
57 views

What is the relation between replica method and “reusable holdout” method?

Among many methods used to detect and avoid overfitting, I am particularly interested in those two: replica method reusable holdout My question is: what is their relation in the context of adaptive ...
0
votes
1answer
252 views

Is it good practice to use both K-Fold cross validation and hold out validation

When using K-Fold cross validation is it a good or bad idea to split the dataset into two, With 70% (for example) being used for K fold CV and 30% used solely for testing in order to check for over ...
16
votes
1answer
847 views

Is Kaggle's private leaderboard a good predictor of out-of-sample performance of the winning model?

While the results of the private test set can not be used to refine the model further, isn't model selection out of a huge number of models being performed based on the private test set results? Would ...
14
votes
4answers
2k views

Predictive models: statistics can't possibly beat machine learning? [closed]

I am currently following a master program focused on statistics/econometrics. In my master, all students had to do 3 months of research. Last week, all groups had to present their research to the rest ...
6
votes
0answers
1k views

Resample random forest OOB to choose number of trees? [duplicate]

My post was inspired by this one (https://stackoverflow.com/questions/29290916/scikit-learn-random-forest-classifier-how-to-produce-a-plot-of-oob-error-agains) Although random forest models do not ...
2
votes
0answers
29 views

Theoretical question on GBM out-of-time performance sensitivity

So this is more of a theoretical question, no dataset or code that I can share. It just came up in a discussion and I was not sure of the answer. Let's say I have 2 GBM models, model A and B, trained ...
2
votes
3answers
4k views

Rolling analysis with out-of sample

I have a model that looks like lm(y ~ lag(x, -1) + lag(z, -1)) So basically, this is a time series regression with exogenous variables, and I want to carry out ...
9
votes
1answer
8k views

How to calculate out of sample R squared?

I know this probably has been discussed somewhere else, but I have not been able to find an explicit answer. I am trying to use the formula $R^2 = 1 - SSR/SST$ to calculate out-of-sample $R^2$ of a ...
1
vote
0answers
2k views

What is the difference between oob (out of bag) error and (1 - accuracy) in RandomForest?

In a Random Forest, I know that the Out Of Bag Error is described as the fraction of number incorrect classifications over number of out of bag samples. Accuracy is defined as the number of correct ...
0
votes
1answer
317 views

Inconsistent Out-of-bag error estimates [closed]

I keep getting different out-of-bag error estimates from the caret package, depending on how the estimates are computed. I can't seem to nail down exactly where the ...
3
votes
0answers
305 views

Nested Cross Validation: Choosing between different best hyperparameters

I know this sort of question has been asked many times, and several answers have been already provided on this platform too (e.g., here, here, and here). Still, there is something about the idea ...
60
votes
8answers
11k views

How can I help ensure testing data does not leak into training data?

Suppose we have someone building a predictive model, but that someone is not necessarily well-versed in proper statistical or machine learning principles. Maybe we are helping that person as they are ...
1
vote
0answers
437 views

prediction using plm for out of sample data in R

I want to predict out of sample data for the same group (lets say state ) for new time window by either fixed or random method.."predict" function is not helping.Here i gave a example of a dataset ...
21
votes
3answers
9k views

Do we need a test set when using k-fold cross-validation?

I've been reading about k-fold validation, and I want to make sure I understand how it works. I know that for the holdout method, the data is split into three sets, and the test set is only used at ...