Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [out-of-sample]

Refers to the practice of assessing model performance on some "test" or "holdout" or "out-of-sample" set of data that was not used for model building.

2
votes
1answer
79 views

Is cross-validation better/worse than a third holdout set?

I see lots of papers that use just train and test datasets, without a third validation set, but they use cross-validation so that every data point is used for training and testing among the different ...
1
vote
3answers
49 views

Can a prediction be better with insignificant variables than with only significant variables (or none at all)?

I have two OLS models and want to do an out of sample prediction for wages on a test set. In the first model I excluded the insignificant variable. The second model has the insignificant variable. The ...
5
votes
0answers
152 views

Is there justification for using cross validation scores as model averaging weights?

Bayesian model averaging uses approximate Bayes factors. Some researchers use AIC to weight models. Is there justification for using, say, the Brier score, median absolute deviation, or other such ...
2
votes
2answers
57 views

How to check if i have strong linear relationship between dependent variable and independent variables in linear regression (OLS)?

I want compare the out of sample prediction from an linear regression model (OLS) and a regression tree. I read that OLS outperforms regression tree if the relationship between the dependent variable ...
0
votes
0answers
11 views

What is the formula that is used to calculated the MSE with Random Forest regression in R?

I am using the package randomForest in R for panel-data on conflict intensity. The dependent variable is the conflict intensity (e.g. the number of battle deaths). Independent variables are population,...
1
vote
1answer
300 views

Out-of bag error in Random Forest

I am trying to code my own, simple version of RandomForest function in R for learning purposes. However I have a hard time understanding the concept of the out-of-bag error. Is it simply done by ...
2
votes
1answer
55 views

Can balanced accuracy be higher than accuracy?

I have classification tree where the balanced accuracy of the test set is higher than the normal accuracy. I thought balanced accuracy can only have at his maximum the same value as the accuracy not ...
-1
votes
0answers
57 views

What to do if random forest still overfit after grid tuning?

I have a random forest and an ols regression. Both models i want use for an out of sample prediction. Before tuning the parameters of the random forest the default settings of the random forest yield ...
0
votes
1answer
808 views

Which one should I use for rolling forecast, dynamic or static?

I'm doing a rolling forecast using a fitted arma-garch model, but I'm confused regarding the rolling method, my window length is 1209 obs, and I roll 100 times, and each time I reset my window to ...
1
vote
0answers
32 views

ARIMA: Produce multi-step, out-of-sample forecasts by feeding in new history without retraining the model? [closed]

I'd like to compare the results of an LSTM model to an ARIMA model. How can I create an ARIMA model in python that trains on the first 70% of data (~2700 observations), and then produces forecasts at ...
1
vote
1answer
51 views

Forecasting with AR(1) and pseudo out-of-sample using R

I'm trying to do Pseudo out-of-sample forecasting using R. And, I also have the following initial data (gdp) ...
0
votes
0answers
28 views

Bootstrapping vs. K-Fold: Is every data point in atleast one of the test set/out of bag - atleast once?

It's easy to see that in K-Fold cross-validation, that split training examples into K parts, in such a way that 1 of the K parts is considered to be the test set, and eventually as you shift which ...
0
votes
0answers
47 views

Classification accuracy in holdout similar to CV if set is randomly sampled, completely wrong otherwise

I'm building a classifier to predict a binary label on a dataset with 30 features and around 60000 samples of measurements from a car assembly process. While experimenting with some baseline models ...
2
votes
1answer
495 views

Precision, Recall and area under ROC curve as sample size increases

The following is a question from an exam paper on evaluating the performance of search engines. To this day I looked in my text book and literally close to 50 web pages and I can't find one convincing ...
3
votes
2answers
129 views

What's the real purpose of cross validation?

As for cross evaluation (CV), I have two questions to ask: 1) CV has nothing to do with parameter selection, but only model evaluation? Specifically, which model? 2) In k-fold CV, what's the final ...
1
vote
1answer
44 views

Asymmetric error measure for forecasts

I am building a model for forecasting some number of activations. My data set has a panel structure. Now, I want to come up with a forecast performance measure to assess the performance of my model ...
8
votes
4answers
16k views

What is a good oob score for random forests with sklearn, three-class classification?

I have learning data consisting of ~45k samples, each has 21 features. I am trying to train a random forest classifier on this data, which is labelled to 3 classes (-1, 0 and 1). The classes are more ...
1
vote
0answers
23 views

Out-of-sample predictive checks for Bayesian TVP models

Comparatively new to Bayesian econometrics so apologies if this is a silly question. I am running a time-varying parameter regression where the parameters are estimated as in Primiceri (2005). My ...
2
votes
1answer
213 views

How do I calculate AUC with leave-one-out CV

In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
2
votes
2answers
746 views

Stacking without splitting data

I learned Stacking used in Ensemble learning. In Stacking, training data is split into two sets. The first set is used for training each model (layer-1, left figure), the second one is used for ...
0
votes
0answers
12 views

Is it appropriate to predict a trained KMeans model on holdout data that would not be included in the training set?

I have a KMeans model that is trained on features that are percentage-transformed descriptions of events. Each observation contains between 1 and 180 events. To help with meaningful comparisons, I ...
0
votes
0answers
28 views

Multivariate Out-of-Sample Evaluation

I have a question about multivariate hypothesis testing in out-of-sample evaluations. Generally, let`s asssume we want to predict three different stock returns in a 1-month ahead forecasting setting. ...
1
vote
1answer
61 views

Reverse prediction in a time series

We know using models like ARIMA we can do out of sample predictions for a Time Series. i.e. we can know what would be the value v at time t. Can we do the reverse of it, and find at what t will be v a ...
0
votes
1answer
29 views

Is there a systematic reason why a model trained on a subset of data does better out-of-sample than the same model trained on the full dataset?

I trained a linear regression model using 3000 data points. (OLS regression, no regularization.) Then I trained another model with the same predictors (about 25), but with a subset ($n=700$) of the ...
2
votes
1answer
257 views

in-sample data vs out-of-sample data

I know that a train-validation-test splits the data into: a training dataset - obviously my "in-sample" data a validation dataset a test data set - obviously my "out-of-sample" data My question is: ...
0
votes
0answers
29 views

Using model from one data set to predict results for another data set

I'm not certain how to phrase this question: I have a dataset of ~45000 execution times of two sets of data. Approximately 35000 of these execution times is ran in one environment, and the remaining ~...
9
votes
2answers
267 views

A ''significant variable'' that does not improve out-of-sample predictions - how to interpret?

I have a question that I think will be quite basic to a lot of users. Im using linear regression models to (i) investigate the relationship of several explanatory variables and my response variable ...
1
vote
0answers
201 views

Why using Out-of-fold predictions as metafeatures in stacking?

So my question is essentially the same as this one: Why do we generate out-of-fold predictions for meta-ensembling/stacking? However, I am not entirely satisfied with the answer (not detailed enough ...
3
votes
1answer
1k views

Using $R^2$ to evaluate out-of-sample performance

In this paper the $R^2$ is used to evaluate out-of-sample predictions for several methods including neural networks and tree based methods (see section 3.3 Evaluation and Validation). How is the out-...
12
votes
1answer
46k views

Difference between “in-sample” and “pseudo out-of-sample” forecasts

Is there an explicit difference between in-sample forecasts and pseudo out-of-sample forecasts. Both is meant in the context of evaluating and comparing forecasting models.
0
votes
1answer
45 views

Is it ok to keep/discard rules based on the holdout set?

We have a POC project that is looking for rules that fit our data (eg "when a=1 and b=2 and c=3 then X=6" sort of thing). We split our data into 6 sets, and we use the first 5 sets as K-fold training ...
1
vote
0answers
45 views

What is the relation between replica method and “reusable holdout” method?

Among many methods used to detect and avoid overfitting, I am particularly interested in those two: replica method reusable holdout My question is: what is their relation in the context of adaptive ...
0
votes
1answer
202 views

Is it good practice to use both K-Fold cross validation and hold out validation

When using K-Fold cross validation is it a good or bad idea to split the dataset into two, With 70% (for example) being used for K fold CV and 30% used solely for testing in order to check for over ...
15
votes
1answer
762 views

Is Kaggle's private leaderboard a good predictor of out-of-sample performance of the winning model?

While the results of the private test set can not be used to refine the model further, isn't model selection out of a huge number of models being performed based on the private test set results? Would ...
14
votes
4answers
2k views

Predictive models: statistics can't possibly beat machine learning? [closed]

I am currently following a master program focused on statistics/econometrics. In my master, all students had to do 3 months of research. Last week, all groups had to present their research to the rest ...
6
votes
0answers
1k views

Resample random forest OOB to choose number of trees? [duplicate]

My post was inspired by this one (https://stackoverflow.com/questions/29290916/scikit-learn-random-forest-classifier-how-to-produce-a-plot-of-oob-error-agains) Although random forest models do not ...
2
votes
0answers
29 views

Theoretical question on GBM out-of-time performance sensitivity

So this is more of a theoretical question, no dataset or code that I can share. It just came up in a discussion and I was not sure of the answer. Let's say I have 2 GBM models, model A and B, trained ...
2
votes
3answers
4k views

Rolling analysis with out-of sample

I have a model that looks like lm(y ~ lag(x, -1) + lag(z, -1)) So basically, this is a time series regression with exogenous variables, and I want to carry out ...
9
votes
1answer
7k views

How to calculate out of sample R squared?

I know this probably has been discussed somewhere else, but I have not been able to find an explicit answer. I am trying to use the formula $R^2 = 1 - SSR/SST$ to calculate out-of-sample $R^2$ of a ...
1
vote
0answers
2k views

What is the difference between oob (out of bag) error and (1 - accuracy) in RandomForest?

In a Random Forest, I know that the Out Of Bag Error is described as the fraction of number incorrect classifications over number of out of bag samples. Accuracy is defined as the number of correct ...
0
votes
1answer
280 views

Inconsistent Out-of-bag error estimates [closed]

I keep getting different out-of-bag error estimates from the caret package, depending on how the estimates are computed. I can't seem to nail down exactly where the ...
3
votes
0answers
277 views

Nested Cross Validation: Choosing between different best hyperparameters

I know this sort of question has been asked many times, and several answers have been already provided on this platform too (e.g., here, here, and here). Still, there is something about the idea ...
58
votes
8answers
10k views

How can I help ensure testing data does not leak into training data?

Suppose we have someone building a predictive model, but that someone is not necessarily well-versed in proper statistical or machine learning principles. Maybe we are helping that person as they are ...
1
vote
0answers
398 views

prediction using plm for out of sample data in R

I want to predict out of sample data for the same group (lets say state ) for new time window by either fixed or random method.."predict" function is not helping.Here i gave a example of a dataset ...
16
votes
3answers
8k views

Do we need a test set when using k-fold cross-validation?

I've been reading about k-fold validation, and I want to make sure I understand how it works. I know that for the holdout method, the data is split into three sets, and the test set is only used at ...
0
votes
0answers
42 views

Can I duplicate every element of my dataset to get around some issues of small datasets?

Take this question with many grains of salt, because it's mostly a theoretical curiosity. I've built a multinomial classifier using GLMnet. The problem is that some of of my input variables have an ...
0
votes
0answers
346 views

SPSS: How to get error measurements (RMSE, MAE, MSE etc.) on holdout sample?

I split my data into "estimation" and "holdout" sample, using "Select Cases" (and select only the cases I want to use as the estimation sample). Then, I created a time series using Expert Modeler in ...
0
votes
1answer
518 views

best practice: preprocessing holdout set at same time as train set, or no?

I am a very new ML programmer, and I have come across a dilemma regarding best practices. Things will "work" either way for me, but I want to know what the best practice is. I am performing text ...
29
votes
4answers
823 views

Has the journal Science endorsed the Garden of Forking Pathes Analyses?

The idea of adaptive data analysis is that you alter your plan for analyzing the data as you learn more about it. In the case of exploratory data analysis (EDA), this is generally a good idea (you are ...
0
votes
0answers
39 views

Out-of-bag sampling from same distribution

The purpose of out-of-bag sampling is to test your model on an unseen data. However, if we have a very large dataset say 5 milllion observations, when the out-of-bag sampling follows the same ...