Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [out-of-sample]

Refers to the practice of assessing model performance on some "test" or "holdout" or "out-of-sample" set of data that was not used for model building.

58
votes
8answers
10k views

How can I help ensure testing data does not leak into training data?

Suppose we have someone building a predictive model, but that someone is not necessarily well-versed in proper statistical or machine learning principles. Maybe we are helping that person as they are ...
29
votes
4answers
823 views

Has the journal Science endorsed the Garden of Forking Pathes Analyses?

The idea of adaptive data analysis is that you alter your plan for analyzing the data as you learn more about it. In the case of exploratory data analysis (EDA), this is generally a good idea (you are ...
21
votes
5answers
3k views

New revolutionary way of data mining?

The following excerpt is from Schwager's Hedge Fund Market Wizzards (May 2012), an interview with the consistently successful hedge fund manager Jaffray Woodriff: To the question: "What are some of ...
16
votes
3answers
8k views

Do we need a test set when using k-fold cross-validation?

I've been reading about k-fold validation, and I want to make sure I understand how it works. I know that for the holdout method, the data is split into three sets, and the test set is only used at ...
15
votes
1answer
762 views

Is Kaggle's private leaderboard a good predictor of out-of-sample performance of the winning model?

While the results of the private test set can not be used to refine the model further, isn't model selection out of a huge number of models being performed based on the private test set results? Would ...
15
votes
0answers
3k views

Confusion with Vowpal Wabbit's multiple-pass behavior when performing ridge-regression [closed]

I have encountered many peculiarities/misunderstandings of Vowpal Wabbit when trying to do online multiple-pass learning. Specifically, I need to solve a Ridge Linear regression problem, with ...
14
votes
4answers
2k views

Predictive models: statistics can't possibly beat machine learning? [closed]

I am currently following a master program focused on statistics/econometrics. In my master, all students had to do 3 months of research. Last week, all groups had to present their research to the rest ...
12
votes
3answers
3k views

Why isn't the holdout method (splitting data into training and testing) used in classical statistics?

In my classroom exposure to data mining, the holdout method was introduced as a way of assessing model performance. However, when I took my first class on linear models, this was not introduced as a ...
12
votes
1answer
46k views

Difference between “in-sample” and “pseudo out-of-sample” forecasts

Is there an explicit difference between in-sample forecasts and pseudo out-of-sample forecasts. Both is meant in the context of evaluating and comparing forecasting models.
11
votes
4answers
956 views

What is the more appropriate way to create a hold-out set: to remove some subjects or to remove some observations from each subject?

I have a dataset with 26 features and 31000 rows. It is the dataset of 38 subjects. It is for a biometric system. So I want to be able to identify subjects. In order to have a testing set, I know I ...
10
votes
1answer
1k views

Does modeling with Random Forests require cross-validation?

As far as I've seen, opinions tend to differ about this. Best practice would certainly dictate using cross-validation (especially if comparing RFs with other algorithms on the same dataset). On the ...
9
votes
2answers
267 views

A ''significant variable'' that does not improve out-of-sample predictions - how to interpret?

I have a question that I think will be quite basic to a lot of users. Im using linear regression models to (i) investigate the relationship of several explanatory variables and my response variable ...
9
votes
1answer
7k views

How to calculate out of sample R squared?

I know this probably has been discussed somewhere else, but I have not been able to find an explicit answer. I am trying to use the formula $R^2 = 1 - SSR/SST$ to calculate out-of-sample $R^2$ of a ...
8
votes
4answers
16k views

What is a good oob score for random forests with sklearn, three-class classification?

I have learning data consisting of ~45k samples, each has 21 features. I am trying to train a random forest classifier on this data, which is labelled to 3 classes (-1, 0 and 1). The classes are more ...
7
votes
3answers
4k views

Bootstrapping estimates of out-of-sample error

I know how to use bootstrap re-sampling to find confidence intervals for in-sample error or R2: ...
7
votes
1answer
3k views

How can a smaller learning rate hurt the performance of a gbm?

I've always subscribed to the folk wisdom that decreasing the learning rate in a gbm (gradient boosted tree model) does not hurt the out of sample performance of the model. Today, I'm not so sure. I'...
6
votes
0answers
1k views

Resample random forest OOB to choose number of trees? [duplicate]

My post was inspired by this one (https://stackoverflow.com/questions/29290916/scikit-learn-random-forest-classifier-how-to-produce-a-plot-of-oob-error-agains) Although random forest models do not ...
5
votes
2answers
5k views

R squared always higher than 1

I am trying to implement an algorithm that solves a linear regression problem with the following objective function (LASSO): $$\min_\beta \frac{1}{2}||y-X\beta||_2^2 + \lambda ||\beta||_1$$ for ...
5
votes
3answers
4k views

Generate synthetic data to match sample data

If I have a sample data set of 5000 points with many features and I have to generate a dataset with say 1 million data points using the sample data. It is like oversampling the sample data to generate ...
5
votes
1answer
4k views

Is a negative OOB score possible with scikit-learn's RandomForestRegressor?

I'm currently implementing scikit-learn's RandomForestRegressor in Python and am scratching my head over why I have occasionally wound up with negative out-of-bag scores from it. As far as I can tell ...
5
votes
1answer
5k views

Computing Out of Bag error in Random Forest

I am implementing a Random Forest classifier as a side-project, and I am a bit unclear on what the correct approach is to compute, say, the OOB estimate for the classifier error rate. My ...
5
votes
0answers
152 views

Is there justification for using cross validation scores as model averaging weights?

Bayesian model averaging uses approximate Bayes factors. Some researchers use AIC to weight models. Is there justification for using, say, the Brier score, median absolute deviation, or other such ...
4
votes
1answer
388 views

Is it possible to talk about “Out-of-sample overfitting” in a model?

I'm analysing various factors to forecast the value of a dichotomic variable and in this context I'm testing many different models (Logistic Regression, DA,PLSDA, various random and non-random "...
4
votes
1answer
413 views

Out-of-sample likelihood example

I'm an amateur in statistics. I was reading a paper in which it is mentioned that A model gives high out-of-sample likelihood. What does that mean? What is exactly "High-out-of-sample likelihood"?...
4
votes
0answers
73 views

How can I help ensure testing data does not leak into training data? [duplicate]

Possible Duplicate: How can I help ensure testing data does not leak into training data? Overfitting is obviously a significant problem in machine-learning...perhaps the most significant problem. ...
3
votes
1answer
11k views

Out of Sample and In Sample testing

I am very confused in testing regressions and know that there are many explanations available online, but I am still not getting anything it in my mind. Suppose I have daily data for past 100 days, I ...
3
votes
1answer
1k views

Out-of-sample vs. test set

Someone asked me if I did out-of-time testing (which I assume is just out-of-sample testing but with a timeline element). But if I have a test set, is that not essentially the same as out-of-sample ...
3
votes
2answers
129 views

What's the real purpose of cross validation?

As for cross evaluation (CV), I have two questions to ask: 1) CV has nothing to do with parameter selection, but only model evaluation? Specifically, which model? 2) In k-fold CV, what's the final ...
3
votes
1answer
2k views

Random Forest - Huge Disparity between OOB Error and test data error

I am building my model in R and am using the randomForest package. My current model has 7 features and I see OOB error rate of about 14%. I also ran the rfcv in the random forest package to see how ...
3
votes
1answer
1k views

Adjusted R squared on a holdout set

The formula for adjusted $R^2$ is: $$ 1 - \frac{(n-1)}{(n-p-1)}(1-R^2) $$ where $r^2$ is the coefficient of determination, $n$ is the number of points, and $p$ is the number of parameters the model ...
3
votes
1answer
48 views

Is there a convincing case to be made against using last years data as a holdout set when building a predictive model?

Building a model on a large subset of your data and testing it on a holdout set is a common practice. I am interested to what extent this is a justifiable approach in various data laden settings. ...
3
votes
1answer
454 views

Do I need an initial train/test split for nested cross-validation?

I have a couple of pipelines: pipeline 1: CV'd feature selection, CV'd hyperparameter selection for classifier A pipeline 2: CV'd feature selection, CV'd hyperparameter selection for classifier B ...
3
votes
1answer
3k views

Out-of-time testing (basic question)

I understand the importance of out-of-sample testing, but could you tell me why I should (or shouldn't) do out-of-time testing ? The only use that comes to mind is if the predictive model applies to ...
3
votes
1answer
1k views

Using $R^2$ to evaluate out-of-sample performance

In this paper the $R^2$ is used to evaluate out-of-sample predictions for several methods including neural networks and tree based methods (see section 3.3 Evaluation and Validation). How is the out-...
3
votes
0answers
277 views

Nested Cross Validation: Choosing between different best hyperparameters

I know this sort of question has been asked many times, and several answers have been already provided on this platform too (e.g., here, here, and here). Still, there is something about the idea ...
3
votes
0answers
87 views

Cross validation or EM for selecting strength of the prior?

Often when I'm looking at bayesian analyses, the influence of the prior is chosen via cross validation. For example, suppose $X$ and $Y$ represent some real valued data that I want perform a bayesian ...
3
votes
0answers
254 views

BIC vs. Out of sample performance

I have two statistical models. Model 1 uses a GLM approach while model 2 uses a time series approach for fitting. I want to compare these two models. Model 1 (i.e. GLM) has a better out of sample ...
3
votes
0answers
222 views

Can holdout validation be systematically biased?

I recently did some experimenting comparing some common method of internal validation. In my field, the use of a single 1:1 holdout validation is extremely common, even with very small datasets, and I ...
2
votes
2answers
57 views

How to check if i have strong linear relationship between dependent variable and independent variables in linear regression (OLS)?

I want compare the out of sample prediction from an linear regression model (OLS) and a regression tree. I read that OLS outperforms regression tree if the relationship between the dependent variable ...
2
votes
2answers
3k views

Pros and cons of cross-validation?

A practicing statistician I know advocates strongly against cross-validation, claiming that he would rather build the model on the entire dataset and make sure the underlying statistical assumptions ...
2
votes
3answers
4k views

Rolling analysis with out-of sample

I have a model that looks like lm(y ~ lag(x, -1) + lag(z, -1)) So basically, this is a time series regression with exogenous variables, and I want to carry out ...
2
votes
2answers
751 views

Stacking without splitting data

I learned Stacking used in Ensemble learning. In Stacking, training data is split into two sets. The first set is used for training each model (layer-1, left figure), the second one is used for ...
2
votes
1answer
55 views

Can balanced accuracy be higher than accuracy?

I have classification tree where the balanced accuracy of the test set is higher than the normal accuracy. I thought balanced accuracy can only have at his maximum the same value as the accuracy not ...
2
votes
1answer
215 views

How do I calculate AUC with leave-one-out CV

In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
2
votes
1answer
261 views

in-sample data vs out-of-sample data

I know that a train-validation-test splits the data into: a training dataset - obviously my "in-sample" data a validation dataset a test data set - obviously my "out-of-sample" data My question is: ...
2
votes
1answer
742 views

In Random Forest, cross-validation can be avoided using the out-of-bag sample? [duplicate]

I have heard that the main reason to use out-of-bag sample(OBB) over CV is that OOB is faster to implement and usually gets the same results(for tuning the hyper-parameters) as implementing CV when ...
2
votes
1answer
891 views

Random Forest - What training set measure is the best predictor of test set accuracy?

I'm running a random forest model on a training sample in R in order to make predictions on a hidden test set. I'm having difficulty in understanding how I should go about improving my model in order ...
2
votes
1answer
160 views

Testing the variance part of a Generalized Linear Model out of sample

Suppose I have a response vector and a factorial design (for simplicity, assume it’s a one-way ANOVA with two treatments). A few Generalized Linear Models (Poisson, Negative Binomial, etc) are fitted ...
2
votes
1answer
955 views

How can I validate a logistic regression model using averaged parameter estimates?

Let me say thanks in advance. I'm working with a set of data that contains reported coyote sightings. I use 2/3 of the data for model calibration along with an equal number of pseudo absences. I ...
2
votes
1answer
80 views

Is cross-validation better/worse than a third holdout set?

I see lots of papers that use just train and test datasets, without a third validation set, but they use cross-validation so that every data point is used for training and testing among the different ...