Questions tagged [out-of-sample]

Refers to the practice of assessing model performance on some "test" or "holdout" or "out-of-sample" set of data that was not used for model building.

0
votes
0answers
11 views

What is the formula that is used to calculated the MSE with Random Forest regression in R?

I am using the package randomForest in R for panel-data on conflict intensity. The dependent variable is the conflict intensity (e.g. the number of battle deaths). Independent variables are population,...
2
votes
1answer
58 views

Can balanced accuracy be higher than accuracy?

I have classification tree where the balanced accuracy of the test set is higher than the normal accuracy. I thought balanced accuracy can only have at his maximum the same value as the accuracy not ...
2
votes
2answers
62 views

How to check if i have strong linear relationship between dependent variable and independent variables in linear regression (OLS)?

I want compare the out of sample prediction from an linear regression model (OLS) and a regression tree. I read that OLS outperforms regression tree if the relationship between the dependent variable ...
1
vote
3answers
51 views

Can a prediction be better with insignificant variables than with only significant variables (or none at all)?

I have two OLS models and want to do an out of sample prediction for wages on a test set. In the first model I excluded the insignificant variable. The second model has the insignificant variable. The ...
1
vote
0answers
35 views

ARIMA: Produce multi-step, out-of-sample forecasts by feeding in new history without retraining the model? [closed]

I'd like to compare the results of an LSTM model to an ARIMA model. How can I create an ARIMA model in python that trains on the first 70% of data (~2700 observations), and then produces forecasts at ...
1
vote
1answer
54 views

Forecasting with AR(1) and pseudo out-of-sample using R

I'm trying to do Pseudo out-of-sample forecasting using R. And, I also have the following initial data (gdp) ...
0
votes
0answers
28 views

Bootstrapping vs. K-Fold: Is every data point in atleast one of the test set/out of bag - atleast once?

It's easy to see that in K-Fold cross-validation, that split training examples into K parts, in such a way that 1 of the K parts is considered to be the test set, and eventually as you shift which ...
0
votes
0answers
47 views

Classification accuracy in holdout similar to CV if set is randomly sampled, completely wrong otherwise

I'm building a classifier to predict a binary label on a dataset with 30 features and around 60000 samples of measurements from a car assembly process. While experimenting with some baseline models ...
1
vote
1answer
44 views

Asymmetric error measure for forecasts

I am building a model for forecasting some number of activations. My data set has a panel structure. Now, I want to come up with a forecast performance measure to assess the performance of my model ...
3
votes
2answers
130 views

What's the real purpose of cross validation?

As for cross evaluation (CV), I have two questions to ask: 1) CV has nothing to do with parameter selection, but only model evaluation? Specifically, which model? 2) In k-fold CV, what's the final ...
1
vote
1answer
336 views

Out-of bag error in Random Forest

I am trying to code my own, simple version of RandomForest function in R for learning purposes. However I have a hard time understanding the concept of the out-of-bag error. Is it simply done by ...
1
vote
0answers
23 views

Out-of-sample predictive checks for Bayesian TVP models

Comparatively new to Bayesian econometrics so apologies if this is a silly question. I am running a time-varying parameter regression where the parameters are estimated as in Primiceri (2005). My ...
2
votes
1answer
224 views

How do I calculate AUC with leave-one-out CV

In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
0
votes
0answers
12 views

Is it appropriate to predict a trained KMeans model on holdout data that would not be included in the training set?

I have a KMeans model that is trained on features that are percentage-transformed descriptions of events. Each observation contains between 1 and 180 events. To help with meaningful comparisons, I ...
0
votes
0answers
28 views

Multivariate Out-of-Sample Evaluation

I have a question about multivariate hypothesis testing in out-of-sample evaluations. Generally, let`s asssume we want to predict three different stock returns in a 1-month ahead forecasting setting. ...
2
votes
1answer
84 views

Is cross-validation better/worse than a third holdout set?

I see lots of papers that use just train and test datasets, without a third validation set, but they use cross-validation so that every data point is used for training and testing among the different ...
1
vote
1answer
64 views

Reverse prediction in a time series

We know using models like ARIMA we can do out of sample predictions for a Time Series. i.e. we can know what would be the value v at time t. Can we do the reverse of it, and find at what t will be v a ...
0
votes
1answer
29 views

Is there a systematic reason why a model trained on a subset of data does better out-of-sample than the same model trained on the full dataset?

I trained a linear regression model using 3000 data points. (OLS regression, no regularization.) Then I trained another model with the same predictors (about 25), but with a subset ($n=700$) of the ...
2
votes
1answer
272 views

in-sample data vs out-of-sample data

I know that a train-validation-test splits the data into: a training dataset - obviously my "in-sample" data a validation dataset a test data set - obviously my "out-of-sample" data My question is: ...
0
votes
0answers
30 views

Using model from one data set to predict results for another data set

I'm not certain how to phrase this question: I have a dataset of ~45000 execution times of two sets of data. Approximately 35000 of these execution times is ran in one environment, and the remaining ~...
9
votes
2answers
267 views

A ''significant variable'' that does not improve out-of-sample predictions - how to interpret?

I have a question that I think will be quite basic to a lot of users. Im using linear regression models to (i) investigate the relationship of several explanatory variables and my response variable ...
1
vote
0answers
206 views

Why using Out-of-fold predictions as metafeatures in stacking?

So my question is essentially the same as this one: Why do we generate out-of-fold predictions for meta-ensembling/stacking? However, I am not entirely satisfied with the answer (not detailed enough ...
2
votes
2answers
769 views

Stacking without splitting data

I learned Stacking used in Ensemble learning. In Stacking, training data is split into two sets. The first set is used for training each model (layer-1, left figure), the second one is used for ...
3
votes
1answer
1k views

Using $R^2$ to evaluate out-of-sample performance

In this paper the $R^2$ is used to evaluate out-of-sample predictions for several methods including neural networks and tree based methods (see section 3.3 Evaluation and Validation). How is the out-...
0
votes
1answer
45 views

Is it ok to keep/discard rules based on the holdout set?

We have a POC project that is looking for rules that fit our data (eg "when a=1 and b=2 and c=3 then X=6" sort of thing). We split our data into 6 sets, and we use the first 5 sets as K-fold training ...
1
vote
0answers
45 views

What is the relation between replica method and “reusable holdout” method?

Among many methods used to detect and avoid overfitting, I am particularly interested in those two: replica method reusable holdout My question is: what is their relation in the context of adaptive ...
0
votes
1answer
203 views

Is it good practice to use both K-Fold cross validation and hold out validation

When using K-Fold cross validation is it a good or bad idea to split the dataset into two, With 70% (for example) being used for K fold CV and 30% used solely for testing in order to check for over ...
14
votes
4answers
2k views

Predictive models: statistics can't possibly beat machine learning? [closed]

I am currently following a master program focused on statistics/econometrics. In my master, all students had to do 3 months of research. Last week, all groups had to present their research to the rest ...
2
votes
0answers
29 views

Theoretical question on GBM out-of-time performance sensitivity

So this is more of a theoretical question, no dataset or code that I can share. It just came up in a discussion and I was not sure of the answer. Let's say I have 2 GBM models, model A and B, trained ...
0
votes
1answer
284 views

Inconsistent Out-of-bag error estimates [closed]

I keep getting different out-of-bag error estimates from the caret package, depending on how the estimates are computed. I can't seem to nail down exactly where the ...
3
votes
0answers
282 views

Nested Cross Validation: Choosing between different best hyperparameters

I know this sort of question has been asked many times, and several answers have been already provided on this platform too (e.g., here, here, and here). Still, there is something about the idea ...
1
vote
0answers
399 views

prediction using plm for out of sample data in R

I want to predict out of sample data for the same group (lets say state ) for new time window by either fixed or random method.."predict" function is not helping.Here i gave a example of a dataset ...
0
votes
0answers
43 views

Can I duplicate every element of my dataset to get around some issues of small datasets?

Take this question with many grains of salt, because it's mostly a theoretical curiosity. I've built a multinomial classifier using GLMnet. The problem is that some of of my input variables have an ...
0
votes
0answers
348 views

SPSS: How to get error measurements (RMSE, MAE, MSE etc.) on holdout sample?

I split my data into "estimation" and "holdout" sample, using "Select Cases" (and select only the cases I want to use as the estimation sample). Then, I created a time series using Expert Modeler in ...
0
votes
1answer
527 views

best practice: preprocessing holdout set at same time as train set, or no?

I am a very new ML programmer, and I have come across a dilemma regarding best practices. Things will "work" either way for me, but I want to know what the best practice is. I am performing text ...
0
votes
0answers
39 views

Out-of-bag sampling from same distribution

The purpose of out-of-bag sampling is to test your model on an unseen data. However, if we have a very large dataset say 5 milllion observations, when the out-of-bag sampling follows the same ...
2
votes
1answer
750 views

In Random Forest, cross-validation can be avoided using the out-of-bag sample? [duplicate]

I have heard that the main reason to use out-of-bag sample(OBB) over CV is that OOB is faster to implement and usually gets the same results(for tuning the hyper-parameters) as implementing CV when ...
0
votes
1answer
810 views

Which one should I use for rolling forecast, dynamic or static?

I'm doing a rolling forecast using a fitted arma-garch model, but I'm confused regarding the rolling method, my window length is 1209 obs, and I roll 100 times, and each time I reset my window to ...
15
votes
1answer
764 views

Is Kaggle's private leaderboard a good predictor of out-of-sample performance of the winning model?

While the results of the private test set can not be used to refine the model further, isn't model selection out of a huge number of models being performed based on the private test set results? Would ...
0
votes
1answer
81 views

Sample to use when doing ARMA order selection, OLS, GARCH estimation

I have a really basic question which I need clarifying on and would appreciate a quick response to this if possible. I am forecasting conditional variance (volatility). I have in-sample from 1996-...
2
votes
0answers
100 views

Is training multiple Random Forests equivalent to a repeated 3-fold cross-validation?

In the book "Elements of Statistical Learning" (Friedman, Hastie, Tibshirani) , the authors suggest that bootstrap and the resulting out-of-bag data for a random forest are equivalent to a 3-fold ...
0
votes
1answer
670 views

Holdout sample for multinomial logistic regression in SPSS

I am having a multiple categorical dependent variable and continuous independent variables. Via a multinomial logistic regression in SPSS I want to test whether the training sample makes good ...
3
votes
1answer
48 views

Is there a convincing case to be made against using last years data as a holdout set when building a predictive model?

Building a model on a large subset of your data and testing it on a holdout set is a common practice. I am interested to what extent this is a justifiable approach in various data laden settings. ...
0
votes
0answers
87 views

evaluating out of sample accuracy

I estimate a linear regression model and compute the variance of residuals in both the training-set and also on an additional test set. Ideally these should not be very different. Does it make sense ...
1
vote
0answers
138 views

Data Partition for In sample and out of sample forecasting in neural network

I got confused with how to do data partition that reflects in sample and out of sample forecast when I do time series forecasting in neural network. What I understand is we have to divide data into: -...
4
votes
1answer
418 views

Out-of-sample likelihood example

I'm an amateur in statistics. I was reading a paper in which it is mentioned that A model gives high out-of-sample likelihood. What does that mean? What is exactly "High-out-of-sample likelihood"?...
1
vote
1answer
99 views

Can I use the Ljung-Box test on out-of-sample prediction errors?

I know that tests like Ljung-Box and Breusch-Godfrey are often used to test the residuals of a fitted ARMA model for whiteness. But say I want to evaluate how well my model describes a new data set ...
5
votes
2answers
5k views

R squared always higher than 1

I am trying to implement an algorithm that solves a linear regression problem with the following objective function (LASSO): $$\min_\beta \frac{1}{2}||y-X\beta||_2^2 + \lambda ||\beta||_1$$ for ...
1
vote
0answers
593 views

How to obtain the same results of a random forest model using caret and randomForest?

I am trying to understand how does building a regression model with caret's train () function differs from randomForest(). For my excercise, I am using the iris dataset. As shown in the code below, I ...
11
votes
4answers
959 views

What is the more appropriate way to create a hold-out set: to remove some subjects or to remove some observations from each subject?

I have a dataset with 26 features and 31000 rows. It is the dataset of 38 subjects. It is for a biometric system. So I want to be able to identify subjects. In order to have a testing set, I know I ...