Questions tagged [out-of-sample]

Refers to the practice of assessing model performance on some "test" or "holdout" or "out-of-sample" set of data that was not used for model building.

Filter by
Sorted by
Tagged with
1
vote
0answers
16 views

Should I standardize forecasted and actual values for Mincer Zarnowitz test?

I have some out-of-sample forecasted values of variance through some GARCH model, and now I am trying to perform a Mincer-Zarnowitz test for validity of my predictions. I first standardized both ...
0
votes
2answers
21 views

Most likely sources of divergence between (adjusted)-R squared and out-of-sample predictive performance

I'm wondering which invalid assumptions are most likely to explain the wild discrepancies between a model's R-squared as a measure of predictive performance, and the actual out-of-sample predictive ...
0
votes
1answer
27 views

How to calculate the confidence interval with weighted data?

I've done some search for similar questions, but they're not the same as what I'm trying to get. Assume that there's a server that handles requests $r$ and returns a set of items $I_{r}$ of random ...
0
votes
0answers
24 views

Deep learning ; LSTM out-of-sample prediction

I am trying to do out-of-sample prediction of housing price index with deep learning LSTM. I've practiced the code with a sample data (apt_data_sc) splitting it with 70%,30% training and test set (...
1
vote
1answer
96 views

In sample splitting for time series data, do we randomly select data?

I'm having a hard to conceptually understanding how to do this. I would like to do my own sample splitting (not the method built into a package). Let's say you have 80 days of weather data. You ...
1
vote
0answers
24 views

Regression hypothesis testing via out-of-sample testing

Let's consider two linear models. Full model: $\mathbb{E}\big{[}Y\big{\vert} X_1,\dots,X_{p+k}\big{]}=[\beta_0 + \beta_1X_1+\dots + \beta_pX_p] + [\beta_{p+1}X_{p+1}+\dots + \beta_{p+k}X_{p+k}]$ ...
2
votes
2answers
89 views

Is there any reason to factor in sample weights when applying a scoring function to a test set?

It's my understanding that sample weights are used to ensure that each observation used to train a machine learning model are given a weight corresponding to its perceived importance/value to the ...
0
votes
0answers
7 views

Will out-of-sample tests be significant when the corresponding in-sample tests are not significant?

I am relatively new to the concept of the out-of-sample tests. I understand that the out-of-sample test is conducted through the following steps. (1) Split the data, (2) Use the data in the first ...
1
vote
1answer
42 views

Out of sample prediction

I have a model in which I estimate the impact of price on acreage. My data is composed of 10 years. So I use these 10 years to estimate the model and get to coefficients. In next step, I want to use ...
2
votes
0answers
13 views

Computation of out-of-sample error

I have a question on how one would theoretically compute the out of sample error of a given hypothesis in a data learning problem. I've been working through Learning From Data: A Short Course (http://...
0
votes
0answers
8 views

Out-of-Sample backtesting program theoretical check-up

Say I have a statistical model for forecasting n-steps ahead. Say that I want to backtest it Out-of -Sample. Do you see anything theoretically wrong in the following backtesting procedure? Define ...
1
vote
0answers
24 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
1
vote
1answer
124 views

Out-of-sample Rolling window forecast with ARIMA(0,0,0) with non-zero mean

I am doing a rolling window out-of-sample forecast and have fitted an ARIMA(0,0,1) model to a first difference time series. People argue that sometimes simpler models are better than more complicated ...
5
votes
3answers
138 views

Forecast accuracy rolling window

What is the best way to get a measure of how well an ARIMA model can predict a timeseries when doing an out-of-sample rolling window? I cant use MPE cause it contains zeroes. What I am looking for is ...
0
votes
0answers
39 views

Ridge Regression worse results with more feature. Does it make sense?

PREMISE I am dealing with a regression problem with time-series data (of option prices data). In my setup, I need to use only piece-wise linear models or linear transformations of data. I took care ...
2
votes
1answer
105 views

Worthwhile to do k-fold cross-validation AND a holdout/test set?

I'm relatively new to machine learning, and most of my experience at this stage comes from working with an automated machine learning tool called DataRobot. In their tool, and in their documentation ...
8
votes
2answers
200 views

When should I use Validation rather than Cross Validation

I am aware that CV was born as a way to validate models when there is a lack of training data, but my understanding is that it is generally better to cross validate rather than just use one validation ...
2
votes
1answer
140 views

OOB vs CV for Random Forest

I know this question has been asked dozens of times, but I want to really clarify what is going on when finding the best forest using OOB Error versus CV with Accuracy. From my understanding, a Random ...
1
vote
1answer
268 views

Out of sample and In sample forecasting - R squared

Can anyone explain why R2 (R-squared) for out of sample forecasting is likely to be smaller than R2 for in-sample forecasting?
0
votes
0answers
20 views

Random forest “out-of-bag” ensemble

I am using the R package RandomForestSRC for random forest applications. In the manual for the main function (rfsrc) they mention a setting called ...
2
votes
0answers
75 views

OOB error prediction in RF if case weights are used

I have a dataset for which grossing-up factors are given. I am using these factors as case weights for a random forest (R package ranger). Until now I was using the OOB prediction error for tuning, ...
2
votes
1answer
341 views

Can balanced accuracy be higher than accuracy?

I have classification tree where the balanced accuracy of the test set is higher than the normal accuracy. I thought balanced accuracy can only have at his maximum the same value as the accuracy not ...
2
votes
2answers
107 views

How to check if i have strong linear relationship between dependent variable and independent variables in linear regression (OLS)?

I want compare the out of sample prediction from an linear regression model (OLS) and a regression tree. I read that OLS outperforms regression tree if the relationship between the dependent variable ...
1
vote
3answers
136 views

Can a prediction be better with insignificant variables than with only significant variables (or none at all)?

I have two OLS models and want to do an out of sample prediction for wages on a test set. In the first model I excluded the insignificant variable. The second model has the insignificant variable. The ...
1
vote
0answers
230 views

ARIMA: Produce multi-step, out-of-sample forecasts by feeding in new history without retraining the model? [closed]

I'd like to compare the results of an LSTM model to an ARIMA model. How can I create an ARIMA model in python that trains on the first 70% of data (~2700 observations), and then produces forecasts at ...
2
votes
1answer
294 views

Forecasting with AR(1) and pseudo out-of-sample using R

I'm trying to do Pseudo out-of-sample forecasting using R. And, I also have the following initial data (gdp) ...
0
votes
0answers
51 views

Classification accuracy in holdout similar to CV if set is randomly sampled, completely wrong otherwise

I'm building a classifier to predict a binary label on a dataset with 30 features and around 60000 samples of measurements from a car assembly process. While experimenting with some baseline models ...
1
vote
1answer
113 views

Asymmetric error measure for forecasts

I am building a model for forecasting some number of activations. My data set has a panel structure. Now, I want to come up with a forecast performance measure to assess the performance of my model ...
3
votes
2answers
202 views

What's the real purpose of cross validation?

As for cross evaluation (CV), I have two questions to ask: 1) CV has nothing to do with parameter selection, but only model evaluation? Specifically, which model? 2) In k-fold CV, what's the final ...
2
votes
1answer
891 views

Out-of bag error in Random Forest

I am trying to code my own, simple version of RandomForest function in R for learning purposes. However I have a hard time understanding the concept of the out-of-bag error. Is it simply done by ...
1
vote
0answers
39 views

Out-of-sample predictive checks for Bayesian TVP models

Comparatively new to Bayesian econometrics so apologies if this is a silly question. I am running a time-varying parameter regression where the parameters are estimated as in Primiceri (2005). My ...
2
votes
1answer
905 views

How do I calculate AUC with leave-one-out CV

In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
2
votes
1answer
152 views

Is cross-validation better/worse than a third holdout set?

I see lots of papers that use just train and test datasets, without a third validation set, but they use cross-validation so that every data point is used for training and testing among the different ...
2
votes
1answer
176 views

Reverse prediction in a time series

We know using models like ARIMA we can do out of sample predictions for a Time Series. i.e. we can know what would be the value v at time t. Can we do the reverse of it, and find at what t will be v a ...
0
votes
1answer
41 views

Is there a systematic reason why a model trained on a subset of data does better out-of-sample than the same model trained on the full dataset?

I trained a linear regression model using 3000 data points. (OLS regression, no regularization.) Then I trained another model with the same predictors (about 25), but with a subset ($n=700$) of the ...
2
votes
1answer
684 views

in-sample data vs out-of-sample data

I know that a train-validation-test splits the data into: a training dataset - obviously my "in-sample" data a validation dataset a test data set - obviously my "out-of-sample" data My question is: ...
0
votes
0answers
205 views

Using model from one data set to predict results for another data set

I'm not certain how to phrase this question: I have a dataset of ~45000 execution times of two sets of data. Approximately 35000 of these execution times is ran in one environment, and the remaining ~...
10
votes
2answers
382 views

A ''significant variable'' that does not improve out-of-sample predictions - how to interpret?

I have a question that I think will be quite basic to a lot of users. Im using linear regression models to (i) investigate the relationship of several explanatory variables and my response variable ...
1
vote
0answers
283 views

Why using Out-of-fold predictions as metafeatures in stacking?

So my question is essentially the same as this one: Why do we generate out-of-fold predictions for meta-ensembling/stacking? However, I am not entirely satisfied with the answer (not detailed enough ...
4
votes
2answers
2k views

Stacking without splitting data

I learned Stacking used in Ensemble learning. In Stacking, training data is split into two sets. The first set is used for training each model (layer-1, left figure), the second one is used for ...
4
votes
1answer
4k views

Using $R^2$ to evaluate out-of-sample performance

In this paper the $R^2$ is used to evaluate out-of-sample predictions for several methods including neural networks and tree based methods (see section 3.3 Evaluation and Validation). How is the out-...
0
votes
1answer
52 views

Is it ok to keep/discard rules based on the holdout set?

We have a POC project that is looking for rules that fit our data (eg "when a=1 and b=2 and c=3 then X=6" sort of thing). We split our data into 6 sets, and we use the first 5 sets as K-fold training ...
2
votes
0answers
67 views

What is the relation between replica method and “reusable holdout” method?

Among many methods used to detect and avoid overfitting, I am particularly interested in those two: replica method reusable holdout My question is: what is their relation in the context of adaptive ...
1
vote
1answer
399 views

Is it good practice to use both K-Fold cross validation and hold out validation

When using K-Fold cross validation is it a good or bad idea to split the dataset into two, With 70% (for example) being used for K fold CV and 30% used solely for testing in order to check for over ...
13
votes
4answers
2k views

Predictive models: statistics can't possibly beat machine learning? [closed]

I am currently following a master program focused on statistics/econometrics. In my master, all students had to do 3 months of research. Last week, all groups had to present their research to the rest ...
2
votes
0answers
30 views

Theoretical question on GBM out-of-time performance sensitivity

So this is more of a theoretical question, no dataset or code that I can share. It just came up in a discussion and I was not sure of the answer. Let's say I have 2 GBM models, model A and B, trained ...
0
votes
1answer
413 views

Inconsistent Out-of-bag error estimates [closed]

I keep getting different out-of-bag error estimates from the caret package, depending on how the estimates are computed. I can't seem to nail down exactly where the ...
4
votes
0answers
347 views

Nested Cross Validation: Choosing between different best hyperparameters

I know this sort of question has been asked many times, and several answers have been already provided on this platform too (e.g., here, here, and here). Still, there is something about the idea ...
1
vote
0answers
498 views

prediction using plm for out of sample data in R

I want to predict out of sample data for the same group (lets say state ) for new time window by either fixed or random method.."predict" function is not helping.Here i gave a example of a dataset ...