Questions tagged [out-of-sample]

Refers to the practice of assessing model performance on some "test" or "holdout" or "out-of-sample" set of data that was not used for model building.

Filter by
Sorted by
Tagged with
0
votes
0answers
23 views

how to explain accuracy score of 0.51 when training and test scores are around 0.79? [closed]

I am trying to solve some machine learning problem, I dont understand how accuracy can be so lower than training score and test score
4
votes
1answer
37 views

Do in-sample distributional assumptions (e.g. normally distributed residuals) apply also to out-of-sample predictions?

I appreciate that out-of-sample error is important (i.e. comparing how close a model's predictions are to leftout data) but are there any distributional assumptions about the out-of-sample predictions?...
0
votes
0answers
32 views

Out-of-sample MSE and MAE for volatility forecasting [duplicate]

I have been searching through the whole CrossValidated but couldn't find the answer. I want to test out-of-sample the volatility forecasts (if it means something ARCH-like ones, MSGARCH, Multifractal ...
1
vote
0answers
26 views

Pros and cons of R squared and normalized RMSE as a scale-free out-of-sample performance metric for regression?

I am mainly considering the nRMSE = RMSE/sd, but if you are more familiar with RMSE/mean or RMSE/min-max that would also be interesting. I am looking for pros/cons for the two, or different use cases ...
1
vote
1answer
18 views

Bagged Decision trees / Random Forests: why ISLR uses validation set instead of OOB to compute out-of-sample MSE?

I am reading the book "An Introduction to Statistical Learning" available here. Chapter 8.3.3 at page 328 of the book computes a bagged decision tree (which is a random forest where we use ...
0
votes
0answers
16 views

Out of sample testing and tail function

Below you see an out of sample rolling window estimation I found here: https://www.r-bloggers.com/2017/11/formal-ways-to-compare-forecasting-models-rolling-windows/ Here is my question: I know the ...
0
votes
0answers
14 views

Cross-validation and subsequent hold-out testing makes sense?

I'm integrating a modelling approach into an auto-tuning framework. Basically what I'm trying to achieve is creating a model over a subset of observation values and use it to predict the remaining ...
2
votes
0answers
16 views

State-of-the-art methods for out-of-sample-extension

I'm using a kernel based dimensionality reduction algorithms, and interested in extending out-of-sample data points for further analysis. I've been using the Nystrom method for this task, and some ...
0
votes
2answers
42 views

Optimal Machine Learning Sample size

I am new in ML area and I want to build a model to score 100,000 people. I wonder if it is abnormal to build (train/test/validation) the model on a dataset of the same size 100,000? Need to mention ...
0
votes
0answers
13 views

Harvey-Newbold test (2000) in R

I want to estimate the best nested model out-of sample and for that purpose, I will implement Harvey-Newbold test. However, I cannot find the respective test in any package. Can anyone help me? If ...
1
vote
1answer
66 views

Out of sample ARCH forecast

I have estimated a conditional mean model for a time series: $ x_t = x_{t-1} + \epsilon_t$. Say I have estimated it using periods 1 to 10. I can do an out of sample conditional mean forecast by ...
0
votes
0answers
11 views

out of sample prediction standard error and confidence limit using bootstrap

I am analyzing growth data to determine if sibling competition affect growth using GLIMMIX procedure in SAS with different class and continuous variables and 2 random effects. I want to predict out of ...
0
votes
0answers
17 views

OOB error reduction

Assuming one has enough data for training, does it make sense to try to reduce the OOB standard error in Random Forest by decreasing the fraction of the data for training if the accuracy R2 is kept ...
1
vote
1answer
143 views

How to calculate the fitted values of out-of-sample predictions from conditional fixed-effects Poisson model?

I am trying to calculate fitted values from xtpoisson fixed effects on out-of-sample data. I know how to calculate fitted values for in-sample predictions (using ...
0
votes
0answers
21 views

Unstable Out-of-Sample Prediction with Gradient Boosting Trees

I have a continuous response, ranging between -100 and 100, but it's highly leptokurtic at 0. I also have a lot of predictors to use. After variable reduction and parameter tuning, the prediction of ...
0
votes
0answers
15 views

In-sample-performance when the in-sample part is divided by k-fold-cross validation

Say, I have the following set up. I have got a dataframe, which is divided into a training and a test set. On the training set I conduct variable selection/Hyperparameter tuning/whatever by k-fold-...
0
votes
0answers
14 views

Out of Sample Model Consistently Outperforming In-Sample

I am finding that my model consistently performs better out-of-sample. I am splitting my testing and training data into equal parts, and doing 100 iterations of re-sampling. It performs better OOS ...
0
votes
0answers
54 views

Model evaluation via k-fold-cross validation after variable selection

I am currently trying to implement some methods (OLS, GLM) and compare their predictive accuracy. For that I want to perform variable selection for each of the methods and then measure the predictive ...
0
votes
1answer
68 views

Nested CV vs CV with a holdout?

I just learned of nested cross-validation and wanted to understand how my current approach is worse/ok. Currently I would: Divide the data into a train/test set (80/20ish). Use k-fold cross-...
2
votes
1answer
72 views

What is the posterior in-sample vs the posterior out-of-sample?

I'm watching this video on Bayesian modelling for the stock market by Thomas Wiecki, Thomas has a slide with two posterior distribution over the mean parameter in his stock return model. Around 18:26 ...
0
votes
0answers
33 views

What I'm dealing with here VAR, VARX or something different?

I have to implement a time-series tool in my company, and I'm not sure what I'm dealing with, when looking at the old tool. We have one target variable (sales) and a lot of different independent ...
0
votes
1answer
23 views

Deciding between linear and non-linear approach in a predictive model when the relationship 'looks linear'

When we have a relatively small number of samples it is easy to see on a plot what is intuitively happening when we fit a regression line; we can see how far each of the individual sample points are ...
1
vote
0answers
59 views

An out of bag testing procedure I developed. Is it biased?

Trying to estimate the error of a prediction model trained on a combined group vs a subset of interest within that group (for example, Men) My procedure is First simulate a random subset like the ...
2
votes
1answer
39 views

Can entropy be used to minimize prediction surprises in machine learning?

Information theory deals with signal/noise identification, while one of its tools, entropy, measures the surprise in random probabilistic outcomes. Has there been any application of using entropy or ...
0
votes
0answers
20 views

“Out-of-sample updating”: Applying estimated rates of change from one sample to another

[If you know the terms I could use to find the literature on this, that information alone would be a very helpful answer.] My model uses a survey -- let's call it O, for "old" -- that ran ...
0
votes
0answers
13 views

Can Gaussian mixture models help an algorithm target a specific cluster?

In the chart below is a Gaussian Mixture model (GMM) based on three time series or datasets that the model was able to easily cluster into three different colored classes. Class 1 is the blue ellipse ...
0
votes
0answers
12 views

How to pull in-sample fitted models towards out-of-sample optimum during training?

A common problem in prediction problems is getting a model fit on in-sample data to predict out-of-sample data with decent accuracy and precision. Assuming that we break a part of the in-sample data ...
3
votes
0answers
32 views

Forecasting with empirical copulas

I estimated the beta copula with 3 variable time series. Now I'd like to make forecasts to evaluate the out-of-sample performance of my model. I know 2 of the 3 variables and I have the dependence ...
1
vote
0answers
58 views

How to evaluate multiple data imputation?

I am performing data imputation of multiple time-series using various ML techniques (such as multiple imputation, iterative imputation). I have a matrix of ~100,000 observations (rows) of 34 stations (...
1
vote
1answer
20 views

Should test MSE be bootstrapped to compare fits?

Suppose you have a training and testing set. You fit two models, A and B, to the training set. They you predict on the testing set. You find (in this contrived example): Test MSE model A: 3 Test MSE ...
0
votes
0answers
161 views

ARIMA interpreting results and how to out-of-sample forecast

I am trying to learn ARIMA using Python and the data in https://www.kaggle.com/c/demand-forecasting-kernels-only. I'm using the sales for Store == 1. Here is how the data looks like: Here is my ARIMA ...
1
vote
0answers
29 views

Should I standardize forecasted and actual values for Mincer Zarnowitz test?

I have some out-of-sample forecasted values of variance through some GARCH model, and now I am trying to perform a Mincer-Zarnowitz test for validity of my predictions. I first standardized both ...
0
votes
2answers
32 views

Most likely sources of divergence between (adjusted)-R squared and out-of-sample predictive performance

I'm wondering which invalid assumptions are most likely to explain the wild discrepancies between a model's R-squared as a measure of predictive performance, and the actual out-of-sample predictive ...
0
votes
1answer
559 views

How to calculate the confidence interval with weighted data?

I've done some search for similar questions, but they're not the same as what I'm trying to get. Assume that there's a server that handles requests $r$ and returns a set of items $I_{r}$ of random ...
1
vote
1answer
715 views

In sample splitting for time series data, do we randomly select data?

I'm having a hard to conceptually understanding how to do this. I would like to do my own sample splitting (not the method built into a package). Let's say you have 80 days of weather data. You ...
3
votes
1answer
72 views

Regression hypothesis testing via out-of-sample testing

Let's consider two linear models. $$\text{Full model}\\\mathbb{E}\big{[}Y\big{\vert} X_1,\dots,X_p, X_{p+1},\dots,X_{p+k}\big{]}=\\\beta_0 + \bigg[\beta_1X_1+\dots + \beta_pX_p\bigg] + \bigg[\beta_{p+...
2
votes
2answers
816 views

Is there any reason to factor in sample weights when applying a scoring function to a test set?

It's my understanding that sample weights are used to ensure that each observation used to train a machine learning model are given a weight corresponding to its perceived importance/value to the ...
1
vote
1answer
77 views

Out of sample prediction

I have a model in which I estimate the impact of price on acreage. My data is composed of 10 years. So I use these 10 years to estimate the model and get to coefficients. In next step, I want to use ...
2
votes
1answer
27 views

Computation of out-of-sample error

I have a question on how one would theoretically compute the out of sample error of a given hypothesis in a data learning problem. I've been working through Learning From Data: A Short Course (http://...
1
vote
0answers
26 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
1
vote
1answer
261 views

Out-of-sample Rolling window forecast with ARIMA(0,0,0) with non-zero mean

I am doing a rolling window out-of-sample forecast and have fitted an ARIMA(0,0,1) model to a first difference time series. People argue that sometimes simpler models are better than more complicated ...
5
votes
3answers
263 views

Forecast accuracy rolling window

What is the best way to get a measure of how well an ARIMA model can predict a timeseries when doing an out-of-sample rolling window? I cant use MPE cause it contains zeroes. What I am looking for is ...
0
votes
0answers
98 views

Ridge Regression worse results with more feature. Does it make sense?

PREMISE I am dealing with a regression problem with time-series data (of option prices data). In my setup, I need to use only piece-wise linear models or linear transformations of data. I took care ...
3
votes
1answer
502 views

Worthwhile to do k-fold cross-validation AND a holdout/test set?

I'm relatively new to machine learning, and most of my experience at this stage comes from working with an automated machine learning tool called DataRobot. In their tool, and in their documentation ...
8
votes
2answers
217 views

When should I use Validation rather than Cross Validation

I am aware that CV was born as a way to validate models when there is a lack of training data, but my understanding is that it is generally better to cross validate rather than just use one validation ...
4
votes
1answer
940 views

OOB vs CV for Random Forest

I know this question has been asked dozens of times, but I want to really clarify what is going on when finding the best forest using OOB Error versus CV with Accuracy. From my understanding, a Random ...
1
vote
1answer
2k views

Out of sample and In sample forecasting - R squared

Can anyone explain why R2 (R-squared) for out of sample forecasting is likely to be smaller than R2 for in-sample forecasting?
2
votes
0answers
124 views

OOB error prediction in RF if case weights are used

I have a dataset for which grossing-up factors are given. I am using these factors as case weights for a random forest (R package ranger). Until now I was using the OOB prediction error for tuning, ...
4
votes
1answer
726 views

Can balanced accuracy be higher than accuracy?

I have classification tree where the balanced accuracy of the test set is higher than the normal accuracy. I thought balanced accuracy can only have at his maximum the same value as the accuracy not ...