Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [out-of-sample]

Refers to the practice of assessing model performance on some "test" or "holdout" or "out-of-sample" set of data that was not used for model building.

1
vote
2answers
805 views

Choosing the right size of an out of sample data

I am beginner in forecasting, especially forecasting with R and I am really willing to improve my knowledge. Recently, I started practicing electricity consumption time series forecasting. The ...
1
vote
0answers
334 views

Out-of-sample vs. in-sample interpretation

I am running predictive regressions on stock returns and as expected less relations hold out-of-sample than in-sample, however in some cases I find a significant relation out-of-sample but not in-...
2
votes
1answer
494 views

Precision, Recall and area under ROC curve as sample size increases

The following is a question from an exam paper on evaluating the performance of search engines. To this day I looked in my text book and literally close to 50 web pages and I can't find one convincing ...
2
votes
0answers
73 views

Is this a valid method to control the FWER?

I have a huge number, say $M$, of hypotheses that are potentially correlated. I have a dataset $D$ of random samples from an unknown distribution and I want to do test the hypotheses for significance ...
1
vote
1answer
221 views

Dataset and outlier question

I'm facing a data dilemma. I would like to have a real data illustration for an outlier detection rule i'm working on. The outlier detection rule targets datasets of continuous (not necessarily ...
0
votes
1answer
2k views

Interpreting tree classification errors in Matlab

I am currently using Matlab to generate random forests. I am using the TreeBagger class with the function oobError. I can plot a ...
21
votes
5answers
3k views

New revolutionary way of data mining?

The following excerpt is from Schwager's Hedge Fund Market Wizzards (May 2012), an interview with the consistently successful hedge fund manager Jaffray Woodriff: To the question: "What are some of ...
4
votes
0answers
73 views

How can I help ensure testing data does not leak into training data? [duplicate]

Possible Duplicate: How can I help ensure testing data does not leak into training data? Overfitting is obviously a significant problem in machine-learning...perhaps the most significant problem. ...
0
votes
1answer
1k views

Using rolling windows to compute out of sample accuracy

I’ve already written this question, but probably I didn’t specified it well, for this reason I write it again. I need to use a random walk model (no-change) yt = yt(1+t) to compute the ratio of RMSFE. ...
1
vote
1answer
3k views

Random walk out of sample forecasting

Probably my question is a bit stupid, but I'm having some problems in writing down in R the out-of-sample forecasting with a Random Walk. I have a multivariate time series (y) and I want to estimate ...
1
vote
1answer
3k views

Estimating out-of sample forecast for an ARIMA model

I’m trying to estimate the out-of sample forecast of an ARIMA model, I tried the code below, but it totally doesn’t work! ...
2
votes
3answers
4k views

Rolling analysis with out-of sample

I have a model that looks like lm(y ~ lag(x, -1) + lag(z, -1)) So basically, this is a time series regression with exogenous variables, and I want to carry out ...
1
vote
2answers
258 views

DLM out of sample errors

I'm using the DLM package to estimate a multivariate time series, I wanna check the out of sample forecasting, by estimating the residuals for 1, 6, 12 months ahead forecast? How can I calculate the 6 ...
58
votes
8answers
10k views

How can I help ensure testing data does not leak into training data?

Suppose we have someone building a predictive model, but that someone is not necessarily well-versed in proper statistical or machine learning principles. Maybe we are helping that person as they are ...
0
votes
1answer
1k views

Error with caret, using “out-of-bag” re-sampling

When I run this code (R 2.13.2, caret 4.98 on windows 7): ...
7
votes
3answers
4k views

Bootstrapping estimates of out-of-sample error

I know how to use bootstrap re-sampling to find confidence intervals for in-sample error or R2: ...