The out-of-sample tag has no wiki summary.
1
vote
0answers
28 views
Out-of-sample vs. in-sample interpretation
I am running predictive regressions on stock returns and as expected less relations hold out-of-sample than in-sample, however in some cases I find a significant relation out-of-sample but not ...
1
vote
0answers
59 views
Precision, Recall and area under ROC curve as sample size increases
The following is a question from an exam paper on evaluating the performance of search engines. To this day I looked in my text book and literally close to 50 web pages and I can't
find one convincing ...
2
votes
0answers
32 views
Is this a valid method to control the FWER?
I have a huge number, say $M$, of hypotheses that are potentially correlated. I have a dataset $D$ of random samples from an unknown distribution and I want to do test the hypotheses for significance ...
1
vote
1answer
93 views
Dataset and outlier question
I'm facing a data dilemma. I would like to have a real data illustration for an outlier detection rule i'm working on. The outlier detection rule
targets datasets of continuous (not necessarily ...
0
votes
1answer
243 views
Interpreting tree classification errors in Matlab
I am currently using Matlab to generate random forests. I am using the TreeBagger class with the function oobError. I can plot a ...
17
votes
5answers
1k views
New revolutionary way of data mining?
The following excerpt is from Schwager's Hedge Fund Market Wizzards (May 2012), an interview with the consistently successful hedge fund manager Jaffray Woodriff:
To the question: "What are some of ...
4
votes
0answers
27 views
How can I help ensure testing data does not leak into training data? [duplicate]
Possible Duplicate:
How can I help ensure testing data does not leak into training data?
Overfitting is obviously a significant problem in machine-learning...perhaps the most significant ...
0
votes
1answer
334 views
Using rolling windows to compute out of sample accuracy
I’ve already written this question, but probably I didn’t specified it well, for this reason I write it again.
I need to use a random walk model (no-change) yt = yt(1+t) to compute
the ratio of RMSFE. ...
1
vote
1answer
819 views
Random walk out of sample forecasting
Probably my question is a bit stupid, but I'm having some problems in writing down in R the out-of-sample forecasting with a Random Walk. I have a multivariate time series (y) and I want to estimate ...
1
vote
1answer
631 views
Estimating out-of sample forecast for an ARIMA model
I’m trying to estimate the out-of sample forecast of an ARIMA model, I tried the code below, but it totally doesn’t work!
...
0
votes
1answer
302 views
Rolling analysis with out-of sample
I have a model that looks like
lm(y ~ lag(x, -1) + lag(z, -1))
So basically, this is a time series regression with exogenous variables, and I want to carry out ...
1
vote
2answers
123 views
DLM out of sample errors
I'm using the DLM package to estimate a multivariate time series, I wanna check the out of sample forecasting, by estimating the residuals for 1, 6, 12 months ahead forecast? How can I calculate the 6 ...
18
votes
7answers
464 views
How can I help ensure testing data does not leak into training data?
Suppose we have someone building a predictive model, but that someone is not necessarily well-versed in proper statistical or machine learning principles. Maybe we are helping that person as they are ...
0
votes
1answer
209 views
Error with caret, using “out-of-bag” re-sampling
When I run this code (R 2.13.2, caret 4.98 on windows 7):
...
5
votes
3answers
340 views
Bootstrapping estimates of out-of-sample error
I know how to use bootstrap re-sampling to find confidence intervals for in-sample error or R2:
...
