Hot answers tagged out-of-sample
14
You are right, this is a significant problem in machine learning/statistical modelling. Essentially the only way to really solve this problem is too retain an independent test set and keep it held out until the study is complete and use it for final validation.
However, inevitably people will look at the results on the test set and then change their model ...
7
Not sure if there'll be any other "ranty" responses, but heres mine.
Cross Validation is in no way "new". Additionally, Cross Validation is not used when analytic solutions are found. For example you don't use cross validation to estimate the betas, you use OLS or IRLS or some other "optimal" solution.
What I see as a glaringly obvious gap in the quote is ...
5
I suppose the only way to guarantee this is that someone else has the test data. In a client-consultant relationship this can be managed fairly easily: the client gives the consultant the training set upon which to build the models, and within this training set the consultant can split the data in whatever way necessary to ensure that overfitting doesn't ...
5
One way to ensure this is to make sure you have coded up all of the things you do to fit the model, even "tinkering". This way, when you run the process repeatedly, say via cross-validation, you are keeping things consistent between runs. This ensures that all of the potential sources of variation are captured by the cross-validation process.
The other ...
5
This is a very good question and a very subtle problem. Of course there are the
bad intentioned mistakes, which derive from someone trying to deceive you. But
there is a deeper question of how to avoid accidental leaking and avoid honest
mistakes.
Let me list some operational good practices. They all stem from honest mistakes
I've made at some point:
...
5
Does this make any sense? Partly.
What does he mean? Please ask him.
Do you have a clue - or perhaps even a name for the proposed method and some references?
Cross Validation. http://en.wikipedia.org/wiki/Cross-validation_(statistics)
Or did this guy find the holy grail nobody else understands? No.
He even says in this interview that his method ...
4
His explanation about a common error in data mining seems sensible. His explanation of what he does does not make any sense. What does he mean when he says "Generally speaking, you are really getting somewhere if the out-of-sample results are more than 50 percent of the in-sample."? Then bad-mouthing SAS and IBM doesn't make him look very smart either. ...
4
You can look for patterns where, on average, all the models
out-of-sample continue to do well.
My understanding of the word patterns here, is he means different market conditions. A naive approach will analyse all available data (we all know more data is better), to train the best curve fitting model, then run it on all data, and trade with it all the ...
2
I'd say "k-fold cross validation" is the right answer from the theoretical point of view, but your question seems more about organizational and teaching stuff so I'll answer differently.
When people are "still learning" it's often thought as if they're learning how to "quickly and dirtily" apply the algorithms and all the "extra" knowledge (problem ...
2
In some cases, such as Biological sequence-based predictors, it is not enough to ensure that cases do not appear in more than one set. You still need to worry about dependency between the sets.
For example, for sequence-based predictors, one needs to remove redundancy by ensuring that sequences in different sets (including the different cross-validation ...
2
If I remember correctly, some of the forecasting contests (such as Netflix or the ones on Kaggle) use this scheme:
There is a training set, with the "answers".
There is test set #1, for which the researcher provides answers. The researcher finds out their score.
There is test set #2, for which the researcher provides answers, BUT the researcher does not ...
2
Here's a brute-force method, which in general I prefer if a) I can't find an appropriate R function in about 3 minutes, and b) I can see that the brute force function's going to be easy to write.
First, I would start by realigning the variables in a data frame so you don't need to use the lag function:
N <- nrow(y)
df <- ...
2
The rwf function in the forecast package may be useful. You can either use it directly, or take a look at the source code:
> library(forecast)
> rwf
function (x, h = 10, drift = FALSE, level = c(80, 95), fan = FALSE,
lambda = NULL)
{
xname <- deparse(substitute(x))
n <- length(x)
freq = frequency(x)
nn <- 1:h
if ...
2
If you're doing something like the example in the matlab documentation, then the plot contains the Out-Of-Bag Error as a function of the total number of trees.
If you want to cross-validate your model (which you should!), then build the TreeBagger object with some portion of your data and apply it the remaining held-out data with the predict() method. ...
2
As a finance professional I know enough context that the statement does not present any ambiguity. Financial time series are often characterized with regime changes, structural breaks, and concept drift, so cross-validation as practiced in other industries is not as successful in financial applications. In the second part he refers to a financial metric, ...
1
There are three approaches:
Use synthetic data. Then of course you know the outliers.
Add outliers to real data by randomization methods.
Use real data that has a rare class, assuming that the rare instances (should be < 1% of the data set, but you can also downsample the class!) are all outliers.
The first option is of course very limited. The ...
1
In the future it would help if you provided error messages, etc., but for this problem it's easy enough to fix anyway. Just some comments before the code:
1) You don't have to subscript the model or predictions.
2) In the code below, betahat is a vector, not a matrix. Your betahat might also be a vector, which would have caused errors in your run (hence ...
1
As you are using a naive (random walk) model, there are no parameters, so it makes no sense to talk of in-sample and out-of-sample, or of training sets and test sets. There is nothing to train.
On the other hand, you may be doing this to compare the results with other methods applied to a training set. I'll assume that's the case.
I'm guessing your data ...
1
This calls for the standard Efron-Gong "optimism" bootstrap. In R you can do this:
require(rms)
# Allow age to interact with sex and age and BP to have nonlinear effects
# using restricted cubic splines (5 and 4 knots)
f <- ols(y ~ rcs(age,5)*sex + rcs(blood.pressure,4), x=TRUE, y=TRUE)
validate(f, B=300)
This will give you the bootstrap ...
1
Bootstrap is neither in-sample or out-of-sample test.
Consider the bootstrap logic:
1. a statistic is computed in the original sample;
2. a resample is constructed by sampling from the sample with replacement (this sample is considered to be a possible sample from the same population)
3. the same statistic a computed
4. step 2 and 3 are repeated and the ...
1
The short answer, if I understand the questions, is "no". Out of sample error is out of your sample and no bootstrapping or other analytical effort with your sample can calculate it.
In answer to your comment on whether the bootstrap can be used in checking a model with data outside a training set: two possible interpretations.
It would be fine, and ...
Only top voted, non community-wiki answers of a minimum length are eligible
