# Tagged Questions

The tag has no wiki summary.

22 views

### Relative Importance using Out-of-Sample Error

I'm currently running some regressions to understand the nature of price changes in a financial market. In this market, price changes can occur for two reasons: local price changes (which affect the ...
42 views

### BIC vs. Out of sample performance

I have two statistical models. Model 1 uses a GLM approach while model 2 uses a time series approach for fitting. I want to compare these two models. Model 1 (i.e. GLM) has a better out of sample ...
122 views

### Adjusted R squared on a holdout set

The formula for adjusted $R^2$ is: $$1 - \frac{(n-1)}{(n-p-1)}(1-R^2)$$ where $r^2$ is the coefficient of determination, $n$ is the number of points, and $p$ is the number of parameters the model ...
143 views

### Computing Out of Bag error in Random Forest

I am implementing a Random Forest classifier as a side-project, and I am a bit unclear on what the correct approach is to compute, say, the OOB estimate for the classifier error rate. My ...
37 views

### Can holdout validation be systematically biased?

I recently did some experimenting comparing some common method of internal validation. In my field, the use of a single 1:1 holdout validation is extremely common, even with very small datasets, and I ...
60 views

### How can I validate a logistic regression model using averaged parameter estimates?

Let me say thanks in advance. I'm working with a set of data that contains reported coyote sightings. I use 2/3 of the data for model calibration along with an equal number of pseudo absences. I ...
119 views

### Kendall's tau for holdouts low and not significant - Conjoint

I have done my Conjoint Analysis (fractional factorial design) but when it comes to validating the model, it shows a Kendall's tau for Holdouts of 0.33 and not significant. But Pearson's R and ...
32 views

### Sequential estimation for pseudo out-of-sample testing

Suppose, I have a time series of some financial variable, say the logarithmic return, denoted by $r_t$, of length $T$. And furthermore, let's say that my econometric investigations led me to accept ...
101 views

### Choosing the right size of an out of sample data

I am beginner in forecasting, especially forecasting with R and I am really willing to improve my knowledge. Recently, I started practicing electricity consumption time series forecasting. The ...
73 views

### Out-of-sample vs. in-sample interpretation

I am running predictive regressions on stock returns and as expected less relations hold out-of-sample than in-sample, however in some cases I find a significant relation out-of-sample but not ...
111 views

### Precision, Recall and area under ROC curve as sample size increases

The following is a question from an exam paper on evaluating the performance of search engines. To this day I looked in my text book and literally close to 50 web pages and I can't find one convincing ...
35 views

### Is this a valid method to control the FWER?

I have a huge number, say $M$, of hypotheses that are potentially correlated. I have a dataset $D$ of random samples from an unknown distribution and I want to do test the hypotheses for significance ...
98 views

### Dataset and outlier question

I'm facing a data dilemma. I would like to have a real data illustration for an outlier detection rule i'm working on. The outlier detection rule targets datasets of continuous (not necessarily ...
341 views

### Interpreting tree classification errors in Matlab

I am currently using Matlab to generate random forests. I am using the TreeBagger class with the function oobError. I can plot a ...
2k views

### New revolutionary way of data mining?

The following excerpt is from Schwager's Hedge Fund Market Wizzards (May 2012), an interview with the consistently successful hedge fund manager Jaffray Woodriff: To the question: "What are some of ...
30 views

### How can I help ensure testing data does not leak into training data? [duplicate]

Possible Duplicate: How can I help ensure testing data does not leak into training data? Overfitting is obviously a significant problem in machine-learning...perhaps the most significant ...
423 views

### Using rolling windows to compute out of sample accuracy

I’ve already written this question, but probably I didn’t specified it well, for this reason I write it again. I need to use a random walk model (no-change) yt = yt(1+t) to compute the ratio of RMSFE. ...
1k views

### Random walk out of sample forecasting

Probably my question is a bit stupid, but I'm having some problems in writing down in R the out-of-sample forecasting with a Random Walk. I have a multivariate time series (y) and I want to estimate ...
817 views

### Estimating out-of sample forecast for an ARIMA model

I’m trying to estimate the out-of sample forecast of an ARIMA model, I tried the code below, but it totally doesn’t work! ...
388 views

### Rolling analysis with out-of sample

I have a model that looks like lm(y ~ lag(x, -1) + lag(z, -1)) So basically, this is a time series regression with exogenous variables, and I want to carry out ...
144 views

### DLM out of sample errors

I'm using the DLM package to estimate a multivariate time series, I wanna check the out of sample forecasting, by estimating the residuals for 1, 6, 12 months ahead forecast? How can I calculate the 6 ...
667 views

### How can I help ensure testing data does not leak into training data?

Suppose we have someone building a predictive model, but that someone is not necessarily well-versed in proper statistical or machine learning principles. Maybe we are helping that person as they are ...