# Tagged Questions

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

253 views

### Calculating prediction intervals when using cross validation

Are standard deviation estimates calculated via: $s_N = \sqrt{\frac{1}{N} \sum_{i=1}^N (x_i - \overline{x})^2}.$ (http://en.wikipedia.org/wiki/Standard_deviation#Sample_standard_deviation) for ...
94 views

### Bayesian, MDL or ML interpretation of cross-validation?

Is there any known Bayesian, ML or MDL interpretation of cross-validation? Can I interpret cross validation as performing the right update on a specifically crafted prior?
115 views

### Estimating model error in $k$-nearest neighbours with strongly spatially autocorrelated training data

In the palaeoclimate world, palaeoecologists have used spatial training sets of say sea-surface temperture (SST) and related this to micro-organisms living at the locations where SST was measured. A ...
146 views

### Implementation of the cross validiation

I'm attending a course in computational statistics, which should be an applied course. We study different methods, which are important in "reality". One of these topics is Cross Validation. I'm faced ...
26 views

### How to pick a trained model instance when its performance fluctuates a lot?

I have a dataset that I have to perform a regression task. I split the dataset into 80% for training and 20% for validation. ...
60 views

### Cross validation for lasso logistic regression

I am writing a routine for logistic regression with lasso in matlab. So the problem is to minimize the negative log-likelihood function with the penalty term ...
51 views

### What's a good range of weights to evaluate for $L_2$ regularized logistic regression?

I want find a weight that minimizes an averaged cross validated misclassification score from a $L_2$ logistic regression classifier. Obviously, the search space for the weights should be bounded below ...
61 views

### Assumptions of two-way ANOVA and k-fold cross validation

I want to compare 3 classifiers (kNN, SVM and CT) by using their classification accuracies on 10 folds, to highlight eventual differences between them. I think it could be done by a two-way ANOVA ...
73 views

### Measuring parameter sensitivity and variability (standard-error) in k-fold cross-validation

I mainly use k-fold cross-validation for parameter tuning and model selection for prediction problems. Now, is there a standard or if not a less-known way to measure the sensitivity of the parameters ...
62 views

### Rule of thumb for tuning the values of the penalty parameter in SVM models

I have recently been running into computational issues in fitting a soft-margin SVM model using the e1071 package in R. The issue is unavoidable since the problem ...
144 views

### “Leave one object out” cross validation

I have a training set with 140 instances and no separate cross validation set. The data set contains 7 measurements from each of 20 objects, hence the 140 instances. Each of 7 measurements have the ...
154 views

### Logistic regression discrimination threshold with cross validation

I'm using logistic regression to perform binary classification with training, CV, and test sets. When is the most appropriate time to pick a discrimination threshold to balance positive and negative ...
35 views

### Is this a valid method to control the FWER?

I have a huge number, say $M$, of hypotheses that are potentially correlated. I have a dataset $D$ of random samples from an unknown distribution and I want to do test the hypotheses for significance ...
96 views

### is it valid to take a mean of p-values during cross-validation, when comparing the predicted output of a model to the actual output?

I am doing a cross-validation study, training a model on an input to predict a target. During training, my model generates an output vector that is guaranteed to be the same size as the corresponding ...
351 views

### K-Fold Cross Validation for mixed-effect models: how to score them?

I'm trying to use k-fold cross validation for model selection for a mixed-effect model (fitted with the lme function). But, what exactly do I use as the score for ...
99 views

### What is the relationship between 10-fold CV RMSE and test set RMSE score?

In my opinion, these two RMSEs are proportion related. I use the training set to get 10-fold CV RMSE and get a model on the whole training set to predict on the test set. As the training set is larger ...
189 views

### Is it possible to have xerror increased in a tree using rpart?

I am new to R and rpart package. When I plot the tree using rpart: ...
278 views

### GAM cross-validation to test prediction error

My questions deals with GAMs in the mgcv R package. Due to a small sample size I want to determine the prediction error using leave-one-out cross-validation. Is this reasonable? Is there a package or ...
137 views

### Cross validation for multivariate imputation

I am currently using the mice: Multivariate Imputation by Chained Equations in R (JSS 2011 45(3)) package. Consider the following example. I am using Sites B to Z and mice() to help infill missing ...
48 views

49 views

### Cross-Validation: Include IV which are significant in one model but insignificant in the other?

I ran regression analyses using SAS with two different data sets containing different individuals but exactly the same IV and DV: let's call them "low_deviance" and "high_deviance" and I would like to ...
51 views

### Does it make sense to minimize negative log likelihood on SVC probability outputs

I'd like to run a grid search cross-validation on the probability outputs of the SVC classifier. In particular I'd like to minimize the negative log likelihood. Is this a reasonable thing to do? I'm ...
15 views

### Cross validation with first and second order effects

Our project involves looking at different chemical properties and developing a model which incorporates first and second order effects to predict properties of new chemicals. The data set consists of ...
37 views

### Conditional cross-validation sampling

When applying cross-validation to evaluate the predictive performance of a binary classification model, is it acceptable to separately sample from cases and non-cases to achieve class proportions in ...
47 views

### Non-nested model verification

My questions are two fold: Is there a generally accepted statistic used to compare non-nested, nonlinear models with different numbers of parameters? I'm thinking RMSE, but wondering what other ...
192 views

### k-fold cross validation vs k times hold-out validation

I am facing the evaluation of a genetic programming algorithm. I am using the Proben1 cancer1 dataset to evaluate the models created by this algorithm. This dataset contains 699 samples, which is ...
77 views

### Which measure could tell me about what is the best predictor in survival analysis?

I have data of cancer survival, like that: > my.data survival stage my_class 27 3 2 221 2 1 43 3 3 ... The survival is the time ...
65 views

### Training with very few positives

I have a binary classification problem where the fraction of positives is very low, e.g. 20 positives in 10,000 examples (0.2%) What is an appropriate cross validation scheme for training a ...
224 views

### How to output training error when using cv.glmnet from the glmnet package in R?

I am currently using the glmnet package in R along with its' cross validation function cv.glmnet. As a reminder, ...
180 views

### Bootstrap or jack-knife for crossvalidation of predictive model?

Is a bootstrap or jack-knife method better for crossvalidation of a multivariate logistic regression based predictive model?
35 views

### On cross-validation schemes for “rectangular” samples

Consider this example. Suppose that for any pair $(x, y)$ of bacterial strain $x$ and (candidate) anti-bacterial agent $y$, we can experimentally determine some measure $f(x, y)$ (say, the ...
69 views

### CV of data after nonparametric regression

I want to do cross validation after running nonparametric regression on my data. Unlike parametric regression where I can first find my parameters and then easily handle the CV set with these ...