Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...
5
votes
0answers
161 views
Calculating prediction intervals when using cross validation
Are standard deviation estimates calculated via:
$
s_N = \sqrt{\frac{1}{N} \sum_{i=1}^N (x_i - \overline{x})^2}.
$
(http://en.wikipedia.org/wiki/Standard_deviation#Sample_standard_deviation)
for ...
5
votes
0answers
85 views
Bayesian, MDL or ML interpretation of cross-validation?
Is there any known Bayesian, ML or MDL interpretation of cross-validation? Can I interpret cross validation as performing the right update on a specifically crafted prior?
5
votes
0answers
306 views
AUC in ordinal logistic regression
I'm using 2 kind of logistic regression - one is the simple type, for binary classification, and the other is ordinal logistic regression. For calculating the accuracy of the first, I used ...
4
votes
0answers
72 views
Implementation of the cross validiation
I'm attending a course in computational statistics, which should be an applied course. We study different methods, which are important in "reality". One of these topics is Cross Validation. I'm faced ...
3
votes
0answers
508 views
Validation: Data splitting into training vs. test datasets
I was naively validating my binomial logit models by testing on a test dataset. I had randomly divided the available data (~2000 rows) into training (~1500) and validation (~500) datasets.
I now ...
3
votes
0answers
47 views
What's a good range of weights to evaluate for $L_2$ regularized logistic regression?
I want find a weight that minimizes an averaged cross validated misclassification score from a $L_2$ logistic regression classifier. Obviously, the search space for the weights should be bounded below ...
2
votes
0answers
28 views
Model variance and bias in cross validation
his question is partly inspired about the answer to this other question: Number of folds for K-fold.
The fundamental question I have is the following: How do different cross-validation methods ...
2
votes
0answers
33 views
Measuring parameter sensitivity and variability (standard-error) in k-fold cross-validation
I mainly use k-fold cross-validation for parameter tuning and model selection for prediction problems. Now, is there a standard or if not a less-known way to measure the sensitivity of the parameters ...
2
votes
0answers
47 views
Rule of thumb for tuning the values of the penalty parameter in SVM models
I have recently been running into computational issues in fitting a soft-margin SVM model using the e1071 package in R.
The issue is unavoidable since the problem ...
2
votes
0answers
111 views
“Leave one object out” cross validation
I have a training set with 140 instances and no separate cross validation set. The data set contains 7 measurements from each of 20 objects, hence the 140 instances. Each of 7 measurements have the ...
2
votes
0answers
139 views
Logistic regression discrimination threshold with cross validation
I'm using logistic regression to perform binary classification with training, CV, and test sets. When is the most appropriate time to pick a discrimination threshold to balance positive and negative ...
2
votes
0answers
32 views
Is this a valid method to control the FWER?
I have a huge number, say $M$, of hypotheses that are potentially correlated. I have a dataset $D$ of random samples from an unknown distribution and I want to do test the hypotheses for significance ...
2
votes
0answers
71 views
is it valid to take a mean of p-values during cross-validation, when comparing the predicted output of a model to the actual output?
I am doing a cross-validation study, training a model on an input to predict a target.
During training, my model generates an output vector that is guaranteed to be the same size as the corresponding ...
2
votes
0answers
288 views
K-Fold Cross Validation for mixed-effect models: how to score them?
I'm trying to use k-fold cross validation for model selection for a mixed-effect model (fitted with the lme function).
But, what exactly do I use as the score for ...
2
votes
0answers
84 views
What is the relationship between 10-fold CV RMSE and test set RMSE score?
In my opinion, these two RMSEs are proportion related. I use the training set to get 10-fold CV RMSE and get a model on the whole training set to predict on the test set. As the training set is larger ...
2
votes
0answers
110 views
Is it possible to have xerror increased in a tree using rpart?
I am new to R and rpart package. When I plot the tree using rpart:
...
2
votes
0answers
233 views
GAM cross-validation to test prediction error
My questions deals with GAMs in the mgcv R package. Due to a small sample size I want to determine the prediction error using leave-one-out cross-validation. Is this reasonable? Is there a package or ...
2
votes
0answers
124 views
Cross validation for multivariate imputation
I am currently using the mice: Multivariate Imputation by Chained Equations in R (JSS 2011 45(3)) package.
Consider the following example. I am using Sites B to Z and mice() to help infill missing ...
2
votes
0answers
44 views
Cross validation for transformed and untransformed outcome
Suppose that I am interested in selecting from one of the following model spaces:
$$\begin{align*} y &= \beta_0 + \beta_1 x_i + \epsilon_i \, \, \text{or} \\
\log (y_i) &= \gamma_0 + ...
2
votes
0answers
128 views
Calculation of seasonal (annual) component of time-series: use of cross-validation?
I've been working for almost a year on electricity load forecasting in collaboration with some climate scientists, using temperature data obtained from models. Instead of using directly temperature ...
2
votes
0answers
331 views
Cross validation procedure - is this right?
Just want to check that I am performing my cross validation procedures right. I'm using a non-linear svm. I do a five fold cross validation (5 splits of test/train on my original training data) and ...
2
votes
0answers
118 views
How to use cross validation to estimate autocorrelation?
I have a stationary time series and I want to calculate the autocorrelation coefficient of order 1. For that I use OLS. I know the autocorrelation parameters change over the time. Thus it is not ...
2
votes
0answers
108 views
Does cross validation work with asymmetric loss functions?
My simple question is does cross validation work with an asymmetric loss function?
I cannot find docs on google to answer.
2
votes
0answers
78 views
Validating a model for a set of DNA sequences
Consider a set of DNA sequences i.e. strings composed from the four
letters A, C, G, T.
We model these sequences as a random vector, $\mathbf{X}=(X_1,\dots,
X_n)$. Each member of the set of ...
2
votes
0answers
74 views
Estimator of mu for gaussian variable with penalty
I have a question about the following problem: estimate $\mu=\mu_1,...\mu_n$ when $Y_i \sim N(\mu_i,\sigma^2)$ using a ridge like penalty
$$
\min_\mu \sum_i(Y_i-\mu_i)^2 + \lambda\sum_i\mu_i^2
$$
...
2
votes
0answers
127 views
Explaining regression performance differences
Context
I have a regression framework and two sets of data. Using leave-one-out cross-validation, the first set gives very good performance and the second set gives rather poor performance. I need to ...
2
votes
0answers
453 views
Model selection in Weka through cross validation for regression problems
Does anyone know an approach to performing model selection in Weka through cross validation for regression problems?
As far as I can tell, the cross validation is implemented in Weka just to assess ...
1
vote
0answers
19 views
LOOCV parameter and feature selection
I have a small data set (30 subjects) and a very big feature space (960 features).
I need to create a classifier to distinguish between two classes and I want to corroborate whether my method makes ...
1
vote
0answers
25 views
Does it make sense to minimize negative log likelihood on SVC probability outputs
I'd like to run a grid search cross-validation on the probability outputs of the SVC classifier. In particular I'd like to minimize the negative log likelihood. Is this a reasonable thing to do? I'm ...
1
vote
0answers
14 views
Cross validation with first and second order effects
Our project involves looking at different chemical properties and developing a model which incorporates first and second order effects to predict properties of new chemicals. The data set consists of ...
1
vote
0answers
27 views
Conditional cross-validation sampling
When applying cross-validation to evaluate the predictive performance of a binary classification model, is it acceptable to separately sample from cases and non-cases to achieve class proportions in ...
1
vote
0answers
23 views
Non-nested model verification
My questions are two fold:
Is there a generally accepted statistic used to compare non-nested, nonlinear models with different numbers of parameters? I'm thinking RMSE, but wondering what other ...
1
vote
0answers
89 views
k-fold cross validation vs k times hold-out validation
I am facing the evaluation of a genetic programming algorithm. I am using the Proben1 cancer1 dataset to evaluate the models created by this algorithm. This dataset contains 699 samples, which is ...
1
vote
0answers
57 views
Which measure could tell me about what is the best predictor in survival analysis?
I have data of cancer survival, like that:
> my.data
survival stage my_class
27 3 2
221 2 1
43 3 3
...
The survival is the time ...
1
vote
0answers
56 views
Training with very few positives
I have a binary classification problem where the fraction of positives is very low, e.g. 20 positives in 10,000 examples (0.2%)
What is an appropriate cross validation scheme for training a ...
1
vote
0answers
157 views
How to output training error when using cv.glmnet from the glmnet package in R?
I am currently using the glmnet package in R along with its' cross validation function cv.glmnet. As a reminder, ...
1
vote
0answers
160 views
Bootstrap or jack-knife for crossvalidation of predictive model?
Is a bootstrap or jack-knife method better for crossvalidation of a multivariate logistic regression based predictive model?
1
vote
0answers
33 views
On cross-validation schemes for “rectangular” samples
Consider this example. Suppose that for any pair $(x, y)$ of bacterial strain $x$ and (candidate) anti-bacterial agent $y$, we can experimentally determine some measure $f(x, y)$ (say, the ...
1
vote
0answers
59 views
CV of data after nonparametric regression
I want to do cross validation after running nonparametric regression on my data. Unlike parametric regression where I can first find my parameters and then easily handle the CV set with these ...
1
vote
0answers
86 views
Generalized Linear Model
How do I identify accuracy data in each class in GLM via cross validation (in Matlab)?
Is R in GLM the overall accuracy?
1
vote
0answers
246 views
optimal choice of smooth.spline parameter?
I'm analyzing a time series (terms of trades) on which I want to perform a trend estimation by nonparametric methods like the above mentioned. By the way, I'm a total beginner with R and using the ...
1
vote
0answers
84 views
Can we use cross-validation to measure how well a distribution fits sample data?
Let's say I have a data set $X = [1,2,3,4,5]$. And I want to measure how close it is to a Gaussian distribution. Is there a way to use cross-validation to do this?
For example, if I do ...
1
vote
0answers
118 views
Cross validation as a measure of SVM generalizability
I am trying to implement the following paper: Learning the Kernel Matrix with Semidefinite Programming using cvx toolbox for matlab. My question is about how exactly I can use this method with ...
1
vote
0answers
24 views
Evaluation and Testsets for NNMF
I am trying to evaluate my recommender system which uses Non-negative Matrix Factorization. Some things that I evaluate are
How does the size of the feature matrix affect the recommendations
How ...
1
vote
0answers
542 views
rpart and the printcp function
I don't really understand how the columns "xerror" and "rel error" are calculated.
I found out that the printcp() function "gives cross-validation estimates of misclassication error (xerror), ...
1
vote
0answers
260 views
Cross validation using experimental design methods?
I have a question about using experimental design methods for cross validation in a non-conventional approach.
I will use an illustration to explain the question. Imagine you have 100 time series ...
1
vote
0answers
73 views
How to use multiple datasets in order to measure the performance of a learning system?
I’m working on a project where I need to test a machine learning system which has a lot of hyper-parameters. Further, in order to gauge the performance of system, I’m planning to use several ...
1
vote
0answers
103 views
Robust Support Vector Regression - robust to outliers
I've been reading/looking around for literature on support vector regressions that are relatively robust to outliers. I understand that standard SVRs can be significantly influenced by a few large ...
1
vote
0answers
113 views
Binary classification of DNA sequence motifs
[I apologize in advance for the length of this question. but it seems
unavoidable.]
I'm trying to do binary classification, using Bayesian methods, and
I'm having some trouble figuring what methods I ...
1
vote
0answers
86 views
Cross validation for a biased estimator of a Gaussian mean
Consider the mean estimator
$$\hat{\mu}(\lambda) = \lambda \frac{1}{n}\sum_{i = 1}^nY_i $$
(for $n$ iid Gaussian variates $Y_i$). After calculating the bias and the variance of this estimator, I ...
