Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
1answer
49 views

Cross validation for logistic regression in R [on hold]

I am doing a K-fold cross validation for a logistic regression. I first used the PCA to reduce the dimensionality. Then I used those principal components to build the regression model. My problem is ...
1
vote
0answers
18 views

Cross-validation for parameter tuning in data mining process (KDD)

In my project I want to compare different classification algorithms to solve a specific problem with a specific dataset. To do this, I divided the dataset in 2 parts. With the first (bigger) part I ...
0
votes
0answers
32 views

AIC versus cross validation in time series

I am interested in model selection in a time series setting. For concreteness, suppose I want to select an ARMA model from a pool of ARMA models with different lag orders. The ultimate intent is ...
1
vote
1answer
16 views

How to do test set evaluation using a regression model in Caret?

I'm used to using Caret to do classification but now I need to use it for regression. I have successfully trained a model on my training set but I'm not sure what ...
0
votes
1answer
12 views

code for cross validation in matlab libsvm

i have posted a question but due to internet problem can't correct it and it was posted sorry for that. now i want to share my problem i want to use cross validation and grid search to classify my ...
-1
votes
0answers
5 views

cross validation code using libsvm [migrated]

i have installes the libsvm to matlab an hour ago. i also read out about is parameters but coul not get how to write a code for cross validation and grid search. i write this: % model_precomputed = ...
4
votes
1answer
67 views

Likelihood ratio test disagrees with cross-validation results

I have computed two logistic models of the same data (for different formulas) in R, and compared them using likelihood ratio test: ...
0
votes
0answers
13 views

leave-one-out error estimate variance

What would be the lower bound for the variance of the leave-one-out error estimate? Not necesary has to hold for all distribution. Any example would be nice. Also the classifier for the example could ...
0
votes
0answers
8 views

tbats{forecast} in R gives strange predictions for some folds in cross validation

My daily data shows weekly and yearly seasonality, so I decide to try the tbats function. When I first fit the model with all the data, it worked fine. However ...
0
votes
0answers
24 views

Accuracy increases using cross-validation and decreases without

I have a question regarding cross validation: I'm using a Naive Bayes classifier to classify blog posts by author. When I validate my dataset without k-fold cross validation I get an accuracy score of ...
0
votes
1answer
23 views

Partitioning for 10-fold cross validation using neural networks in MATLAB

I am working on an assignment which is set to recognize on of 6 basic human emotions based on facial expression data. The data set looks like this: input data: Nx136, where N is the total number of ...
0
votes
0answers
7 views

Is there an algorithm to determine if too few features are selected for k-nearest-neighbor?

Is there an algorithm to determine if too few features are selected for k-nearest-neighbor when no test set is available, --when the input vectors are unknowns? Here's my problem, I have a massive ...
0
votes
1answer
19 views

Cross-validation ($3$-fold) for optimizing ($C$, $\gamma$) in RBF-SVM

Let $\mathcal{X}$ be a training set which will feed a binary SVM with RBF kernel. $\mathcal{X}$ consists of $10$ positive examples and $100$ negative examples. I am interested in optimizing the ...
0
votes
0answers
17 views

Bias-variance trade-off regarding K-Fold Cross-Validation

In the book An Introduction to Statistical Learning (4th Edition), while discussing bias-variance trade-off in the context of k-fold cross-validation vs. ...
0
votes
0answers
29 views

K fold cross validation with many levels factors. Why cv.glmnet can do it and cv.glm cannot?

I notice that you can have many problems with cross validation if you have a categorical predictor which has many unbalanced levels. It happens often that the levels present in the training set are ...
1
vote
1answer
38 views

RMSEP vs RMSECV vs RMSEC vs RMSEE

I am getting real confused now, What is the difference between, RMSEP (Root Mean Square Error of Prediction), RMSECV(Root Mean Square Error of Cross Validation), RMSEC (Root Mean Square of ...
6
votes
4answers
96 views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
1
vote
1answer
21 views

Correlated cases and Cross Validation

I'm posting to ask if there is a method of cross-validation for correllated data that is already well implemented in R language. Some quick search on such method shows some techniques like h-block ...
0
votes
0answers
20 views

Cross-validation for best subset selection

Rephrasing my question: I have predictors $\mathbf{X}=(X_1,X_2,X_3,...)$ and want to find the best subset for predicting some variable Y. My interest lies not in linear prediction, but using nearest ...
0
votes
0answers
15 views

Could iterative linear model use cross validation?

I explain my question in the following example. The data is y1,y2...yn(i.e. y(1)=0.9,y(2)=1.1,y(3)=2.05,y(4)=3.1,y(5)=4.9... like Fibonacci sequence added some normal noise). Then I try to use two ...
0
votes
0answers
29 views

K-fold cross validation

I recently ran a k-fold cross validation on a data set/model that I was interested in evaluating the performance of. In doing so, I received a value of 0.46. I'm assuming this low value indicates poor ...
0
votes
0answers
39 views

elastic net from glmnet to select variables

Recently I am reading papers . One section is about using elastic net to select significant features associated with drug sensitivity. Since all data sets are public available on the website I would ...
1
vote
0answers
7 views

Cross Validation: Which classifier to use in the end - more difficult setting with the EM algorithm

Referring to already discussed question, I solve something more difficult. During the cross validation, I obtain say $n$ models. The discussed question assumes that the best way is to train a new ...
0
votes
1answer
58 views

Implementation of nested cross-validation

I'm trying to figure out if my understanding of nested cross-validation is correct, therefore I wrote this toy example to see if I'm right: ...
0
votes
0answers
30 views

Cross validation and glmer logistic regression (response = 1 and 0)

I'm trying to cross validate the following model: ...
0
votes
1answer
45 views

Leave-One-Subject-Out cv method

I would like to use a Leave-One-Subject-Out cv on my datasets (I have dataset including 38, 15, 10 participants, respectively). I don't know the hyperparamenters C and gamma of my SVM so I have to ...
1
vote
1answer
27 views

How to apply cross-validation for time series analysis using a regression-based approach?

I'd like to know how to use cross-validation for time series analysis using a regression-based approach without incurring in under- or over-fitting. In particular, assume we have an input time series ...
2
votes
2answers
32 views

Cross-validation: Which classifier to use in the end?

This might sound like a very simple question, but I haven't been able to find an answer to it, yet: Assuming I am working on a binary classification task and I am using k-fold cross-validation to ...
2
votes
1answer
72 views

Cross-validation and logistic regression

I'm interested in building a set of candidate models in R for an analysis using logistic regression. Once I build the set of candidate models and evaluate their fit to the data using AICc (...
0
votes
0answers
7 views

What is a good goal for a 5CV in SVR?

I'm running into a complexity x accuracy problem. For example, in a 5CV working with an Epsilon-SVR: ...
0
votes
0answers
12 views

Apply Cross Validation to a prediction linear regression model

Hello I am slightly confused on how to apply a k-fold cross validation to a prediction linear regression model. Also how can i look at the data that the prediction model outputs before i apply cross ...
0
votes
1answer
40 views

cross validation vs. sample size

I am interested in performing GLM regression on a data with sample size of 37. After splitting the data into training (30 obs.) and testing (7 obs.) sets, I'm wondering wether it's better to build one ...
0
votes
1answer
24 views

Holdout set for image task

I need to validate whether one or two templates/shapes are present in an image. Fitting two templates has a better maximum likelihood then fitting one template which is a clear symptom of overfitting. ...
0
votes
0answers
26 views

Model selection and performance evaluation using cross-validation for time series with missing values

So my task is to select and evaluate a statistical model (random forest, boosted trees, neural networks etc.) for a time series with missing values around 10 years long. One of the goals of that is to ...
0
votes
0answers
16 views

How do I report the weights of the most influential features for logistic regression?

I am currently using logistic regression to compute the probability of some event. I randomly split my training/test data and perform cross-validation on the training data, getting a "best model" for ...
0
votes
1answer
11 views

What does ROC-EER in percent stand for?

Ive tried to understand what the ROC Curve represents and what EER (Equal Error Rate) means. And I somehow think I got to understand some of the explanations I read on the internet and videos I ...
0
votes
0answers
18 views

Metrics for regression error

Suppose I undertake a least squares regression on some data. I end up with a function such as $\hat{f}(x,y,\ldots)=\hat{\beta_0}+\hat{\beta_1}\cdot x+\hat{\beta_2}\cdot y+\hat{\beta_3}\cdot ...
4
votes
2answers
43 views

Are cross-validated prediction errors i.i.d?

Say, we test an arbitrary regression or classification procedure on $n$ independent samples with leave-one-out cross-validation. This results in an estimate of the prediction error $e_n$ for each ...
1
vote
0answers
35 views
1
vote
0answers
33 views

Confidence intervals sum of dependent variables

How to construct confidence intervals for sum of dependent random variables. Specifically, I want a high probability claim regarding the difference between the empirical mean and the true mean of the ...
1
vote
2answers
62 views

Does the order of input matter in cross-validation in linear regression?

Please imagine the following problem: A linear regression problem with one input variable (X) and one output (Y) The number of input data is 50 instances. The input data is sorted in increasing ...
8
votes
1answer
114 views

Are there any contemporary uses of jackknifing?

The question: Bootstrapping is superior to jackknifing; however, I am wondering if there are instances where jackknifing is the only or at least a viable option for characterizing uncertainty from ...
3
votes
2answers
44 views

Does k-fold cross validation always imply k uniformly sized subsets?

I'm a bit confused on a minor point that I'm trying to discern due to a cross-validation strategy I've come across in my work that creates k-folds but the folds are not of equal length (for example ...
0
votes
0answers
31 views

PCA and cross-validation [duplicate]

I am fairly new to the machine learning, and I have been going over all the great posts about cross-validation today and I have a question regarding PCA and cross-validation, I don't have enough ...
0
votes
1answer
24 views

Cross validation on clinical datasets [closed]

I am very new to R programming. In my project I need to perform a Cross validation for the clinical datasets (small). I want to know what will be the results. I am unable to recognize the results. I ...
0
votes
0answers
27 views

Statistical comparison of multiple prediction models

I have a rather limited data set where for 100 subjects 30 attributes were measured before surgery and one attribute ($y$) was measured after surgery. About 20% of values are missing. The goal is to ...
1
vote
1answer
31 views

K-fold validation, how to use MSE and STD for model selection

When using K-fold validation for model selection I'm wondering what's the best approach to select a model using both the mean square error (MSE) and the standard deviation of errors among folds (STD). ...
0
votes
0answers
13 views

Non idependence within groups

I have to train a machine learning model for classifying two groups. Unfortunately, my positive group has a small number and many cases are not independent from each other (observations taken in ...
1
vote
0answers
14 views

Unbalanced groups and classification errors

I would like to adopt a general strategy for dealing with an very unbalanced dataset, where my "positive" group corresponds to 1/40 of all the observations. The reason why I ask it is because all the ...
0
votes
0answers
20 views

logistic regression test error rate in intercept-only model

I'm using logistic regression with LOOCV and am balancing the classes for the two responses. I noticed that with my model, the test error rate is decent (0.22) and the predictor variable is ...