Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
10 views

What is the commonly used mehtod for measuring variance of accuracy mean using k-fold cross validation?

I know there are planty of questions about standard deviation, though I didn't find an answer tuned to my particular need and also I could really use your help! I'm performing 18-Fold Cross ...
3
votes
0answers
18 views

How to deal with factors with rare levels in cross-validation?

Suppose in a regression analysis in R, I have a factor type independent variable with 3 levels in my train dataset. But in the test data set that same factor variable has 5 levels. Therefore I can not ...
1
vote
1answer
26 views

What model fit / predictive accuracy measure can be used to cross validate a Cox PH model with censored data?

How would you go about validating a Cox PH model with censored data? I am trying to run a Cox PH model on a dataset with observations that failed, and observations that are censored. Normally, I use ...
0
votes
1answer
29 views

bootstrapping vs. “repeated cross validation”

For a research project, I conducted the following methodology. The dataset was of size $N$. $B$ times, I: took a random $N/2$ rows and trained my model, which finds the optimal size $M$ of a ...
0
votes
0answers
10 views

How to include only true positives and false positives, that is ignore false negative classifications in a confusion matrix?

I have performed a 10 fold cross validation on my data set using binary decision trees. I've got 6 trees (to detect one of the six basic human emotions from facial data points) trained for each fold. ...
1
vote
1answer
13 views

Matlab - why crossval function inputs a full trained model?

The question is regarding the Matlab implementation. As we can see here, the crossval function expects to receive a full trained model. For example, my data ...
1
vote
0answers
37 views

Combining bootstrap and cross validation

I recently read this paper: Estimating misclassification error with small samples via bootstrap cross-validation, by Fu et al. (BMC Bioinformatics, 2005). The authors talk about combining cross ...
0
votes
0answers
12 views

What is the time complexity of binary classification of SVM?

One of the earliest solution to the SVM problem is SMO applied to dual form.What is the time complexity of SMO algorithm? What is the best known time complexity to solve SVM algorithm (non linear)?
0
votes
0answers
35 views

using Cross Validation in matlab with neural networks

I want to make a cross validation on neural network, I tried to pass the labels to crossval function, with the help of ...
0
votes
1answer
15 views

Designing an experiment for a marketing campaign using Incremental Response Modelling

I have the following hypothetical question, can anyone provide some clarification? I'm looking at designing an experiment or modelling what steps can be taken to maximise the Net Incremental Revenue ...
1
vote
1answer
53 views

Explain “validation” process of repeated k-fold cross-validation?

My understanding is currently that the canonical repeated k-fold cross-validation (CV) process might do the following if $n=100$ observations in sample, $k=5$ folds, $i= 10$ iterations (see iteration ...
0
votes
1answer
26 views

Comparing classification algorithms using cross validation and caret's train

I am having issues understanding some concepts of algorithm comparison/parameter optimization/cross-validation in R Let's say I want to compare two classification algorithms, such as Random Forests ...
0
votes
0answers
9 views

Blocks of variable size in k-fold cross-validation

I would like to make a custom k-fold cross-validation method for my data, by generating folds of auto-correlated observations. This would create many folds of variable size for test errors as well as ...
1
vote
1answer
30 views

Early stopping vs cross validation

I'm currently using early stopping in my work to prevent over fitting. Specifically those taken form Early Stopping But When?. I'm now wanting to compare to other classification algorithms where it ...
1
vote
1answer
110 views

Multicollinearity, plm, and omitting variables

I'm fitting a fixed effect model with plm and know that I'm dealing with multi-collinearity between two of the independent variables. I working on identifying ...
1
vote
1answer
18 views

Cross validation of a survival model- what to make of “random effects” of parameter estimates?

This is a question surrounding k-fold cross validation for time to event data. I am interested in what to do with the knowledge that certain variables fail to perform as well within some of the ...
0
votes
2answers
40 views

Datapoint Classification Accuracy

I am interested in finding ways to quantify the certainty of correct classificaions for single datapoints. This is interesting for me since for clinical studies where we for instance would classify ...
0
votes
0answers
17 views

Value of the loss function and Classification Errors in gbm package (R)

I have a simple problem of classification (0s and 1s) using adaboost loss function. When I check the components of a boosted model using the gbm package I see: ...
0
votes
0answers
89 views

Crossvalidation in hierarchical bayesian models (HBMs)

I am trying to find a way to cross-validate Hierarchical Bayesian Models used for predicting and modelling abundance in Species Distribution Models. For this purpose, I have tried posterior predictive ...
0
votes
0answers
22 views

Learning Curve Meaning

I have made a learning curve that looks like this: Why wouldn't it be more like both training and cross-validation score begin low and both gradually increase with more samples? Why does one start ...
1
vote
1answer
34 views

Data splitting in regression

Edit: The key point I'm attempting to understand is whether during a regression model building exercise, do I need separate datasets to: search for predictors and settle on a functional form ...
0
votes
1answer
24 views

Contribution of a sample to cross validation error

I was wondering how to asses which sample in the data, during K fold cross-validation drives the bias that may be observed in the results. My training data consists of 40 samples. And I try to ...
1
vote
1answer
20 views

In cross validation, higher the value of k, lesser the training data for this formula?

Is it right to say that smaller the value of k in cross validation based on the following formula, more the number of records in test data/smaller the number of records in training data. According to ...
2
votes
1answer
45 views

One vs all Linear SVM Cross validation -Parameter selection

I'm performing one vs all classification (SVM) for a dataset. Since I'm using a linear SVM the parameters I need to tune and select are-Tolerance and C. I'm a bit confused on how to go about doing ...
1
vote
0answers
22 views

Cross-validation for parameter tuning in data mining process (KDD)

In my project I want to compare different classification algorithms to solve a specific problem with a specific dataset. To do this, I divided the dataset in 2 parts. With the first (bigger) part I ...
0
votes
0answers
48 views

AIC versus cross validation in time series

I am interested in model selection in a time series setting. For concreteness, suppose I want to select an ARMA model from a pool of ARMA models with different lag orders. The ultimate intent is ...
1
vote
1answer
32 views

How to do test set evaluation using a regression model in Caret?

I'm used to using Caret to do classification but now I need to use it for regression. I have successfully trained a model on my training set but I'm not sure what ...
0
votes
1answer
23 views

code for cross validation in matlab libsvm

i have posted a question but due to internet problem can't correct it and it was posted sorry for that. now i want to share my problem i want to use cross validation and grid search to classify my ...
4
votes
1answer
84 views

Likelihood ratio test disagrees with cross-validation results

I have computed two logistic models of the same data (for different formulas) in R, and compared them using likelihood ratio test: ...
0
votes
0answers
18 views

leave-one-out error estimate variance

What would be the lower bound for the variance of the leave-one-out error estimate? Not necesary has to hold for all distribution. Any example would be nice. Also the classifier for the example could ...
0
votes
0answers
10 views

tbats{forecast} in R gives strange predictions for some folds in cross validation

My daily data shows weekly and yearly seasonality, so I decide to try the tbats function. When I first fit the model with all the data, it worked fine. However ...
0
votes
0answers
34 views

Accuracy increases using cross-validation and decreases without

I have a question regarding cross validation: I'm using a Naive Bayes classifier to classify blog posts by author. When I validate my dataset without k-fold cross validation I get an accuracy score of ...
0
votes
1answer
53 views

Partitioning for 10-fold cross validation using neural networks in MATLAB

I am working on an assignment which is set to recognize on of 6 basic human emotions based on facial expression data. The data set looks like this: input data: Nx136, where N is the total number of ...
0
votes
0answers
11 views

Is there an algorithm to determine if too few features are selected for k-nearest-neighbor?

Is there an algorithm to determine if too few features are selected for k-nearest-neighbor when no test set is available, --when the input vectors are unknowns? Here's my problem, I have a massive ...
0
votes
1answer
24 views

Cross-validation ($3$-fold) for optimizing ($C$, $\gamma$) in RBF-SVM

Let $\mathcal{X}$ be a training set which will feed a binary SVM with RBF kernel. $\mathcal{X}$ consists of $10$ positive examples and $100$ negative examples. I am interested in optimizing the ...
0
votes
0answers
29 views

Bias-variance trade-off regarding K-Fold Cross-Validation

In the book An Introduction to Statistical Learning (4th Edition), while discussing bias-variance trade-off in the context of k-fold cross-validation vs. ...
0
votes
0answers
62 views

K fold cross validation with many levels factors. Why cv.glmnet can do it and cv.glm cannot?

I notice that you can have many problems with cross validation if you have a categorical predictor which has many unbalanced levels. It happens often that the levels present in the training set are ...
1
vote
1answer
116 views

RMSEP vs RMSECV vs RMSEC vs RMSEE

I am getting real confused now, What is the difference between, RMSEP (Root Mean Square Error of Prediction), RMSECV(Root Mean Square Error of Cross Validation), RMSEC (Root Mean Square of ...
6
votes
4answers
120 views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
1
vote
1answer
24 views

Correlated cases and Cross Validation

I'm posting to ask if there is a method of cross-validation for correllated data that is already well implemented in R language. Some quick search on such method shows some techniques like h-block ...
0
votes
0answers
26 views

Cross-validation for best subset selection

Rephrasing my question: I have predictors $\mathbf{X}=(X_1,X_2,X_3,...)$ and want to find the best subset for predicting some variable Y. My interest lies not in linear prediction, but using nearest ...
0
votes
0answers
15 views

Could iterative linear model use cross validation?

I explain my question in the following example. The data is y1,y2...yn(i.e. y(1)=0.9,y(2)=1.1,y(3)=2.05,y(4)=3.1,y(5)=4.9... like Fibonacci sequence added some normal noise). Then I try to use two ...
0
votes
0answers
31 views

K-fold cross validation

I recently ran a k-fold cross validation on a data set/model that I was interested in evaluating the performance of. In doing so, I received a value of 0.46. I'm assuming this low value indicates poor ...
0
votes
0answers
52 views

elastic net from glmnet to select variables

Recently I am reading papers . One section is about using elastic net to select significant features associated with drug sensitivity. Since all data sets are public available on the website I would ...
1
vote
0answers
8 views

Cross Validation: Which classifier to use in the end - more difficult setting with the EM algorithm

Referring to already discussed question, I solve something more difficult. During the cross validation, I obtain say $n$ models. The discussed question assumes that the best way is to train a new ...
0
votes
1answer
98 views

Implementation of nested cross-validation

I'm trying to figure out if my understanding of nested cross-validation is correct, therefore I wrote this toy example to see if I'm right: ...
0
votes
0answers
38 views

Cross validation and glmer logistic regression (response = 1 and 0)

I'm trying to cross validate the following model: ...
0
votes
1answer
68 views

Leave-One-Subject-Out cv method

I would like to use a Leave-One-Subject-Out cv on my datasets (I have dataset including 38, 15, 10 participants, respectively). I don't know the hyperparamenters C and gamma of my SVM so I have to ...
1
vote
1answer
29 views

How to apply cross-validation for time series analysis using a regression-based approach?

I'd like to know how to use cross-validation for time series analysis using a regression-based approach without incurring in under- or over-fitting. In particular, assume we have an input time series ...
2
votes
2answers
39 views

Cross-validation: Which classifier to use in the end?

This might sound like a very simple question, but I haven't been able to find an answer to it, yet: Assuming I am working on a binary classification task and I am using k-fold cross-validation to ...