Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
11 views

How to reduce the time in cross validation of a training set with 15k rows?

This is the code and it is taking a lot of time. Any other technique which i could follow? ...
1
vote
1answer
28 views

Learning curves - Why does the training accuracy start so high, then suddenly drop?

I implemented a model in which I use Logistic Regression as classifier and I wanted to plot the learning curves for both training and test sets to decide what to do next in order to improve my model. ...
2
votes
1answer
27 views

Lasso and Ridge tuning parameter scope

In ridge and lasso linear regression, an important step is to choose the tuning parameter lambda, often I use grid search on log scale from -6->4, it works well on ridge, but on lasso, should I take ...
-1
votes
0answers
16 views

Leave One Out Cross Validation Weka [on hold]

If I want to use the LOO method in weka do I have to set the number of folders to the size of the dataset?
0
votes
0answers
10 views

using cross validation to produce and test a model [duplicate]

I'm a bit confused about cross validation purpose. lets say I have a linear model. one option is to get sample an split to train and test data sets. train on on train set, evaluate error on test set. ...
0
votes
1answer
38 views

What should be validation strategy?

I am building CTR(https://en.wikipedia.org/wiki/Click-through_rate) Click prediction model with different (61) variables.Dependent variable is weather 0/1( click).I have build logistic regression ...
0
votes
0answers
14 views

Is it common to perform a calcuation in several passes (similar to a k-fold cross validation)

i'm writting a paper where i want to plot the results of a similarity calcuation of a larger text corpus. (The result is a square matrix containing all the similarities between the documents) For each ...
0
votes
0answers
7 views

How to assign defined training set, val set and test set for training a Neural net in NNtoolbox?

To find an optimal number of hidden neurons and layers in my code using feedforward net, I use cross validation technique and cvpartition function to split data. Now my aim is to use this split data ...
3
votes
2answers
51 views

Completely different results after each cross validation

I'm running some classification algorithms in MATLAB and validating them with a 10-fold cross validation. The problem is that every time I execute the cross validation, it gives a very different ...
3
votes
2answers
52 views

Meaning of cross validation

This is a very fundamental question but I want to make sure I get this right. K-fold cross validation will only help in predicting the accuracy and other metrics of the model but not really improve ...
0
votes
0answers
16 views

Class selection from a probability vector (multiclass problem) [closed]

Is there a way to choose the class from a probability vector instead of the typical of approach of choosing the class with the max probability? For example: Say that a model produces the probability ...
1
vote
1answer
21 views

Selecting a loss-function for k-fold cross-validation over shrinkage parameter

I am doing a penalized regression with categorical (ordinal) outcomes. I would like to select the shrinkage parameter $\lambda$ on the basis of cross-validation (CV). In this case, I have 50k ...
0
votes
0answers
20 views

How Cross validation for GLASSO works?

I just found a written R function to do cross validation for glasso to choose best lambda. ...
0
votes
2answers
32 views

Overfitting over test set in terms of model selection?

I didn't find any similar question so I just post the question here. Suppose after training and validation, the model performs poorly on the test set. Then what we do is to consider another model (Is ...
0
votes
1answer
26 views

KNN classifier + cross validation

how can I find the mean and standard deviation of error rate or accuracy of a k- fold cross validation performing K-nearest-neighbour classification model for each fold?
0
votes
0answers
15 views

Naive Bayes + k- fold Cross Validation

How can I find the mean and standard deviation of the accuracy of k-fold cross validation when the classifier method is Naive Bayes?
0
votes
0answers
8 views

R caret: leave subject out cross validation with data subset for training? [migrated]

I want to perform leave subject out cross validation with R caret (cf. this example) but only use a subset of the data in training for creating CV models. Still, the left out CV partition should be ...
3
votes
3answers
200 views

Neural network working well on datasets near the training set, but poorly on farther datasets. Why?

I've been using a siamese neural network for the binary classification of biological data. Each entry of the datasets I'm using has a position coordinate. My problem is that, even if my neural ...
1
vote
1answer
47 views

GridSearchCV and KFold

I noticed that in some cases, a GridSearchCV is applied on the output of KFold. For example, like in the code below. Why is it needed? I thought that something equivalent to KFold is already applied ...
0
votes
0answers
8 views

What is an “almost stable” inducer?

I found a very interesting paper by Ron Kohavi titled "A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection" (International Joint Conference on Artificial ...
3
votes
2answers
54 views

Out of Bag Error makes CV unnecessary in Random Forests?

I am fairly new to random forests. In the past, I have always compared the accuracy of fit vs test against fit vs train to detect any overfitting. But I just read here that: "In random forests, ...
0
votes
0answers
16 views

How to interpret the the train result?

I using the caret trained my dataset using naive bayesian as method with an repeated 10-fold cross validation. I seem to get a lot of different output, but can't ...
2
votes
1answer
33 views

How to choose probability to predict success in logistic regression?

I'm working through a logistic regression example from the lab on logistic regression in Intro to Statistical Learning. When they try to test how accurate their model is they do, ...
0
votes
1answer
14 views

Cross validation error dependency?

Let's say we are running CV with K folds. Can you give an intuitive explanation on why the errors per fold are dependent? I was asked this and after thinking about it I kind of see it but need some ...
0
votes
0answers
23 views

How to select a model from leave one out cross-validation

I have a set of 400 positive vectors and hundreds of millions of negative ones. I have split the data into a training and test set each of 200 positive vectors and lots of negative ones. I would ...
1
vote
1answer
68 views

Is AUC via CV a good procedure for selecting optimal model?

I'm fitting a logit classifier with LASSO and cross-validation, and struggling to select the optimal model using AUC -instead of the more usual loss like binomial deviance or classification error. I ...
0
votes
0answers
23 views

ROC calculation in LOOCV context - caret

I am not sure how caret handle the ROC calculation when used with LOOCV. From what I understand, in the more common case where a 10-fold cross validation is used, the ROC value is calculated for each ...
1
vote
2answers
65 views

Are K-Fold Cross Validation , Bootstrap ,Out of Bag fundamentally same?

Can Anyone tell me how K-Fold Cross Validation ,Bootstrap and Out of Bag Approach differ as they use 1)Separate data into training data and testing data 2)Make model using training data and ...
0
votes
0answers
36 views

Assumptions behind cross-validation

According to "no free lunch theorem" (also here and here), we cannot deduce just from the data alone (without any domain knowledge) which classifier is better. Of course, we use cross-validation to do ...
1
vote
1answer
23 views

How to partition leave-one-subject-out (not leave-one-example-out) cross-validation in MATLAB?

I am currently extracting 16 features from 7 samples all of different length. Now I would like to apply the data using multiple classification algorithms with cross validation. I already done this by ...
1
vote
1answer
38 views

cross validation after lasso

I used cross validation to select lambda. Then I performed lasso and get non zero coefficients (features). Shall I perform cross validation for these non zero coefficients as a kind of validation?
2
votes
1answer
54 views

Can we gain by merging validation and test set?

Reading this, Cross-validation including training, validation, and testing. Why do we need three subsets? I realized that if we can reduce the variance of the model performance, I wouldn't need the ...
0
votes
1answer
35 views

Is there a way to return the standard error of cross-validation predictions using caret `train`

In the book Applied Predictive Modelling Ch 4., there is the following table: The standard error here is used in the following graph, and to use the "one-standard error method" to find the optimal ...
0
votes
0answers
12 views

How is it correct to optimize a binary classifier output threshold with ROC and LPOCV?

Hello everyone and thank you in advance for you help! I'm building a screening tool with a machine learning algorithm. The model provides a probabilistic prediction (i.e. logistic regression, ...
2
votes
0answers
29 views

How to extract the predictions and probabilities of each training sample in a cross-validation result in caret (R)?

I'm learning the caret package in R for classifications by Naive Bayes. I'm following the tutorial from: http://topepo.github.io/caret/training.html Thanks for the great tutorial! But I have one ...
0
votes
0answers
32 views

LOOCV v/s K fold Cross validation bias

Why LOOCV(Leave-One-Out Cross-Validation) has less bias than K fold Cross Validation ? Please explain with example if possible
0
votes
1answer
30 views

Accuracy on the test set do not change. Why?

I train a SVM classifier using 36 features. If I use all the features, the train accuracy is about 0.96, the test accuracy is about 0.77. Then I change the number of features. The train accuracy drops ...
3
votes
2answers
55 views

K- cross-validation

In Max Kuhn's Book "Applied Predictive Modeling" this is writen about K-cross validation: As k gets larger, the difference in size between the training set and the resampling subsets gets ...
1
vote
0answers
14 views

When using loss matrix in rpart in R, xerror does not start at 1 [closed]

I am trying to use a loss matrix in rpart penalizing false positives 10 times as much as false negatives, but when I fit my data and then use printcp, my xerror values start at 10 and not 1. I am ...
0
votes
0answers
17 views

R model developing & validating - Open to Discussion [duplicate]

Throughout my R journey I have noticed the way we can use given data to develop and validate a model. Assume that you have given data for a problem train.csv test.csv Method A Combine ...
0
votes
0answers
29 views

How to perform cross validation clustered input data?

How do i perform knn cross validation with input data that has been clustered using k-means. I seem to be unable to find the correct function which is able to do so. ...
0
votes
0answers
13 views

What sort of cross validation is this?

I've always tested my classification techniques using non-standardised trial and error but I'm interested to see which category my techniques fall under. They seem to fall under several but I'm not ...
1
vote
0answers
34 views

How do I add cross validation for a random forest regression?

The error percentage of regression changes with change in the train and test data which I am deciding randomly. Cross validation can overcome this but how do I apply it for my regression model?
0
votes
0answers
9 views

How to understand this R Squared value?

It is the first time for me to run 5 fold CV technique using JMP to classify 86 cases into two categories using 8 variables.The value of R Squared in the cross validation box is different from the ...
1
vote
2answers
65 views

Cross Validation - purpose, need and utility [duplicate]

The question might sound like an old one but I haven't got satisfactory answers for a number of questions I have about CV. I looked at several questions on CV here, here, here and here and yet things ...
0
votes
0answers
7 views

Which parameters detect the best quality of tree when splitting the data into training set/validation set?

Which parameters detect the best quality of tree when splitting the data into training set/validation set using JMP partition model?
0
votes
0answers
18 views

R Squared value in K fold cross validation

I'm using JMP to create partition model for classifying 86 cases into two different categories according to 11 significant variables using 5 fold cross validation. What is the value of Squared that ...
0
votes
0answers
48 views

How to calculate accuracy in cross-validation?

I have a classification problem consisting of two classes. I have around 10000 data pionts and 20 features. I'm doing nested 10-fold cross-validation. I am unsure about calculating the accuracy. I ...
0
votes
1answer
13 views

Can one use k-fold cv and holdout analysis together?

I would like to start by saying i have just started using cross-validation, so please bear with me if the questions seems very trivial. I am reviewing someones work where the person has used k-fold ...
0
votes
0answers
39 views

What if I'm getting always around the same error on Training, Validation and Test set?

currently playing around with H20 and R using a Feed-Forward Neural Net. I'm doing parameter space exploration - than means I'm trying to tweak the various parameters of the NN in a set of nested ...