Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
19 views

Can Pearson correlation be used as a measure of fit?

In the context of multiple linear regression, is it acceptable to use Pearson correlation to discriminate how well a model fits a data set? Let's say that I have some experimental values that come ...
0
votes
0answers
17 views

MATLAB: leave-one-out-cross-validation for PCA

I'm trying to write my own function for PCA (of course there's a lot already written but I'm just interested in implementing stuff by myself). The main problem I encountered is the cross-validation ...
0
votes
0answers
11 views

Rpart and tree models — cross validation beyond cp value

I normally cross-validate my tree-models (rpart) only on cp-value, e.g. by using caret or xpred or the internal rpart cross validation function. However, there are other rpart parameters -- for ...
0
votes
0answers
11 views

What is the proper way to compare two estimated densities using sample data?

Say if have a dataset $X \subset \mathbb R^d$. I have two candidate probabilistic models M1 and M2 (e.g., M1 is a mixture of 2 gaussians and M2 is a mixture of 3 gaussians). I want to know which model ...
1
vote
0answers
19 views

Compare averaged GLM with boosted regression trees using cross validation : d2 and RMSE calculation

I want to compare BRT and averaged glm models on test sets by calculating the explained deviance and RMSE. How can I calculate d2 and RMSE from predictions? I use the following functions: gbm1 ...
1
vote
1answer
48 views

How to define the maximum k of the kNN classifier?

I am trying to use kNN classifier to perform some supervised learning. In order to find the best number of 'k' of kNN, I used cross validation. For example, the following codes load some Matlab ...
1
vote
1answer
30 views

How to implement a hold-out validation in R

Let's say I'm using the Sonar data and I'd like to make a hold-out validation in R. I partitioned the data using the createFolds ...
2
votes
1answer
64 views

Choosing the right regression and validating it

I have two questions: Which regression should I choose and why? Based on the R squared value, exponential regression seems to be a better fit. But I am not really sure, if I should just go ...
0
votes
1answer
22 views

R's equivalente of scikit's KFold

I'm new to R and I'm trying to set up a basic k folds CV loop. In Python I'd use scikit's KFold. ...
0
votes
0answers
15 views

Creating folds for k-fold CV in R using Caret [migrated]

I'm trying to make a k-fold CV for several classification methods/hiperparameters using the data available at ...
0
votes
0answers
5 views

Rpart cross-validation errror

Despite reading the rpart technical documentation, it is still unclear to me how the cross-validation error (xerror) is computed. Using the standard 10 fold cross-validation, does that mean a) ...
0
votes
0answers
10 views

R rpart cross validation and 1 SE rule, why is the column in cptable called “xstd”?

The rpart() function in R returns cptable that includes columns xerror and xstd. Here is an arbitrary example. ...
0
votes
1answer
49 views

How can I perform 10-fold cross validation by manually constructing datasets?

I am working in text classification in RapidMiner where, because of the nature of my problem, I cannot use the built-in k-fold cross validation strategy, so I decided to create 10 copies of my dataset ...
0
votes
1answer
17 views

Cross validation - feature information outside the fold

i am building a model that predicts likelihood of the presence of an infectious disease across 10000 villages. There is spatial dimension - each villages is surrounded by e.g. 8 other villages, and ...
1
vote
0answers
9 views

Creating a table with individual trials from a frequency table in R (inverse of table function) [migrated]

I have a frequency table of data in a data.frame in R listing factor levels and counts of successes and failures. I would like to turn it from frequency table into ...
10
votes
3answers
158 views

Training, testing, validating in a survival analysis problem

I've been browsing various threads here, but I don't think my exact question is answered. I have a dataset of ~50,000 students and their time to dropout. I am going to be performing proportional ...
0
votes
0answers
25 views

libsvm: Cross Validation with imbalanced classes

I am using libsvm for a 2-class classification problem. For my testing I use the C-SVM with RBF kernel. My main problem seems to be that the classes are highly imbalanced. While I have 35000 datasets ...
3
votes
1answer
63 views

Cross-Validation, how to actually implement the concept

I have 100 samples from a normal distribution with variance equal to 1, and mean equal to 0. I have selected 90 samples randomly out of the 100 and i have estimated their mean (the average of the 90 ...
0
votes
0answers
11 views

How to cross validate with optimal sample size selection?

My problem is as follows: I got a training set composed of texts, where 10% of observations are 1, and 90% are 0. To make it simpler, lets say that 1000 is 1, and 9000 is 0. I implement Naive Bayes ...
0
votes
1answer
29 views

Grid Search for hyperparameter and feature selection

So I need to select my hyperparameters and also my features. A full grid search of the space of hyperparameters and features is too computationally intensive, so what I am doing instead is for each ...
1
vote
0answers
24 views

How to determine appropriate number of features and also which features to select?

So I have a dataset which I am using K fold cross validation on to select the number of features and which features should be selected. As I understand it, I would set the number of features to be ...
0
votes
0answers
35 views

Overfitting times series data

I am trying to use Robert Shiller's stock market data to estimate future 5 year returns. My first naive attempt used M5P in Weka (decision tree regression) with 60% of the data for training and 40% ...
1
vote
2answers
61 views

Why is Leave One Out Cross Validation (LOOCV) variance about the mean estimate for error high?

In Leave One Out Cross Validation, each of the training sets look very similar to each other, differing in only one observation. When you want to estimate the test error, you take the average of the ...
0
votes
0answers
21 views

Leave-one-out cross-validation in LiblineaR

So I've been encountering this weird issue with LiblineaR where I get different results from just running LiblineaR with cross set to the size of my dataset and actually looping through the dataset, ...
1
vote
0answers
27 views

What is the meaning of the term “enrichment” when performing cross-validation?

Trying to understand a discussion of a 5-fold cross-validation process to validate a predictive model and its results, there is a particular phrase which has me stumped, i.e.: The predictions of ...
0
votes
0answers
22 views

Why do I get more than 25 symbols in plot in 25-fold cross-validation

I have a group of observations, 25, And I want to perform a 25-fold cross-validation (leave-one-out). Here is my code, ...
2
votes
2answers
68 views

In k-fold cross validation does the training subsample include test set?

In this Wikipedia page in subsection for K-fold cross validation it says "In k-fold cross-validation, the original sample is randomly partitioned into k equal size subsamples. Of the k subsamples, a ...
3
votes
1answer
40 views

Cross Validation in Unbalanced Datasets

Is there a specific way of sampling which maintains the ratio of samples in an unbiased set? e.g., lets say I want to do k-fold cross-validation on my training set And my training set is very ...
0
votes
0answers
19 views

How to do cross validation for repeated measurement (proc mixed)

I had run the mixed model for repeated mesurement using SAS proc mixed, I got the model(s) already for my dependent variable. However, proc mixed doesn't provide R^2. Is there any cross validation ...
1
vote
1answer
52 views

How is AUC of decision tree calculated?

I have a dataset which only has one continuous variable, and I try to use decision tree algorithm to build a model which classify the +ve and -ve label from the dataset. I run 10-fold ...
0
votes
0answers
25 views

Neural networks in multi-class setting: How to train/test with cross-validtion; how to evaluate?

I am currently working on a project where neural networks are used for email categorization. There are two things where I still have some problems understanding the general approach: cross-validation ...
0
votes
0answers
30 views

Cross-validated bootstrap samples

In one of the paper I am writing, I am looking at cross-validation on bootstrap samples. I wrote the following explanation.One of the reviewer wrote that he didn't understand where is the correlation. ...
1
vote
0answers
31 views

Sizing of training and validation sets in machine learning: Is there a proven optimum, or merely heuristics?

When I watch presentations where machine learning algorithms were used, the amount of data put in the training and validation sets seems to be somewhat arbitrary. Sometimes it's 80-20, sometimes it's ...
0
votes
1answer
79 views

Superiority of LASSO over forward selection/backward elimination in terms of the cross validation prediction error of the model

I obtained three reduced models from a original full model using forward selection backward elimination L1 penalization technique (LASSO) For the models obtained using forward selection/backward ...
0
votes
0answers
38 views

Abnormally high accuracy with repeated 10-fold CV and ordinal regression

I am using repeated 10-fold CV to calculate the accuracy of my ordinal regression model. I have 6 predictors, 10 ordered response categories, and a total of 1166 data points. For the ordinal model, ...
3
votes
2answers
211 views

Significance testing of cross-validated classification accuracy: shuffling vs. binomial test

I have a dataset with 2 classes and a certain way to build a binary classifier. I want to measure its performance and to test if it is significantly above the chance level. I measure its performance ...
1
vote
2answers
78 views

How to report a SVM model to a 3rd party after cross-validation?

I have a binary classification problem. I trained my dataset using a Support Vector Machine (SVM). Now I want to report the model I trained to a 3rd party so that they can use. For the primal probem ...
2
votes
1answer
56 views

How to select tuning parameter for regularized regressions for interpretation?

I'm using linear regression to predict a continuous variable using a large number (~200) of binary indicator variables. I have around 2,500 data rows. There are a couple of issues here: When I run ...
0
votes
1answer
47 views

Cross-validation for mixed-effect logistic regression? [duplicate]

I would like to use cross-validation to test how predictive my mixed-effect logistic regression model is (model run with glmer). Is there an easy way to do this using a package in R? I've only seen ...
5
votes
3answers
247 views

Confidence interval for cross-validated classification accuracy

I'm working on a classification problem that computes a similarity metric between two input x-ray images. If the images are of the same person (label of 'right'), a higher metric will be calculated; ...
0
votes
1answer
29 views

When accuracy in the cross-validation process less,is reducing the features a good idea?

I am doing a project for classifying the presence of cars/bikes in an image.I have extracted the features from the images(data-set of cars and images not belonging to that of cars) and applied K-means ...
3
votes
2answers
115 views

k folds cross validation on a multi-class dataset

Cross validation is one of the most important tools because it gives us an honest assessment of the true accuracy of our system. In other words, the cross-validation process provides a much more ...
0
votes
0answers
26 views

How are final model coefficients estimated when using k-fold cross validation? [duplicate]

It is i'm sure quite a simple question but i was not really able to find an explanation of this online and does not seem to be explicitly called out in the typical expositions i've seen. I get how ...
0
votes
2answers
35 views

Selecting features and estimating their out-of-sample performance with cross-validation

I have only a small dataset. I want to 1. select the most predictive features out of a large candidate pool and 2. get an estimate of their expected predictive performance. In the elements of ...
3
votes
2answers
217 views

Cross-validation and feature selection of a multivariate regression

I've been trying to create a multivariate regression model to fit my training data into the prediction of a value. I've put my data into a matrix X with ...
1
vote
0answers
162 views

Backward stepwise regression with cross validation in R

I would like to do model selection using backward stepwise procedure and cross validation. https://www.otexts.org/fpp/5/3 I have used stepAIC in ...
4
votes
3answers
148 views

Can you compare different clustering methods on a dataset with no ground truth by cross-validation?

Currently, I am trying to analyze a text document dataset that has no ground truth. I was told that you can use k-fold cross validation to compare different clustering methods. However, the examples I ...
1
vote
0answers
48 views

Cross-Validation: Conditional test error vs expected test error

My textbook on cross-validation is Element of Statistical Learning (Hastie et. al., 2nd ed.) In sections 7.10.1 and 7.12, they talk about the difference between conditional test error ...
4
votes
3answers
190 views

Is a lower training accuracy possible in overfitting (one class SVM)

I am using the heart_scale data from LibSVM. The original data includes 13 features, but I only used 2 of them in order to plot the distributions in a figure. Instead of training the binary ...
1
vote
0answers
72 views

is it possible to use partitioned data(train&test) together with cross-fold validation?

I have used SPSS Clementine in order to train a classifier, for this I have used a partition node with 2 parts(train and test),then using a c5-tree and cross-fold validation. I did this because I ...