Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
3 views

Comparing predictors based on ROC AUC and cross-validation error

I am analysing how well some continuous variables (e.g. weight, height) predict the occurrence of a given disease after surgery. I have computed the area under the curve of the receiver-operator ...
1
vote
1answer
19 views

Cross validation with test data set

I am a bit confused about the application of cross-validation. So, if I have a big data set, I will split my data into test and training data and and perform validation on the test data. But if I have ...
0
votes
0answers
10 views

Testing accuracy keeps on increasing. No signs of overfitting

I am trying to do some cross-validating in R. As I am increasing the cost parameter in LiblineaR, the testing accuracy should increase initially, but start to decrease. However, as my cost parameter ...
0
votes
0answers
11 views

Including Feature Selection in Cross Validation - Application to Bag of Words

I am working on a prediction problem where I was given a 6,000 record dataset with the value of the dependent variable included ("train"), and a 2,000 record dataset with the same independent ...
2
votes
0answers
18 views

May I use the whole dataset to prove the existence of a confounding variable in a machine learning framework if I don't use the labels?

I have a certain dataset that I am analyzing with machine learning techniques. I believe there is a certain variable (not used for training or testing the classifiers but is still known) that has an ...
0
votes
0answers
9 views

Logistic model cross validation error in SAS [on hold]

I am using sas and want to fit a logistic regression model to predict the prob of long male life and construct a 2*2 table for cross validation rate. But I don't know why my code does not work. ...
1
vote
1answer
20 views

out-of-bag error estimate for Boosted Trees

In Random Forest, each tree is grown in parallel on a unique boostrap sample of the data. Because each boostrap sample is expected to contain about 63% of unique observations, this lefts roughly 37% ...
2
votes
1answer
23 views

What is the chance level accuracy in unbalanced classification problems?

Suppose one has a balanced classification problem (50% of 0's and 50% of 1's). In such a case, the so called chance-level accuracy of classifier would be 50%. What is the chance-level accuracy if the ...
0
votes
0answers
16 views

What does “10-fold cross validation” mean when only a training set is provided? [duplicate]

Recently I encounter "10 cross validation" several times in literature reading as well as using a data mining tool "Weka". I checked wikipedia, it states "One round of cross-validation involves ...
-1
votes
1answer
22 views

Somebody explain Training, Testing and Validation Test of Artificial Neural Network [duplicate]

What is the procedure of Training, Testing and Validation Test? Explain it thoroughly. Or give some link for related articles
5
votes
1answer
42 views

What if high validation accuracy but low test accuracy in research?

I have a specific question about validation in machine learning research. As we know, the machine learning regime asks researchers to train their models on the training data, choose from candidate ...
1
vote
0answers
36 views

Set the seed to a specific number?

i'm quite new to stata and statistics, and I have some questions I hope some of you can answer. My first question is regarding "seed". I have a assignment where i'm asked to use 300 repetitions and ...
0
votes
0answers
21 views

Overfitting in the validation set

When running an algorithm for training a system it is common to consider a lot of models and using the validation set for selecting one of them. In my case I am running a mini-batch gradient descent ...
1
vote
0answers
22 views

Cross validation before or after stepwise modeling [duplicate]

I have a dataset of 1931 observations and I intend to predict a binary outcome out of that. There is a list of 128 predictors (both binary and continuous). First I ran logistic regression modeling ...
0
votes
1answer
29 views

LOOCV $R^2$ higher than regular $R^2$ in RF

I am working with RF and the caret package, and I am having a confusion because sometimes the LOOCV $R^2$ is higher than the regular $R^2$. Is it right? How can I interpret this? Here an example ...
0
votes
0answers
39 views

Is this the wrong way to do cross-validation?

I am building an ARIMA model and did a grid search to find which values to use for my AR and MA components using the AIC criteria (this was using all of my data). The results are in this graphic: ...
2
votes
0answers
34 views

How to make use of less data of a particular class for better modeling?

I have a dataset, say 9000 rows, with some features. Around 8000 belong to class 1 and 1000 to class 0. So, if I am creating a model with any method say SVM, LR, Random forest the model has a tendency ...
0
votes
0answers
20 views

Confounding factor in cross-validation

I have been exploring a dataset using support vector machines. I am solving a binary classification problem and using stratified K-fold cross-validation for performance estimation (the SVM ...
0
votes
1answer
32 views

Help for interpreting SVM cross-validation results

I am using support vector machines for an unbalanced binary problem (0: 25%, 1: 75%). I do K-fold cross-validation with $K=10$. The metrics I get are: 80% classification accuracy on average for the ...
0
votes
0answers
13 views

Class weights in unbalanced SVM classification

The answer to this question says that class weights for unbalanced SVM classification can be picked so that that sums of the weights for each class are equal. Should this be done before ...
3
votes
1answer
32 views

How is the confusion matrix reported from K-fold cross-validation?

Suppose I do K-fold cross-validation with K=10 folds. There will be one confusion matrix for each fold. When reporting the results, should I calculate what is the average confusion matrix, or just sum ...
0
votes
1answer
22 views

Isn't leave-1-out insufficient for proper classification evaluation?

I encountered several papers that used some classification method (for instance, LDA), with leave-1-out validation, and posted the classification results as an aggregation of all results (for all ...
0
votes
1answer
18 views

Cross-Validation in binary classification using only 10 positive samples (SVM)

I have a binary classification problem for which only $10$ positive samples are available for training. Negatives are in general in abundance, but I choose to use solely $70$ ($7$ negatives per one ...
0
votes
0answers
16 views

K-cross validation and Naive Bayes

I am doing an exercise of machine learning, and I have built a Gaussian Naive Bayes classifier (i.e., I have defined values of mean and standard deviation) using scikit-learn. Now I am supposed to ...
0
votes
1answer
28 views

Estimating the variance of the noise in Gaussian Process prediction

I've been trying to use leave-one-out cross-validation to estimate the $\sigma_n$, the variance of the signal noise when doing prediction according to $E[f_*] = k_*^T(K+\sigma_n^2I)^{-1}y$ (GPML ...
0
votes
1answer
19 views

Cross validated $R^2$ and the adjusted $R^2$?

What are the similarities and differences between cross validated $R^2$ and adjusted $R^2$?
0
votes
0answers
12 views

Different results with each test/ train set

I'm new to machine learning and is facing a very basic problem. I have around 500 labelled data with 8 features. I'm trying to build surrogate models on this data using linear regression. I want to do ...
0
votes
0answers
12 views

Rebalancing class-imbalance in test set?

A friend of mine has code, which rebalances the classes of the test set before running the algorithm and calculating the accuracy. This causes the distribution of the two classes to be 50%/50% instead ...
3
votes
1answer
19 views

KFold Cross Validation Package/Library in C++?

I need to do some cross validation work in C++. Is there any existing package/library that you'd recommend? I performed a search on Google but prefer to get advice from field experts here. Thank you ...
5
votes
1answer
58 views

At what point does cross-validation become overtraining?

I've often worked on projects for which the data is plentiful enough that I can do k-fold cross validation (k=5 or k=10, typically). In my experience, I've used this as a way to compare different ...
0
votes
0answers
37 views

caret: using random forest and include cross-validation

I used the caret package to train a random forest, including repeated cross-validation. I’d like to know whether the OOB, as in the original RF by Breiman, is used or whether this is replaced by the ...
1
vote
0answers
41 views

Should I perform linear regression multiple times to train my dataset?

I am working on Boston data set from MASS library. I separated the training and test data (70 / 30) In order to train my data, should I run linear regression multiple times on training data? Is this ...
0
votes
1answer
21 views

Assessing model performance of stochastic algorithm

I'm looking at how I currently evaluate my classification models and wondering if it could be improved. I've got a stochastic algorithm (Genetic Programming), which for non-classification problems is ...
1
vote
1answer
40 views

Proper cross validation for stacking models

Lets assume that we have dataset that contains continuous variable $Y$ which we want to predict and 10 predictors $X_{1}, ..., X_{10}$. The number of observations is $n=1000$. I have questions about ...
-1
votes
1answer
40 views

Clustering Data of 8 dimensions

I am working on a data clustering and don't know how I can achieve it with R ! I am working on a data set of 50 observations each of 8 variables. What i want is to have clusters gathering the ...
1
vote
0answers
17 views

Should cross-validation to compare models be performed with the same partitions?

If I want to compare two regression models using cross-validation, should I use the same partitions of training and test data for both of the models? For example, suppose I fit a linear model with ...
2
votes
1answer
23 views

Is it a good idea to evaluate cross-validation using correlation coefficient?

I am doing cross-validation of my model. I was looking for a metric, that would be able to compare the predictions with the independent data, and I thought that correlation coefficient r would be very ...
0
votes
1answer
41 views

What is test and what is training data in this SVM formula?

I am studying how to use Gaussian RBF kernels for mapping 2D data to 3D. In this link: Use Gaussian RBF kernel for mapping of 2D data to 3D, @MaxS provides an answer on this topic, but I can't ...
1
vote
0answers
26 views

K-means validation

If anyone knows a suitable approach to validate cluster solution, I will be glad if the person share with me. I am conducting a research using k-means and partition gave me two groups. The second part ...
3
votes
0answers
54 views

How to avoid an overfitting?

The standard way to avoid an over-fitting is to use a "validation set". It means that we split the data into two parts. The first part we use to fit (train) and the second part we use to validate. ...
1
vote
0answers
41 views

Does overlapping standard deviation between null and candidate models imply statistical insignificance? [closed]

Let me give an example. Suppose I'm trying to solve a classification problem. I am using 10-fold cross validation to evaluate performance and I am testing two classifiers ($C_1/C_2$). Let's say that ...
5
votes
1answer
128 views

In a model with several parameters, which one should be tuned via cross validation first?

I have a loss function like $$\eqalign{ L(U, V, P, Q) = & \alpha_1 (R - U \cdot V^T )^2 + \alpha_2 (D - U \cdot P^T )^2 + \alpha_3 (S - V \cdot Q^T )^2 \\ &+ \lambda_1 (\parallel U ...
0
votes
1answer
25 views

After adding additional features, same accuracy on test data, but higher accuracy on training data, how should I interpret ?

I've done 5-fold cross-validation and the model is SVM. 300 features: 0.53 on test, 0.55 on training; 700 featuers: 0.53 on test, 0.67 on training. Does this mean that the additional 400 features ...
0
votes
0answers
17 views

Leave one out cross validation error term interpretation

I have a dataset that involved 70 participants and 7 variables (1 y variable and 6 explanatory variable). I have used leave one out cross validation to assess the model and have resulted in an answer ...
0
votes
0answers
31 views

Cross-Validation gives different result on the same data

I have done Cross-Validation by crossval function in matlab on my data, but when I run the Cross-Validation many times, it give me a different results, so is that ...
1
vote
1answer
16 views

K-fold cross validation for hierarchical data sets

I'm currently working on a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). Does anyone know how to write R code for this type of ...
0
votes
1answer
31 views

Jackknife vs. LOOCV

Is there really any difference between the jackknife and leave one out cross validation? The procedure seems identical am I missing something?
0
votes
0answers
22 views

K-fold cross validation and hierarchical data structure and lme4 package

I'm currently trying to locate R code to conduct a k-fold cross validation for a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). In ...
0
votes
0answers
27 views

What is the commonly used mehtod for measuring variance of accuracy mean using k-fold cross validation?

I know there are planty of questions about standard deviation, though I didn't find an answer tuned to my particular need and also I could really use your help! I'm performing 18-Fold Cross ...
5
votes
1answer
58 views

How to deal with factors with rare levels in cross-validation?

Suppose in a regression analysis in R, I have a factor type independent variable with 3 levels in my train dataset. But in the test data set that same factor variable has 5 levels. Therefore I can not ...