Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
14 views

K-cross validation and Naive Bayes

I am doing an exercise of machine learning, and I have built a Gaussian Naive Bayes classifier (i.e., I have defined values of mean and standard deviation) using scikit-learn. Now I am supposed to ...
0
votes
1answer
16 views

Estimating the variance of the noise in Gaussian Process prediction

I've been trying to use leave-one-out cross-validation to estimate the $\sigma_n$, the variance of the signal noise when doing prediction according to $E[f_*] = k_*^T(K+\sigma_n^2I(^{-1}y$ (GPML ...
0
votes
1answer
18 views

Cross validated $R^2$ and the adjusted $R^2$?

What are the similarities and differences between cross validated $R^2$ and adjusted $R^2$?
0
votes
0answers
11 views

Different results with each test/ train set

I'm new to machine learning and is facing a very basic problem. I have around 500 labelled data with 8 features. I'm trying to build surrogate models on this data using linear regression. I want to do ...
0
votes
0answers
12 views

Rebalancing class-imbalance in test set?

A friend of mine has code, which rebalances the classes of the test set before running the algorithm and calculating the accuracy. This causes the distribution of the two classes to be 50%/50% instead ...
3
votes
1answer
16 views

KFold Cross Validation Package/Library in C++?

I need to do some cross validation work in C++. Is there any existing package/library that you'd recommend? I performed a search on Google but prefer to get advice from field experts here. Thank you ...
1
vote
0answers
18 views

R: Cross Validation with caret and linear regression

I want to choose best (predictive) linear model with 10-fold Cross-Validation by stepwise selection and minimizing MSE in each step. The R-function I use for this is train from the R-package caret. ...
5
votes
1answer
57 views

At what point does cross-validation become overtraining?

I've often worked on projects for which the data is plentiful enough that I can do k-fold cross validation (k=5 or k=10, typically). In my experience, I've used this as a way to compare different ...
0
votes
0answers
21 views

caret: using random forest and include cross-validation

I used the caret package to train a random forest, including repeated cross-validation. I’d like to know whether the OOB, as in the original RF by Breiman, is used or whether this is replaced by the ...
1
vote
0answers
38 views

Should I perform linear regression multiple times to train my dataset?

I am working on Boston data set from MASS library. I separated the training and test data (70 / 30) In order to train my data, should I run linear regression multiple times on training data? Is this ...
0
votes
1answer
17 views

Assessing model performance of stochastic algorithm

I'm looking at how I currently evaluate my classification models and wondering if it could be improved. I've got a stochastic algorithm (Genetic Programming), which for non-classification problems is ...
1
vote
1answer
34 views

Proper cross validation for stacking models

Lets assume that we have dataset that contains continuous variable $Y$ which we want to predict and 10 predictors $X_{1}, ..., X_{10}$. The number of observations is $n=1000$. I have questions about ...
-1
votes
1answer
36 views

Clustering Data of 8 dimensions

I am working on a data clustering and don't know how I can achieve it with R ! I am working on a data set of 50 observations each of 8 variables. What i want is to have clusters gathering the ...
1
vote
0answers
17 views

Should cross-validation to compare models be performed with the same partitions?

If I want to compare two regression models using cross-validation, should I use the same partitions of training and test data for both of the models? For example, suppose I fit a linear model with ...
2
votes
1answer
21 views

Is it a good idea to evaluate cross-validation using correlation coefficient?

I am doing cross-validation of my model. I was looking for a metric, that would be able to compare the predictions with the independent data, and I thought that correlation coefficient r would be very ...
0
votes
1answer
40 views

What is test and what is training data in this SVM formula?

I am studying how to use Gaussian RBF kernels for mapping 2D data to 3D. In this link: Use Gaussian RBF kernel for mapping of 2D data to 3D, @MaxS provides an answer on this topic, but I can't ...
1
vote
0answers
23 views

K-means validation

If anyone knows a suitable approach to validate cluster solution, I will be glad if the person share with me. I am conducting a research using k-means and partition gave me two groups. The second part ...
3
votes
0answers
50 views

How to avoid an overfitting?

The standard way to avoid an over-fitting is to use a "validation set". It means that we split the data into two parts. The first part we use to fit (train) and the second part we use to validate. ...
1
vote
0answers
38 views

Does overlapping standard deviation between null and candidate models imply statistical insignificance? [closed]

Let me give an example. Suppose I'm trying to solve a classification problem. I am using 10-fold cross validation to evaluate performance and I am testing two classifiers ($C_1/C_2$). Let's say that ...
5
votes
1answer
125 views

In a model with several parameters, which one should be tuned via cross validation first?

I have a loss function like $$\eqalign{ L(U, V, P, Q) = & \alpha_1 (R - U \cdot V^T )^2 + \alpha_2 (D - U \cdot P^T )^2 + \alpha_3 (S - V \cdot Q^T )^2 \\ &+ \lambda_1 (\parallel U ...
0
votes
1answer
25 views

After adding additional features, same accuracy on test data, but higher accuracy on training data, how should I interpret ?

I've done 5-fold cross-validation and the model is SVM. 300 features: 0.53 on test, 0.55 on training; 700 featuers: 0.53 on test, 0.67 on training. Does this mean that the additional 400 features ...
0
votes
0answers
12 views

Leave one out cross validation error term interpretation

I have a dataset that involved 70 participants and 7 variables (1 y variable and 6 explanatory variable). I have used leave one out cross validation to assess the model and have resulted in an answer ...
0
votes
0answers
30 views

Cross-Validation gives different result on the same data

I have done Cross-Validation by crossval function in matlab on my data, but when I run the Cross-Validation many times, it give me a different results, so is that ...
1
vote
1answer
13 views

K-fold cross validation for hierarchical data sets

I'm currently working on a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). Does anyone know how to write R code for this type of ...
0
votes
1answer
29 views

Jackknife vs. LOOCV

Is there really any difference between the jackknife and leave one out cross validation? The procedure seems identical am I missing something?
0
votes
0answers
21 views

K-fold cross validation and hierarchical data structure and lme4 package

I'm currently trying to locate R code to conduct a k-fold cross validation for a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). In ...
0
votes
0answers
27 views

What is the commonly used mehtod for measuring variance of accuracy mean using k-fold cross validation?

I know there are planty of questions about standard deviation, though I didn't find an answer tuned to my particular need and also I could really use your help! I'm performing 18-Fold Cross ...
5
votes
1answer
51 views

How to deal with factors with rare levels in cross-validation?

Suppose in a regression analysis in R, I have a factor type independent variable with 3 levels in my train dataset. But in the test data set that same factor variable has 5 levels. Therefore I can not ...
1
vote
1answer
32 views

What model fit / predictive accuracy measure can be used to cross validate a Cox PH model with censored data?

How would you go about validating a Cox PH model with censored data? I am trying to run a Cox PH model on a dataset with observations that failed, and observations that are censored. Normally, I use ...
0
votes
1answer
31 views

bootstrapping vs. “repeated cross validation”

For a research project, I conducted the following methodology. The dataset was of size $N$. $B$ times, I: took a random $N/2$ rows and trained my model, which finds the optimal size $M$ of a ...
0
votes
0answers
13 views

How to include only true positives and false positives, that is ignore false negative classifications in a confusion matrix?

I have performed a 10 fold cross validation on my data set using binary decision trees. I've got 6 trees (to detect one of the six basic human emotions from facial data points) trained for each fold. ...
1
vote
1answer
17 views

Matlab - why crossval function inputs a full trained model?

The question is regarding the Matlab implementation. As we can see here, the crossval function expects to receive a full trained model. For example, my data ...
1
vote
0answers
47 views

Combining bootstrap and cross validation

I recently read this paper: Estimating misclassification error with small samples via bootstrap cross-validation, by Fu et al. (BMC Bioinformatics, 2005). The authors talk about combining cross ...
0
votes
0answers
19 views

What is the time complexity of binary classification of SVM?

One of the earliest solution to the SVM problem is SMO applied to dual form.What is the time complexity of SMO algorithm? What is the best known time complexity to solve SVM algorithm (non linear)?
0
votes
0answers
45 views

using Cross Validation in matlab with neural networks

I want to make a cross validation on neural network, I tried to pass the labels to crossval function, with the help of ...
0
votes
1answer
22 views

Designing an experiment for a marketing campaign using Incremental Response Modelling

I have the following hypothetical question, can anyone provide some clarification? I'm looking at designing an experiment or modelling what steps can be taken to maximise the Net Incremental Revenue ...
1
vote
1answer
68 views

Explain “validation” process of repeated k-fold cross-validation?

My understanding is currently that the canonical repeated k-fold cross-validation (CV) process might do the following if $n=100$ observations in sample, $k=5$ folds, $i= 10$ iterations (see iteration ...
0
votes
1answer
42 views

Comparing classification algorithms using cross validation and caret's train

I am having issues understanding some concepts of algorithm comparison/parameter optimization/cross-validation in R Let's say I want to compare two classification algorithms, such as Random Forests ...
0
votes
0answers
9 views

Blocks of variable size in k-fold cross-validation

I would like to make a custom k-fold cross-validation method for my data, by generating folds of auto-correlated observations. This would create many folds of variable size for test errors as well as ...
1
vote
1answer
32 views

Early stopping vs cross validation

I'm currently using early stopping in my work to prevent over fitting. Specifically those taken form Early Stopping But When?. I'm now wanting to compare to other classification algorithms where it ...
1
vote
1answer
126 views

Multicollinearity, plm, and omitting variables

I'm fitting a fixed effect model with plm and know that I'm dealing with multi-collinearity between two of the independent variables. I working on identifying ...
1
vote
1answer
20 views

Cross validation of a survival model- what to make of “random effects” of parameter estimates?

This is a question surrounding k-fold cross validation for time to event data. I am interested in what to do with the knowledge that certain variables fail to perform as well within some of the ...
0
votes
2answers
45 views

Datapoint Classification Accuracy

I am interested in finding ways to quantify the certainty of correct classificaions for single datapoints. This is interesting for me since for clinical studies where we for instance would classify ...
0
votes
0answers
27 views

Value of the loss function and Classification Errors in gbm package (R)

I have a simple problem of classification (0s and 1s) using adaboost loss function. When I check the components of a boosted model using the gbm package I see: ...
0
votes
0answers
95 views

Crossvalidation in hierarchical bayesian models (HBMs)

I am trying to find a way to cross-validate Hierarchical Bayesian Models used for predicting and modelling abundance in Species Distribution Models. For this purpose, I have tried posterior predictive ...
0
votes
0answers
22 views

Learning Curve Meaning

I have made a learning curve that looks like this: Why wouldn't it be more like both training and cross-validation score begin low and both gradually increase with more samples? Why does one start ...
1
vote
1answer
35 views

Data splitting in regression

Edit: The key point I'm attempting to understand is whether during a regression model building exercise, do I need separate datasets to: search for predictors and settle on a functional form ...
0
votes
1answer
24 views

Contribution of a sample to cross validation error

I was wondering how to asses which sample in the data, during K fold cross-validation drives the bias that may be observed in the results. My training data consists of 40 samples. And I try to ...
1
vote
1answer
21 views

In cross validation, higher the value of k, lesser the training data for this formula?

Is it right to say that smaller the value of k in cross validation based on the following formula, more the number of records in test data/smaller the number of records in training data. According to ...
2
votes
1answer
54 views

One vs all Linear SVM Cross validation -Parameter selection

I'm performing one vs all classification (SVM) for a dataset. Since I'm using a linear SVM the parameters I need to tune and select are-Tolerance and C. I'm a bit confused on how to go about doing ...