Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
5 views

K-fold Cross Validation in SPSS for Multiple Linear Regression [duplicate]

I have a data set of 72 variables and have performed multiple linear regression in SPSS. I'm looking to use k-fold cross validation to evaluate the model's performance but I'm having some ...
0
votes
0answers
11 views

Confused about cross validation for model stacking

I'm reading section 8.8 of Elements of Statistical Learning, and though I keep reading the section on calculating the ensemble weights I'm missing something. It says that the stacking weights are ...
0
votes
0answers
13 views

Significance of multivariate models and correction for multiple comparisons

I have performed a multivariate binary classification using a number of features (or variables), I will call them features from sets (A), (B) and (C). I have calculated the P value of this ...
0
votes
0answers
8 views

K-fold Cross Validation Toolbox in ArcMap

K-fold Cross Validation Toolbox in ArcMap? I have a raster map in arcmap that create with many points by different algorithms and i need to validate this models. but in arcmap existe cross validation. ...
6
votes
1answer
355 views

What is cross-validation?

I'm having trouble understanding what cross-validation is. Also, what is the connection between cross-validation and the issue of model overfitting?
0
votes
1answer
14 views

20-fold cross validation of a dataset composed of1000 observations?

I have a dataset made up of 1000 observations and I want to split that data and use it for estimation of parameters in a way that maximum of 50 observations form one fold and my estimation process ...
1
vote
1answer
35 views

Cross validation in semi-supervised learning

With semi-supervised learning a labeled set $X_L$ and unlabeled set $X_U$ are given. If the learning algorithm has several free-parameters we are forced to perform cross-validation to try to guess ...
0
votes
0answers
24 views

time series rolling cross-validation for parameter selection and model comparison

I want to do two things using rolling cross-validation for time series (as in the famous Hyndman's post): select parameters for model A, and compare it's predictive performance with a model B. I'm ...
0
votes
1answer
30 views

Possible overfitting?

Hi I have a limited dataset with 100 examples with 15 features. I trained a linear svm with 80 samples after I did a 5-fold cross-validation and found the best parameter values for C. Then I tested ...
0
votes
0answers
13 views

With posterior inclusion probability how do I settle on the final predictive model?

After using the spike-and-slab prior to perform Bayesian model selection, I get the posterior distribution of my variables, from which I calculate the inclusion probability for each variable. How do ...
0
votes
0answers
6 views

Relative magnitudes of mean squared errors in cross-validation and test data for large regression trees

When pruning a regression tree using cost-complexity pruning, is there any reason to expect that the mean squared errors for the cross-validated data is larger than the mean squared errors for the ...
0
votes
1answer
24 views

Outer crossvalidation cycle in caret package (R)?

Could somebody provide a nice example code how to best implement an outer crossvalidation cycle using the caret package in R? The package provides a convenient trainControl() argument to ajust the ...
1
vote
0answers
45 views

Cross-Validation: how to choose k for small datasets (n=900)?

In a binary classification task, I have a small training set (n=900, 9 features). The two groups are not symmetric (1 = 560, 0 = 340). I also have a test set (n=400) where I don't know the class ...
2
votes
2answers
79 views

Backward feature selection with CV model selection

I am thinking about doing the following to a data set with $N$ samples and $m$ features 1) Train using semi-supervised learning and cross validate on labeled data using LOO-CV to select the best ...
0
votes
0answers
44 views

Advice for feature selection or feature extraction with semi-supervised learning

I am trying to solve a semi-supervised learning problem using LaplacianSVM. However, before applying LapSVM I would like either to perform feature selection or feature extraction. Furthermore, after ...
1
vote
1answer
69 views

How do you evaluate a generative model?

Evaluating a discriminative model is relatively easy: compare the predictions with ground truth, using cross-validation. Unfortunately this strategy can't be used for generative models. Surely this ...
0
votes
0answers
31 views

Is 100% accuracy using randomForest indicative of anything wrong?

I am getting a 100% accurate result on randomForest model in R for loan default data even when my training set and test set are completely non-overlapping. I am using abt 8 parameters/features for ...
0
votes
2answers
67 views

Split clustered data into calibration and validation sample (Cross validation)

I have a dataset with >800 cases ($n$) from >30 ($k$) different organisations (clustered data). The number of cases within each organisation differ (unbalanced data; e.g.: organisation 1 = 30 cases, ...
1
vote
1answer
61 views

R caret package - number of principal components when preprocessing using PCA

I am using the caret package in R for training of binary SVM classifiers. For reduction of features I am preprocessing with PCA using the built in feature [preProc=c("pca")] when calling train(). How ...
1
vote
0answers
17 views

Confidence intervals for the Log Loss metric for model comparison?

Quite a few Kaggle competitions have used or are using the Logarithmic Loss metric as the quality measure of a submission. I'm wondering if there are other ways besides N-fold cross-validation to ...
0
votes
0answers
79 views

Cross Validation to find Misclassification rate of Explanatory Variables

I am trying to create a function that will allow me to identify which explanatory variable (x) in a logistic regression of a data set has the lowest rate of error in predicting a response variable (y) ...
1
vote
2answers
74 views

How to do cross-validation when comparing different feature selection methods?

I am using SVM for a prediction task. My sample size is small, only N=140. Suppose I want to compare the prediction accuracy when using two different feature selection methods. Would it be better to: ...
1
vote
0answers
49 views

Interpreting output from cvFit(), understanding cross-validation in classification tree model

I am trying to understand how to interpret the output for cvFit(). The data is from UCI's ML repository. This is my model ...
0
votes
0answers
28 views

cvFit mean predicted error interpretation for nls models

I have been using the cvFit() function of the cvTools library in order to test my models (nls() ones), but I would like to know more precisely what the cvFit() returns to me when it's done. It only ...
1
vote
1answer
51 views

Glmnet Caret Package with small number of observations

I have a regression problem where I’m attempting to train a data set with 70 predictors, but only 35 observations with glmnet in the caret package. I’m trying to determine the best resampling method. ...
3
votes
2answers
95 views

cross-validation to predict distribution of errors on finite test sets

In one use of k-fold cross-validation for evaluating classifiers, one trains k models, each on n(k-1)/k examples, and tests each on n/k examples. The average accuracy on those k test sets of size n/k ...
3
votes
0answers
47 views

LASSO prediction model question

I am trying to create a prediction model with 33 predictors (brain metabolite levels in various regions) and 8 observations (cognitive test scores) with p>>n problem using LASSO in MATLAB (...
8
votes
1answer
233 views

Can we use leave one out mean and standard deviation to reveal the outliers?

Suppose I have normally distributed data. For each element of the data I want to check how many SDs it is away from the mean. There might be an outlier in the data (likely only one, but might be also ...
1
vote
0answers
15 views

Cross-validating a survival model with right censoring?

Is it sensible and possible to cross-validate a survival model? Does it depend on whether there is censoring? If not, why? If the answer depends on the model, then answer for common survival models ...
0
votes
0answers
9 views

How to account for different ratio of samples during training and detection using a support vector machine (svm)?

Consider the following object recognition case: Detection of objects in an image using a sliding window approach in combination with a svm model. During sliding window search using multiple scale ...
0
votes
0answers
24 views

How to find a good model for an object recognition case using a support vector machine (svm)?

Consider the following example of an object recognition case: I'm trying to detect objects in an image using histograms of oriented gradients (hog) features. The feature vector resulting from hog is ...
0
votes
0answers
23 views

How to CORRECT un-reliable and un-stability in the prediction results

Currently, I meet such questions when building Random Forest model using my data set. My full data set: X_lab: 839 * 469 and y_lab: 839 * 1 which is for all labelled data and X_unl: 20346 * 469 which ...
0
votes
0answers
12 views

filter feature selection output and cross validation

If I use a filter method for ranking the features like Relief. suppose I have 100 features with 1000 sample and I used cross validation 3-fold . therefore I have 3 ranks for may features . at the end ...
1
vote
0answers
20 views

Distinguishing between different notions of $R^2$

What is the distinction between $R^2_{pop}$ – the population R-squared $R^2_{out}$ – the out-of-sample R-squared $R^2_{c.v.}$ – the squared population cross-validity coefficient ? These ...
0
votes
1answer
49 views

What is cross validation?

Yes, I know that after we fit (train) a predictive model on a training dataset, we need to test the fitted model on the testing-dataset. The motivation behind this procedure is clear: If our model ...
2
votes
2answers
63 views

Using K-fold cross validation to select a model's parameters

I think I understand completely the concept of cross validation, but there is one aspect I've never seen detailed. Let's assume I have a logistic regression model with four parameters I want to train. ...
1
vote
2answers
76 views

Different results from several “passes” of Random Forest on same dataset

I've been playing around with the German Credit dataset available in Kuhn & Johnson's caret package for ...
0
votes
0answers
42 views

Loss Functions and Evaluation Metrics

Do you have to evaluate with the same (or equivalent) loss function for model selection purpose? Say you have bunch of models to select. A loss function of one model in training stage is different ...
0
votes
0answers
22 views

Why would a reasonable range of the regularization parameters $\lambda$ be up to the maximum eigenvalue of the kernel matrix?

I was wondering, how do you choose a reasonable range for the regularization parameter $\lambda$ for regularized least squares when doing k-fold cross validation? I was told that a reasonable range ...
3
votes
4answers
117 views

Sample selection algorithms to ensure that training & validation sets are representative

Currently, I am encountering a question, which is how to selection representative samples (training set and test set, even validation set) from the whole data set? I would like build a classification ...
1
vote
0answers
35 views

Cross validation with unequal sample size for the left out sets

I am trying to do cross validation on several (20) subsets of samples, which all have unequal sample size. I cannot subsample so that sizes are equal. Example: batch 1: 500 samples batch 2: 400 ...
1
vote
1answer
39 views

10-fold cross-validation (high variation)

I am using 10-fold validation method to validate my model. I am using CART model and my sample size $\approx$ 50. Features $\approx$ 9. The 10-fold validated accuracy (averages) is about 76%. However, ...
0
votes
0answers
15 views

cross-validate hierarchical model for binomial data that is often sparse

I have binomial data (e.g, 130 successes in 4000 trials). In many, if not most, cells of interest, there were few trials and thereby few successes (e.g., 0 successes in 35 trials, 1 successes in 18 ...
2
votes
1answer
38 views

How to perform Cross-Validation for glasso to select lambda in R

I am using glasso for variable selection. To get the best possible value of lambda cross validation is recommended. However, I am not able to find how to perform cross validation for glasso in R. ...
0
votes
1answer
42 views

Probabilistic importance value for Caret linear SVM classifier

In a linear SVM model, inside caret I would like to get the variable importance after recursive feature elimination, so according to the documentation: ...
-1
votes
1answer
43 views

How to choose the right model after k-fold cross validation is done?

I'm using naive bayes to classify tweet into three classes. and i want to use k-fold cross validation to predict the right model, but i'm confused how to choose the right model after k-fold validation ...
1
vote
0answers
39 views

R - Warnings when using cv.lm

I've tried to apply cv.lm function instead of my own script which performs K-Fold cross validation and though the results are matching the formulas and my own thing, I keep getting warnings when using ...
0
votes
0answers
33 views

How do I get a classification report for my cross validated scores using sklean

I am running a logistic regression model using sklearn with 2 classes (1 and 0). Here is my code: ...
0
votes
0answers
12 views

Standard errors for the CV error curve using the boot package

Does anybody know how to obtain the standard errors for the CV error curve using the boot package? I understand the boot package can compute the K-fold CV for a fitted model, but I'd like to know if ...
0
votes
0answers
13 views

Comparing training and test data before cross-validation

When creating a train and test set to be used for cross validation, is it standard practice (or worthwhile) to, say, run Welch's t-tests on each feature between the two data sets to ensure that they ...