Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
3 views

What does ROC-EER in percent stand for?

Ive tried to understand what the ROC Curve represents and what EER (Equal Error Rate) means. And I somehow think I got to understand some of the explanations I read on the internet and videos I ...
0
votes
0answers
12 views

Metrics for regression error

Suppose I undertake a least squares regression on some data. I end up with a function such as $\hat{f}(x,y,\ldots)=\hat{\beta_0}+\hat{\beta_1}\cdot x+\hat{\beta_2}\cdot y+\hat{\beta_3}\cdot ...
-1
votes
0answers
7 views

Can I get input/output slacks for efficient units? [on hold]

My DEA results indicate input/output slacks for efficient units. Is there some error? If no, then can I interpret it as'for unit which are efficient, slacks represent excess use of resources which ...
4
votes
2answers
38 views

Are cross-validated prediction errors i.i.d?

Say, we test an arbitrary regression or classification procedure on $n$ independent samples with leave-one-out cross-validation. This results in an estimate of the prediction error $e_n$ for each ...
0
votes
0answers
24 views

What is the difference between two implementations of a random forest? [closed]

Question 1: What is the difference between these two implementations of a random forest? Both models RF1 and RF2 use repeated cross validation. The only difference I see is that the number of trees is ...
1
vote
0answers
22 views
1
vote
0answers
29 views

Confidence intervals sum of dependent variables

How to construct confidence intervals for sum of dependent random variables. Specifically, I want a high probability claim regarding the difference between the empirical mean and the true mean of the ...
1
vote
2answers
46 views

Does the order of input matter in cross-validation in linear regression?

Please imagine the following problem: A linear regression problem with one input variable (X) and one output (Y) The number of input data is 50 instances. The input data is sorted in increasing ...
8
votes
1answer
63 views

Are there any contemporary uses of jackknifing?

The question: Bootstrapping is superior to jackknifing; however, I am wondering if there are instances where jackknifing is the only or at least a viable option for characterizing uncertainty from ...
3
votes
2answers
34 views

Does k-fold cross validation always imply k uniformly sized subsets?

I'm a bit confused on a minor point that I'm trying to discern due to a cross-validation strategy I've come across in my work that creates k-folds but the folds are not of equal length (for example ...
0
votes
0answers
29 views

PCA and cross-validation [duplicate]

I am fairly new to the machine learning, and I have been going over all the great posts about cross-validation today and I have a question regarding PCA and cross-validation, I don't have enough ...
0
votes
1answer
19 views

Cross validation on clinical datasets [closed]

I am very new to R programming. In my project I need to perform a Cross validation for the clinical datasets (small). I want to know what will be the results. I am unable to recognize the results. I ...
0
votes
0answers
26 views

Statistical comparison of multiple prediction models

I have a rather limited data set where for 100 subjects 30 attributes were measured before surgery and one attribute ($y$) was measured after surgery. About 20% of values are missing. The goal is to ...
1
vote
1answer
21 views

K-fold validation, how to use MSE and STD for model selection

When using K-fold validation for model selection I'm wondering what's the best approach to select a model using both the mean square error (MSE) and the standard deviation of errors among folds (STD). ...
0
votes
0answers
13 views

Non idependence within groups

I have to train a machine learning model for classifying two groups. Unfortunately, my positive group has a small number and many cases are not independent from each other (observations taken in ...
1
vote
0answers
10 views

Unbalanced groups and classification errors

I would like to adopt a general strategy for dealing with an very unbalanced dataset, where my "positive" group corresponds to 1/40 of all the observations. The reason why I ask it is because all the ...
0
votes
0answers
15 views

logistic regression test error rate in intercept-only model

I'm using logistic regression with LOOCV and am balancing the classes for the two responses. I noticed that with my model, the test error rate is decent (0.22) and the predictor variable is ...
1
vote
1answer
20 views

How useful is estimate of accuracy for cross-validation in case of imbalance in class distribution

I have about 4000 instances of one class and 38000 instances of another. I used the DAAG library and I got the following result: ...
0
votes
0answers
54 views

How to validate sentiment classification and compare different algorithms

I need to compare SVM and NB about sentiment classification by evaluating accuracy, precision and recall measures. I have 1500 manually classified documents, and I would know which is the best way to ...
2
votes
1answer
28 views

Test Error less than cross-validation error-implications?

If the test-set RMSE error of a model is less than cross-validated RMSE error, how can I interpret this? Is this abnormal? Does it imply a mistake in the methodology?
2
votes
1answer
37 views

CV for model parameter tuning AND then model evaluation

I have a basic question on using cross-validation for model parameter tuning (model training) and model evaluation (testing) similar to this Model Tuning and Model Evaluation in Machine Learning I ...
0
votes
0answers
4 views

Possible causes of large F-score fluctuation between iterations with random data preprocessing?

I'm trying to create prediction models for a specific dataset and obtain their out of sample error to test their generalization power. To do so, I'm using hold out cross validation, randomly selecting ...
0
votes
0answers
28 views

Reporting variance of the repeated k-fold cross-validation

I have been using repeated k-fold cross validation and been reporting the mean (of the evaluation metric e.g., sensitivity, specificity) computed as the grand mean across the folds of different runs ...
0
votes
0answers
42 views

Best machine learning methtod for classificating datasets with non-independent cases within the groups

I have to perform binary classification of my data with supervised machine learning, but I have some difficulties working with my data set. It consists many genetic mutations that have parameters ...
1
vote
3answers
37 views

How to compare 2 predictive models where one uses predictor with missing values

I am developing a model to predict y from a dataset (N=20,000) that contains x1, x2. Say I ...
1
vote
1answer
26 views

Statistically significant comparison of classifiers

I am working on a movie review sentiment analysis project, and comparing various classifiers on the same dataset. The data for the two classes is balanced, so I'm using accuracy on 3-fold cross ...
-1
votes
0answers
10 views

which could be better for rationale weight assignment to different properties, PCA or Shannon Entropy?

Rationale weight assignment to the factors normally done by either using Shannon entropy or by using PCA, but I am bit confused which could be better between these two analysis, please help?
4
votes
1answer
74 views

How to choose a kernel for kernel PCA?

What are the ways to choose what kernel would result in good data separation in the final data output by kernel PCA (principal component analysis), and what are the ways to optimize parameters of the ...
0
votes
1answer
67 views

Metrics for multi-class problems in R caret package for various method tags

The caret package for R provides a variety of error metrics predominantly aimed at 2-class classification models with limited error metrics. Here is a multi-class function to allow caret:::train to ...
6
votes
7answers
752 views

Why not validate on the entire training set?

We have a dataset with 10,000 manually labeled instances, and a classifier that was trained on all of this data. The classifier was then evaluated on ALL of this data to obtain a 95% success rate. ...
2
votes
2answers
41 views

Cluster analysis as a preliminary analysis

I want to produce four groups (high/high, high/low, low/high and low/low) using two continues variables and compare these groups in terms of a few dependent variables. I know that cluster analysis ...
1
vote
0answers
18 views

RandomForest - why isn't it predicting well with manually-selected test sets?

I am using python sklearn.ensemble to do a RandomForestClassifier on about 800K rows of data, coupled with sklearn.cross_validation to generate the train/test sets. When it completes, it says on the ...
1
vote
1answer
37 views

Running tests over samples from cross-validation on multiple datasets. To average or not to average?

Let us say that I am trying to investigate whether we can reliably decode (i.e., predict) some information from some data. The particular scenario is predicting the object a person is seeing from the ...
0
votes
0answers
24 views

Analyzing variance of major parameters of SVM model

I am using SVM to classify a two class problem using the set of features from a dataset of 474 samples making 237 training, 237 test samples. I have cross validated by making 100 random combination of ...
0
votes
0answers
29 views

K Fold Cross Validation, Variable Selection in LDA

I'm currently working on a multi-class classification problem and I attempt to use lda for the same. I have 2 questions here. 1) Is it possible to perform k-fold ...
2
votes
2answers
40 views

Cross validation with Pearson correlation coefficient - is testing one tailed enough?

My method of cross validation is to first split my sample into two sub-samples, with 80% respectively 20% of the observations, and then to correlate the predicted values of my model (created with the ...
3
votes
1answer
27 views

Is it better to pre-filter the entire data set or just the training sub-set?

I am currently working on a classifier for the qualitative spectral analysis of alloys. One of the problems that I faced is preparation of samples for the classifier training. Samples have to me ...
1
vote
0answers
10 views

Multiple Imputation and Matrix Completion

It is quite common that data sets will contain missing values in them. Suppose we want to try to fill in the missing values. For this we have techniques such as single/multiple imputation and matrix ...
0
votes
0answers
39 views

K-fold Cross Validation. R squared value?

I am using multiple linear regression with a data set of 72 variables and using 5-fold cross validation to evaluate the model. I am unsure what values I need to look at to understand the validation ...
0
votes
1answer
27 views

Confused about cross validation for model stacking

I'm reading section 8.8 of Elements of Statistical Learning, and though I keep reading the section on calculating the ensemble weights I'm missing something. It says that the stacking weights are ...
0
votes
0answers
21 views

Significance of multivariate models and correction for multiple comparisons

I have performed a multivariate binary classification using a number of features (or variables), I will call them features from sets (A), (B) and (C). I have calculated the P value of this ...
0
votes
0answers
10 views

K-fold Cross Validation Toolbox in ArcMap

K-fold Cross Validation Toolbox in ArcMap? I have a raster map in arcmap that create with many points by different algorithms and i need to validate this models. but in arcmap existe cross validation. ...
6
votes
1answer
365 views

What is cross-validation?

I'm having trouble understanding what cross-validation is. Also, what is the connection between cross-validation and the issue of model overfitting?
0
votes
1answer
16 views

20-fold cross validation of a dataset composed of1000 observations?

I have a dataset made up of 1000 observations and I want to split that data and use it for estimation of parameters in a way that maximum of 50 observations form one fold and my estimation process ...
1
vote
1answer
47 views

Cross validation in semi-supervised learning

With semi-supervised learning a labeled set $X_L$ and unlabeled set $X_U$ are given. If the learning algorithm has several free-parameters we are forced to perform cross-validation to try to guess ...
0
votes
1answer
32 views

Possible overfitting?

Hi I have a limited dataset with 100 examples with 15 features. I trained a linear svm with 80 samples after I did a 5-fold cross-validation and found the best parameter values for C. Then I tested ...
0
votes
0answers
15 views

With posterior inclusion probability how do I settle on the final predictive model?

After using the spike-and-slab prior to perform Bayesian model selection, I get the posterior distribution of my variables, from which I calculate the inclusion probability for each variable. How do ...
0
votes
0answers
6 views

Relative magnitudes of mean squared errors in cross-validation and test data for large regression trees

When pruning a regression tree using cost-complexity pruning, is there any reason to expect that the mean squared errors for the cross-validated data is larger than the mean squared errors for the ...
0
votes
1answer
29 views

Outer crossvalidation cycle in caret package (R)?

Could somebody provide a nice example code how to best implement an outer crossvalidation cycle using the caret package in R? The package provides a convenient trainControl() argument to ajust the ...
1
vote
0answers
48 views

Cross-Validation: how to choose k for small datasets (n=900)?

In a binary classification task, I have a small training set (n=900, 9 features). The two groups are not symmetric (1 = 560, 0 = 340). I also have a test set (n=400) where I don't know the class ...