Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

1
vote
1answer
27 views

Glmnet Caret Package with small number of observations

I have a regression problem where I’m attempting to train a data set with 70 predictors, but only 35 observations with glmnet in the caret package. I’m trying to determine the best resampling method. ...
0
votes
0answers
15 views

cross-validation to predict distribution of errors on finite test sets

In one use of k-fold cross-validation for evaluating classifiers, one trains k models, each on n(k-1)/k examples, and tests each on n/k examples. The average accuracy on those k test sets of size n/k ...
3
votes
0answers
24 views

LASSO prediction model question

I am trying to create a prediction model with 33 predictors (brain metabolite levels in various regions) and 8 observations (cognitive test scores) with p>>n problem using LASSO in MATLAB (...
5
votes
1answer
87 views

Can we use leave one out mean and standard deviation to reveal the outliers?

Suppose I have normally distributed data. For each element of the data I want to check how many SDs it is away from the mean. There might be an outlier in the data (likely only one, but might be also ...
0
votes
0answers
29 views

SVM accuracy evaluation

Could you please answer some of my questiona regarding SVM-RFE (svm with recursive feature elimination). I am using SVM-RFE with linear kernel for the binary classification and feature selection ...
1
vote
0answers
10 views

Cross-validating a survival model with right censoring?

Is it sensible and possible to cross-validate a survival model? Does it depend on whether there is censoring? If not, why? If the answer depends on the model, then answer for common survival models ...
0
votes
0answers
3 views

How to account for different ratio of samples during training and detection using a support vector machine (svm)?

Consider the following object recognition case: Detection of objects in an image using a sliding window approach in combination with a svm model. During sliding window search using multiple scale ...
0
votes
0answers
9 views

How to find a good model for an object recognition case using a support vector machine (svm)?

Consider the following example of an object recognition case: I'm trying to detect objects in an image using histograms of oriented gradients (hog) features. The feature vector resulting from hog is ...
0
votes
0answers
20 views

How to CORRECT un-reliable and un-stability in the prediction results

Currently, I meet such questions when building Random Forest model using my data set. My full data set: X_lab: 839 * 469 and y_lab: 839 * 1 which is for all labelled data and X_unl: 20346 * 469 which ...
0
votes
0answers
8 views

filter feature selection output and cross validation

If I use a filter method for ranking the features like Relief. suppose I have 100 features with 1000 sample and I used cross validation 3-fold . therefore I have 3 ranks for may features . at the end ...
1
vote
0answers
17 views

Distinguishing between different notions of $R^2$

What is the distinction between $R^2_{pop}$ – the population R-squared $R^2_{out}$ – the out-of-sample R-squared $R^2_{c.v.}$ – the squared population cross-validity coefficient ? These ...
0
votes
1answer
36 views

What is cross validation?

Yes, I know that after we fit (train) a predictive model on a training dataset, we need to test the fitted model on the testing-dataset. The motivation behind this procedure is clear: If our model ...
1
vote
1answer
27 views

Using K-fold cross validation to select a model's parameters

I think I understand completely the concept of cross validation, but there is one aspect I've never seen detailed. Let's assume I have a logistic regression model with four parameters I want to train. ...
0
votes
2answers
44 views

Different results from several “passes” of Random Forest on same dataset

I've been playing around with the German Credit dataset available in Kuhn & Johnson's caret package for ...
0
votes
0answers
18 views

Loss Functions and Evaluation Metrics

Do you have to evaluate with the same (or equivalent) loss function for model selection purpose? Say you have bunch of models to select. A loss function of one model in training stage is different ...
0
votes
0answers
19 views

Why would a reasonable range of the regularization parameters $\lambda$ be up to the maximum eigenvalue of the kernel matrix?

I was wondering, how do you choose a reasonable range for the regularization parameter $\lambda$ for regularized least squares when doing k-fold cross validation? I was told that a reasonable range ...
3
votes
4answers
81 views

Sample selection algorithms to ensure that training & validation sets are representative

Currently, I am encountering a question, which is how to selection representative samples (training set and test set, even validation set) from the whole data set? I would like build a classification ...
1
vote
0answers
23 views

Cross validation with unequal sample size for the left out sets

I am trying to do cross validation on several (20) subsets of samples, which all have unequal sample size. I cannot subsample so that sizes are equal. Example: batch 1: 500 samples batch 2: 400 ...
1
vote
1answer
31 views

10-fold cross-validation (high variation)

I am using 10-fold validation method to validate my model. I am using CART model and my sample size $\approx$ 50. Features $\approx$ 9. The 10-fold validated accuracy (averages) is about 76%. However, ...
0
votes
0answers
10 views

cross-validate hierarchical model for binomial data that is often sparse

I have binomial data (e.g, 130 successes in 4000 trials). In many, if not most, cells of interest, there were few trials and thereby few successes (e.g., 0 successes in 35 trials, 1 successes in 18 ...
0
votes
0answers
16 views

How to perform Cross-Validation for glasso to select lambda in R

I am using glasso for variable selection. To get the best possible value of lambda cross validation is recommended. However, I am not able to find how to perform cross validation for glasso in R. ...
0
votes
1answer
19 views

Probabilistic importance value for Caret linear SVM classifier

In a linear SVM model, inside caret I would like to get the variable importance after recursive feature elimination, so according to the documentation: ...
-1
votes
1answer
32 views

How to choose the right model after k-fold cross validation is done?

I'm using naive bayes to classify tweet into three classes. and i want to use k-fold cross validation to predict the right model, but i'm confused how to choose the right model after k-fold validation ...
0
votes
0answers
14 views

R - Warnings when using cv.lm

I've tried to apply cv.lm function instead of my own script which performs K-Fold cross validation and though the results are matching the formulas and my own thing, I keep getting warnings when using ...
0
votes
0answers
16 views

How do I get a classification report for my cross validated scores using sklean

I am running a logistic regression model using sklearn with 2 classes (1 and 0). Here is my code: ...
0
votes
0answers
10 views

Standard errors for the CV error curve using the boot package

Does anybody know how to obtain the standard errors for the CV error curve using the boot package? I understand the boot package can compute the K-fold CV for a fitted model, but I'd like to know if ...
0
votes
0answers
9 views

Random State in Matlab Cross validation [migrated]

I have got stuck in doing crossvalidation in svm in matlab. I wanted to generate 5 fold stratified cross-validation and want to reset the random number generator at each run in order to be sure ...
0
votes
0answers
7 views

Comparing training and test data before cross-validation

When creating a train and test set to be used for cross validation, is it standard practice (or worthwhile) to, say, run Welch's t-tests on each feature between the two data sets to ensure that they ...
0
votes
0answers
8 views

Average accuracy score for leave-one-out cross validation

I need to show that average accuracy score for LOOCV is less or equal to (Number of support vectors trained)/(Size of training set). I have no hint on how to even start this. Any ideas? Thank you a ...
2
votes
1answer
15 views

AIC: relative versus absolute predictive error

I've read two interpretations of Akaike's Information Criterion (AIC) that seem to be in conflict, and I was hoping that someone could help me understand how to reconcile them. Interpretation 1: ...
2
votes
1answer
17 views

Cross-validation for nonlinear models that are linear in the parameters

I'd like to know if it's correct to the CV function in the forecast R package (http://cran.r-project.org/web/packages/forecast/forecast.pdf) to cross-validate a nonlinear model that is linear in the ...
1
vote
0answers
44 views

Deviance measure in glmnet package

for my current reseach I'm using the Lasso method via the glmnet package in R on a binomial dependent variable. In glmnet the optimal lambda is found via cross-validation and the resulting models ...
0
votes
0answers
31 views

Which R packages offer the foldid (or simliar) parameter for cross-validation of group lasso?

My situation: small sample size: 116 binary outcome variable long list of explanatory variables: 50 (both continuous and categorical) explanatory variables did not come from the top of my head; ...
3
votes
1answer
40 views

Why use stratified cross validation? Why does this not damage variance related benefit?

I've been told that is beneficial to use stratified cross validation especially when response classes are unbalanced. If one purpose of cross-validation is to help account for the randomness of our ...
0
votes
0answers
35 views

Using third validation set in Cross Validation?

(Note there's 2 paragraphs of background information before I get to the question) I've got a Neural Network classifier, trained with an EA to classify data. I previously used a holdout framework ...
0
votes
1answer
52 views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...
1
vote
3answers
30 views

How to split dataset for time-series prediction?

I have historic sales data from a bakery (daily, over 3 years). Now I want to build a model to predict future sales (using features like weekday, weather variables, etc.). How should I split the ...
0
votes
0answers
11 views

Cross Validation and perfcurv in Matlab

I am trying to use perfcurv in a cross validation code. However at some point all the members of the test dataset are of the same class (0). My problem is a binary classification problem. Therefore ...
0
votes
0answers
10 views

CrossValidated KNN classifier

I am working with acoustic data. I just have 110 sound samples for training and test. So for solving the problem of having few test samples i want to use cross validation method. I am using KNN ...
0
votes
0answers
16 views

comparing models based on holdout method and n-fold CV

What to do when model choice based on n-fold CV results doesn't agree with holdout method results? I’m comparing ~100 models using both n-fold CV (jackknifing) and holdout method with expanding ...
0
votes
1answer
31 views

Feature selection when bagging trees/random forest

I want to get a better understanding of feature selection and how the number of features affect performance when bagging trees. I am using Matlab's treebagger and I ...
0
votes
0answers
21 views

cross validation for parameter-tuning a metaheuristic

For a certain problem, I've come up with a novel metaheuristic. The question I'd like to answer is "Does my metaheuristic perform better than previous methods over most problem instances?". My ...
0
votes
1answer
6 views

Select the most confident variable that has two features

Suppose now I have a group of students, and for each student two measurements are given: one is the height of the student and the other is the weight of the student. Then my question is how I can ...
1
vote
0answers
46 views

disadvantages of Neural network method

Hello Dear Researchers! I want to list the advantages and disadvantages of Neural network methods for classification or estimation purposes. I have already found the advantages of NN method in many ...
1
vote
2answers
50 views

Varying LIBSVM predictions based on test series labels

So I have a pretty well testing SVC train series which puts me into the mid 80 percentile without outrageous C/g values. My current C value is 2.0 and gamma is 0.5. Good numbers across the range ...
0
votes
0answers
30 views

Complete Logistic Regression framework using K-Cross validation

I'm implementing a logistic regression model in a low event rate data. I have gone through many webpages (including stackoverflow, including my questions) but none answer or describe the end-to-end ...
1
vote
0answers
104 views

Ensuring exploratory study's validity with pseudo-simple random sampling

The context of my questions is as follows. I'm performing a cross-sectional secondary research study, involving open source software (OSS) projects. I collect data (information about the projects) ...
0
votes
0answers
12 views

What should be the fitness function while using Particle Swarm optimisation

I am using Particle Swarm Optimisation for optimising the parameters of a Neural network (for multi-class classification problem). But what should be the fitness function for it ? I have tried ...
0
votes
0answers
53 views

R- Improving linear regression fit

I am trying to construct a predictive model in R. I am using the glm() in R to fit the model. I am getting a very high residual error after fitting the model. My target values are in the range of ...
0
votes
0answers
27 views

R- Improving linear regression fit [closed]

I am trying to construct a predictive model in R. I am using the glm() in R to fit the model. I am getting a very high residual error after fitting the model. My target values are in the range of ...