Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
1answer
16 views

Why does the CV estimate of Test Error Underestimate Actual Test Error?

It is my understanding that the k-fold cross-validation estimate of test error usually underestimates actual test error. I’m confused why this is the case. I see why the training error is usually ...
0
votes
1answer
20 views

Bootstrapping test set?

Let's say I have a classification problem with a small and fixed test set. If I train a classifier and report the accuracy on this test set, I know that this estimate has a high variance. Does it make ...
2
votes
1answer
42 views

What does the cost (C) parameter mean in SVM?

I am trying to fit a SVM to my data. My dataset contains 3 classes and I am performing 10 fold cross validation (in LibSVM): ...
0
votes
1answer
18 views

What happens inbetween switching test and training sets in k-fold cross validation

The general idea of k-fold cross validation is to partition your test data from your training data, then within your training data you make another partition where all but one of those partitions are ...
0
votes
0answers
24 views

cross validation in weka and model accuracy

I am developing a sentence classifier using Weka. The number of class is 48. I trained the classifier using AdaBoost.M1 where weak classifier is multi-layered perceptron. And each class label is just ...
0
votes
0answers
4 views

Validation for Household Relocation Model without validation set

We are working on several household relocation models based on census data. This data contained the location of many households over the course of three years, along with several other variables. The ...
1
vote
0answers
20 views

validating a gaussian process fitted to data

I am relatively new to applying Gaussian processes to data. I come from a math background but the most popular literature on it seems to be from a machine learning perspective and not from a ...
0
votes
1answer
15 views

What metric to use as the cross validation error in the training set for a binary classification problem?

When I am running cross validation on the training set for a binary classification problem, what metric should I use if I am only interested in obtaining the largest AUC (area under receiver operating ...
4
votes
0answers
28 views

Variability in LASSO models for predicting rare events

I want to build a model for predicting a rare (ca 10%) event in my dataset of around 300 samples and 15 candidate predictors (of these, I know that five, when looked at individually in the whole ...
0
votes
0answers
6 views

How do I proceed if crossvalidation in Structural Equiation Modeling results in model misfit?

In context of a university course (Psychology) I am supposed to run a SEM Analysis, improve model fit and to cross validate the new model. So I split my data set (N=1000) in two halves (70% versus 30%)...
0
votes
0answers
15 views

Are my regression lines from the same population? [duplicate]

I have two sets of data looking at weight change over time. Subjects within each data sets have weights at multiple time points. The first is a small set of data I used to determine an equation to ...
2
votes
2answers
34 views

How to apply classifiers from k-folding to data not used in the k-folding?

When I am using k-folding to split my labelled data (labelled as signal or background) and train k classifiers on it, I believe I am not allowed to assume that the distributions of the classifier ...
0
votes
0answers
7 views

Comparing the performance of two classifiers for statistical significant differences

I am aware that there are similar questions asked on Cross Validated, however, this differs slightly. Say that I have one dataset, which I want to use to train two classifiers A and B (supervised ...
3
votes
2answers
64 views

Confusion about when to use least-squares regression analysis

I am going through an article titled On the misuse of regression in earth science. On page 65, the author say as follows about the least-squares method. It is usual to require that the ...
0
votes
0answers
7 views

what should be validation parameter for Logistic Regression(LR) in online learning plus rare event scenario?

We have been following below paper to predict CTR( Click probability) of different ad items. This will be used to serve different ads based on probability values. http://olivier.chapelle.cc/pub/...
1
vote
0answers
9 views

MARS with cross validation, do I need testing

If you build a MARS model with cross validation (say 5-fold), do you still need to split the data into training and test and test your model? I thought you are training on four-folds and testing on ...
24
votes
1answer
786 views

Cross-validation misuse (reporting performance for the best hyperparameter value)

Recently I have come across a paper that proposes using a k-NN classifier on an specific dataset. The authors used all the data samples available to perform k-fold cross validation for different k ...
0
votes
0answers
10 views

What an error rate in regression means ? [closed]

As I cannot say that the error in regression means precision as in classification. What an error rate after the system has been cross-validate in regression means ?
0
votes
1answer
32 views

cross validation for discrete time survival analysis [closed]

I would be very grateful if you could let me know how to do cross validation when estimating a discrete time survival analysis in R. ID TIME EVENT x1 x2 x3 x4 x5 1 1 0 1.281 ...
0
votes
0answers
20 views

Cross-validating an ordinal logistic regression in R (using rpy2) [migrated]

I'm trying to create a predictive model in Python, comparing several different regression models through cross-validation. In order to fit an ordinal logistic model (...
0
votes
0answers
17 views

ACF does not support ADF conclusion

I have 2 stocks price series : Below are the two price series and their ACF respectively. ...
0
votes
0answers
18 views

Doing cross-validation when diagnosing a classifier through learning curves

I have a theoretical question on the correct way to make learning curves to diagnose a classifier. To see a generic example of these curves one can refer to this (min 34 onward) lecture by Andrew Ng ...
1
vote
0answers
11 views

Is having more features definitely equal to having a higher chance of overfitting?

I am doing a EEG data classification problem. Currently I am using the ANOVA test to help me select K best input features (with K a parameter to tune) and feeding the selected features into a logistic ...
1
vote
1answer
35 views

Nested cross-validation and feature selection: when to perform the feature selection?

I am trying to predict a behavioral variable using neuroimaging data using supporting vector regression. Since there are ~ 400.000 voxels (=features) in an image and I have a limited sample size I ...
0
votes
0answers
9 views

Parameter tuning/model selection using cross validation

I have been trying to get into more details of resampling methods and implemented them on a small data set of 1000 rows.the data was split into 800 training set and 200 validation set. I used K fold, ...
0
votes
0answers
67 views

Different results with “xgboost” vs. “caret” in R

I am new to R programming language and I need to run "xgboost" for some experiments. The problem is that I need to cross-validate the model and get the accuracy and I found two ways that give me ...
0
votes
0answers
19 views

Problem OOB in random forest

I have a 6 levels of one group and i have to do a random forest classification. My problem is that OOB in test set is too low and cv give me a almost 0 error so i don't understand if this can be a big ...
3
votes
0answers
45 views

Cross-validation scheme used in the Introduction to Statistical Learning, Chapter 6, Lab 3

I've been really enjoying the Introduction to Statistical Learning textbook so far, and I'm currently working my way through chapter 6. I realize that I am very confused by the process used in lab 3 ...
4
votes
0answers
19 views

Data Transformation Question - Multiplying data proportional to demographics

I have a bunch of data that is tied to demographic variables (Age, Sex, Income, Education, etc.). However, the data is sent by one person in a household for the entire house. It's numerical data and I ...
1
vote
2answers
19 views

How does k-fold cross validation fit in the context of training/validation/testing sets?

My main question is with regards trying to understand how k-fold cross-validation fits in the context of having training/validation/testing sets (if it fits at all in such context). Usually, people ...
1
vote
1answer
29 views

When do I cross-validate?

I'm writing code to perform classification on novel data sets in our lab, and I'm confusing myself as to when I should be performing cross-validation. From this question I understand that I should ...
1
vote
2answers
67 views

Am I performing feature selection correctly?

I'd like to design a feature extraction, selection, and classification scheme to use on novel data sets. For each row in a table I calculate 10 features. I then select which features are relevant (...
0
votes
0answers
15 views

Difference between Model selection/Parameter tuning using cross validation

I have been trying to resolve my doubts around the purpose and usage procedure of the cross validation.I have gone through some thoroughly written responses on the cross validated and books like ...
0
votes
0answers
15 views

collapse a cross-validation matrix to a single value

I want to compare the sensitivity of a clustering solution to the inclusion of different subjects in the dataset using leave-one-out cross-validation. I use Variation of Information (VI) to compare ...
-3
votes
0answers
42 views

How to select the final/best model and the important predictors from LASSO outputs in R? [duplicate]

I am trying to learn about regularization techniques The R commands generates the following plots: I would like to know how to select the final/best model and the important predictors included ...
0
votes
0answers
10 views

Specific value of tuning parameter “s” (the path) for finding lasso coefficients [migrated]

I would like to assign automatically the best s value to predict.lars which is given by the CV process (...
1
vote
2answers
72 views

Time series cross validation by reversing the series

I am trying to forecast revenue of a company, using neural networks. The response is a time series of monthly revenues from 11/2008 to 05/2016, and there are about 45 predictors (including lagged ...
1
vote
0answers
19 views

On learning curve - how to split datasets

The other questions doesn't address this point in a simple manner and I found some differences in the approaches from different tutorials. Assume a dataset with N samples, what is the strategy to plot ...
-1
votes
0answers
21 views

how to perform on a validation set?

Hello suppose i am creating a feed forward neural net and also training it with backpropagation. I am having three different sets training testing and validation.How do I apply training on the ...
2
votes
1answer
30 views

Since we use Bootsrap to approximate the SE, can we use Bootstrap to find prediction errors?

Instead of using Cross-validation, or K-fold Cross-validation, can we use Bootstrap to generate random samples and use one of them as test set, and others as training set?
1
vote
0answers
22 views

Amount of error with CV.GLMNET

I'm predicting a continuous variable whose values are in $[0,1000]$. When I plot the cross validation of cv.glmnet, I obtain this : I don't understand why the CV ...
0
votes
0answers
43 views

Cross validation of regression results: Interpretation of significant differences

I estimated a simple regression model twice in a randomly split sample of about $n = 800$ cases (calibration sample: $n_A = 400$; validation sample: $n_B = 400$). I then tested whether the regression ...
1
vote
1answer
30 views

Comparing performance of two k-fold cross-validated models

I'm developing two k-fold cross-validated models, based on two different data sets, but using the same variables. I plan to then apply both models to each data set and calculate a few model ...
0
votes
3answers
42 views

what is the need of k-fold cross validation even though it has nothing to do with how generalized our model is?

I recently studied K-fold cross validation, but I didn't get why this result even matter, since it does not help to improve the generalization ability of our model and the model will work the same on ...
0
votes
0answers
23 views

Random forest classfication: the Monte Carlo approach to train/test split

I'm trying to build a classification random forest. The problem is, that the output (dependend) variable is skewed (the more important class, let's call it < true >, happens ~20% of time) and the ...
0
votes
0answers
13 views

Confidence interval via cross-validation

I am running an experiment with 10-fold cross-validation. For each fold, I calculate my score (Kendall's tau) and its confidence interval (via bootstrap resampling). Then, I average the score values ...
0
votes
0answers
17 views

statistics on Cross Validated posts [migrated]

since CV is about statistics and machine learning, and this forum has been gaining popularity, can the organizers please share some statistics about CV. I'm sure a lot of folks would be interested in ...
0
votes
0answers
13 views

How can I compare model fit of LASSO versus OLS?

I want to compare the accuracy of a linear model which uses three predictors and which I estimate with OLS with a model which uses alternative predictors and which I estimate with LASSO. Number of ...
0
votes
1answer
22 views

Combining two or more (SVC) models in Python/scikit.learn

I have some data which I use SVC models with 10 fold cross validation and a parameter grid search on (scikit.learn). I observed that the predictions of some folds have low accuracy, whereas remained ...
0
votes
0answers
25 views

How to compare experimental results of multiple models using significance tests?

I am using 5-fold cross validation to evaluate the errors of 10 models. Therefore, there are 50 results (each model has 5 error values generated from 5 random evaluation partitions). My question is: ...