Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
15 views

Are my regression lines from the same population? [duplicate]

I have two sets of data looking at weight change over time. Subjects within each data sets have weights at multiple time points. The first is a small set of data I used to determine an equation to ...
0
votes
2answers
16 views

How to apply classifiers from k-folding to data not used in the k-folding?

When I am using k-folding to split my labelled data (labelled as signal or background) and train k classifiers on it, I believe I am not allowed to assume that the distributions of the classifier ...
0
votes
0answers
5 views

Comparing the performance of two classifiers for statistical significant differences

I am aware that there are similar questions asked on Cross Validated, however, this differs slightly. Say that I have one dataset, which I want to use to train two classifiers A and B (supervised ...
3
votes
2answers
62 views

Confusion about when to use least-squares regression analysis

I am going through an article titled On the misuse of regression in earth science. On page 65, the author say as follows about the least-squares method. It is usual to require that the ...
0
votes
0answers
7 views

what should be validation parameter for Logistic Regression(LR) in online learning plus rare event scenario?

We have been following below paper to predict CTR( Click probability) of different ad items. This will be used to serve different ads based on probability values. http://olivier.chapelle.cc/pub/...
1
vote
0answers
9 views

MARS with cross validation, do I need testing

If you build a MARS model with cross validation (say 5-fold), do you still need to split the data into training and test and test your model? I thought you are training on four-folds and testing on ...
23
votes
1answer
764 views

Cross-validation misuse (reporting performance for the best hyperparameter value)

Recently I have come across a paper that proposes using a k-NN classifier on an specific dataset. The authors used all the data samples available to perform k-fold cross validation for different k ...
0
votes
0answers
10 views

What an error rate in regression means ? [on hold]

As I cannot say that the error in regression means precision as in classification. What an error rate after the system has been cross-validate in regression means ?
0
votes
1answer
32 views

cross validation for discrete time survival analysis [closed]

I would be very grateful if you could let me know how to do cross validation when estimating a discrete time survival analysis in R. ID TIME EVENT x1 x2 x3 x4 x5 1 1 0 1.281 ...
0
votes
0answers
20 views

Cross-validating an ordinal logistic regression in R (using rpy2) [migrated]

I'm trying to create a predictive model in Python, comparing several different regression models through cross-validation. In order to fit an ordinal logistic model (...
0
votes
0answers
16 views

ACF does not support ADF conclusion

I have 2 stocks price series : Below are the two price series and their ACF respectively. ...
0
votes
0answers
17 views

Doing cross-validation when diagnosing a classifier through learning curves

I have a theoretical question on the correct way to make learning curves to diagnose a classifier. To see a generic example of these curves one can refer to this (min 34 onward) lecture by Andrew Ng ...
1
vote
0answers
11 views

Is having more features definitely equal to having a higher chance of overfitting?

I am doing a EEG data classification problem. Currently I am using the ANOVA test to help me select K best input features (with K a parameter to tune) and feeding the selected features into a logistic ...
1
vote
1answer
35 views

Nested cross-validation and feature selection: when to perform the feature selection?

I am trying to predict a behavioral variable using neuroimaging data using supporting vector regression. Since there are ~ 400.000 voxels (=features) in an image and I have a limited sample size I ...
0
votes
0answers
9 views

Parameter tuning/model selection using cross validation

I have been trying to get into more details of resampling methods and implemented them on a small data set of 1000 rows.the data was split into 800 training set and 200 validation set. I used K fold, ...
0
votes
0answers
67 views

Different results with “xgboost” vs. “caret” in R

I am new to R programming language and I need to run "xgboost" for some experiments. The problem is that I need to cross-validate the model and get the accuracy and I found two ways that give me ...
0
votes
0answers
19 views

Problem OOB in random forest

I have a 6 levels of one group and i have to do a random forest classification. My problem is that OOB in test set is too low and cv give me a almost 0 error so i don't understand if this can be a big ...
3
votes
0answers
44 views

Cross-validation scheme used in the Introduction to Statistical Learning, Chapter 6, Lab 3

I've been really enjoying the Introduction to Statistical Learning textbook so far, and I'm currently working my way through chapter 6. I realize that I am very confused by the process used in lab 3 ...
4
votes
0answers
19 views

Data Transformation Question - Multiplying data proportional to demographics

I have a bunch of data that is tied to demographic variables (Age, Sex, Income, Education, etc.). However, the data is sent by one person in a household for the entire house. It's numerical data and I ...
1
vote
2answers
19 views

How does k-fold cross validation fit in the context of training/validation/testing sets?

My main question is with regards trying to understand how k-fold cross-validation fits in the context of having training/validation/testing sets (if it fits at all in such context). Usually, people ...
1
vote
1answer
27 views

When do I cross-validate?

I'm writing code to perform classification on novel data sets in our lab, and I'm confusing myself as to when I should be performing cross-validation. From this question I understand that I should ...
1
vote
2answers
65 views

Am I performing feature selection correctly?

I'd like to design a feature extraction, selection, and classification scheme to use on novel data sets. For each row in a table I calculate 10 features. I then select which features are relevant (...
0
votes
0answers
15 views

Difference between Model selection/Parameter tuning using cross validation

I have been trying to resolve my doubts around the purpose and usage procedure of the cross validation.I have gone through some thoroughly written responses on the cross validated and books like ...
0
votes
0answers
15 views

collapse a cross-validation matrix to a single value

I want to compare the sensitivity of a clustering solution to the inclusion of different subjects in the dataset using leave-one-out cross-validation. I use Variation of Information (VI) to compare ...
-3
votes
0answers
40 views

How to select the final/best model and the important predictors from LASSO outputs in R? [duplicate]

I am trying to learn about regularization techniques The R commands generates the following plots: I would like to know how to select the final/best model and the important predictors included ...
0
votes
0answers
10 views

Specific value of tuning parameter “s” (the path) for finding lasso coefficients [migrated]

I would like to assign automatically the best s value to predict.lars which is given by the CV process (...
1
vote
2answers
71 views

Time series cross validation by reversing the series

I am trying to forecast revenue of a company, using neural networks. The response is a time series of monthly revenues from 11/2008 to 05/2016, and there are about 45 predictors (including lagged ...
1
vote
0answers
19 views

On learning curve - how to split datasets

The other questions doesn't address this point in a simple manner and I found some differences in the approaches from different tutorials. Assume a dataset with N samples, what is the strategy to plot ...
-1
votes
0answers
21 views

how to perform on a validation set?

Hello suppose i am creating a feed forward neural net and also training it with backpropagation. I am having three different sets training testing and validation.How do I apply training on the ...
2
votes
1answer
29 views

Since we use Bootsrap to approximate the SE, can we use Bootstrap to find prediction errors?

Instead of using Cross-validation, or K-fold Cross-validation, can we use Bootstrap to generate random samples and use one of them as test set, and others as training set?
1
vote
0answers
21 views

Amount of error with CV.GLMNET

I'm predicting a continuous variable whose values are in $[0,1000]$. When I plot the cross validation of cv.glmnet, I obtain this : I don't understand why the CV ...
0
votes
0answers
43 views

Cross validation of regression results: Interpretation of significant differences

I estimated a simple regression model twice in a randomly split sample of about $n = 800$ cases (calibration sample: $n_A = 400$; validation sample: $n_B = 400$). I then tested whether the regression ...
1
vote
1answer
30 views

Comparing performance of two k-fold cross-validated models

I'm developing two k-fold cross-validated models, based on two different data sets, but using the same variables. I plan to then apply both models to each data set and calculate a few model ...
0
votes
3answers
42 views

what is the need of k-fold cross validation even though it has nothing to do with how generalized our model is?

I recently studied K-fold cross validation, but I didn't get why this result even matter, since it does not help to improve the generalization ability of our model and the model will work the same on ...
0
votes
0answers
23 views

Random forest classfication: the Monte Carlo approach to train/test split

I'm trying to build a classification random forest. The problem is, that the output (dependend) variable is skewed (the more important class, let's call it < true >, happens ~20% of time) and the ...
0
votes
0answers
13 views

Confidence interval via cross-validation

I am running an experiment with 10-fold cross-validation. For each fold, I calculate my score (Kendall's tau) and its confidence interval (via bootstrap resampling). Then, I average the score values ...
0
votes
0answers
17 views

statistics on Cross Validated posts [migrated]

since CV is about statistics and machine learning, and this forum has been gaining popularity, can the organizers please share some statistics about CV. I'm sure a lot of folks would be interested in ...
0
votes
0answers
12 views

How can I compare model fit of LASSO versus OLS?

I want to compare the accuracy of a linear model which uses three predictors and which I estimate with OLS with a model which uses alternative predictors and which I estimate with LASSO. Number of ...
0
votes
1answer
21 views

Combining two or more (SVC) models in Python/scikit.learn

I have some data which I use SVC models with 10 fold cross validation and a parameter grid search on (scikit.learn). I observed that the predictions of some folds have low accuracy, whereas remained ...
0
votes
0answers
24 views

How to compare experimental results of multiple models using significance tests?

I am using 5-fold cross validation to evaluate the errors of 10 models. Therefore, there are 50 results (each model has 5 error values generated from 5 random evaluation partitions). My question is: ...
1
vote
1answer
28 views

Model nature in stepwise regression

Suppose that we perform forward stepwise regression and use cross-validation to choose the best model size. Using the full data set to choose the sequence of models is the WRONG way to do cross-...
0
votes
1answer
21 views

Running Time for CV.GLMNET [closed]

I am running a cv.glmnet model (Poisson Model for rates) in R: elasticnet_poisson_model <- cv.glmnet(x=design.x,y=y.,offset=log z.,family='poisson',alpha=elasticnet.alpha,standardize=FALSE,lambda=...
0
votes
0answers
15 views

Cross validation for obtaining maximum median SRC Vaue

My dataset is comprised of 5 types of image's sets along with their distortions scores (DMOS) .Let us call the types as Type1,...
1
vote
1answer
29 views

Score for classification of dataset composed by different class with class imbalance

I am searching for a classification score, preferably provided by Python scikit-learn, to evaluate classification in a cross-validation routine. This classification score must be suitable for: ...
1
vote
0answers
19 views

Time series cross-validation, calculate RMSE for different forecasting horizions

Following Rob J. Hyndman suggestions on how to do cross validation for time series ( http://robjhyndman.com/hyndsight/tscvexample/ ), I modified his original code to evaluate how RMSE (not MAE) ...
1
vote
1answer
38 views

Gold Standard data for training but not validation

My goal is to determine which among the three classification algorithms perform better { Logist Reg or Neural Network or SVM }. I have a training dataset and the ...
0
votes
1answer
29 views

R e1071 SVM always gives me (in average) below changes cross validation accuracy with random data

I am running e1071 linear SVM on my neuroimaging data. ( by function svm() ) When I was doing permutation tests, I found, in average, the cross validation (CV) accuracies with shuffle labels were ...
1
vote
0answers
23 views

Sample dependency in Neural Net Training cross-validation

I've created a Monte Carlo simulation that randomly divides my data into "test" and "training"-Samples and then trains a neural network. The ratio of 0 and 1 (19.62%) Category is stabilized on ...
3
votes
0answers
42 views

K-fold cross-validation for time series with dynamic target variable (Scikit)

I would like to do a K-fold cross-validation on time series data (market data) with a two class classification target. My test folds must be forward looking and of a fixed size ...
0
votes
1answer
35 views

How do we obtain a final/best model when using k-fold cross-validation?

My past assumption of cross-validation (in particular k-fold CV) was that in order to given same chance to each sample in our dataset to appear in training , we use k-fold CV. Under my assumption we ...