Questions tagged [cross-validation]

Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

Is there such a thing as a "Random Forest of Random Forests"? [duplicate]

This was a question I had and I have tried to find an answer for a while but without any conclusive answers. In short - can you create a random forest in which each "tree" in the "...
0 votes
0 answers
25 views

Cross Validation - leave one out or k-fold?

I have read the disadvantage/advantages of using leave-one-out cross validation, but I have not been able to find anything telling how to choose if to use it. I have got a very small dataset, made out ...
0 votes
1 answer
604 views

Why is my training curve and validation curve on the same level and how to find a proper architecture for an ANN?

I have build a neural network model. All in all it doesn't work properly, but since it is part of my Master Thesis I've to evaluate it and find some explanations. The learning curve looks like this: ...
1 vote
1 answer
73 views

How do I access the p-values of individual predictors using caret::train? [closed]

I can't figure out how to access the p-values for my predictor variables after using k-fold cross-validation with caret::train. Does anyone know? Below is an example using the Boston data set that ...
0 votes
1 answer
173 views

prophet cross validation

I'm confused by the 'cross_validation' of prophet. In the following cross validation process, were parameters learned and saved to the model? is this cross validation used to train model or just to ...
0 votes
0 answers
16 views

(XGBOOST) 5-fold cross validation test aucpr is always lower than train aucpr, is that overfitting?

I am using XGBOOST to construct a prediction model, but no matter what I do (including set gamma, subsample, eta), I will get results similar to the following picture. I did see all train and test ...
1 vote
0 answers
19 views

Why isn't RandomSearchCV returning the optimum parameters for the XGBoost Model, and how can I avoid Overfitting?

I have a dataset for energy consumer customers and binary target variables with which I want to predict the churn for the customers. Counts of target values Not Churn 0: 14153 Churn 1: 1520 I have ...
0 votes
0 answers
33 views

Quantifying the difference between micro and macro averaging in cross-validation

I am studying the effect of micro vs macro averaging the results of a cross-validation run. Using true positive rate as a running example, given a cross-validation run of $K$ folds, the final $TPR$ ...
1 vote
2 answers
985 views

Decision tree: where and how to split an attribute on numerical dataset?

I am new to data mining and am manually implementing decision tree classification on a dataset with all continues values. A very small sample dataset of 4 attributes (columns) would be like this: <...
1 vote
0 answers
18 views

Should I use cross validation for simple linear regression model?

I have a data set with 181 observations. I have 9 predictors and I have developed different regression models using ordinary linear regression and stepwise linear regression. Now I'm trying to decide ...
0 votes
0 answers
10 views

Displaying Cross validated R square in dredge output [closed]

I am comparing different linear models from dredge with the following commands : globalmodel<-lm(response ~ . , data=dataset) ...
1 vote
0 answers
6 views

CT-Scan Classification Overfitting Problem

I am currently trying to train pretrained convolutional neural networks trained on the imagenet dataset to be able to classify ct-scans into two classes. Viral Pneumonia and Normal. I am using K-fold ...
1 vote
1 answer
29 views

What are "volatile" learning curves indicative of? [duplicate]

I have a dataset set with ~40 features onto which I'm applying a multi-layer perceptron for regression purposes. The train, validation, and test sets are made up of 3M, 800K, and 800K examples each, ...
0 votes
0 answers
8 views

PRESS statistic and k-fold cross-validation

How is the PRESS statistic calculated in a k-fold cross-validation? I know how it is done in the leave-one-out scenario. Is it still summed over all training samples, just that there are now k-many ...
0 votes
1 answer
18 views

Cross Validation on whole data for model comparision

I have good an imbalanced very small dataset (58 instances) and whould like to create a multiclass classification model. I'd like to use cross-validation in order to make the most out of the data I ...
1 vote
1 answer
19 views

Splitting medical dataset by patient

I am currently trying to train a CNN model to classify CT-scans. I split the dataset using K-fold cross-validation and since the dataset I am using contains multiple slices per patient, I split the ...
1 vote
1 answer
1k views

Engineering lag features for the test set in time-series machine learning

I am trying to do time series forecasting through machine learning. I want to engineer lag features, but was wondering what would be the best way to go about generating these features for the test set ...
0 votes
1 answer
693 views

Can I use Rolling and Window cross-validation techniques to check feature importance and forecasting error stability?

I want to propose a simple experiment. Let's say I have a time series data, where I first split data into train and test sets and then work with my training set to pick the best model to do forecast ...
1 vote
1 answer
2k views

Overfitting in Cross Validation for Hyperparameter Selection

I am using 3-fold cross validation for hyperparameter selection of my XGBOOST model. To be specific, I use xgboost.cv for cross validation instead of sklearn. I use random search for hyperparameter ...
1 vote
1 answer
372 views

k-fold cross validation with multiple classes

I'm working on an image retrieval system (not classification). I have 5,000 images as the data set. 500 images of this dataset are the query images used for retrieval evaluation. these 500 images ...
94 votes
6 answers
39k views

On the importance of the i.i.d. assumption in statistical learning

In statistical learning, implicitly or explicitly, one always assumes that the training set $\mathcal{D} = \{ \bf {X}, \bf{y} \}$ is composed of $N$ input/response tuples $({\bf{X}}_i,y_i)$ that are ...
1 vote
0 answers
44 views

Out-of-sample prediction error using nested cross-validation

I am applying Lasso regression and the R function glmnet::cv.glmnet() to obtain a prediction model based on 90% of the data. I have set aside 10% as a hold-out set and obtain predicted probabilities ...
1 vote
1 answer
2k views

Partitioning with cross validation?

I am new to data analytics having only started exploring the field this week. I have downloaded KNIME and am working with a single dataset to try out different classification algorithms. I am ...
1 vote
2 answers
58 views

Cross Validation for Time Series Classification (Not Forecasting!)

Is it possible to use regular k-fold cross validation where the folds contain entire time series in time series classification? I'm asking because most sources discussing cross validation with time ...
5 votes
3 answers
458 views

Model Selection: AIC/BIC and Cross-Validation gives different conclusion

In general, there vast number of ways to select model/feature in machine learning or statistics. For example, empirical method like Cross-Validation, Bootstrap methods or in sample penalty such as AIC,...
1 vote
1 answer
566 views

Saving each step of Backward selection in R

I'm trying to recreate Leo Breiman's work http://www.stat.washington.edu/courses/stat527/s13/readings/BreimanSpector_1992.pdf and I'm experiencing some major difficulties in R. I've made it that far ...
0 votes
0 answers
10 views

Approximately Unbiased P-value vs Bootstrap Probability: which one should i choose?

Some references first: How is approximately unbiased bootstrap better than a regular bootstrap with regards to hierarchical clustering? Suzuki et al. 2004 https://www.researchgate.net/publication/...
1 vote
1 answer
2k views

How to report cross validation?

When we use cross validation (for example 10 fold) I can obtain 10 sensitivity, specificity and accuracy measures and also 10 ROC curves with the associated AUC. What should I do for reporting my ...
18 votes
3 answers
8k views

High variance of leave-one-out cross-validation

I read over and over that the "Leave-one-out" cross-validation has high variance due to the large overlap of the training folds. However I do not understand why that is: Shouldn't the performance of ...
3 votes
2 answers
206 views

Does it make sense to use data augmentation on the Validation set? (note, this is not the same as asking to augment the test set)

Curious, do people use data augmentation on the validation set? I am aware there is a debate for the test set -- but the validation set is usually a split form the train set, so wouldn't it make sense ...
1 vote
1 answer
31 views

Is cross-validation with no data leakage sufficient to replace train-test split?

I would like to seek expert advice on the topic above. I was taught to follow this workflow: Split dataset into training and testing Use training dataset to develop model Set hyperparameter in model ...
0 votes
1 answer
422 views

Interview question: overfitting, MSE over validation and test sets

Q: Consider two linear models, and a dataset split into a training set, a validation set, and a test set in a 70-15-15% proportion. The two models produce a comparable and low mean squared error (MSE) ...
1 vote
1 answer
251 views

How to choose the number of steps ahead when comparing time series CV to the AIC or the BIC?

I would like to empirically evaluate the performance of the AIC, BIC and Cross Validation as model selection criteria for time series forecasting, i.e. which one of these criteria leads to the best ...
0 votes
1 answer
26 views

Perform Rose Method, then Logistic Regression and do k -fold cross validation

I have unbalanced data so I want to oversample obs from the minority class and then apply Logistic regression to the training set. After that, I would like to perform cross-validation. My question is: ...
0 votes
1 answer
14 views

use Cross-validation to find the better model

...
2 votes
2 answers
436 views

Selecting multiple hyper-parameters via successive nested cross-validation

Selecting multiple hyper-parameters via successive nested cross-validation I am currently working in a classification task on motion data. Each sample to classify is represented by a set of features ...
3 votes
1 answer
168 views

Validating uncertainty quantification

Regression performance is often evaluated by means of cross-validation. However, classical cross-validation only regards the mean of the identified parameters. How can one quantify the quality of the ...
3 votes
1 answer
43 views

How to handle outcome variables during imputation of missing data in model building and assessment process?

Der community I have a question about the appropriate handling of the imputation of missing data to get an unbiased estimate of prediction accuracy during model building and assessment. While ...
8 votes
3 answers
11k views

Why using cross validation is not a good option for Lasso regression?

I watched the lecture about Lasso and at the end of this module (between 00:40 and 01:25) she explains how to choose the regularization parameter Lambda and it sounds like using (K-fold)Cross ...
272 votes
13 answers
210k views

Is there any reason to prefer the AIC or BIC over the other?

The AIC and BIC are both methods of assessing model fit penalized for the number of estimated parameters. As I understand it, BIC penalizes models more for free parameters than does AIC. Beyond a ...
0 votes
0 answers
16 views

Implementing a bandwidth selection algorithm with cross validation

I'm trying to implement a bandwidth selection algorithm based on cross-validation for kernel density estimation. I learned from the article that cross-validation needs to minimize Mean Integrated ...
37 votes
3 answers
61k views

10-fold Cross-validation vs leave-one-out cross-validation

I'm doing nested cross-validation. I have read that leave-one-out cross-validation can be biased (don't remember why). Is it better to use 10-fold cross-validation or leave-one-out cross-validation ...
1 vote
0 answers
9 views

What error estimates is used for 5x2 cv paired t-test?

I am performing a 5x2 cv paired t-test. I have read most of the paper "Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms" by Remco R. Bouckaert and Eibe ...
8 votes
1 answer
417 views

Predicted R squared

When calculating the predicted $R^2$ value for a linear model using the equation $R^2 = 1 - \frac{PRESS}{TSS}$ should the currently left out sample also be excluded when working out the mean value ...
1 vote
2 answers
588 views

Validation set in presence of cross-validation

I am new to machine learning and want to ask regarding a confusion I have. I have a data set which is labeled and I want to do supervised learning. My question is related to cross-validation and ...
128 votes
10 answers
70k views

Bias and variance in leave-one-out vs K-fold cross validation

How do different cross-validation methods compare in terms of model variance and bias? My question is partly motivated by this thread: Optimal number of folds in $K$-fold cross-validation: is leave-...
2 votes
1 answer
49 views

Is "sensitivity at fixed specificity" a valid metric for comparing different classifiers?

For a given dataset, a common way to compare 2 classifiers is to compare their average validation accuracies using cross-validation. Is it valid to replace the accuracy with other classification ...
0 votes
0 answers
16 views

How to compare 2 classifiers using a classification metric?

Let's assume we have 2 binary classifiers (A and B) and some labeled dataset, and we want to compare A and B. Let's assume we use the ROC AUC as the metric (although it could be the accuracy or ...
13 votes
2 answers
5k views

What is Combinatorial Purged Cross-Validation for time series data?

I'm trying to understand the "Combinatorial Purged Cross-Validation" technique for time series data described in Marcos Lopez de Prado's "Advances in Financial Machine Learning" book (p. 163). The ...
5 votes
2 answers
2k views

SMOTE data balance - before or during Cross-Validation

I'm using Random Forest in the CARET package to tag a binary outcome with 1/10 ratio, thus I need to balance the dataset. I know two ways: Use SMOTE as a stand-alone function and then pass it to the ...

1
2 3 4 5
65