Questions tagged [cross-validation]
Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.
3,249
questions
0
votes
0
answers
5
views
sklearn.model_selection.cross_validate function random split? [closed]
For the sklearn.model_selection.cross_validate, when I write a loop to pass on difference model to the cross validate, like
...
- 23
0
votes
0
answers
17
views
Conceptual question: Evaluate model performance
What I think I understood about evaluating model performance:
When a model is fit on some data, the performance of this model on the same data is not a good indicator of how the model would fit on ...
- 525
1
vote
0
answers
10
views
Reliably evaluate model performance with very few positive samples
I do a binary classification in the domain of predictive maintenance.
Setup
My dataset is highly imbalanced with only 17 samples of the positive class, but an nearly indefinite amount of negative ...
- 525
2
votes
1
answer
37
views
Does Cross Validation Reduces Underfitting
I know that Cross-Validation helps reduce Overfitting in the data by its property of testing and training on all the instances. But is there any chance that it also reduces Underfitting? I mean What I ...
- 21
0
votes
0
answers
11
views
Is there such a thing as a "Random Forest of Random Forests"? [duplicate]
This was a question I had and I have tried to find an answer for a while but without any conclusive answers. In short - can you create a random forest in which each "tree" in the "...
- 5,874
0
votes
0
answers
26
views
Cross Validation - leave one out or k-fold?
I have read the disadvantage/advantages of using leave-one-out cross validation, but I have not been able to find anything telling how to choose if to use it.
I have got a very small dataset, made out ...
- 109
0
votes
0
answers
16
views
(XGBOOST) 5-fold cross validation test aucpr is always lower than train aucpr, is that overfitting?
I am using XGBOOST to construct a prediction model, but no matter what I do (including set gamma, subsample, eta), I will get results similar to the following picture.
I did see all train and test ...
- 101
1
vote
0
answers
19
views
Should I use cross validation for simple linear regression model?
I have a data set with 181 observations. I have 9 predictors and I have developed different regression models using ordinary linear regression and stepwise linear regression. Now I'm trying to decide ...
- 11
0
votes
0
answers
10
views
Displaying Cross validated R square in dredge output [closed]
I am comparing different linear models from dredge with the following commands :
globalmodel<-lm(response ~ . , data=dataset)
...
- 1
1
vote
0
answers
6
views
CT-Scan Classification Overfitting Problem
I am currently trying to train pretrained convolutional neural networks trained on the imagenet dataset to be able to classify ct-scans into two classes. Viral Pneumonia and Normal.
I am using K-fold ...
- 21
1
vote
1
answer
29
views
What are "volatile" learning curves indicative of? [duplicate]
I have a dataset set with ~40 features onto which I'm applying a multi-layer perceptron for regression purposes. The train, validation, and test sets are made up of 3M, 800K, and 800K examples each, ...
- 565
0
votes
0
answers
8
views
PRESS statistic and k-fold cross-validation
How is the PRESS statistic calculated in a k-fold cross-validation? I know how it is done in the leave-one-out scenario. Is it still summed over all training samples, just that there are now k-many ...
- 1
0
votes
1
answer
18
views
Cross Validation on whole data for model comparision
I have good an imbalanced very small dataset (58 instances) and whould like to create a multiclass classification model. I'd like to use cross-validation in order to make the most out of the data I ...
- 109
1
vote
1
answer
21
views
Splitting medical dataset by patient
I am currently trying to train a CNN model to classify CT-scans.
I split the dataset using K-fold cross-validation and since the dataset I am using contains multiple slices per patient, I split the ...
- 21
1
vote
0
answers
19
views
Why isn't RandomSearchCV returning the optimum parameters for the XGBoost Model, and how can I avoid Overfitting?
I have a dataset for energy consumer customers and binary target variables with which I want to predict the churn for the customers.
Counts of target values
Not Churn 0: 14153
Churn 1: 1520
I have ...
- 31
1
vote
0
answers
45
views
Out-of-sample prediction error using nested cross-validation
I am applying Lasso regression and the R function glmnet::cv.glmnet() to obtain a prediction model based on 90% of the data. I have set aside 10% as a hold-out set and obtain predicted probabilities ...
- 11
5
votes
3
answers
460
views
Model Selection: AIC/BIC and Cross-Validation gives different conclusion
In general, there vast number of ways to select model/feature in machine learning or statistics. For example, empirical method like Cross-Validation, Bootstrap methods or in sample penalty such as AIC,...
- 404
0
votes
0
answers
10
views
Approximately Unbiased P-value vs Bootstrap Probability: which one should i choose?
Some references first:
How is approximately unbiased bootstrap better than a regular bootstrap with regards to hierarchical clustering?
Suzuki et al. 2004 https://www.researchgate.net/publication/...
- 1
1
vote
1
answer
31
views
Is cross-validation with no data leakage sufficient to replace train-test split?
I would like to seek expert advice on the topic above.
I was taught to follow this workflow:
Split dataset into training and testing
Use training dataset to develop model
Set hyperparameter in model
...
- 13
0
votes
1
answer
26
views
Perform Rose Method, then Logistic Regression and do k -fold cross validation
I have unbalanced data so I want to oversample obs from the minority class and then apply Logistic regression to the training set. After that, I would like to perform cross-validation. My question is: ...
- 3
0
votes
1
answer
14
views
0
votes
0
answers
16
views
Implementing a bandwidth selection algorithm with cross validation
I'm trying to implement a bandwidth selection algorithm based on cross-validation for kernel density estimation. I learned from the article that cross-validation needs to minimize Mean Integrated ...
- 527
1
vote
0
answers
9
views
What error estimates is used for 5x2 cv paired t-test?
I am performing a 5x2 cv paired t-test. I have read most of the paper "Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms" by Remco R. Bouckaert and Eibe ...
- 11
3
votes
1
answer
43
views
How to handle outcome variables during imputation of missing data in model building and assessment process?
Der community
I have a question about the appropriate handling of the imputation of missing data to get an unbiased estimate of prediction accuracy during model building and assessment. While ...
- 31
2
votes
1
answer
50
views
Is "sensitivity at fixed specificity" a valid metric for comparing different classifiers?
For a given dataset, a common way to compare 2 classifiers is to compare their average validation accuracies using cross-validation.
Is it valid to replace the accuracy with other classification ...
- 935
0
votes
0
answers
16
views
How to compare 2 classifiers using a classification metric?
Let's assume we have 2 binary classifiers (A and B) and some labeled dataset, and we want to compare A and B. Let's assume we use the ROC AUC as the metric (although it could be the accuracy or ...
- 935
0
votes
2
answers
36
views
KFold Cross Validation with Train/Test/Validation-Set
I want to do KFold Cross Validation on a specific model and I am wondering what data to use.
In my project I have got a Train, Test and Validation set (this was already provided).
Now I want to to ...
1
vote
1
answer
23
views
Validation in deep learning networks
https://es.mathworks.com/help/deeplearning/ref/trainnetwork.html?s_tid=doc_ta#bu6sn4c-traininfo
In the above link, there is an example for train network with augmented images. The number of iterations ...
- 47
0
votes
0
answers
33
views
Quantifying the difference between micro and macro averaging in cross-validation
I am studying the effect of micro vs macro averaging the results of a cross-validation run. Using true positive rate as a running example, given a cross-validation run of $K$ folds, the final $TPR$ ...
- 222
1
vote
0
answers
20
views
Cross-validation on the HP filter for penalty selection
I'm looking for a bit more clarification on the selection of the $\lambda$ value for the Hodrick-Prescott Filter, aka HP filter. Which stems from a time-series in the form of
$$y_t=g_t+\epsilon_t$$
...
- 189
2
votes
1
answer
33
views
Using cross validation to actually fit the final model [closed]
I usually use cross validation only for the tuning part and once I have my hyperparameters, I fit the final model using the actual model with these hyperparameters like this:
...
- 45
1
vote
2
answers
61
views
Cross Validation for Time Series Classification (Not Forecasting!)
Is it possible to use regular k-fold cross validation where the folds contain entire time series in time series classification? I'm asking because most sources discussing cross validation with time ...
0
votes
0
answers
17
views
Can I perform hyperparameter tuning inside K-fold CV?
I am performing leave one subject out cross-validation, using one subject as the independent final test set to get the performance of my model.
Can I perform hyperparameter tuning inside each of my K-...
0
votes
1
answer
71
views
What is the uncertainty of Leave-one-out-cross-validation method?
I have used the LOOCV to validate my model. As we know, the LOOCV method is a special case of cross-validation where the number of folds equals the number of instances in the data set. Thus, the ...
- 1
0
votes
0
answers
22
views
Decision tree feature importances on a test set
Many tree based models come with a built in feature importance method usually based on impurity decrease (ex. sklearn RandomForest). Would it be possible to calculate feature importances on a test set ...
- 1
1
vote
1
answer
34
views
Difference between cross validation vs model accuracy measures
I have a time series ARIMA model and I want to validate the accuracy my prediction. But I dont understand the difference of using cross validation vs model accuracy measures such as MAPE, MAE, MSE and ...
- 11
2
votes
0
answers
41
views
What is the bias adjustment needed on k-folds CV? (made by cv.glm on R's boot package)
I'm trying to find the equation defined on the documentation of the cv.glm function in the boot package: "When $K$ is less ...
- 21
2
votes
1
answer
32
views
Does using grid search for hyperparemeters make test set redundant?
The purpose of train, validate and test data splits addresses the issue of data leakage when tuning for the model's hyperparameters. Does Grid Search then eliminates the need for test set? Because ...
- 21
3
votes
0
answers
21
views
How to manage out of sample data in the long run?
For example, you are interested in testing an investment strategy and there is data from 1950 to 2022. So you split it into a train and test set, say 1950-2000 and 2000-2022. Then you build your model ...
- 31
0
votes
1
answer
22
views
References for cross-validation implementations in Pytorch
I'm interested in good references on cross-validation implementations for feed-forward neural networks in pytorch from scratch.
Thanks in advance.
- 187
0
votes
0
answers
10
views
Are stacked meta models for time series forecasting still considered forecasters?
I am working on building a meta model that is based on the principle of stacked generalisation (1). In a nutshell, this method works by using building a meta model based on the predictions of various ...
- 63
1
vote
1
answer
62
views
Significance test for comparing different 10-fold cross-validated Machine Learning Regressions
Is there a recommended significance test for comparing different 10-fold cross validated regressions?
For instance, I want to compare the performance of LASSO against Random Forest for my dataset.
...
0
votes
0
answers
20
views
Difference K-fold versus Blocked Cross-Validation?
In the paper "Evaluating time series forecasting models: an empirical study
on performance estimation methods" by Cerqueira et al (2020), they mention k-fold cross-validation. Which they ...
- 180
0
votes
0
answers
17
views
learning curve for RF and LR comparison and selection
I am plotting learning curve to check how the model perform on training data set and the effect of the training size on the accuracy. I am using two models, random forest and logistic regression.
From ...
4
votes
2
answers
297
views
Stratification of the continuous y (target) variable in regression setting
Is it wise to stratify the continuous y (target) variable when you split your training and testing data from the total sample in regression setting? Here is the approach in python to do implement ...
- 153
1
vote
0
answers
18
views
Interpreting learning curves
There is really few examples online regarding interpreting learning curves and they are all of different type.It is quite confusing to me honestly.May I just ask:
How should we interpret them?What ...
- 11
0
votes
0
answers
13
views
After training a model, how does test set error inform decision making?
I split a data set into three subsets: training, validation, and test sets. I use my training data for fitting and validation to check for overfitting. I then have a final model that I then propose to ...
- 1,063
2
votes
1
answer
83
views
Should folds in k-fold CV actually be representative?
I have read somewhere that the k of the k-fold CV should be picked in such a manner as to have representative validation sets (folds). This seems to me to be contradictory since the leave-one-out CV ...
- 33
0
votes
1
answer
55
views
Forcing covariates to always be part of a Lasso model
I want to use a Lasso to predict outcomes for different policy scenarios. At the optimal degree of regularization obtained by cross-validation, one important variable in whose impact I'm interested in ...
- 1
2
votes
0
answers
16
views
Statistical test for temporal cross validation
I estimated the performance of my forecasting model and that of a baseline on 10 folds using temporal cross-validation. With which test do I assess if my model is significantly better than the ...
- 423