Questions tagged [cross-validation]
Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.
2,988
questions
0
votes
0answers
19 views
LOOCV AUC better than original model
I wonder if its possible that the AUC of my LOOCV is better than the AUC obtained from the original model? I am doing feature selection within the LOOCV as well. However, the LOOCV ROC curve look more ...
0
votes
0answers
7 views
LOOCV with groups of different sizes
I am doing a limma analysis of a data set comprising 4 groups with 50 samples in each. In total I am having 5 different comparisons: Group1 vs Group2; Group1 vs Group3 and so on... Limma gives me a ...
0
votes
0answers
5 views
How to combine the estimate and confidence intervals of the fitted parameters that you obtain in every run of cross validation?
I am fitting quite a complex model to experimental data points (lots of equations, but just two parameters).
The nonlinear least square regression tool (I use fitnlm in Matlab) outputs the estimates ...
0
votes
0answers
13 views
Leave-one-out cross-validation and stratified bootstrapping together
I had asked a related question related here
I was told that I am using a kind of bootstrapping. I didn't realise it then. Based on the responses, I tried to understand what exactly was going on.
I ...
0
votes
0answers
7 views
Is there a way to do cross-validation using caret in a linear mixed model? [closed]
I have a data set with 2 treatments and 1 random effect and I am not sure if I can get one model with all this data doing cross-validation.
Thank you,
0
votes
0answers
18 views
Cross-validation / Threshold moving when training is balanced but test is imbalanced?
I have a binary text classification problem where texts of class 0 account for ~95% of cases and class 1 for ~5%. I put some effort until having a decently sized, balanced manually labeled subset (7k) ...
0
votes
0answers
6 views
Does adding a model complexity penalty to the objective function allow you to skip cross-validation?
It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of ...
0
votes
0answers
11 views
Are hyperparameters chosen from cross-validation slightly biased towards greater regularization?
I intend to fit a single model to the entire dataset after selecting hyperparameters by k-fold cross-validation. So on each round of training, my model is fit to $\frac{k-1}{k}n$ of my dataset, and ...
0
votes
0answers
11 views
Is there a difference between these two validation measurements?
Say I have one large data set and 10 smaller data sets.
What is the difference between the following two validation approaches, if any:
Fit the model on the large data set once, and then test it on ...
1
vote
2answers
37 views
What is the point of doing cross validation like this?
I have some data and some models that I'd like to cross-validate.
Here is my approach.
Take my data, which has roughly 10.000 rows.
Generate 10 test sets by simulating, with replacement, 1.000 rows ...
1
vote
1answer
48 views
What is this cross-validation technique called?
Say I have a data set.
I split the data into 10 chunks.
I fit the model on the 1st chunk, and test it on the 9 other chunks.
The average result over the 9 chunks is then the average performance of the ...
5
votes
2answers
131 views
How do I cross validate when I don't have a test set?
Situation: I have two models, fitted on the same data.
Goal: I want to meaure the out-of-sample performance of the two models.
Problem: I don't have a test set, and the original data is too small to ...
0
votes
0answers
27 views
Is upsampling a tiny class before cross-validation valid?
I'm working with a dataset containing several classes. The largest class has over 500 samples, and the smallest classes have fewer than 10 samples. I know that you should perform upsampling inside the ...
2
votes
0answers
20 views
What could be the cause of too much fluctuations of validation loss and accuracy?
I am doing a binary-classification problem and I have 7600 images, 3800 for each class. I am wondering why the accuracy and loss plots look like these. Is this fine? or is this a symptom of some ...
0
votes
0answers
12 views
Machine Learning on MICE-imputed data
I'm working on a project with medical data where some of it is missing. We decided to impute the data using MICE and I found enough literature about how to choose $m$ (the number of imputations) and $...
0
votes
1answer
22 views
k-fold CV Bias-Variance Trade-Off
I am confused from a seemingly contradict explanation in the book ISL.
In page 183 (5.1.4 Bias-Variance Trade-Off for k-Fold Cross-Validation),
I found the sentence: advantage of k-fold CV is that it ...
2
votes
1answer
26 views
Confidence interval vs. cross validation to estimate performance of a machine learning model
When training and testing a machine learning model, if I split the dataset just once, I may end up with the "good" portion so I can have a good performance. I may also end up with the "...
0
votes
0answers
14 views
Cross-validation and subsequent hold-out testing makes sense?
I'm integrating a modelling approach into an auto-tuning framework. Basically what I'm trying to achieve is creating a model over a subset of observation values and use it to predict the remaining ...
0
votes
0answers
9 views
Do I understand the confusion matrix and validation curve results from a boosted regression tree model correctly?
I created a model using boosted regression tree classification and I want to know if I am interpreting the results of the confusion matrix and validation curve correctly. The dataset consisted of 100,...
0
votes
1answer
18 views
1
vote
1answer
44 views
Optimise Random Forest Model using GridSearchCV in Python
I am working on a classification problem where I am applying various machine learning models.
I have used DecisionTreeClassifier from Sklearn on my dataset using the following steps:
Calculated alpha ...
1
vote
2answers
38 views
Can test set performance be too poor?
I've trained a binary classifier (PCA-LDA) on some clinical data for which I have 18 patients, each with ~250 observations. I am performing leave-one-patient-out cross-validation, each left out ...
0
votes
1answer
36 views
Can I run glm after MI with Elastic-Net non-zeroed coefficients from 'miselect'?
I have data with n = 80 and 10 predictors, and ran MI using MICE, followed by Variable Selection for Multiply Imputed Data using ‘miselect’ and finally have 4 non-zeroed coefficients.
Since ...
1
vote
1answer
27 views
Selecting number of input variables in a PCA-LDA model
Let's say that you have a dataset with a huge number of variables, so that a linear discriminant analysis on the original data may not be a good idea.
If you first use PCA a dimensionality reduction ...
0
votes
0answers
12 views
joint two-stage estimation with ML
I'm working with data of around 15 variables and half a million observations. To avoid selection bias, I'm trying to incorporate a two-stage joint estimation. I've seen this performed as a censored ...
0
votes
0answers
27 views
Given a dataset of $l$ examples, what is the standard 10-fold cross validation procedure in order to build the best model?
Given a dataset of $l$ examples, how do I build the best model with the standard 10-fold cross validation procedure?
Is it correct to:
Split the dataset into training and test datasets ( 90% / 10% );
...
0
votes
0answers
15 views
How to interpret the output of cross-validation for SVR
I wrote this code to run a SVR with cross validation:
...
0
votes
0answers
28 views
Combinatorial symmetric CV vs Combinatorial purged CV
Reading "Advances in Financial Machine Learning", and the author proposes 2 methods of CV: "combinatorial symmetric cross validation" (11.6) and "combinatorial purged cross ...
0
votes
0answers
17 views
The correct way to analyse crossvalidation fold perfomances (when using correlation)
My scenario.
I have datasets containing between 5,000 an 400,000 predictors (i.e., columns) and between 3,000 and 14,000 cases (i.e., rows) without any strata/subgroups.
I perform a nested, n-times-...
0
votes
0answers
12 views
Is the test set in Combinatorial Purged Cross-Validation in-sample or out-of-sample?
I am trying to understand the Combinatorial Purged Cross-Validation (CPCV) method of Marcos Lopez de Prado's "Advances in Financial Machine Learning" book. There are a few things that I do ...
0
votes
0answers
18 views
Is validation performance sufficient for hyperparameters choice on a small dataset for images multi-classificaiton problem
Problem: multiclass (3-6 classes) images classification (DeepLearning). Dataset size <2000 samples. One class is rare <50 samples.
We've conducted several sets of stratified cross-validations ...
1
vote
1answer
21 views
Can nested cross validation be considered bootstrapping?
I am using nested cross validation with 5 inner and outer folds. Each of the folds are created using stratified shuffle splits from scikit-learn. Because I am using, can this be considered ...
5
votes
2answers
26 views
Internal validation steps
I am currently developing models for a prediction analysis. I have read through much of Harrell's Regression Modeling Strategies text and am confused on one point regarding internal validation.
...
2
votes
0answers
24 views
Using cross-validation for model selection and model comparison
Let us suppose that we have two classifiers: SVM and CART. For each one of them, a set of hyperparameters is considered (C=0.001,0.01,... for SVM cp=... for CART).
The question is, can I use k-fold ...
3
votes
2answers
23 views
Normalisation of data along with KFold Cross Validation
I am working on a project of disease prediction where I have to create survival prediction using the given data by applying machine learning classifiers.
Along with the survival prediction I will then ...
0
votes
1answer
17 views
Why the CV in smaller test set perform better than a bigger test set?
I have a question about cross-validation.
I use random forests (RF) for regression forecast. I use a daily data from day 1 to day 365. In this case, I have the dataset from the year 2011 to the year ...
1
vote
0answers
17 views
Cross validation and undersampling
I have a dataset with very skewed distribution (approx. 90 with class 0 and 10 with class 1).
I have considered to use undersampling to reduce size of the majority class.
I would like to know the ...
1
vote
1answer
16 views
Test size with small dataset with class imbalance
what would a good split train/test be, having a class imbalance problem in the dataset and a small number of observations (<5000 obs)? Would it make sense to consider k-fold cross validation (e.g. ...
0
votes
1answer
24 views
Predict the first observations of a time series when order of the model is higher
Suppose you have you have a time series with 365 observations, one for each day of the year, and you split the first 183 rows in training set and the latest 182 in test set.
Suppose you create an AR (...
0
votes
0answers
9 views
If a model performance is the exact same every run is it overfitting?
I benchmark scikit-learn models (random forest, gradient boosting, SVM, KNN) using 5-fold nested cross validation on a regression classification problem.
I've noticed if I re-run the script the model ...
0
votes
0answers
19 views
Low val_accuracy in first fold of k-fold cross validation at every try
I have some problems at implementing kfold for sentiment analysis,I have used kfold with for-loop technique but ı get smallest accuracy first fold whenever run my code .After first fold val_accuracy ...
0
votes
0answers
13 views
Why are random forest predictions over a smaller range than the true target variable? [duplicate]
I have been running a random forest regression to predict a normalized target variable (where scores range between 0 and 1). However, whilst I am getting a reasonable level of overall performance, ...
0
votes
1answer
20 views
How to stabilize model performance?
I am performing a regression classification to predict genes that are likely to cause disease. I have 600 rows of training genes by 8 features. Although only 50 genes have a score >0.9 (on a scale ...
0
votes
0answers
29 views
How to compare AUC across models, with cross validation?
I am working on a feature selection problem using an LDA classifier. I choose to start with the forward selection method, wherein starting with no features, at each step one feature is considered that ...
0
votes
1answer
8 views
Proper way to perform cross-validatation for selecting best parameters to build a calibrated model and assessing the error of the model?
I want to find the best possible:
Post process to apply on my data (for example, whether or not perform PCA or scaling, to remove some features...). Some of this options have parameters to tune (like ...
0
votes
0answers
50 views
Model smoothness selection for GAMs: GCV vs. REML vs. ML?
I am studying patterns of bird abundances with certain habitat variables and how they vary over time. I am interested in using GAMs with smooth terms for some of the variables. I am, however, confused ...
0
votes
1answer
48 views
Linear Regression Back-testing Paradox
I have run a linear regression with OLS for the period 2009 - 2017 and then complete back testing
The model is :
y=1.0527x - 0.082
Where
y= IOS (percentages)
x = Bank Rate (percentages)
When i plot ...
0
votes
1answer
15 views
Kfold model selection with high train test score deviance
For model hyper parameter tuning, standard k-fold cross validation is being performed. Scoring is done using coefficient of determination, with higher test scores preferred. Dataset size is (100K, 30)....
0
votes
0answers
7 views
Is a comparison of scores meaningful on different settings?
I want to compare two encodings of categorical data, one hot encoding with one learnt in a neural network. Their performances will be tested in a binary classification task. However, it seems that one ...
0
votes
0answers
11 views
Cross validation for Collaborative filter-based recommendation systems
I am an absolute beginner and am trying to implement collaborative filter for furniture ecommerce (think wayfair). I need some guidance about cross-validation strategy.
Situation:
I am working on a ...
