Questions tagged [cross-validation]
Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.
2,856
questions
0
votes
0answers
7 views
Cross-validation or test performance for decision making on explanatory analysis
I am working on explanatory analysis, that is to get the best feature set to represent a class (feature selection/classification problem) and taking the class characteristic from the selected features....
0
votes
0answers
14 views
Cross Validation for PCA Using Krzanowski's Method
I am trying to implement Krzanowski cross validation for PCA. I am referencing this article: Reference pages 1243-1244. When I run my code, I am getting Q2 values that continue to increase with each ...
0
votes
1answer
17 views
Questions regarding implementation of cross-validation
I have a paper coming up and I would like to clear some questions regarding cross-validation,
because I could not find this information explicitly stated anywhere in literature or it differs.
I ...
0
votes
0answers
9 views
when i use random search for SVM RBF kernel it returns C=100, does it good or bad value? [closed]
i am using randomizedsearchCV with rand_grid = {'C': [0.1,1, 10, 100], 'gamma': [1,0.1,0.01,0.001],'kernel': ['rbf']} for ...
0
votes
0answers
11 views
how to apply 5-by-2 cross-validation to show the statistical significance of my model compared to random model in matlab?
I want to show that my model is better than random chance in classifying 8 classes.
I read that performing 5-by-2 cross validations is a good measure but I didn't understand it well and all the code I ...
0
votes
0answers
33 views
+50
Least Square Cross Validation for Density Estimation with Histograms
In a 1981 paper by Rudemo an easy to compute expression for the integrated squared error of a histogram relative to the true distribution is derived (Eq. 2.8 of the paper and the last equation in this ...
0
votes
0answers
19 views
+50
How to balance transformation decisions, feature selection, and model tuning vs time in text analytics?
Being to new text analytics, I haven't gotten the hang of my typical ML workflow given how long processes take to run in the commonly large feature space of text analytics. I would like to know what ...
0
votes
1answer
22 views
Cross-validation using Caret in R: Why are coefficients from FinalModel identical to those from lm()?
I think I must be missing some fundamental part of the logic of cross-validation, or machine learning in general.
Using the caret package in R, I ran a repeated k-...
0
votes
0answers
9 views
Which model to use? (cross validation with early stopping)
In this example, to keep things simple we use only 1 training and validation set, and we are trying to find the best regularization parameter for ridge regression. The square loss below is on the ...
1
vote
2answers
72 views
Machine learning strategy for imbalanced data with high number of examples
I am working on a classification problem, with unbalanced classes :
Number of positive examples: ~200k; Number of negative ...
0
votes
0answers
11 views
Questions about scikit-learn.org “Nested vs non-nested CV” example
I think I'm starting to get an idea about nested cross-validation, but I have a couple questions about this specific example.
Isn't what we calculate in the outer loop a less biased performance ...
2
votes
0answers
14 views
Which R-Squared formula is used in carets' k-fold-cross validation function
I am currently trying to implement a linear model and calculate some out-of-sample accuracy metrics while using k-fold-cross validation. I try to do this using R's caret package, so this question ...
0
votes
1answer
17 views
What is the difference between external and internal cross validation?
What is the difference between external and internal cross validation? From here I found:
Internal statistical cross-validation assesses the expected
performance of a prediction method in cases (...
0
votes
1answer
8 views
Cross-validation for hyperparameter tuning
I've read as many topics regarding hyperparameter tuning as I could, and I developed the following algorithm for hyperparameter tuning & final model building
Split the data in train set (80%) &...
0
votes
0answers
19 views
Training/test splits (Monte Carlo sensitivity analysis) or Cross-validation
I am using SVM in Matlab (fitcsvm function) to train a classifier for a problem with two classes. Further, I have three features, e.g. A1, A2 and A3, available for each observation composing my full ...
0
votes
1answer
26 views
How large should my data set be before I can do a train-test split (cross validation)?
Data set
I am currently working on a data set that I would like to fit a regression on. It consists of 81 observations and 9 variables. The variables consisting of 1 response variable and 8 predictor ...
0
votes
2answers
50 views
Is it Valid to Grid Search Cross Validation for Model Hyperparameter Selection then a separate Cross Validation for Generalisation Error?
The question has to do with Model Selection and Evaluation
I'm trying to wrap my head around the scale of how different nested cross validation would be from the following:
Let's say I am attempting ...
3
votes
2answers
36 views
Should we use AUC as an indicator of overfitting when dataset is highly imbalanced?
In my problem, there are 2 class labels, but one label only counts for 1% of the total data. I first divided my data set by train_test_split such that only 10% are test set, then I performed 10-Fold ...
2
votes
2answers
57 views
Cross validation on a single model (not model comparison)
I understand the method of cross validation to be to leave out some part of a dataset (whether that be one data point at a time = LOO, or subsets = K fold), and train the model on some data, test the ...
0
votes
0answers
19 views
Small sized training set and results varying based on cross-validation split
I need to try to build a classifier based on around 30 instances. The outcome can also be that the dataset is not large enough for this purpose, however I'm not sure on how can I justify this outcome ...
0
votes
0answers
12 views
Using Cross Validation for Highly Seasonal Data with small sample
I'm having trouble getting good scores on cross validated metrics on time series regression models. Essentially, I am trying to model product purchases based on amount of money spent on different ...
0
votes
0answers
17 views
Is this the correct procedure for hyperparameter tuning and model evaluation?
As a follow up to my previous post (What is the correct procedure for nested cross-validation?) I wanted to see if my following nested cross-validation procedure is valid:
Split my data into train/...
0
votes
0answers
13 views
Ranking of models on dev set does not match ranking on test
Using my train/dev/test split, I optimized four different model's hyper parameters over the development set. Suppose the ranking based on F1 score on the development set was:
model 1
model 2
model 3
...
0
votes
1answer
20 views
Calculating pooled p-values from cross validation folds
I want to calculate the pooled p-value of a regression coefficient across K fold cross validation. I have a model
$$Y \sim \mathrm{Intercept} + \mathrm{Cov}_1 + \mathrm{Cov}_2 + \mathrm{Cov_3} + X$$
...
1
vote
0answers
76 views
Should I perform nested CV with Grid Search to make my ensemble model robust?
I'm doing classification of 8 types of hand gestures with stacking models. For that I initially split the data into training and test sets. Then I used GridSerachCV ...
0
votes
0answers
19 views
Biase of ASE estimation Kernel Regression
I'm trying to calculate the bias of the estimator $p(h)=n^{-1}\displaystyle\sum_{i=1}^{n}(Y_{j}-\hat{m}_{h}(X_{j})^{2}w(X_{j})$ of the averaged squared error. The result I find in the literature is ...
0
votes
0answers
17 views
Treating “probability thresholds” (classification problems) as a hyperparameter
I found this link over here :
https://topepo.github.io/caret/using-your-own-model-in-train.html (section 13.8)
My understanding is that the probability threshold over here is essentially being ...
0
votes
0answers
21 views
What is the correct procedure for nested cross-validation?
I am trying to use scikit-learn to make a classifier and then predict the accuracy of the classifier. My dataset is relatively small and I am unsure of the best parameters. Hence I turned to nested ...
1
vote
2answers
49 views
Can cross validation MSE have multiple minima as function of lambda?
I was running some LassoCV and RidgeCV and wanted to know whether it is possible for CV MSE functions of lambda for say Ridge regression can have multiple minima.
e.g. multiple values of lambda such ...
0
votes
1answer
20 views
What to report in cross validated elastic net regression?
Let's assume I want to construct a regression model to predict a specific outcome variable but I don't have enough data to do a proper train-test set split (n = 200). I have 7 predictor variables (...
0
votes
0answers
12 views
AUC for crossvalidation
I have a medical research scenario where I am trying to predict disease progression. I need to produce a model to integrate into clinical decision support (and evaluate further). In addition to ...
0
votes
0answers
18 views
What is the rational behind bootstrapping a model?
At this point, I understand what bootstrapping is and how it works. What I would like to understand better is the exact properties of the method regarding its test error and, as a related question, ...
1
vote
1answer
16 views
RFE on small dataset: What kind of cross validation should I use?
I’m using recursive feature elimination (RFE) to rank features. My dataset contains 50 observations and 20 predictors. Is there a specific type of cross-validation method I should be using to estimate ...
2
votes
2answers
69 views
Neural Network vs regression in prediction
I collected a sample of 600 observation (time series data) with 100 predictors variables in order to predict another one. I want to use some prediction models but I know that, unfortunately, ...
0
votes
0answers
163 views
Recreating Grid Search (k fold) Cross Validation Functions in R
I am working on a binary classification problem with an imbalanced dataset. I have decided to use a random forest model. I then trained the random forest model (grid search cross validation) using ...
0
votes
0answers
13 views
Sample Split for Cross validation
I want to split a sample to create a multiple regression model using training dataset.
Problem is, I have three dependent variables (numberA, numberB, numberC) and want to do three regression models. ...
0
votes
0answers
13 views
Proper way to incorporated CalibratedClassifierCV in cross-validation in Scikit
I'm creating some classifiers for a binary classification problem. I want to find out three things:
Which algorithm I should use.
Which set of hyperparameters I
should use.
If I should calibrate the ...
4
votes
1answer
50 views
When is cross validation necessary to estimate a parameter?
In 2013, @Donbeo asked whether there were any theoretical results supporting use of Cross Validation to choose the lasso penalty, and was scolded in the comments for asking "a pretty generic ...
0
votes
0answers
33 views
Does k-fold cross-validation induce data leakage in time series data?
I created a predicative model using neural networks and applied in on a time series. This is how I split my data:
...
0
votes
0answers
13 views
How to chose a set of feature weights from cross-validation?
I'm fairly new to machine learning so if there are missing links I'd appreciate the help.
My goal is to train a model and then use it on real world data.
I am going to use only one algorithm (logistic ...
3
votes
2answers
411 views
Cross validation confusion on the concept
I am getting confused with the concept of cross validation in ML. Suppose I have divided my training set in 3 folds: A,B,C. When I am training my model using 3 fold cv, I am training it as follows:
1:...
3
votes
1answer
43 views
Why is my Random Forest Regression performing worse in cross validation than on a baseline?
So I am trying to use a Random Forest Regression on a dataset with a mix of categorical and numeric data types. The predictors are in X_train and ...
0
votes
0answers
31 views
Need to implement train_test split before validation_curve/learning_curve in Sklearn?
I'm doing a regression using sklearn, and I wonder if there is a need to split my data set X and the target variable y into x_train, x_test, y_train, y_test before implementing the validation_curve ...
0
votes
0answers
27 views
Averaging Brier score [duplicate]
To score a RandomForestClassifier using GridSearchCV for multiclass classification, I decided to use Brier score.
However, I ...
0
votes
0answers
33 views
What is the limit to consider something is overfitting?
I'm running random forests for imbalanced multiclass classification. Because of this, I'm trying many variations of RF: basic RF, balanced RF, weighted RF, undersampled RF and SMOTE RF (oversampled).
...
0
votes
0answers
9 views
Applying cross-validation to find the best length of n-gram [duplicate]
Having seen questions on stackoverflow and stats.stackexchange, I have not found a hands-on example of using cross-validation for finding the best length of n-gram.
I am writing a plagiarism detection ...
1
vote
1answer
33 views
Should the cross validation have the natural rate of unbalanced data set
I have a classification problem with unbalanced classes where the natural proportion of the positive class is 0.02% but the total number of cases is high (+300k) which allows to change it when ...
0
votes
0answers
8 views
The number of steps in Recursive Partitioning for Heterogeneous Causal Effects (Athey and Imben 2016) algorithm
How many steps does the Recursive Partitioning for Heterogeneous Causal Effects (Athey and Imben 2016) algorithm have?
Does the algorithm use the cross-validation only in step 2? or there is another ...
3
votes
1answer
45 views
Does using a random train-test split lead to data leakage?
I am trying to understand data leakage in modeling practice.
If we had a dataset of patient instances from 2000-2018 (with all patient visits included), and used a randomly selected train-test split (...
1
vote
0answers
11 views
How does Cross-validation prevent the Recursive Partitioning for Heterogeneous Causal Effects from over fitting?
I have a question regarding the steps in the Casual Tree algorithm based on Dr. Athey's paper, entitled " Recursive partitioning for heterogeneous causal effects".
Does the algorithm use ...
