Questions tagged [cross-validation]
Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.
3,226
questions
0
votes
0
answers
6
views
learning curve for RF and LR comparison and selection
I am plotting learning curve to check how the model perform on training data set and the effect of the training size on the accuracy. I am using two models, random forest and logistic regression.
From ...
4
votes
2
answers
207
views
Stratification of the continuous y (target) variable in regression setting
Is it wise to stratify the continuous y (target) variable when you split your training and testing data from the total sample in regression setting? Here is the approach in python to do implement ...
- 131
0
votes
0
answers
8
views
Moving window size for time series analysis : Minimum description length and Fractal data dimension [closed]
I'm doing the time-series analysis as a hobby and am attempting to discover the optimal window size. After some study, I found 2 techniques that had been mentioned on the internet:
MDL (minimum ...
1
vote
0
answers
17
views
Interpreting learning curves
There is really few examples online regarding interpreting learning curves and they are all of different type.It is quite confusing to me honestly.May I just ask:
How should we interpret them?What ...
- 11
-1
votes
0
answers
14
views
Solution for Session Crash in google Colab by applying KFold Cross Validation in Python [closed]
Is there any way to modify this below-mentioned code? As I am applying 5-Fold cross-validation by using the below code in Google Colab free version. My data frame consists of 5000 rows and 9000 ...
0
votes
0
answers
9
views
After training a model, how does test set error inform decision making?
I split a data set into three subsets: training, validation, and test sets. I use my training data for fitting and validation to check for overfitting. I then have a final model that I then propose to ...
- 1,003
2
votes
0
answers
58
views
Should folds in k-fold CV actually be representative?
I have read somewhere that the k of the k-fold CV should be picked in such a manner as to have representative validation sets (folds). This seems to me to be contradictory since the leave-one-out CV ...
- 21
0
votes
1
answer
46
views
Forcing covariates to always be part of a Lasso model
I want to use a Lasso to predict outcomes for different policy scenarios. At the optimal degree of regularization obtained by cross-validation, one important variable in whose impact I'm interested in ...
- 1
2
votes
0
answers
13
views
Statistical test for temporal cross validation
I estimated the performance of my forecasting model and that of a baseline on 10 folds using temporal cross-validation. With which test do I assess if my model is significantly better than the ...
- 423
0
votes
0
answers
5
views
Meaning of the sample variance computed from a k-fold CV
Let's consider a k-fold cross-validation to estimate the generalization error of a model. I would like to clarify the relationship between the following quantities:
the variance of the CV estimator
...
- 869
0
votes
1
answer
23
views
evaluating scoring metrics during hyperparameter tuning
I'm struggling with a couple of concepts related to hyperparameter tuning.
I'm developing a model (gradient boosted tree) in python using sklearn. Currently, I'm in the process of using sklearn's ...
1
vote
2
answers
45
views
How to multiply two models together in R?
I am constructing a two-stage GAM to model capture rate across my study site while accounting for zero inflation.
...
- 11
0
votes
0
answers
7
views
Options for two-stage GAM univariate model selection when AICc values <2Δ
I am constructing a two-stage GAM (stage 1 presence/absence with binomial and logit link, stage 2 abundance with poisson and log link) to model capture rate across my study site. As this is a pilot ...
- 11
0
votes
0
answers
10
views
Why does MobileNet Architecture start with a Standard Convolution?
I am trying to understand the design choices behind the MobileNet architecture. (pdf available on the right).
The authors use Depthwise Separable Convolutions as a replacement for Classical ...
- 1
0
votes
0
answers
12
views
Correction for multiple testing when testing multiple machine learning models on test set
Suppose I optimize the parameters of k models on a training set (via cross-validation), and then test all of them on a separate test set. Is there some sort of correction for multiple testing that ...
- 21
0
votes
0
answers
25
views
Should I use the predictions from my model or use the predict function on my model?
I have a small dataset (with about 5 covariates and 30 rows) and I am trying to make some predictions using R. I was advised to use Leave-One-Out Cross Validation (LOOCV) with a random forest due to ...
- 185
0
votes
1
answer
29
views
Cross Validation after using train-test to decide optimal algorithm to use?
I am interested in training different algorithms on a data set and observing performance metrics.
Currently, my approach is to train different algorithms on train data, and then evaluate performance ...
- 13
0
votes
0
answers
13
views
Hyperparameters chosen by CV on train dataset don't perform well on validation/test dataset
I've used the following objective function to assess best hyper-parameters using Hyperopt():
...
- 123
2
votes
1
answer
120
views
Appropriate way to get cross validated performance metrics
For cross-validation of a logistic regression classifier, it seems to me that there are a number of different approaches to calculating each performance metric:
The performance metric is calculated ...
- 144
-1
votes
0
answers
10
views
Can you use k-fold cross validation on a time series neural network? [duplicate]
From my understanding, if I am trying to predict one step ahead in a time series, the usual k-fold system will not work because the order of the training data is important.
I know this work suggested ...
- 1
1
vote
1
answer
16
views
Threshold optimization with cross validation
I have an imbalanced dataset; 95% negative class and 5% positive class. I split my data into train (80%) and test (20%) sets. I am using 5-fold cross-validation on the train set to determine the ...
- 21
1
vote
1
answer
12
views
Train a Final Machine Learning Model with Tensorflow
Based on a previous question and on this article, it is suggested that you split the data between train and test (or train/validate/test). But once you have control of your model, you should retrain ...
- 127
0
votes
0
answers
38
views
Overfitting the (non-nested) cross validation set
This is a follow up query to this query
In one line: I wish to understand why is it that we will severely over fit the cross validation set (and hence need nested cross validation to correctly account ...
- 219
0
votes
1
answer
25
views
How is the train_score from sklearn.model_selection.cross_validate calculated?
I split the data 80/20 as follows:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
...
- 101
1
vote
1
answer
85
views
response mixed effect model
I have created a mixed-effect model which involves drug response, here I have 2-factor level Drug variable of response that is Control or Drug A. The model that I employed is m1 which basically check ...
- 111
0
votes
0
answers
19
views
My random forrest regressor was overfitting so I tried to use randomsearchcv but I still got a worse result, what should I change? [duplicate]
I tried to fix my overfitting with randomized search cross-validation. These are my params:
I set 100 estimators but that is irrelevant for the overfitting. I read log2 was best for regressors ...
- 15
2
votes
1
answer
42
views
Cross-validation: error estimation and bias
When obtaining the error estimation of a model over a dataset using k-fold cross-validation, lower values of the error estimation necessarily imply a lower bias? Are both concepts, error estimation ...
- 55
0
votes
0
answers
17
views
Validity of external cross validation using data generated by the fit model?
Context:
A paper I'm reading uses PDEs to characterise the effects of cancer treatments on the tumour microenvironment. The exact wording used in the paper is:
The predictive power of the [...
- 13
3
votes
2
answers
183
views
AUROC too high in image classification
I'm dealing with an image classification problem, with a multiclass imbalanced dataset (the bigger class has 4000 samples and the smaller has 110 samples) with 50 classes and 24000 samples.
I'm using ...
- 378
1
vote
1
answer
27
views
Issues regarding cross-validation and metrics for comparing approaches in machine learning for image classification with imbalanced datasets
I'm trying to compare the performances of N classifiers for multiclass image classification, with an imbalanced dataset with 50 classes.
I'm considering now the following basic metrics:
accuracy:
...
- 378
1
vote
0
answers
59
views
Why did cross validation accuracy suddenly decrease? and then increase again?
I am training a classification model in Tensorflow. Here is a screenshot of the graphs I obtained after training.
The blue one is the training accuracy, while the orange one is the cross validation ...
- 111
0
votes
0
answers
13
views
Developing an optimized model after nested cross validation
I have been assigned to learn about k-fold cross-validation for my class. As an extension, I wanted to learn more about nested cross-validation.
I understand that nested cross-validation involves ...
- 103
2
votes
1
answer
24
views
Is it valid to cross-validate linear regression models out of chronological order in a time series context?
For simplicity I'll stick to the univariate case with one exogenous regressor. Suppose I have
X = (x1, x2, ..., xn)
Y = (y1, y2, ..., yn)
where i < j indicates xi came before xj chronologically, ...
- 21
1
vote
1
answer
26
views
RepeatedKfold - Use full data or only train data?
I am working on a binary classification using random forest with a dataset size of 977 records and 6 columns. class ratio is 77:23 (imbalanced dataset)
Since, my dataset is small, I learnt that it is ...
- 1,726
0
votes
0
answers
26
views
Using SSE to find the optimal model does not yield the correct solution
I am generating an AR(3) model with coefficients
$(\phi_1, \phi_2, \phi_3) = (0.35, 0.5, 0.08)$. Then I use the statsmodels module
from Python and its Yule-Walker implementation to get back the ...
- 101
0
votes
1
answer
34
views
Can I remove variance of OLS estimators through averaging coefficients from cross-validation?
Imagine working on a linear multiple regression problem with a design-matrix $\Phi$ based upon some independent variables $x_k, k\in{}[1, r]$. The goal is to find an equation that explains the "...
- 11
1
vote
0
answers
146
views
Obtaining approximate posterior probabilities with Bayesian cross-validation
(Apologies to anyone that may have been following this question: I have decided to rewrite it to make it more succinct. As a result, comments below now appear out of context.)
Given a set of models $\{...
- 1,125
0
votes
2
answers
114
views
Is it possible to evaluate a given model without having access to its fit method?
I have a data set with one real-valued feature and a real-valued target. Someone has used this data set to fit a model (a regression). I get a results of this fit, which is a single function mapping ...
- 973
0
votes
0
answers
18
views
Validating features that were already selected by train-test of a LASSO model
I have 50 tagged samples. I have selected features out of lots of possible features (10,000 maybe) I would like to test their ability to predict the tags.
I tried to train lasso/ridge models on a ...
2
votes
1
answer
48
views
Is cross-validation necessary when computing significance of coefficients?
I'm unclear on if its important to perform cross-validation when determining if a dependent variable has a significant effect on my independent variable in multilinear regression.
Specifically, I'm ...
- 23
2
votes
0
answers
32
views
Any theory on whether good choices of $k$ depend on $N$ and $D$ in KNN classification?
I am well aware that cross validation is a usual method for selecting hyperparameters. However, I am looking for theoretical guidance on how to pick $k$, the number of neighbors, for a $k$-nearest-...
- 579
0
votes
0
answers
24
views
How to use intermediate neural network layers for model chaining
I have a neural network structured like this:
inputs -> neural network layers -> intermediary output -> more layers -> output
I would like to use the features from the intermediary ...
2
votes
1
answer
35
views
Is it logical to combine cross-validation estimator like RidgeCV with cross_val_score in sklearn?
I was going through solutions for a regression problem competition on Kaggle here. Many solutions for the problem are combining cross-validation estimators like RidgeCV, LassoCV with cross_val_score ...
- 31
0
votes
0
answers
20
views
R: How to determine lmer baseline in k-fold cross validation
For testing whether a predictor improves a linear mixed-effects model fitted with lme4, I used to fit a baseline model and the baseline model with an additional ...
- 33
2
votes
2
answers
138
views
Random forest regressor accuracy reduces when the input data is not shuffled
I am using a random forest regressor and I split the independent variables with shuffle = True, I get a good r squared but when I don't shuffle the data the ...
- 139
0
votes
0
answers
15
views
How to calculate RMSE with missing values except filling them with zeros? [duplicate]
I have a real and predicted matrix of the form np.array and I calculate the RMSE using Sklearn:
...
- 1
2
votes
0
answers
34
views
How to name train + validation set
Usually in machine learning pipelines, we use a train set, a validation set and a test set. Quite often, we first split the test set from the rest, and then we split the "rest" into train ...
- 21
0
votes
0
answers
15
views
Seeking A Scale-Independent Alternative To Q2 For Model Selection When The Response Varies Over Multiple Orders of Magnitude
I am using constrained polynomial regression to predict y = f(x). I have prior knowledge about the relationship that allows me to add constraints to the optimization problem for the first and second ...
- 1
2
votes
1
answer
72
views
Why does XGBoost with cross-validation perform worse on test holdout than unvalidated model?
I have an XGBoost model that I fit on some X data directly out of the box:
...
- 61
1
vote
0
answers
40
views
Bootstrapped Latin Partition
I'm having trouble understanding the Bootstrapped Latin partition method (as presented in Statistical validation of classification and calibration models using bootstrapped Latin partitions and ...
- 11