Questions tagged [cross-validation]

Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

learning curve for RF and LR comparison and selection

I am plotting learning curve to check how the model perform on training data set and the effect of the training size on the accuracy. I am using two models, random forest and logistic regression. From ...
user avatar
4 votes
2 answers
207 views

Stratification of the continuous y (target) variable in regression setting

Is it wise to stratify the continuous y (target) variable when you split your training and testing data from the total sample in regression setting? Here is the approach in python to do implement ...
user avatar
0 votes
0 answers
8 views

Moving window size for time series analysis : Minimum description length and Fractal data dimension [closed]

I'm doing the time-series analysis as a hobby and am attempting to discover the optimal window size. After some study, I found 2 techniques that had been mentioned on the internet: MDL (minimum ...
user avatar
1 vote
0 answers
17 views

Interpreting learning curves

There is really few examples online regarding interpreting learning curves and they are all of different type.It is quite confusing to me honestly.May I just ask: How should we interpret them?What ...
user avatar
-1 votes
0 answers
14 views

Solution for Session Crash in google Colab by applying KFold Cross Validation in Python [closed]

Is there any way to modify this below-mentioned code? As I am applying 5-Fold cross-validation by using the below code in Google Colab free version. My data frame consists of 5000 rows and 9000 ...
user avatar
0 votes
0 answers
9 views

After training a model, how does test set error inform decision making?

I split a data set into three subsets: training, validation, and test sets. I use my training data for fitting and validation to check for overfitting. I then have a final model that I then propose to ...
user avatar
  • 1,003
2 votes
0 answers
58 views

Should folds in k-fold CV actually be representative?

I have read somewhere that the k of the k-fold CV should be picked in such a manner as to have representative validation sets (folds). This seems to me to be contradictory since the leave-one-out CV ...
user avatar
0 votes
1 answer
46 views

Forcing covariates to always be part of a Lasso model

I want to use a Lasso to predict outcomes for different policy scenarios. At the optimal degree of regularization obtained by cross-validation, one important variable in whose impact I'm interested in ...
user avatar
  • 1
2 votes
0 answers
13 views

Statistical test for temporal cross validation

I estimated the performance of my forecasting model and that of a baseline on 10 folds using temporal cross-validation. With which test do I assess if my model is significantly better than the ...
user avatar
  • 423
0 votes
0 answers
5 views

Meaning of the sample variance computed from a k-fold CV

Let's consider a k-fold cross-validation to estimate the generalization error of a model. I would like to clarify the relationship between the following quantities: the variance of the CV estimator ...
user avatar
  • 869
0 votes
1 answer
23 views

evaluating scoring metrics during hyperparameter tuning

I'm struggling with a couple of concepts related to hyperparameter tuning. I'm developing a model (gradient boosted tree) in python using sklearn. Currently, I'm in the process of using sklearn's ...
user avatar
1 vote
2 answers
45 views

How to multiply two models together in R?

I am constructing a two-stage GAM to model capture rate across my study site while accounting for zero inflation. ...
user avatar
  • 11
0 votes
0 answers
7 views

Options for two-stage GAM univariate model selection when AICc values <2Δ

I am constructing a two-stage GAM (stage 1 presence/absence with binomial and logit link, stage 2 abundance with poisson and log link) to model capture rate across my study site. As this is a pilot ...
user avatar
  • 11
0 votes
0 answers
10 views

Why does MobileNet Architecture start with a Standard Convolution?

I am trying to understand the design choices behind the MobileNet architecture. (pdf available on the right). The authors use Depthwise Separable Convolutions as a replacement for Classical ...
user avatar
  • 1
0 votes
0 answers
12 views

Correction for multiple testing when testing multiple machine learning models on test set

Suppose I optimize the parameters of k models on a training set (via cross-validation), and then test all of them on a separate test set. Is there some sort of correction for multiple testing that ...
user avatar
  • 21
0 votes
0 answers
25 views

Should I use the predictions from my model or use the predict function on my model?

I have a small dataset (with about 5 covariates and 30 rows) and I am trying to make some predictions using R. I was advised to use Leave-One-Out Cross Validation (LOOCV) with a random forest due to ...
user avatar
  • 185
0 votes
1 answer
29 views

Cross Validation after using train-test to decide optimal algorithm to use?

I am interested in training different algorithms on a data set and observing performance metrics. Currently, my approach is to train different algorithms on train data, and then evaluate performance ...
user avatar
  • 13
0 votes
0 answers
13 views

Hyperparameters chosen by CV on train dataset don't perform well on validation/test dataset

I've used the following objective function to assess best hyper-parameters using Hyperopt(): ...
user avatar
2 votes
1 answer
120 views

Appropriate way to get cross validated performance metrics

For cross-validation of a logistic regression classifier, it seems to me that there are a number of different approaches to calculating each performance metric: The performance metric is calculated ...
user avatar
-1 votes
0 answers
10 views

Can you use k-fold cross validation on a time series neural network? [duplicate]

From my understanding, if I am trying to predict one step ahead in a time series, the usual k-fold system will not work because the order of the training data is important. I know this work suggested ...
user avatar
1 vote
1 answer
16 views

Threshold optimization with cross validation

I have an imbalanced dataset; 95% negative class and 5% positive class. I split my data into train (80%) and test (20%) sets. I am using 5-fold cross-validation on the train set to determine the ...
user avatar
1 vote
1 answer
12 views

Train a Final Machine Learning Model with Tensorflow

Based on a previous question and on this article, it is suggested that you split the data between train and test (or train/validate/test). But once you have control of your model, you should retrain ...
user avatar
  • 127
0 votes
0 answers
38 views

Overfitting the (non-nested) cross validation set

This is a follow up query to this query In one line: I wish to understand why is it that we will severely over fit the cross validation set (and hence need nested cross validation to correctly account ...
user avatar
0 votes
1 answer
25 views

How is the train_score from sklearn.model_selection.cross_validate calculated?

I split the data 80/20 as follows: from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) ...
user avatar
  • 101
1 vote
1 answer
85 views

response mixed effect model

I have created a mixed-effect model which involves drug response, here I have 2-factor level Drug variable of response that is Control or Drug A. The model that I employed is m1 which basically check ...
user avatar
0 votes
0 answers
19 views

My random forrest regressor was overfitting so I tried to use randomsearchcv but I still got a worse result, what should I change? [duplicate]

I tried to fix my overfitting with randomized search cross-validation. These are my params: I set 100 estimators but that is irrelevant for the overfitting. I read log2 was best for regressors ...
user avatar
2 votes
1 answer
42 views

Cross-validation: error estimation and bias

When obtaining the error estimation of a model over a dataset using k-fold cross-validation, lower values of the error estimation necessarily imply a lower bias? Are both concepts, error estimation ...
user avatar
0 votes
0 answers
17 views

Validity of external cross validation using data generated by the fit model?

Context: A paper I'm reading uses PDEs to characterise the effects of cancer treatments on the tumour microenvironment. The exact wording used in the paper is: The predictive power of the [...
user avatar
  • 13
3 votes
2 answers
183 views

AUROC too high in image classification

I'm dealing with an image classification problem, with a multiclass imbalanced dataset (the bigger class has 4000 samples and the smaller has 110 samples) with 50 classes and 24000 samples. I'm using ...
user avatar
  • 378
1 vote
1 answer
27 views

Issues regarding cross-validation and metrics for comparing approaches in machine learning for image classification with imbalanced datasets

I'm trying to compare the performances of N classifiers for multiclass image classification, with an imbalanced dataset with 50 classes. I'm considering now the following basic metrics: accuracy: ...
user avatar
  • 378
1 vote
0 answers
59 views

Why did cross validation accuracy suddenly decrease? and then increase again?

I am training a classification model in Tensorflow. Here is a screenshot of the graphs I obtained after training. The blue one is the training accuracy, while the orange one is the cross validation ...
user avatar
0 votes
0 answers
13 views

Developing an optimized model after nested cross validation

I have been assigned to learn about k-fold cross-validation for my class. As an extension, I wanted to learn more about nested cross-validation. I understand that nested cross-validation involves ...
user avatar
2 votes
1 answer
24 views

Is it valid to cross-validate linear regression models out of chronological order in a time series context?

For simplicity I'll stick to the univariate case with one exogenous regressor. Suppose I have X = (x1, x2, ..., xn) Y = (y1, y2, ..., yn) where i < j indicates xi came before xj chronologically, ...
user avatar
  • 21
1 vote
1 answer
26 views

RepeatedKfold - Use full data or only train data?

I am working on a binary classification using random forest with a dataset size of 977 records and 6 columns. class ratio is 77:23 (imbalanced dataset) Since, my dataset is small, I learnt that it is ...
user avatar
  • 1,726
0 votes
0 answers
26 views

Using SSE to find the optimal model does not yield the correct solution

I am generating an AR(3) model with coefficients $(\phi_1, \phi_2, \phi_3) = (0.35, 0.5, 0.08)$. Then I use the statsmodels module from Python and its Yule-Walker implementation to get back the ...
user avatar
0 votes
1 answer
34 views

Can I remove variance of OLS estimators through averaging coefficients from cross-validation?

Imagine working on a linear multiple regression problem with a design-matrix $\Phi$ based upon some independent variables $x_k, k\in{}[1, r]$. The goal is to find an equation that explains the "...
user avatar
  • 11
1 vote
0 answers
146 views

Obtaining approximate posterior probabilities with Bayesian cross-validation

(Apologies to anyone that may have been following this question: I have decided to rewrite it to make it more succinct. As a result, comments below now appear out of context.) Given a set of models $\{...
user avatar
  • 1,125
0 votes
2 answers
114 views

Is it possible to evaluate a given model without having access to its fit method?

I have a data set with one real-valued feature and a real-valued target. Someone has used this data set to fit a model (a regression). I get a results of this fit, which is a single function mapping ...
user avatar
  • 973
0 votes
0 answers
18 views

Validating features that were already selected by train-test of a LASSO model

I have 50 tagged samples. I have selected features out of lots of possible features (10,000 maybe) I would like to test their ability to predict the tags. I tried to train lasso/ridge models on a ...
user avatar
2 votes
1 answer
48 views

Is cross-validation necessary when computing significance of coefficients?

I'm unclear on if its important to perform cross-validation when determining if a dependent variable has a significant effect on my independent variable in multilinear regression. Specifically, I'm ...
user avatar
  • 23
2 votes
0 answers
32 views

Any theory on whether good choices of $k$ depend on $N$ and $D$ in KNN classification?

I am well aware that cross validation is a usual method for selecting hyperparameters. However, I am looking for theoretical guidance on how to pick $k$, the number of neighbors, for a $k$-nearest-...
user avatar
  • 579
0 votes
0 answers
24 views

How to use intermediate neural network layers for model chaining

I have a neural network structured like this: inputs -> neural network layers -> intermediary output -> more layers -> output I would like to use the features from the intermediary ...
user avatar
2 votes
1 answer
35 views

Is it logical to combine cross-validation estimator like RidgeCV with cross_val_score in sklearn?

I was going through solutions for a regression problem competition on Kaggle here. Many solutions for the problem are combining cross-validation estimators like RidgeCV, LassoCV with cross_val_score ...
user avatar
0 votes
0 answers
20 views

R: How to determine lmer baseline in k-fold cross validation

For testing whether a predictor improves a linear mixed-effects model fitted with lme4, I used to fit a baseline model and the baseline model with an additional ...
user avatar
  • 33
2 votes
2 answers
138 views

Random forest regressor accuracy reduces when the input data is not shuffled

I am using a random forest regressor and I split the independent variables with shuffle = True, I get a good r squared but when I don't shuffle the data the ...
user avatar
  • 139
0 votes
0 answers
15 views

How to calculate RMSE with missing values except filling them with zeros? [duplicate]

I have a real and predicted matrix of the form np.array and I calculate the RMSE using Sklearn: ...
user avatar
  • 1
2 votes
0 answers
34 views

How to name train + validation set

Usually in machine learning pipelines, we use a train set, a validation set and a test set. Quite often, we first split the test set from the rest, and then we split the "rest" into train ...
user avatar
0 votes
0 answers
15 views

Seeking A Scale-Independent Alternative To Q2 For Model Selection When The Response Varies Over Multiple Orders of Magnitude

I am using constrained polynomial regression to predict y = f(x). I have prior knowledge about the relationship that allows me to add constraints to the optimization problem for the first and second ...
user avatar
2 votes
1 answer
72 views

Why does XGBoost with cross-validation perform worse on test holdout than unvalidated model?

I have an XGBoost model that I fit on some X data directly out of the box: ...
user avatar
1 vote
0 answers
40 views

Bootstrapped Latin Partition

I'm having trouble understanding the Bootstrapped Latin partition method (as presented in Statistical validation of classification and calibration models using bootstrapped Latin partitions and ...
user avatar
  • 11

1
2 3 4 5
65