Questions tagged [cross-validation]
Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.
2
votes
1answer
8 views
How to perform cross validation in semi-supervised learning
Suppose in semi-supervised learning, we have labeled set $X_L$ and unlabeled set $X_U$
Is it ok to validate model performance on labeled data only?
How to do cross-validation in transductive learning,...
0
votes
0answers
7 views
How to calculate One Standard Error rule tuning parameter for prediction error during k-fold CV
I'm trying to wrap my head around exactly how this rule goes into place, so I can use it by hand in other model selection setups.
So here's some R code to get it started:
...
0
votes
0answers
15 views
Reporting performance after hyperparameter tuning
I have a really small dataset ~500 and I am wondering how to report performance. I need to perform hyperparameter tuning, but I was wondering whether the approach is okay.
Approach 1: Can I for ...
1
vote
0answers
9 views
Is it a good idea to do Cross-Validation for LASSO with a small sample size?
I have a dataset consist of 40 rows and 15 terms as variables. I need to develope a "prediction model" based on LASSO classification. Thus, I want know the best significant terms with their ...
0
votes
0answers
7 views
Nested CV with Stratified Kfold
I am using nested CV for model evaluation and my target variable has imbalanced classes.
With Sklearn, I am using GridSearchCV and cross_val_score to perform the nested cross validation. Each takes ...
0
votes
1answer
32 views
Disease prediction based on genes
I am a biologist and I am dealing with biological function prediction.
I have a model that annotates "disease" function for genes in a genome based on a set of "disease" genes as input.
Note: ...
0
votes
0answers
40 views
Classification accuracy in holdout similar to CV if set is randomly sampled, completely wrong otherwise
I'm building a classifier to predict a binary label on a dataset with 30 features and around 60000 samples of measurements from a car assembly process.
While experimenting with some baseline models ...
0
votes
0answers
10 views
How does Cross Validation work in Matlab
I am doing some research on Lasso classification method. I have a 40x15 dataset and I want to develop a binomial equation without dividing data into train and test set (because of small sample size). ...
0
votes
0answers
25 views
validation vs test vs training accuracy, which one to compare for claiming model overfitting?
I have read on the several answers here and on the internet that cross-validation helps to indicate that if the model will generalize well or not and about overfitting.
But I am confused that which ...
0
votes
1answer
24 views
Beta Regression Model Selection with CV Lasso in R
Is there a package that will do a cross-validatation with regularization for beta regression in R? I'm looking for an equivalent of glmnet for the betareg package.
0
votes
0answers
13 views
How exactly Matlab performs the Lasso classification?
I am doing some research on Lasso classification method. I have a 40x15 dataset and I want to develop a binomial equation without dividing data into train and test set (because of small sample size).
...
0
votes
0answers
22 views
LassoCV regression on price returns doesn' t work
I'm trying to use LassoCV to get a linear model for the log return of an asset price.
So what I am doing is:
Download historical prices for near 61 assets
Calculates the log return and difference of ...
0
votes
1answer
24 views
Why Lasso classification results changes in Matlab?
I am doing lasso classification using Lassoglm command in Matlab. I have a problem and that is, every time I run the program for my dataset I get different variables to have non-zero coefficients. I ...
0
votes
2answers
28 views
Criteria to perform Cross Validation
Now I know why we do Cross Validation. And how we do it. I have few questions around it though:
Why can't we do that on every dataset we work on? Like, why shouldn't it be a standard, or a rule of ...
1
vote
0answers
21 views
Negative Binomial Regression Questions
First post!
I'm a biologist with limited background in applied statistics and R. Basically know enough to be dangerous, so I'd appreciate it someone could confirm/deny that I'm on the right path.
My ...
2
votes
2answers
82 views
Can the alpha, lambda values of a glmnet object output determine whether ridge or Lasso?
Given a glmnet object using train() where trControl method is "cv" and number of iterations is 5, I obtained that the bestTune alpha and lambda values are alpha=0.1 and lambda= 0.007688342. On ...
0
votes
0answers
11 views
How do I get the results from my Cross-validation on Python? [closed]
I have created a small script to use cross-validation in a dataset.
...
0
votes
0answers
34 views
How can I use k-fold cross-validation to determine whether a linear regression model performs significantly better than chance?
I have an experiment in which I present a subject with 1820 inputs. For each input, a response is produced in ~25,000 separate output variables. I have a function F which produces a feature vector ...
2
votes
0answers
28 views
Overfitting cross-validation scores
I've ran 1000 iterations of XGBClassifier parameters search using RandomizedSearchCV on the titanic dataset. That's just for context, but the question applies to any CV search method, any model and ...
0
votes
0answers
5 views
choice of the lambda parameter in the logit multinomial ridge model
I can not find clear literature on how to choose the penalty parameter in the logit multinomial ridge model. As read in the linear models and trying to adapt it to the model I need it would be through ...
0
votes
0answers
9 views
Chossing between high number of components in PCR vs linear regresion
Let's say my original data set has 18 variables.
If the result of the cross-validation error is lowest on the 17 components of PCR is that a good indication that you most likely choose the ...
0
votes
1answer
14 views
Dataset partitioning for k-fold cv
In general, is the size of the validation set for k-fold CV given by n/k and that of the training set given by n(k-1)/k and if so, why? (this is based on the ISLR book)
Alternatively, what are some ...
0
votes
0answers
8 views
Do I need to do glmnet after doing a cv.glmnet?
I'm studying now about the model selection from the ISLR book.
I'm don't understand about whether should I do glmnet() after I do ...
0
votes
0answers
19 views
How are the training and cross-validation metrics calculated in H2O?
I am working with the GBM algorithm in H2O in R. I am using 100% of the data as the training data, and then using 5-fold cross-validation to train and validate my model using 100% of the data.
My ...
1
vote
1answer
40 views
How to validate models beyond checking for overfitting
I have an unusual problem, which is that my model is performing too well and I am struggling to trust it.
The data is a table of "snapshots" about moments in games of chess. For example, a game that ...
0
votes
0answers
29 views
Random partition techniques for lasso and elastic net
I think this is the correct place for this question.
I have implemented lasso, elastic net and a different estimator on a real data set. I used 10-fold cross-validation (CV) one time without ...
3
votes
2answers
44 views
Do I need a validation set if I am doing 10-fold cross validation?
I am looking at a dataset with ~120 observations and I am investigating it using two sets of explanatory variables, one has about 12 features, the other about 8. This is for a regression analysis.
...
0
votes
0answers
18 views
Use R2 in hypothesis testing
I am conducting a large simulation study, where different statistical models are set up from training data and cross validated on a validation set. I want to assess, under which conditions which ...
0
votes
1answer
20 views
How does cross-validation work exactly?
I'm having a hard time figuring out how exactly cross validation works in practice:
To do K-fold cross validation on a data set, you divide your data into K sets. Then for each fold $i$, $1 \leq i \...
1
vote
1answer
43 views
k-fold cross validation with multiple classes
I'm working on an image retrieval system (not classification). I have 5,000 images as the data set. 500 images of this dataset are the query images used for retrieval evaluation. these 500 images ...
0
votes
1answer
35 views
Regression for curve fitting
For a curve generated from dataset points, split the curve into parts and obtain the best-fit degree of polynomial,coeffcients and the interval/range of the split through implementation in python.I am ...
0
votes
0answers
9 views
Getting Significance Information from glm object
I am creating 1000 logistic models in R with glm(), using a different 75% of my data each time. Currently, I am doing this via for-loop (which I have been told is ...
0
votes
1answer
16 views
Avoiding information leakage in CV folds with scaling
Chapter 6 (Algorithm Chains and Pipelines) in the book Introduction to ML with Python made me aware of a common mistake when scaling data for cross validation: leaking information into the test set by ...
0
votes
1answer
35 views
FIND KNOTS IN REGRESSION
To find the knots automatically in piecewise polynomial regression, which concept is BEST, cubic splines or k fold cross-validation in python
3
votes
2answers
33 views
Bias-variance tradeoff associated with cross validation methods
I was reading about the bias-variance tradeoff associated with cross validation methods on James et al, Introduction to Statistical Learning (Page 183-184).
When we perform LOOCV, we are in effect ...
0
votes
0answers
12 views
Leave One Out Cross-Validation in Python
For me is not clear the way to implement LOOCV in Python, I have the next Python scripts:
1) ACC = 76.92 %
...
0
votes
0answers
16 views
LDA attribute similar to fitted values from glm
I am comparing the results of LDA vs logistic regression as an exercise in understanding their differences and similarities. When fitting a glm model
...
0
votes
0answers
5 views
Having two sets of test data with different acceptance criteria
I've almost completed Google's Machine Learning Crash Course, and I've also tried this OCR tutorial (not from Google).
OCR seems like a good initial project for a Neural Net, and I was wondering if ...
1
vote
0answers
18 views
80-20 better than full dataset for LightGBM
Recently I have been using LightGBM as regressor in order to predict, on a dataset of 20 thousand observations.
I have two modes, 1) Production and 2) Testing. The first one just trains a model with ...
2
votes
1answer
53 views
What if cross-validation fails to prevent overfitting
I'm training a random forest model with AUC as performance metric. I've splitted my data to train set (70%) and test set (30%) and performed cross-validation on train set to tune the hyperparameters. ...
1
vote
0answers
15 views
Cross-validation of (Cox) survival model with very high censoring rate
I am currently working on survival analysis of data with very high censoring rate (~99%), and the number of events is only about 500, using R.
I would like to ask in such case, whether the validate() ...
0
votes
0answers
11 views
What are the functions of training data, validation data and test data and how to partition?
Usually we partition the whole data-set into three parts: training data, validation data and test data. For the training data it is used to train the parameters of the variables in the model; and the ...
1
vote
1answer
24 views
Randomization in Cross Validation
I have made some researches about cross validation for machine learning. I faced two types of randomization techniques used for splitting the dataset into K-folds.
...
0
votes
0answers
15 views
I am getting 0 percent accuracy when using 3-fold cross validation
I want to train a CNN that will be k-fold cross-validated. for that, I have divided my signature data set in three equal part. using the two parts training is happening and from the remaining part, ...
1
vote
1answer
46 views
Generalization performance in Bayesian errors-in-covariates model
I'm working on a model with this basic structure:
The square nodes are data, and the round nodes are parameters and/or latent variables. The left plate represents the "training observations" we ...
1
vote
1answer
26 views
Is it valid to compare the likelihood of different models in Gaussian process regression?
When applying different kernel's through scikit-learn's Gaussian process regression, I observe certain instances with positive log-likelihood outputs which indicate a likelihood that is greater than 1....
1
vote
0answers
16 views
Which metric should be used for learning rate reduction on plateau?
I am testing three different neural network architectures on a dataset to see which architecture performs the best.
My methodology for every architecture is to
Split the data into train/test
Take ...
1
vote
0answers
11 views
Rank error metric for time series
Suppose I have a collection of MAE across multiple time series (say, 10), and 3 models. However, MAE cannot be compared across time series.
I compare the errors in this way: assign ranks to models, ...
0
votes
0answers
8 views
Calculating F1 with crossvalidation - Average over folds or compute at end?
When performing crossvalidation, is it appropriate to calculate metrics (e.g. accuracy, F1) for each fold and then average these, or to calculate them at the end using the prediction results from the ...
0
votes
1answer
28 views
Splitting into train and test sets keeping class proportions
I have a dataset for a binary classification task which has 90 percent 'yes' and 10 percent 'no'. Let's say I want to take 25 percent of the data as the test set (which the model will not see). How ...
