Cross Validated

Questions tagged [cross-validation]

Ask Question

Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.

2,376 questions
Newest
Active
Bounties
0
Unanswered
Filter by
Sorted by
Tagged with
-1
votes
0answers
20 views

Finding the best prediction model for data in R

I have a data set which I want to predict D based on A, B and ...
0
votes
0answers
16 views

rstanarm loo: implications of k-pareto values above 1

(this is the first time I post here, so please excuse any formatting or other errors) I have estimated a linear regression model using stan_glm and I am using loo() to evaluate the model fit. I ...
0
votes
1answer
9 views

k-fold CV-scheme stratifying response and considering groups

I have the following small dataset (n~140): 1-8 samples per patient A small fraction of samples belonging to the negative class (no tumor), ~ 10-20% A larger fraction of samples belonging to the ...
1
vote
1answer
25 views

Predictions based on k-fold Cross Validation, which model is used (Caret)

I am sorry if there is an obvious or intuitive answer to this, which I missed. We have tuned the hyperparameters of a RF using Grouped 10 - Fold CV (repeated 5 times), to obtain the values for mtry ...
0
votes
1answer
37 views

Modeling non-linear (short) time series and cross-validate them

beginner data scientist here. Time series analysis is a completly new area for me, so please correct me if i write something that makes no sense. I have many multivariante short time series, between ...
1
vote
2answers
34 views

Is it possible to test overfitting with randomized data?

I have built machine learning models for a classification problem with four classes. They run at around 70% nested cross validation accuracy. I am looking to do further testing to check of ...
0
votes
0answers
25 views

Way to stop model from overfitting in automated training pipeline?

I'm currently training a gradient boosting model for which I want to create an automated training pipeline containing hyperparameter optimization with hyperopt and also cross-validation. While trying ...
0
votes
0answers
13 views

PLSDA Validation Methods - Metabolomics [closed]

I'm both new to this forum and to the ways of statistical analysis. Bear with me. I'm working on a metabolomic data set in https://www.metaboanalyst.ca/ and using cross validation methods to determine ...
0
votes
1answer
36 views

Grouped 7-fold Cross Validation in R

I am searching for a grouped 7-fold cross validation function. I couldn't find it in the caret package. I got 70 subjects performing 7 trials (Outcome variable: categorical with 7 values) = 490 ...
0
votes
0answers
8 views

Error in the mean of uncertain results from cross validation folds

I have a dataset and some candidates I want to make predictions for. I have done 5-fold cross validation to assess the model's performance. However, when I make predictions for the new samples I am ...
0
votes
1answer
31 views

Splitting data into test/train set vs. using k-fold cross validation

So, I am working on a binary classification problem (using R) and I am having some confusion on when/how to use data splitting and k-fold cv. I have about 50 labeled samples and I want to train ...
4
votes
0answers
29 views

Predictive Accuracy of a Survival Model using Concordance

Is there a preferred approach for evaluating the predictive power of a survival regression model (e.g., Weibull Accelerated Failure Time)? The metric of choice, for now, is concordance or c-index. I'm ...
4
votes
2answers
833 views

Can K-fold cross validation cause overfitting?

I am learning $k$-fold cross validation. Since each fold will be used to train the model (in $k$ iterations), won't that cause overfitting?
2
votes
1answer
50 views

What is after doing k-fold cross-validation?

When we do K-fold cross validation, we are testing how well our model is able to get trained by some data and then predict data it hasn't seen. I selected 9 fold for training, and 1 fold for ...
0
votes
0answers
27 views

Feature importance with nested cross validation?

I am appplying nested cross validation to estimate the performance of different models used for a regression task. The approach I am using for training-validation-testing is described in the pic below:...
1
vote
0answers
12 views

Splitting Data With High Percentage of Repeated Values [duplicate]

I am building a model in which my data set has a high percentage of repeated values. I am concerned that if I do traditional hold-out or k-fold cross-validaton, that I will get unreliable results as ...
0
votes
0answers
23 views

Are there any evaluation tests for LASSO logistic regression?

As we have Hosmer lemshow test,McFadden R2, Concordance tests for Logistic regression. Do we have any such tests for evaluating the performance of LASSO?
0
votes
0answers
17 views

optimising algorithms in R for regression parameters

I am seeking more guidance than actual answer and if someone can point me to the right direction, it would be great. Suppose I have an equation of loss which is ...
0
votes
0answers
29 views

Cross validation for glm gives AUC 1, but predictor values overlap between groups

I want to apply cross validation for a glm and get a classifier evaluation. Two variables in my dataset have identical values, but prediction for them is different and that results in perfect ...
1
vote
1answer
15 views

Testing set accuracy by using cross validation using xgboost with caret

I am working on an xgboost model using caret. I'm using cross validation, but don't know if I'm understanding it correctly. As I understand, it creates multiple training and test sets. Does this mean ...
0
votes
0answers
9 views

How To Use Both Early Stop And Cross validation to do variable selection and parameter selection

Here this the case I met, I was training GBM Classifier for a binary classification problem, I need to do forward stepwise variable selection and then do parameter selection. Both step I user CV as ...
0
votes
0answers
18 views

Not splitting test set and just using competition public score

With relatively small (~700) training set, would it be better if I train model with XGboost+k-fold validation and just use public score to evaluate the model's performance on unseen data? I wonder if ...
0
votes
1answer
28 views

Cross validation best practice for competition purpose

I'm fairly new to DS scene and I have been learning about theories and doing practices on kaggle/participate in private competition. For real world problems, my understanding is that you split out ...
0
votes
0answers
12 views

Estimating k-means performance by sub-sampling

I have 200 similar images, each with approximately 64000 pixels. I have run k-means on the average of those 200 to segment the average image into 8 clusters. Now, to measure the stability of ...
4
votes
2answers
253 views

Should I cross-validate metrics that were not optimised?

I want to compare two models. Say I have two objective functions on the same data $f(X,y,\theta)$ and $g(X,y,\theta)$ that both evaluate the models performance in ways that I am interested in ($\theta$...
0
votes
0answers
25 views

Is the use of Nested Cross Validation and train- test CV necessary or an overkill?

I have been relatively obsessed lately in the proper way of selecting a model (including tuning hyper parameters) and then assessing model performance. I have read various posts and the approach I ...
0
votes
0answers
20 views

Difference between % Var explained and R2 in Random Forest model?

I am learning how to use Random Forest to predict forest biomass(AGB) in R. Which values should I choose to evaluate the fitted model? the OOB % Var explained, the correlation or R2 of predicted ...
0
votes
0answers
21 views

Train a neural network over full-dataset after crossvalidation in Matlab

I am working on artificial neural networks using MATLAB for application in Movement Analysis. I am trying to implement a K-fold CV procedure to evaluate a model for movement classification. My Dataset ...
0
votes
0answers
16 views

What is the difference between high variance in k-fold cross validation and high variance in machine learning model?

I have this question because of what is mentioned on a book: Does this high variance mean that the model you are evaluating has high variance (overfitting)?
0
votes
0answers
13 views

Bias and variance in the model o in the predictions?

This topic confuses me. In the literature or articles, when talking about bias and variance in automatic learning, specifically in cross-validation, do they refer to the high bias (underfitting) and ...
0
votes
0answers
19 views

Train, validate and test on different folds

To elaborate on the title. Does it make sense to do the following: Split the data into train and test set. Cross-validate on the training set. Remove redundant variables, tune parameters of the model ...
2
votes
2answers
28 views

How should I perform cross-validation of 10-fold?

Suppose that the circles in red are negative samples and the circles in blue are positive samples and that the green boxes are the validation set and the blank boxes are the training set. In addition, ...
1
vote
1answer
25 views

Can MAD (median absolute deviation) or MAE (mean absolute error) be used to calculate prediction intervals?

From my understanding, RMSE (root mean square error) estimated through cross-validation can be used to calculate the prediction interval of a mixed-effect linear model with gaussian error. In my case, ...
0
votes
0answers
18 views

Kfold validation of skewed data

I'm working on a data which is skewed,I was trying to use cross validation but since most of my data is in just 1 of 6 classes when using cross validation some folds don't contain samples from all 6 ...
2
votes
1answer
31 views

Elastic net/LASSO with soft labels

Sometimes you do not have firm Y/N labels, but e.g. 80% probability of Y as a label. E.g. this happens, if you train a model on a small amount of labelled data, predict for a large amount of ...
0
votes
0answers
19 views

How does MAE as objective function impact gradient boosting training compared to MSE?

I have a regression problem where I want to minimize MAE as a business metric. I'm using LightGBM. I initially used the default objective function for regression ...
1
vote
0answers
17 views

Why does leave one out cross validation have less bias compared to k fold cross validation but higher variance? [duplicate]

I was reading ISLR and came across this claim. Can someone elaborate mathematically.
1
vote
0answers
14 views

How to deal with training data after cross validation?

I have imbalanced data and used undersampling to construct several logistic regression models, using the way very similar to EasyEnsemble. The parameters, like regularization, number of models were ...
0
votes
1answer
24 views

Cross Validation Score difference between test/train individual r-square

I am new to Machine Learning, as part of my learning, I created a Linear Regression Model using two attributes for Boston Housing Price dataset. I am having doubts calculating : MSE and R-Square. So, ...
0
votes
0answers
14 views

Nested cross-validation for performance assessment and simple CV for model building?

I am not sure if I understood Dikran Marsupial in this thread correctly: Dikran Marsupial comment on Jan 22 2019 at 11:38: Dikran suggests that nested CV is only used for performance estimates of ...
0
votes
0answers
11 views

How important are hyper-parameters in SVM based RFE feature selection?

How important are hyper-parameters in svm based RFE feature selection? For feature selection using RFE (recursive feature elimination / selection), I have seen some publications where only "external" ...
0
votes
0answers
8 views

Cross Validation with Panel Data of Variable Length

I have a problem of wanting to predict $y_t$ from $x_t$ and some lags given a series of $\{(y_t, x_t)\}_{t=1}^T$. To build my prediction algorithm I was hoping to use cross validation techniques such ...
0
votes
1answer
13 views

Feature selection on full training set, does information leak if using Filter Based Feature Selection or Linear discriminate analysis?

In order to test a potential classification set, usually some data is kept as a holdout set, and not used for inner-cross-validation or model training. However, what happens if too many features ...
0
votes
0answers
19 views

Time series data validation error is significantly lower than training error

I have a time series dataset that covers daily observations (closing price) for several stocks, and I would like to build models to forecast the closing prices for the future 7 days using their ...
0
votes
1answer
25 views

Validation method with outliers

I have a method for prediction in a regression problem with outliers. I'd like to make a validation of my approach. How to make it considering that I need to have outliers in train and test datasets?
1
vote
0answers
19 views

Ideal score of a model on training and cross validation data

The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here. I have trained a classifier model for binary classification task. Now ...
2
votes
0answers
26 views

Is cross validation a general method for evaluating any kind of model?

Can I apply CV to models other than those from machine learning or statistical methods such as a biophysical energy balance model? k-fold cross-validation is used to evaluate models built by ...
1
vote
0answers
10 views

MSE series derived from cross validation on time series

This answer suggests a way of doing leave-one-out cross-validation on time series data: An approach that's sometimes more principled for time series is forward chaining, where your procedure ...
1
vote
1answer
20 views

Which cross-validation measure of model fit performs best when the objective is probability estimation in classification tasks?

Suppose a binary outcome $Y=0$ or $Y=1$ where $P(Y=1|X)=f(X)$ is a function of $X$. The goal is to estimate $f$ as closely as possible using a classifier that returns a probability estimate (e.g. ...
0
votes
0answers
9 views

Cross validation for timeseries interpolation

I am trying to calculate the MSE from interpolating few data points obtained from a sensor. I am considering 100 datapoints and applying 10 fold cross validation to it. For explaining what i am doing ...

1 2 3 4 5 … 48