New answers tagged

0 votes

Difference K-fold versus Blocked Cross-Validation?

Yes, the default k-fold splitter in sklearn is the same as this 'blocked' cross validation. Setting shuffle=True will make it ...
user avatar
  • 264
0 votes

How to statistically combine regression coefficients derived from subsamples of data

Not sure this will answer your question, but to my understanding I would do this: If I want to find the relationship between variables and target/response, then following are the methods I am aware of ...
user avatar
  • 1
2 votes

Does cross validation reduce underfitting?

The short answer is No. Cross validation does not "reduce the effects of underfitting" — or overfitting, for that matter. I agree with the comments that your question seems to miss the point ...
user avatar
  • 264
0 votes

How do I access the p-values of individual predictors using caret::train?

This may be what you're after: summary(model$finalModel)
user avatar
1 vote

What are "volatile" learning curves indicative of?

This means that your optimization hasn't really settled, yet. There could be many causes for that. If you are sure that the validation set is from the same population as the training set, then the ...
user avatar
  • 3,708
0 votes
Accepted

Cross Validation on whole data for model comparision

Part of your approach is right: you should divide your dataset into train and test by using stratified sampling. However, the test set should remain unseen during the modelling process to make sure ...
user avatar
1 vote

Splitting medical dataset by patient

There's no whatsoever problem with unbalanced test folds (other than that you need to think how to properly aggregate the results - on scan vs. on patient level - but that's a consequence of the data ...
user avatar
0 votes

On the importance of the i.i.d. assumption in statistical learning

One area where i.i.d. assumption is critical in practice (other that inference) is data collection. If you do not collect data in a random manner then you will have a sampling bias and your data will ...
user avatar
  • 933
0 votes

Cross Validation for Time Series Classification (Not Forecasting!)

I understand that your problem is that your dataset is a time series but that you are not interested in making forecast (i.e. making prediction of future state from past state). In this case, you ...
user avatar
4 votes

Model Selection: AIC/BIC and Cross-Validation gives different conclusion

AIC is asymptotically equivalent to leave-1-out cross-validation (LOOCV) It's not equivalent to 10-fold cross-validation, which is what you're comparing it to. It's only asymptotically equivalent, so ...
user avatar
  • 5,541
10 votes

Model Selection: AIC/BIC and Cross-Validation gives different conclusion

Nothing strange in here. If all the model selection methods always gave the same results, we wouldn't have multiple criteria, but just pick arbitrary one. AIC and BIC explicitly penalize the number ...
user avatar
  • 115k
4 votes

Model Selection: AIC/BIC and Cross-Validation gives different conclusion

Maybe you should concentrate more on the methods that are intended precisely for feature selection, rather than model selection. Model selection methods like cross-validation or AIC try to compare ...
user avatar
  • 3,708
0 votes

High variance of leave-one-out cross-validation

There are two "kinds" of variance in LOOCV. one is the variance in the result, and another is the variance in the model...
user avatar
0 votes

Does it make sense to use data augmentation on the Validation set? (note, this is not the same as asking to augment the test set)

You can use augmentation data in training, validation and test sets. The only thing to avoid is using the same data from the training set in validation or test sets. For example, if you generate 3 ...
user avatar
  • 143
0 votes
Accepted

Is cross-validation with no data leakage sufficient to replace train-test split?

The confusion might stem from a clash of workflows. Your workflow includes a step Repeat with different hyperparameter set until performance of train is near to validation whereas the typical workflow ...
user avatar
1 vote
Accepted

Perform Rose Method, then Logistic Regression and do k -fold cross validation

As some links I posted discuss, class imbalance usually is not a problem. Therefore, attempts to solve a non-problem are somewhere between superfluous and damaging. Consequently, it is difficult to ...
user avatar
  • 32.6k
0 votes

use Cross-validation to find the better model

What bothers me the most about the first model is that you have two variables (Post_attendance1, Discussion) with a very high p-value. You should remove them, and then, you basically get the same ...
user avatar
0 votes

Why using cross validation is not a good option for Lasso regression?

Here is a very simple explanation of why there is a difference between modeling for research and modeling for prediction. I'll get into how this relates to cross-validation and Lasso by the end. Let's ...
user avatar
  • 1
0 votes

10-fold Cross-validation vs leave-one-out cross-validation

The existing answers focus on getting good estimates of the out of sample prediction error. This is not the only perspective on the LOOCV versus K-fold CV decision. In particular, some readers may ...
user avatar

Top 50 recent answers are included