New answers tagged cross-validation
0
votes
Difference K-fold versus Blocked Cross-Validation?
Yes, the default k-fold splitter in sklearn is the same as this 'blocked' cross validation. Setting shuffle=True will make it ...
- 264
0
votes
How to statistically combine regression coefficients derived from subsamples of data
Not sure this will answer your question, but to my understanding I would do this:
If I want to find the relationship between variables and target/response, then following are the methods I am aware of
...
- 1
2
votes
Does cross validation reduce underfitting?
The short answer is No. Cross validation does not "reduce the effects of underfitting" — or overfitting, for that matter.
I agree with the comments that your question seems to miss the point ...
- 264
0
votes
How do I access the p-values of individual predictors using caret::train?
This may be what you're after:
summary(model$finalModel)
1
vote
What are "volatile" learning curves indicative of?
This means that your optimization hasn't really settled, yet. There could be many causes for that. If you are sure that the validation set is from the same population as the training set, then the ...
- 3,708
0
votes
Accepted
Cross Validation on whole data for model comparision
Part of your approach is right: you should divide your dataset into train and test by using stratified sampling. However, the test set should remain unseen during the modelling process to make sure ...
- 16
1
vote
Splitting medical dataset by patient
There's no whatsoever problem with unbalanced test folds (other than that you need to think how to properly aggregate the results - on scan vs. on patient level - but that's a consequence of the data ...
0
votes
On the importance of the i.i.d. assumption in statistical learning
One area where i.i.d. assumption is critical in practice (other that inference) is data collection. If you do not collect data in a random manner then you will have a sampling bias and your data will ...
- 933
0
votes
Cross Validation for Time Series Classification (Not Forecasting!)
I understand that your problem is that your dataset is a time series but that you are not interested in making forecast (i.e. making prediction of future state from past state). In this case, you ...
- 1
4
votes
Model Selection: AIC/BIC and Cross-Validation gives different conclusion
AIC is asymptotically equivalent to leave-1-out cross-validation (LOOCV)
It's not equivalent to 10-fold cross-validation, which is what you're comparing it to.
It's only asymptotically equivalent, so ...
- 5,541
10
votes
Model Selection: AIC/BIC and Cross-Validation gives different conclusion
Nothing strange in here.
If all the model selection methods always gave the same results, we wouldn't have multiple criteria, but just pick arbitrary one.
AIC and BIC explicitly penalize the number ...
Tim♦
- 115k
4
votes
Model Selection: AIC/BIC and Cross-Validation gives different conclusion
Maybe you should concentrate more on the methods that are intended precisely for feature selection, rather than model selection. Model selection methods like cross-validation or AIC try to compare ...
- 3,708
0
votes
High variance of leave-one-out cross-validation
There are two "kinds" of variance in LOOCV. one is the variance in the result, and another is the variance in the model...
- 1
0
votes
Does it make sense to use data augmentation on the Validation set? (note, this is not the same as asking to augment the test set)
You can use augmentation data in training, validation and test sets.
The only thing to avoid is using the same data from the training set in validation or test sets.
For example, if you generate 3 ...
- 143
0
votes
Accepted
Is cross-validation with no data leakage sufficient to replace train-test split?
The confusion might stem from a clash of workflows. Your workflow includes a step Repeat with different hyperparameter set until performance of train is near to validation whereas the typical workflow ...
- 56.4k
1
vote
Accepted
Perform Rose Method, then Logistic Regression and do k -fold cross validation
As some links I posted discuss, class imbalance usually is not a problem. Therefore, attempts to solve a non-problem are somewhere between superfluous and damaging. Consequently, it is difficult to ...
- 32.6k
0
votes
use Cross-validation to find the better model
What bothers me the most about the first model is that you have two variables (Post_attendance1, Discussion) with a very high p-value. You should remove them, and then, you basically get the same ...
- 11
0
votes
Why using cross validation is not a good option for Lasso regression?
Here is a very simple explanation of why there is a difference between modeling for research and modeling for prediction. I'll get into how this relates to cross-validation and Lasso by the end.
Let's ...
- 1
0
votes
10-fold Cross-validation vs leave-one-out cross-validation
The existing answers focus on getting good estimates of the out of sample prediction error. This is not the only perspective on the LOOCV versus K-fold CV decision. In particular, some readers may ...
- 5,056
Top 50 recent answers are included