Skip to main content

New answers tagged

4 votes

Train-validation-test split for small and unbalanced dataset?

A useful way to quantify the difficulty of the task is to compute the effective sample size as discussed here. Here the ESS is $3np(1-p)$ where $p=0.07$; ESS=19.5. With that amount of information ...
Frank Harrell's user avatar
4 votes

Train-validation-test split for small and unbalanced dataset?

I agree with Stephan Kolassa generally. If you have only 7 observations from one class, the basis for hyperparameter selection and performance assessment is severely limited. Your data only carries a ...
Christian Hennig's user avatar
8 votes

Train-validation-test split for small and unbalanced dataset?

However, my number of class 1 rows is so low that the way they get shuffled into validation or test set causes huge fluctuations in performance metrics. Here is another way of looking at things: your ...
Stephan Kolassa's user avatar
1 vote

Proper usage of K-fold cross validation and finalizing model

Yes, when you do cross validation, you should not include the test data in this procedure. The test data is for your evaluation and reporting loss, accuracy, etc. computed on this set makes most sense....
picky_porpoise's user avatar
1 vote
Accepted

Is my understanding/approach to nested cross-validation, final model tuning correct?

I want to make a statement of the generalizability of the approach to different independent training and testing datasets [...] my understanding is that I would do nested cross-validation. I think ...
MuhammedYunus's user avatar
4 votes
Accepted

How to interpret the results of a classifier when train/test method gives much better results than cross validated one?

What does these varying scores represent, particularly the low scores of cross validation? Together, they represent the fact that error estimtes based on a small number of tested cases are highly ...
cbeleites unhappy with SX's user avatar
3 votes

Bayesian Justification of Cross-validation

Several ideas could serve to justify model selection based on the ELPD instead of the model-posterior. Direct relation to frequentist-like MSE comparison for regression models with Gaussian ...
Johan de Aguas's user avatar

Top 50 recent answers are included