Search Results

.: Performance of Error Estimators for Classification Current Bioinformatics, 2010, 5, 53-67 Beleites, C. et al.: Variance reduction in estimating classification error using sparse datasets Chemom Intell … As regression-type error measures do not have the "hardening" step of cutting decisions with a threshold, they often have less variance than their classification counterparts. …

Both are legitimate concerns about data leakage. (Side note: and it is not that rare to have a situation where multiple factors need to be taken into account for independent splitting, e.g. crossed ra …

 What does these varying scores represent, particularly the low scores of cross validation? Together, they represent the fact that error estimtes based on a small number of tested cases are highly v …

.: Performance of Error Estimators for Classification Current Bioinformatics, 2010, 5, 53-67. is a good starting point. … G.: Variance reduction in estimating classification error using sparse datasets. …

One additional thought, not fully explored: Maybe we should think whether overfitting can amplify existing bias. In a linear model (where we understand better what happens), we observe overfitting wit …

In addition to the excellent points in Camille's and EdM's answers: Purpose of the model/study Whether something sensible can be done or not depends crucially on the purpose of the modeling. For any k …

If your paper is not for a machine learning/classification audience, I'recommend to make this distinction very clear. …

There's nothing to keep you from calculating several metrics, so evaluate all metrics that are relevant for your application. A model is rarely (if ever?) characterized well with a single metric. E.g …

When I look at textbooks on classification and machine learning, many of the examples focus on data that is often twisted up such as to avoid linear separation. I have an example picture below. … The common description of the classification problem is that data can be twisted in this manner, and hence kernel methods or random forests are better at dealing with nonlinear decision boundaries. …

To add a bit to @usεr11852's answer: variance due to finite test sample size in each fold: A fraction of tested cases such as accuracy follows a binomial distribution. You can therefore do a rough ch …

Have a look into one-class classification. … In medical diagostics, discriminative classification (the "usual" classifiers) would typically be appropriate for differential diagnostics. …

As it is just a first data reduction step before the "actual" classification, using a few too many PCs will likely not hurt the performance. … Usually the PCA step is done because you need to stabilize the classification. That is, in a situation where additional cases do influence the model. …

There are some trivial answers to consider that may serve as corner cases: If you have sufficiently few samples (if need be, 0) your model will not only be unstable but even become mathematically imp …

This would allow for pixels to contain mixtures such as half grasses half shrubs - whereas classification models cannot deal well with this situtation. … Let's say I were to pick RandomForest, which I think is one classification algorithm; would I still need to cross-validate my data with a resampling technique? …

Whatever kind of resampling (CV or bootstrap) verification/validation scheme you use, when you know or suspect that your data is clustered such as by organization in your case, you need to split at th …