0
$\begingroup$

I'm working on a non-standard dataset with deep learning, and I want to prove that my method is better.

We have a dataset composed from dataset A and dataset B where each one is composed of biological measurements from 10 subjects (the same ones).

In each run, we merge randomly 5 subjects dataset for training from A and we keep 4 to validation from A. The test dataset is from dataset B and not related to the same subject.

Ex: we test subject 2 so : training: 3,5,8,6,1 validation : 4,7,9,10 test : 2

The process is repeated 10 times for each subjects and we average the accuracies for each classifier (2 sets of accuracies). The accuracies of the classifiers (my model and the baseline) refute the normal hypothesis.

Now, we want to compare the 2 classifiers.

My questions are:

  • Is the data considered paired or unpaired?
  • Which statistical test should I run?
  • Is alpha = 0.1 seem acceptable?
$\endgroup$
0
$\begingroup$

It is hard to say whether it is paired or unpaired. It depends on the relation between your dataset A and B. If these are the same subjects then it is paired. In either case, there are lots of different options for statistical tests on classification results. There is an overview by Raschka that summarizes the common ones: https://arxiv.org/pdf/1811.12808.pdf

4.2 TestingtheDifferenceofProportions......................... 34
4.3 ComparingTwoModelswiththeMcNemarTest................... 35
4.4 Exactp-ValuesviatheBinomialTest......................... 37
4.5 MultipleHypothesesTesting ............................. 38
4.6 Cochran’s Q Test for Comparing the Performance of Multiple Classifiers . . . . . . 39
4.7 TheF-testforComparingMultipleClassifiers..................... 41
4.8 ComparingAlgorithms ................................ 42
4.9 ResampledPairedt-Test ............................... 43
4.10k-foldCross-validatedPairedt-Test ......................... 44
4.11 Dietterich’s 5x2-Fold Cross-Validated Paired t-Test . . . . . . . . . . . . . . . . . 44
4.12Alpaydin’sCombined5x2cvF-test ......................... 45
4.14NestedCross-Validation ............................... 45

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ 1) Are those compatibles with non-normal distributions? 2) the training dataset and validation dataset and test dataset doesn't include the same subjects. Are you sure that it is paired? $\endgroup$ – zeronoid Mar 16 '19 at 22:15
  • $\begingroup$ "Are those compatibles with non-normal distributions" -> Of course. They are for 0/1 loss (i.e., classification error or accuracy). " the training dataset and validation dataset and test dataset doesn't include the same subjects. Are you sure that it is paired?" => then it's not paired. In my answer, I just mentioned a hypothetical scenario. Somehow, "dataset A and dataset B where each one is composed of biological measurements from 10 subjects (the same ones)." made me think you meant pairs of subject (i.e., before after treatment) $\endgroup$ – resnet Mar 17 '19 at 1:46
  • $\begingroup$ Could you recheck the question, I added some modifications. just to make sure $\endgroup$ – zeronoid Mar 17 '19 at 11:21
  • $\begingroup$ "Now, we want to compare the 2 classifiers." ==> in this case, this is a paired setting. I would not recommend a paired t-test for this though. your data is too small, and normal approximation assumption also does not work well for this. you can try the dietterich's CV paired t-test. there is a python implementation online linked in that article $\endgroup$ – resnet Mar 17 '19 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.