I'm working on a non-standard dataset with deep learning, and I want to prove that my method is better.
We have a dataset composed from dataset A and dataset B where each one is composed of biological measurements from 10 subjects (the same ones).
In each run, we merge randomly 5 subjects dataset for training from A and we keep 4 to validation from A. The test dataset is from dataset B and not related to the same subject.
Ex: we test subject 2 so : training: 3,5,8,6,1 validation : 4,7,9,10 test : 2
The process is repeated 10 times for each subjects and we average the accuracies for each classifier (2 sets of accuracies). The accuracies of the classifiers (my model and the baseline) refute the normal hypothesis.
Now, we want to compare the 2 classifiers.
My questions are:
- Is the data considered paired or unpaired?
- Which statistical test should I run?
- Is alpha = 0.1 seem acceptable?