Problem background: As part of my research, I have written two algorithms that can select a set of features from a data set (gene expression data from cancer patients). These features are then tested to see how well they can classify an unseen sample as either cancer or non-cancer. For each run of the algorithm, a solution (a set of features) is generated and tested on XZ unseen samples. Percentage accuracy of the solution is calculated like this: (correct classifications / XZ) * 100.
I have two algorithms: algorithm AX & algorithm BY
I have run each algorithm 10 times on each data set. So, algorithm AX has 10 results from data set A, 10 from data set B and 10 from data set C. Overall, algorithm AX has 30 results.
My problem: I would like to see if algorithm A'sX's combined performance across all three data sets is statistically significantly different from algorithm B'sY's combined performance.
Is it possible for me to combine results for algorithm AX from each data set into a single set of results? This way, I would have 30 standardized results for algorithm AX and 30 standardized results for algorithm BY. I can then use the t-test to see if there is a significant difference between the two methods.
**Edit -Edit - These are Evolutionary Algorithms, so they return a slightly different solution each time they are run. However, if there's a feature in a sample that when present can strongly classify the sample as either being cancer or non-cancer, then that feature will be selected almost every time the algorithm is run.
This is the reasonI get slightly different results for repeating each experimentof the 10 times. Then you can do stats forruns due to the following reasons:
- These algorithms are randomly seeded.
- I use repeated random sub-sampling validation (10 repeats).
- The datasets that I use (DNA microarray and Proteomics) are very difficult to work with in the sense that there are many local optima the algorithm can get stuck in.
- There are lots of inter-feature and inter-subset interactions that I would like to detect.
- I train 50 chromosomes and they are not restricted to any particular length. They are free to grow and shrink (although selection pressure guides them towards shorter lengths). This also introduces some variation to the final result.
Having said, the algorithm almost always selects a data set. E.g.particular subset of features!
Here's a sample of my results (only 4 runs out of 10 for data set A, Ieach algorithm is shown here):
Dataset/run Algorithm X Algorithm Y
A 1 90.91 90.91
A 2 90.91 95.45
A 3 90.91 90.91
A 4 90.91 90.91
B 1 100 100
B 2 100 100
B 3 95.65 100
B 4 95.65 86.96
C 1 90.32 87.10
C 2 70.97 80.65
C 3 96.77 83.87
C 4 80.65 83.87
As you can calculate the meansee, standard deviation and standard errorI've put together results for algorithm Atwo algorithms from three datasets. I can do the same for algorithm B.Kruskal-Wallis test on this data but will it be valid? I can thenask this because:
- I'm not sure accuracies in different data sets are commensurable. If they are not, then putting them together like I've done would be meaningless and any statistical test done on them would also be meaningless.
- When you put accuracies together like this, the overall result is susceptible to outliers. One algorithm's excellent performance on one dataset may mask it's average performance on another dataset.
I cannot use t Test to compare algorithm A with algorithm B-test in this case either, which I've done already.this is because:
- Commensurability - the t-test only makes sense when the differences over the data sets are commensurate.
- t-test requires that the differences between the two algorithms compared are distributed normally, there's no way (at least that I'm aware of) to guarantee this condition in my case.
- t-test is affected by outliers which skew the test statistics and decrease the test’s power by increasing the estimated standard error.
My problem is that I would like to know the performance of algorithm A compared to algorithm B across all three data sets. IWhat do you think? How can then conclude whichI do a comparison between algorithm is better for feature selection and classification for these data sets.**X & Y in this case?
