I am comparing several methods of ordering training patterns for on-line neural network training. Let's call those methods I-VII.
Then, I have a sample of 20 data sets within a given "data set type" - for instance, with patterns distributed uniformly, let's call that "Uniform Set Type". In other words, I will have 20 randomly generated instances of this type of data set.
Now, for each ordering method I-VII I let it run 50 times on each instance of a data set type. This would give 20 instances x 50 repetitions = 1000 runs total per each method I-VII, forming the whole body of knowledge about performance over "Uniform Set Type" of the methods under test.
The measure of performance I use is MSE. In addition, I also compute another measure for each data set instance like this:
- See what was the single lowest MSE achieved on this data set instance by any method I-VII, during any of the 50 repetitions
- Divide the MSEs by the lowest MSE. This gives me values starting at 1.0, corresponding to the lowest error. Let's call this new measure "Relative error".
With this data, I see two ways to compare the methods:
- Calculate mean and SD of MSE over all data set instances, all repetitions for each method (sample size 20 x 50 = 1000), then use a statistical test to compute p-values concerning whether the mean MSE between any two methods is different
- Calculate mean and SD of MSE for each data set instance (20 means and SDs, each based on a sample size of 50, since 50 repetitions are performed per data set instance), and then use a statistical test that would tell whether there is a significant difference between methods based on comparison of respective results over each data set instance.
The problem with approach (1) is that if a method achieved a big bad error on one of the data set instances, this will significantly affect the single mean used for the test, assuming I use pure MSE and not the "Relative error". This is because the MSEs over data set instances might lie anywhere between 0.0001 and 2.0. I can give an example how this would be a problem if asked; omitting it now for brevity.
To solve this problem, I could use the "Relative error". Or I could simply take approach (2) - this time, it seems, there would be no such problem with either pure MSE or "Relative error" - since I am comparing the results instance by instance, which intuitively feels like the better approach to me.
My questions, ranked most important to least, are these:
- Which approach, (1) or (2) is better suited to compare the methods?
- Which statistical tests should I use for approaches (1) and (2)?
- Is there any other approach you would recommend in this scenario?
- Any opinion regarding whether MSE or "Relative error" might be the more valuable measure under any of the approaches?