In WEKA Experimenter I have a problem that seems counterintuitive and I would like to know whether I correctly interpret the results.
My experiment specifics:
- Type of experiment: Regression Analysis
- Data: A small, all numerical, dataset of 73 instances.
- Feature Selection: Multiple datasets have been set up, each with a different feature selection technique.
- Techniques: Linear Regression, Multilayer Perceptron and Support Vector Machine.
- Validation: Leave-One-Out Cross-Validation
- Runs: 10
After finishing the experiment in WEKA I get the following results when I test for the Mean Absolute Error (MAE):
Some information on the table: the first function is the Linear Regression, the second is the Multilayer Perceptron and the last one is the Support Vector Machine.
In the book Data Mining Practical Machine Learning Tools and Techniques, I've read the following:
The symbol placed beside a result indicates that it is statistically better (v) or worse (*) than the baseline scheme at the specified significance level (0.05, or 5%).
Yet, when looking at the above table, for example on the second line, I see that my baseline - the Linear Regression - performs worse in terms of MAE when compared to the other two techniques. However, the WEKA Experimenter environment is telling me the other two techniques are worse off compared to the baseline (i.e. the difference is statistically significant).
Am I interpreting this result wrongly? Or does this really seem counterintuitive? I'd like to know a bit more about the reasoning behind this if possible.
