1
$\begingroup$

In WEKA Experimenter I have a problem that seems counterintuitive and I would like to know whether I correctly interpret the results.

My experiment specifics:

  • Type of experiment: Regression Analysis
  • Data: A small, all numerical, dataset of 73 instances.
  • Feature Selection: Multiple datasets have been set up, each with a different feature selection technique.
  • Techniques: Linear Regression, Multilayer Perceptron and Support Vector Machine.
  • Validation: Leave-One-Out Cross-Validation
  • Runs: 10

After finishing the experiment in WEKA I get the following results when I test for the Mean Absolute Error (MAE):

Results WEKA Experimenter

Some information on the table: the first function is the Linear Regression, the second is the Multilayer Perceptron and the last one is the Support Vector Machine.

In the book Data Mining Practical Machine Learning Tools and Techniques, I've read the following:

The symbol placed beside a result indicates that it is statistically better (v) or worse (*) than the baseline scheme at the specified significance level (0.05, or 5%).

Yet, when looking at the above table, for example on the second line, I see that my baseline - the Linear Regression - performs worse in terms of MAE when compared to the other two techniques. However, the WEKA Experimenter environment is telling me the other two techniques are worse off compared to the baseline (i.e. the difference is statistically significant).

Am I interpreting this result wrongly? Or does this really seem counterintuitive? I'd like to know a bit more about the reasoning behind this if possible.

$\endgroup$
1
$\begingroup$

You are interpreting it correctly, I have recently faced the same case where Weka tells you that it is better a model with higher error. I have researched about it but I couldn't find any explanation.

As far as I'm concerned the lower the error the better the model thus Weka is doing it wrong, it just considers the higher the better even for the errors.

When facing this problem just use your knowledge and don't trust Weka because I think it's wrong.

$\endgroup$
  • $\begingroup$ If anyone could find an explanation I'll be very interested in understanding this issue $\endgroup$ – davidivad Nov 13 '15 at 12:23
  • $\begingroup$ Thanks for the answer, I've noticed some different issues with the Weka Experimenter as well (e.g. results between the Explorer & Experimenter mismatching where the Explorer seemed to have it correct). $\endgroup$ – Floris Devriendt Nov 14 '15 at 15:34
  • $\begingroup$ I guess we should use the Expermineter carefully and trust the Explorer for getting the final results $\endgroup$ – davidivad Nov 14 '15 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.