Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have 3 algorithms (A, B, C) which predict "author name" of each text article. I have tested all the algorithms on 3 different corpora (X, Y, Z) (different number of articles from 3 newspapers). I have got the following result.

The result provides the number of "Correct" and "Incorrect" answers and the "Accuracy" (correct_answer/total) for each Algorithm on each Corpus:

Algorithm A =>    Corpus X: 80%,    Corpus Y: 93%,    Corpus Z: 94%
Algorithm B =>    Corpus X: 78%,    Corpus Y: 90%,    Corpus Z: 89%
Algorithm C =>    Corpus X: 90%,    Corpus Y: 83%,    Corpus Z: 97%

I have to compare the results by T-test and sign-test in order to define which one is the best algorithm. But I do not know which kind of T-test I have to use (independent, two-tailed, paired…).

Also I have heard that I have to use one-way ANOVA instead of t-test.

share|improve this question

migrated from Jul 25 '13 at 2:09

This question came from our site for professional mathematicians.

Maybe this question might be a better fit for stats.SE – Daniel Moskovich Jul 24 '13 at 15:19
I could imagine a one-way ANOVA with just three observations in each group, the "groups" being the three algorithms, BUT the three "errors" in one group don't seem really independent since the same three corpora are used all three times. And the corpora may not be a random sample from any population. So corpora are "crossed" with algorithms, rather than "nested" within algorithms. I'd also want to know how big the samples were: is the 80% four out of five, or maybe 80 out of 100? So maybe just shoving this into a standard algorithm isn't the best thing to do. – Michael Hardy Jul 24 '13 at 15:38
I think this should be "migrated" rather than simply closed. (I don't know how that's done, though. But now that mathoverflow uses stackexchange software, that may be possible?) – Michael Hardy Jul 24 '13 at 15:39
@MichaelHardy Just click "off topic" and click "belongs on another stackexchange site"! – Daniel Moskovich Jul 24 '13 at 15:58
@DanielMoskovich : But it now says "put on hold" instead of "migrated". Have those who run such things on mathoverflow simply not found out about migration yet? – Michael Hardy Jul 24 '13 at 16:00

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.