Problem Description

I want to enhance a photo and I have 3 software to do that, out of which 2 are my own. I show users 2 photos and ask them to choose a better one. Now, I compare software 1 vs. 2, 1 vs. 3. Software 2 and 3 are mine. Now, I can clearly see from the votes, that people choose 2 and 3, over 1, in both sessions.

Task 1: Show that 2 is better than 1, then 3 is better than 1.

A published paper (not on statistics) handled the same task and used a paired t-test. I can do the same, and show that 2 is the better method. I intend to that in MATLAB as follows:

[h,p]=ttest(votes_for_software1,votes_for_software2) %similarly for 1 vs. 3 

I do the above and I get,

h = 1 and p = 7.2372e-04

This confirms (at least I think so) that a mean value of the observed size or greater magnitude will occur with a probability p (interpretation taken from here).

So different people are looking at same set of photos and voting them, am I right in using a paired t-test? Also, am I doing the overall process right? (including the implementation).

Task 2: Is there any way I can compare software 2 and 3 (both of my software)?

The data compared in the comparison of software 1 vs. 2 and 1 vs. 3 remains same. So, I calculate the mean rank obtained for all the three software (lower the better), and I can see that, rank_3 < rank_2 < rank_1 -> Software 2 of mine is the best.

How could show this statistically?


1 Answer 1


In general, you are getting it right. I would suggest two modifications:

  1. If I understand you correctly, the two values the variables can have is 1 (better picture) and 0 (worse picture). This means that the data type is not really interval/ratio/absolute but rather ordinal. In that case I would use a sign test which test whether the subjects choose one over the other more often than not. It has the additional advantage, that it makes less assumptions about the distribution of the data. For the t-test you either need normally distributed data (not true in your case) or a lot of data points (depends on how many subjects you have).

  2. You are performing two test on the same data. For that reason you should account for multiple comparisons. The simplest way (Bonferroni correction) is to multiply each p-value by two.

One additional thought: What do you do if the subjects rank the data inconsistently, i.e. 1 < 2 < 3 < 1?

Concerning the second task: Why don't you show the subjects photos from 2 and 3 and let the rank which one is better? I think this would be the cleanest way.

  • $\begingroup$ Interesting. Actually I use t-test on the number of votes given by n people. So the input to t-test would be two columns of the voting matrix and hence it would be integer. But if sign test does not make any assumption, I can easily carry out that test with a 0/1 type voting matrix. I have total of 1000 votes (i.e. 1000x2 matrix), if that adds any information. 2. Can you elaborate more on how can I do multiple comparison (regarding implementation)? 3. I cannot repeat the experiments, sorry. 4. The sign test lacks the statistical power when compared to t-test(wiki). What's this? $\endgroup$
    – Autonomous
    Dec 6, 2014 at 21:56
  • $\begingroup$ I don't understand what do you mean by <<What do you do if the subjects rank the data inconsistently, i.e. 1 < 2 < 3 < 1?>>. Users are shown two photos at a time, they have to vote to one of them. Also 1 vs. 2 and 1 vs. 3 are disjoint sessions. $\endgroup$
    – Autonomous
    Dec 6, 2014 at 21:59
  • $\begingroup$ How about Wilcoxon signed-rank test test? $\endgroup$
    – Autonomous
    Dec 6, 2014 at 23:55
  • $\begingroup$ Inconsitent rating: People are shown two photos at a time. The content is the same, but the processing has been done with different software. Now a single user could rate software 1 better than software 2, software 2 better than software 3 and software 3 better than software 1. This is inconsistent since software 3 should be worse than software 1. There is nothing in the experimental paradigm, that excludes this situation. $\endgroup$
    – fabee
    Dec 7, 2014 at 15:09
  • $\begingroup$ Wilcoxon signed-rank test would work as well. $\endgroup$
    – fabee
    Dec 7, 2014 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.