How do I test which of two algorithm is perceived better? I am trying to evaluate which of two algorithms is preferred by people. The algorithms generate images and I've shown the generated images to different people and asked them to vote on whether they prefer the result from Algorithm 1 or Algorithm 2 (1 vote each). I remember learning about this in school but can't remember what the "statistically sound" way is to tell if one algorithm is actually preferred over the other or not. The data I collected looks like this.
          Algorithm #1 Algorithm #2
Image #1  4            2
Image #2  6            5
Image #3  3            3
Image #4  8            12
Image #5  1            4

How do I compute if Algorithm #1 is better than #2 (as perceived by the user)? 
Does it matter that some different people have voted on each image?
 A: Fundamentally, I think you are asking if there is a statistically sigificant difference between the preference for the two algorithms (Potentially given a number of factors like the image being scored, and the person scoring the algorithm). 
To keep this simple, I might propose to try to test the following null hypothesis: 
The expected odds that a person prefers algorithm 1 is no different than the expected odds they prefer algorithm 2. I provide some R code to test this hypothesis based on the numbers you gave.
#Generate data
rawResults <- data.frame(Image <- c(1:5), PreferAlg1 = c(4,6,3,8,1), TotalVoters = c(6,11,6,20,5))

#create a long form of this for logistic regression. If a person preferred alg1, then PreferAlg1 gets a 1, otherwise a 0
longResults <- data.frame(Image = c(), PreferAlg1 = c())
for(i in 1:nrow(rawResults)){
  #the people who prefer alg 1
  preferA1 <- data.frame(Image = i, PreferAlg1 = rep(1, rawResults$PreferAlg1[i]))
  #the people who prefer alg2
  preferA2 <- data.frame(Image = i, PreferAlg1 = rep(0, (rawResults$TotalVoters[i]-rawResults$PreferAlg1[i])))
  tempResult <- rbind(preferA1, preferA2)
  longResults <- rbind(longResults, tempResult)
}

#next, run a logistic regression to see if a person's preference for alg1 is significantly different than their expected preference for alg2, where preference here is expressed in odds.

logitFit <- glm(formula = PreferAlg1 ~ 1, family = binomial(link = "logit"), data = longResults)

summary(logitFit)

I do not find that there is a significant difference.
This is a simplistic test. I think that an alternative, like a t-test would be asking the wrong question since it would ask about the mean vote per algorithm. Logistic regerssion asks something similar, but it comes closer to what we are interested in, which is "the chance someone picks an algorithm." Logistic regression will also more naturally extend to include the interviewer information.
Regarding your question about whether it matters who voted on what question, this is a good point. There might be a pattern in their answering between images. This could be included using a random effect, but since the data for this is not posted, I have not run that analysis. Accounting for the voter effect will improve the precision of your estimates on the main effect of interest: the power of alg1 vs alg2.
It could also be interesting to test if there is a difference in the performance of the algorithms on the different pictures. Do you see a pattern there? For this, I think it would be useful to have the voter IDs as a random effect again.
