In the example you give, only one person has reviewed and given a score of 5/5. At this point, I would say you don't have enough information to give an informative estimate of the mean (or median). Possible scores are 1,2,3,4, or 5, so all you could say is that the mean average is somewhere between 1 and 5 and that one person on planet earth really likes the book.
However, if you have more people review, you can construct a confidence interval for that true mean review score. That way you could give a confidence level and some upper and lower bounds for the rating. (e.g. 95% confident that the book's rating is between 4.2 and 4.8). These bounds become tighter the more reviewers you have, so they do take into account the number of scores received.
However, typical Gaussian based confidence interval theory only holds up when you have a random sample from some population. Here the population is not well defined, perhaps those people who have bought the book through that website. Also, I would not say online reviewers are a random sample at all. I've found that book reviews (as with many online reviews) attract those people at the extremes who either love or hate the product. But perhaps it's best not to dwell too much on these issues...
I think what you're hinting at is the idea that if one person gave a book 5/5, this should probably not be considered better than an average of, say, 4.5/5 that's been reviewed by 200 people. And you mentioned "average", so perhaps you just want a one number summary that can be sorted easily.
I'm not too familiar with the Wilson score interval, but it looks like it is similar to the gaussian confidence interval, but it's construction is based on the score statistic.
You might want to look into some kind of weighted average that penalizes you for having a small sample.