Learning how judges rate piano performances - How to learn the correct distribution in a regression problem I'm working on the following problem, but I'm confused as to what kind of problem this is, how to select an appropriate model, and what metric I should be optimizing.
The problem is as follows:
Team of experts is given a recording of someone performing a piano piece.  As a group, they give the recording a score from 1 - 100.  There is lots of data (100k+) in the form (audio of recording $x^{[i]}$, score $y^{[i]}$).  However, the experts aren’t perfectly reliable.  There is a smaller data (2) set of the form (recording number $x^{[i]}$,  , score $y^{[i]}$) where for each recording number, I have about 20 different scores.  For this smaller data set, many groups of experts were given the same recordings.  These scores seem to follow a binomial distribution, with a different mean for each recording number.
I want to learn the mapping from an audio recording $x^{[i]}$ to a score distribution over $y^{[i]}$ that would match the distribution formed if the experts were given this recording many times to evaluate.
Currently, I’m training a neural network to do a regression problem and get a mean squared error of about 7.0 on the validation set.  By treating the problem as a classification problem instead I’m able to get a probability distribution.  When I compute the expected value of each distribution for every validation example, and use the expected value to compute mean squared error, I get a mse of about 7.5.
However, this still all feels wrong to me as


*

*Using classification gives me a distribution, but doesn’t capture the linear nature of the data

*Using regression captures the linear nature of the data, but doesn’t give a distribution

*I’m not even sure I have the right set of data to compute this distribution in the first place.


Any advice on how to set up this problem, how to better ask this question, and which metric I should use?
 A: I have only suggestions, and no experience with this. But you can parametrize the distribution, with a binomial distribution, those parameters are n and p, and use those as the output layer. The optimization criterion should be something that measures the "distance" between two probability measures, so the Kullback-Leibner distance comes to mind for that. The binomial distribution may be appropriate "by the look of it", but by parametrising in this way you should be aware that the "predicted" distributions will only take on the very specific form of a binomial distribution, and if you want to allow for $n$ != 100 you will have problems to solve because the domain is not equal to {1, 2, .. 100}. 
Maybe you can take something a bit more flexible, such as a mixture model consisting of a mix of Beta-binomial distributions? Or just one Beta-binomial distribution to start with? The beta-binomial distribution seems a bit more appriopriate in any case since its support is finite discrete (it is actually an infinite mixture of binomial distributions with a beta-distributed p). 
All this is related to Variational Inference, and I think if you find someone really fluent in that, he or she might have a really clever suggestion for framing this problem in a way that allows applying all kinds of cool new techniques. 
