I'm working on the following problem, but I'm confused as to what kind of problem this is, how to select an appropriate model, and what metric I should be optimizing.

The problem is as follows:

Team of experts is given a recording of someone performing a piano piece. As a group, they give the recording a score from 1 - 100. There is lots of data (100k+) in the form (audio of recording $x^{[i]}$, score $y^{[i]}$). However, the experts aren’t perfectly reliable. There is a smaller data (2) set of the form (recording number $x^{[i]}$, , score $y^{[i]}$) where for each recording number, I have about 20 different scores. For this smaller data set, many groups of experts were given the same recordings. These scores seem to follow a binomial distribution, with a different mean for each recording number.

I want to learn the mapping from an audio recording $x^{[i]}$ to a score distribution over $y^{[i]}$ that would match the distribution formed if the experts were given this recording many times to evaluate.

Currently, I’m training a neural network to do a regression problem and get a mean squared error of about 7.0 on the validation set. By treating the problem as a classification problem instead I’m able to get a probability distribution. When I compute the expected value of each distribution for every validation example, and use the expected value to compute mean squared error, I get a mse of about 7.5.

However, this still all feels wrong to me as

  1. Using classification gives me a distribution, but doesn’t capture the linear nature of the data
  2. Using regression captures the linear nature of the data, but doesn’t give a distribution
  3. I’m not even sure I have the right set of data to compute this distribution in the first place.

Any advice on how to set up this problem, how to better ask this question, and which metric I should use?

  • $\begingroup$ Your notation confuses me a bit. What is $x^{[i]}$? What is $i$? Do you have variables for the experts? You mention 'experts', 'groups of experts' and 'as a group, they give the recording a score'? This confuses me. What is the most basic measurement, do you know data from a single expert, or from groups of experts? Do you know the composition of those groups? Do they change and do you know how? Etc. $\endgroup$ – Sextus Empiricus May 4 '18 at 7:27
  • $\begingroup$ I use $x^{[i]}$ here to refer to the ith audio recording, and $y^{[i]}$ to refer to the score for the ith audio recording. Sorry about the confusion over the experts. The identities, group size, and mechanism by which the experts assign a score are all unknown to me. I was leaving the details in for completeness but I see now it just confuses the problem. Perhaps it's best to treat 'the experts' as a single expert. $\endgroup$ – mboss May 8 '18 at 17:03
  • $\begingroup$ You mention something like a difference between 'lots of data (100k+) in the form ....' and 'a smaller data (2) set of the form....', what is the difference between them? Is $x^{[i]}$ data of the recording, like a signal or some other information about the recording, or just a label to the recording? $\endgroup$ – Sextus Empiricus May 8 '18 at 17:15

I have only suggestions, and no experience with this. But you can parametrize the distribution, with a binomial distribution, those parameters are n and p, and use those as the output layer. The optimization criterion should be something that measures the "distance" between two probability measures, so the Kullback-Leibner distance comes to mind for that. The binomial distribution may be appropriate "by the look of it", but by parametrising in this way you should be aware that the "predicted" distributions will only take on the very specific form of a binomial distribution, and if you want to allow for $n$ != 100 you will have problems to solve because the domain is not equal to {1, 2, .. 100}.

Maybe you can take something a bit more flexible, such as a mixture model consisting of a mix of Beta-binomial distributions? Or just one Beta-binomial distribution to start with? The beta-binomial distribution seems a bit more appriopriate in any case since its support is finite discrete (it is actually an infinite mixture of binomial distributions with a beta-distributed p).

All this is related to Variational Inference, and I think if you find someone really fluent in that, he or she might have a really clever suggestion for framing this problem in a way that allows applying all kinds of cool new techniques.

  • $\begingroup$ Thanks for your advice. What would your thoughts be on instead first optimizing for classification accuracy, then adding a data partition for a secondary objective of optimizing for empirical calibration error a la On Calibration of Modern Neural Networks My hesitation is that I've been getting relatively good results using NNs, that switching to a mixture model (probably over some latter layer output of the NN) would not be able to compare in accuracy. $\endgroup$ – mboss Oct 3 '17 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.