I want to use a neural network (or any other method, for that matter) to perform regression from a high-dimensional space (10k dimensions) to a low-dimensional space (3 dimensions). To train this, I have many, many input-output training pairs $(x, y)$. So to train for regression, I could just send each training data $x$ through the network, compute the L2 error between its predicted value of $y$ and the ground truth value over my 3 output nodes, and back propagate the derivative of that error.

However, my problem arises because, for each training pair, $(x, y)$, there exist several other training pairs with exactly the same value of $x$. So, my list of training pairs looks like: $(x1, y1a), (x1, y1b), (x1, y1c) ...., (x2, y2a), (x2, y2b), (x2, y2c) ...., (x3, y3a), (x3, y3b), (x3, y3c) ...., .....)$.

This is because every value of $x$ has multiple "plausible" values of $y$, which are all equally important. Therefore, my network will be trained to output different values for the same input.

My question is: How will my network cope with this problem? If I were to pass $x1$ through the network, will it end up outputting just to $y1a$? Or will it average the values of $y1a, y2a, y3a, ...$? Or will it do something else?

And is there an alternative way I can pose this problem? I thought of doing it as a classification, but I would have to discretise my output space, which I want to avoid. And in any case, this would have the same problem as above...


1 Answer 1


See this paper$^1$ that might help you. That paper relates that "supervised learning is a classic data mining problem where one wishes to be able to predict an output value associated with a particular input vector. We present a new twist on this classic problem where, instead of having the training set contain an individual output value for each input vector, the output values in the training set are only given in aggregate over a number of input vectors."

[1] Musicant DR, Christensen JM, Olson JF. Supervised learning by training on aggregate outputs. Seventh IEEE International Conference on Data Mining (ICDM 2007): IEEE; 2007. p. 252-61.

  • 2
    $\begingroup$ Can you summarise the contents as the link may go dead in the future. $\endgroup$
    – mdewey
    Sep 29, 2016 at 11:19
  • $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$ Sep 29, 2016 at 12:18
  • $\begingroup$ @gung I already edited this to include the topic of the paper, but I see that has been removed-so-please confirm that the practice is that only the original author can substantially supplement answers, so that I do not waste time in the future, please. $\endgroup$
    – Carl
    Sep 29, 2016 at 15:26
  • $\begingroup$ @Carl, we should let the OP make those kinds of substantive edits to their post (unless they invite you to do so). If the OP doesn't edit this, it can be deleted. $\endgroup$ Sep 29, 2016 at 16:51
  • $\begingroup$ gung, @Carl looked up the paper and provided a quotation. That was objective and constructive. IMO it should have been accepted, for the following reasons. SE encourages the community to improve answers as well as questions: this is the opposite of your intimation that we need to leave such edits to the original posters. Of course we need to take care not to change the originally intended meaning of a post, but that scarcely seems to be a problem in this case. $\endgroup$
    – whuber
    Sep 29, 2016 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.