4
$\begingroup$

I am working on a machine learning problem where I have to predict a set of $N$ numbers (proportions) for each data point, all of them summing to one. One toy example to illustrate my problem would be predicting at a daily level the percentage of volume of water rained in each of the states of the US over the total rain in the country - in this example $N=50$ (the number of states) and $\sum_{n=1}^{50}{\hat{y}_n}=1$

I was thinking on designing a neural net with $N$ outputs and apply a Softmax in the output, then backpropagate the MSE or the RMSE... I am a bit unsure about the convergence guarantees (potential vanishing gradient). I would also like to know if you would approach the problem in another way.

Improve this question
$\endgroup$
9
  • 1
    $\begingroup$ Depending on your independent variables, I should imagine that the better problem would be to build a model to predict the amount of rain in each area. If you then would like to have percentages, simply divide by the total predicted rain over all areas. $\endgroup$ – Him Apr 1 '20 at 14:50
  • 1
    $\begingroup$ But that is very dangerous, given that only an outlier in one of the communities would bias all your distribution... I would rather like to adjust all the probabilities at the same time with the sum-1 constraint $\endgroup$ – ivallesp Apr 1 '20 at 14:55
  • $\begingroup$ It may be worth noting that, at least for classification problems, NNs often need special care with calibration. I would think that this problem would be equally bad for regression on a compositional variable. Just something to think about if you end up going that route. $\endgroup$ – Him Apr 1 '20 at 14:55
  • $\begingroup$ "an outlier in one of the communities would bias all your distribution" I'm not sure how constraining the model to sum to 1 would alleviate this. Would you care to elaborate? $\endgroup$ – Him Apr 1 '20 at 14:56
  • $\begingroup$ As you increase $N$, the probability of having a very large value in one of the sub-models increases. This approach optimizes the different sub-models separately while the approach I am looking for would optimize a calibrated output (with a Softmax for example). Hence the model I am looking for should account for this risk of overestimation given that overestimating one of the outputs affects the prediction of all the outputs. $\endgroup$ – ivallesp Apr 1 '20 at 15:07
5
+50
$\begingroup$

You have what is called . There is quite some literature on how to model this. Take a look through the tag, or search for the term.

Typically, one would choose a reference category and work with log ratios, or similar. One paper I personally know about predicting compositional data is Snyder at al. (2017, IJF). They use a state space approach, not an NN, but their transformation may still be useful to you.

Improve this answer
$\endgroup$
1
$\begingroup$

Answering my past self... One elegant solution is to use the cross-entropy with "soft-targets" as loss. This means that your targets will not be in one-hot-encodding format, but they will still sum to one. The original cross-entropy formula formula applies.

The cross-entropy loss with soft targets is widely used in the knowledge-distillation field: ref.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.