I am interested in understanding how Neural Networks could be used to both learn from and predict proportions.

That is, say matrix $X$ is the training features data with $N$ cases and $k$ features. Also, matrix $Y$ are the training outputs, with $N$ cases and $h$ variables, where each row holds $h$ proportions that sum to $1$. The problem is defined as learning from those proportions in $Y$, so that with test data for which proportions of the $1 \cdots h$ variables are not known, they can be predicted.

When I search online for this problem I find a lot of mixed and somewhat out-of-the track suggestions. In part, confusions arise from the fact that some people call this type of issue "proportion prediction", which is ambiguous with the predicting of class proportions in traditional multiclass problems.

Naturally, the modelling of this type of problem has been explored before in other fields and with other approaches, which is indicated in the this SE question. But I am having difficulty finding details about how would this be done in a more typical Supervised Machine Learning fashion - I found it particularly hard to find much about doing it using Neural Networks.

In the above linked question, the OP mentions that they eventually settled with a multi-output Neural Network (in my notation, a network with $h$ output targets) with soft-targets cross-entropy as the loss function. My objective questions are:

  • would that be enough to correctly model a Neural Network to predict proportions? I am not convinced that the generated target outputs, while forced to sum up to 1, would indeed represent the proportions trying to be predicted.

  • if they do and that approach is enough, I cannot understand why, and therefore would love an explanation of why, cross-entropy with soft-targets would correctly guarantee that, in a multi-output network architecture, the predicted target values end up representing predicted proportions.

  • 4
    $\begingroup$ The softmax activation is usually used in conjunction with classification models but one could also use softmax to get predictions $h$ in [0,1] s.t. the sum of the predictions is 1. You can use softmax independently of binomial cross-entropy; if you have a different loss in mind, especially one related to the probability model, that’s a great place to start. $\endgroup$ – Sycorax Jun 23 at 21:12
  • $\begingroup$ @Sycorax Thanks, that helps a lot - then the key here is activation, not the loss function in itself. But one clarifying question, which I guess is what is behind my original discomfort with the multi-output approach: would that be enough to account for (the obviously existing, since they compose 1) dependence between the multiple targets? $\endgroup$ – Jul Jun 24 at 6:32

Yes, that kind of set-up with $h$ outputs to which the softmax function is applied should work. I.e. before you apply the softmax-function you have $h$ outputs, let's call them $\hat{z}_i$ for $i=1,\ldots,h$ in $(-\infty, \infty)$ and by doing $\hat{y}_i = \frac{\exp{\hat{z}_i}}{\sum_{j=1}^h \exp{\hat{z}_j}}$, you enforce that $\sum_{i=1}^h \hat{y}_i = 1$ (as you can easily check). Using the categorical cross-entropy as the loss function and the $h$ probabilities $y_i$ as targets (instead of one-hot-encoding a single category out of the h categories) then backpropagates the right information into the neural network.

How successful this will be is another question and depends on there being enough information in the predictors to predict the probabilities, how much data there is and further details of how you implement it all.

It can also be non-ideal, if some of the probabilities to predict are differently noisy, then only predicting the probabilities is not so great. E.g. if what we predict is "What proportion of people out of $N$ picked category $i$?", but sometimes $N=10$ and sometimes $N=100$, then

  • By just using categorical cross-entropy on the proportion, we give both types of observations the same weight in updating the neural network. This makes sense, if you have approx. equal but unknown denominators for the proportions you are modelling.
  • However, if you really reflect that a proportion estimated on just 10 people is much less certain (or a much noisier estimate) than one estimated on 100 people, it becomes clear that those with $N=100$ should have more weight.
  • You could reflect that in this example using the negative of the multinomial log-probability mass function as the loss function.
  • Consider the plot below, where I plot the marginal loss for one proportion, where we observed either 5 out of 10, or 50 out of 100. You can see that the loss landscape is much flatter for N=10 compared with N=100, so you'd get much smaller gradients (less strong updating of the neural network = the records contain less information) from records with N=10 compared with N=100. enter image description here
  • 1
    $\begingroup$ This is a good answer (+1) Could you elaborate on the concept in your second paragraph? I think the idea is valuable, but more detail or an example would make it more clear how the size of $N$, the multinomial loss, and prediction quality come together. $\endgroup$ – Sycorax Jun 23 at 21:34
  • 1
    $\begingroup$ Thanks for your answer. I was wondering the same that @Sycorax asked for. I am intrigued about the connection you made between imbalance, the multinomial log-prob and prediction quality $\endgroup$ – Jul Jun 23 at 23:34
  • $\begingroup$ Also, bringing here the same reaction I had to @Sycorax's comment to my question. I guess what is behind my original discomfort with the multi-output approach is: would that be enough to account for (the obviously existing, since they compose 1) dependence between the multiple targets? $\endgroup$ – Jul Jun 24 at 6:49
  • $\begingroup$ @Sycorax updated. Jul: Either of the loss functions do result in appropriate gradients to teach the neural network about that (i.e. they reflect that if one proportion is higher others should be lower). Given enough training data, useful inputs and a sufficiently expressive neural network (in practice, the model that you have e.g. in the fast.ai library seems often pretty good, as just pointed out in yet another tabular NN paper), the neural network should be able to fit the underlying patterns pretty well. $\endgroup$ – Björn Jun 24 at 6:58
  • $\begingroup$ I'm not claiming a NN is the best answer to your particular problem. If your inputs are tabular, then xgboost (or LightGBM) with a multi:softprob loss function for proportions or multi:softmax-loss function for multinomial with a record for each N (or the H2O version, which seems to have a multinomial loss). NNs really, really shine if you have a lot of outputs with different losses (or, even, for multi-label classification with many correlated labels), or when inputs or outputs are non-tabular (images, text...). $\endgroup$ – Björn Jun 24 at 7:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.