Can proportions over variables be learned/predicted with Neural Networks using multiple-outputs?

Question

I am interested in understanding how Neural Networks could be used to both learn from and predict proportions.

That is, say matrix $X$ is the training features data with $N$ cases and $k$ features. Also, matrix $Y$ are the training outputs, with $N$ cases and $h$ variables, where each row holds $h$ proportions that sum to $1$. The problem is defined as learning from those proportions in $Y$, so that with test data for which proportions of the $1 \cdots h$ variables are not known, they can be predicted.

When I search online for this problem I find a lot of mixed and somewhat out-of-the track suggestions. In part, confusions arise from the fact that some people call this type of issue "proportion prediction", which is ambiguous with the predicting of class proportions in traditional multiclass problems.

Naturally, the modelling of this type of problem has been explored before in other fields and with other approaches, which is indicated in the this SE question. But I am having difficulty finding details about how would this be done in a more typical Supervised Machine Learning fashion - I found it particularly hard to find much about doing it using Neural Networks.

In the above linked question, the OP mentions that they eventually settled with a multi-output Neural Network (in my notation, a network with $h$ output targets) with soft-targets cross-entropy as the loss function. My objective questions are:

• would that be enough to correctly model a Neural Network to predict proportions? I am not convinced that the generated target outputs, while forced to sum up to 1, would indeed represent the proportions trying to be predicted.

• if they do and that approach is enough, I cannot understand why, and therefore would love an explanation of why, cross-entropy with soft-targets would correctly guarantee that, in a multi-output network architecture, the predicted target values end up representing predicted proportions.

Answer 1

Yes, that kind of set-up with $h$ outputs to which the softmax function is applied should work. I.e. before you apply the softmax-function you have $h$ outputs, let's call them $\hat{z}_i$ for $i=1,\ldots,h$ in $(-\infty, \infty)$ and by doing $\hat{y}_i = \frac{\exp{\hat{z}_i}}{\sum_{j=1}^h \exp{\hat{z}_j}}$, you enforce that $\sum_{i=1}^h \hat{y}_i = 1$ (as you can easily check). Using the categorical cross-entropy as the loss function and the $h$ probabilities $y_i$ as targets (instead of one-hot-encoding a single category out of the h categories) then backpropagates the right information into the neural network.

How successful this will be is another question and depends on there being enough information in the predictors to predict the probabilities, how much data there is and further details of how you implement it all.

It can also be non-ideal, if some of the probabilities to predict are differently noisy, then only predicting the probabilities is not so great. E.g. if what we predict is "What proportion of people out of $N$ picked category $i$?", but sometimes $N=10$ and sometimes $N=100$, then

• By just using categorical cross-entropy on the proportion, we give both types of observations the same weight in updating the neural network. This makes sense, if you have approx. equal but unknown denominators for the proportions you are modelling.
• However, if you really reflect that a proportion estimated on just 10 people is much less certain (or a much noisier estimate) than one estimated on 100 people, it becomes clear that those with $N=100$ should have more weight.
• You could reflect that in this example using the negative of the multinomial log-probability mass function as the loss function.
• Consider the plot below, where I plot the marginal loss for one proportion, where we observed either 5 out of 10, or 50 out of 100. You can see that the loss landscape is much flatter for N=10 compared with N=100, so you'd get much smaller gradients (less strong updating of the neural network = the records contain less information) from records with N=10 compared with N=100.