Skip to main content
Cross Validated

Return to Answer

expanded on N=10 vs N=100 example
Source Link
Björn
Björn
  • 35.2k
  • 3
  • 44
  • 104

Yes, that kind of set-up with $h$ outputs to which the softmax function is applied should work. I.e. before you apply the softmax-function you have $h$ outputs, let's call them $\hat{z}_i$ for $i=1,\ldots,h$ in $(-\infty, \infty)$ and by doing $\hat{y}_i = \frac{\exp{\hat{z}_i}}{\sum_{j=1}^h \exp{\hat{z}_j}}$, you enforce that $\sum_{i=1}^h \hat{y}_i = 1$ (as you can easily check). Using the categorical cross-entropy as the loss function and the $h$ probabilities $y_i$ as targets (instead of one-hot-encoding a single category out of the h categories) then backpropagates the right information into the neural network.

How successful this will be is another question and depends on there being enough information in the predictors to predict the probabilities, how much data there is and further details of how you implement it all. 

It can also be non-ideal, if some of the probabilities to predict are differently noisy (e, then only predicting the probabilities is not so great. E.g. if this is what we predict is "What proportion of people out of $N$ picked category $i$?", but sometimes $N=10$ and sometimes $N=100$), then only predicting the probabilities is not so great (e.g. in the example with picking categories, using the negative of the multinomial log-probability mass function as the loss function would give the neural network more information on the uncertainty).

  • By just using categorical cross-entropy on the proportion, we give both types of observations the same weight in updating the neural network. This makes sense, if you have approx. equal but unknown denominators for the proportions you are modelling.
  • However, if you really reflect that a proportion estimated on just 10 people is much less certain (or a much noisier estimate) than one estimated on 100 people, it becomes clear that those with $N=100$ should have more weight.
  • You could reflect that in this example using the negative of the multinomial log-probability mass function as the loss function.
  • Consider the plot below, where I plot the marginal loss for one proportion, where we observed either 5 out of 10, or 50 out of 100. You can see that the loss landscape is much flatter for N=10 compared with N=100, so you'd get much smaller gradients (less strong updating of the neural network = the records contain less information) from records with N=10 compared with N=100. enter image description here

Yes, that kind of set-up with $h$ outputs to which the softmax function is applied should work. I.e. before you apply the softmax-function you have $h$ outputs, let's call them $\hat{z}_i$ for $i=1,\ldots,h$ in $(-\infty, \infty)$ and by doing $\hat{y}_i = \frac{\exp{\hat{z}_i}}{\sum_{j=1}^h \exp{\hat{z}_j}}$, you enforce that $\sum_{i=1}^h \hat{y}_i = 1$ (as you can easily check). Using the categorical cross-entropy as the loss function and the $h$ probabilities $y_i$ as targets (instead of one-hot-encoding a single category out of the h categories) then backpropagates the right information into the neural network.

How successful this will be is another question and depends on there being enough information in the predictors to predict the probabilities, how much data there is and further details of how you implement it all. It can also be non-ideal, if some of the probabilities to predict are differently noisy (e.g. if this is what proportion of people out of $N$ picked category $i$, but sometimes $N=10$ and sometimes $N=100$), then only predicting the probabilities is not so great (e.g. in the example with picking categories, using the negative of the multinomial log-probability mass function as the loss function would give the neural network more information on the uncertainty).

Yes, that kind of set-up with $h$ outputs to which the softmax function is applied should work. I.e. before you apply the softmax-function you have $h$ outputs, let's call them $\hat{z}_i$ for $i=1,\ldots,h$ in $(-\infty, \infty)$ and by doing $\hat{y}_i = \frac{\exp{\hat{z}_i}}{\sum_{j=1}^h \exp{\hat{z}_j}}$, you enforce that $\sum_{i=1}^h \hat{y}_i = 1$ (as you can easily check). Using the categorical cross-entropy as the loss function and the $h$ probabilities $y_i$ as targets (instead of one-hot-encoding a single category out of the h categories) then backpropagates the right information into the neural network.

How successful this will be is another question and depends on there being enough information in the predictors to predict the probabilities, how much data there is and further details of how you implement it all. 

It can also be non-ideal, if some of the probabilities to predict are differently noisy, then only predicting the probabilities is not so great. E.g. if what we predict is "What proportion of people out of $N$ picked category $i$?", but sometimes $N=10$ and sometimes $N=100$, then

  • By just using categorical cross-entropy on the proportion, we give both types of observations the same weight in updating the neural network. This makes sense, if you have approx. equal but unknown denominators for the proportions you are modelling.
  • However, if you really reflect that a proportion estimated on just 10 people is much less certain (or a much noisier estimate) than one estimated on 100 people, it becomes clear that those with $N=100$ should have more weight.
  • You could reflect that in this example using the negative of the multinomial log-probability mass function as the loss function.
  • Consider the plot below, where I plot the marginal loss for one proportion, where we observed either 5 out of 10, or 50 out of 100. You can see that the loss landscape is much flatter for N=10 compared with N=100, so you'd get much smaller gradients (less strong updating of the neural network = the records contain less information) from records with N=10 compared with N=100. enter image description here
Source Link
Björn
Björn
  • 35.2k
  • 3
  • 44
  • 104

Yes, that kind of set-up with $h$ outputs to which the softmax function is applied should work. I.e. before you apply the softmax-function you have $h$ outputs, let's call them $\hat{z}_i$ for $i=1,\ldots,h$ in $(-\infty, \infty)$ and by doing $\hat{y}_i = \frac{\exp{\hat{z}_i}}{\sum_{j=1}^h \exp{\hat{z}_j}}$, you enforce that $\sum_{i=1}^h \hat{y}_i = 1$ (as you can easily check). Using the categorical cross-entropy as the loss function and the $h$ probabilities $y_i$ as targets (instead of one-hot-encoding a single category out of the h categories) then backpropagates the right information into the neural network.

How successful this will be is another question and depends on there being enough information in the predictors to predict the probabilities, how much data there is and further details of how you implement it all. It can also be non-ideal, if some of the probabilities to predict are differently noisy (e.g. if this is what proportion of people out of $N$ picked category $i$, but sometimes $N=10$ and sometimes $N=100$), then only predicting the probabilities is not so great (e.g. in the example with picking categories, using the negative of the multinomial log-probability mass function as the loss function would give the neural network more information on the uncertainty).