How to account for varying levels of “confidence” among data points during training I am training a neural network (VGG16) on some image data for an image classification task. Each image has a “number of clicks” and a “number of views” feature. From these features, I’m defining a binary label as a threshold value on clicks/views.
The issue is that not all images have been viewed the same number of times. For example, we could have two images, one with 10 clicks out of 100 views and another with 1000 clicks out of 10000 views.  Both have the same clicks/views ratio but the latter has much lower variance.
It seems this variance (or confidence in the success rate) is meaningful and should somehow be accounted for in the training process. Is there a standard way to incorporate this information?
 A: Here's how I'd address this. From your description it sounds like you're trying to predict a "conversion rate" for each image, and once you have the conversion rate you treat the clicks / views as a binomial experiment with the conversion rate as p.
Maximum likelihood estimation (MLE) is a standard practice in statistical parameter estimation, which I think is very relevant in this case. Except, you don't want to estimate a single set of parameters, but train a model to predict parameters given some input. Still, MLE can address the issue you mentioned. Usually MLE is done by maximizing the log-likelihood of the samples (or equivalently minimizing the negative log likelihood). In this case you could use the negative log-likelihood as the loss function, with sample $ i $ contributing $ -(clicks_i \cdot \ln(prediction_i) + (views_i - clicks_i) \cdot \ln(1 - prediction_i)) $ to the loss term (this is the negative log-likelihood of the binomial PMF, omitting the combinatorial coefficient which doesn't depend on the prediction). You'll notice this is very similar to the binary cross-entropy loss function, except that each term (and sample) is weighted according to the number of clicks / views in a way that pushes the model to maximally explain the observed data.
This addresses the issue you raised because the likelihood of 100/1000 clicks/views given a prediction of e.g. 0.11 is different from the likelihood of 1000/10000 clicks/views given the same prediction. Note, however, that large disparities in number of views can make some samples dominate the loss term.
