is negative log loss affected by oversampling? I'm working on a multiclass classification problem where negative log loss is the evaluation metric. My initial train set and my static test set have similar class distribution and my validation (20% of train) nll is in line with my test nll.
When I oversample some classes in my train data by just adding new examples at my disposal, thus changing the train set class distribution and consequently making the train set class distribution different from the test set distribution, I notice that the test nll worsens. My hypothesis is that nll is affected by class distribution. Does that make sense? And is there a way to adjust predictions to account for the difference in class distribution between oversampled train and test?
 A: Yes, it is affected: say you have $K$ classes, and denote the NLL for class $k$ by $\textrm{NLL}_k$. ($\textrm{NLL}_k$ is a random variable, the random variable for the NLL of a random example conditioned on it being of class $k$. If you want, pretend that the only thing that affects NLL is the class, and then it's just a constant for each $k$.)
Then, when evaluating NLL on a set of size $N$ with instances from each class $(n_1, \dots, n_K)$, your overall NLL is $\sum_k \frac{n_k}{N} \textrm{NLL}_k$, since NLL is additive across instances. If you train on a dataset with a particular class distribution $(\frac{n_k}{N})$, and test on a different one, the learner will be optimizing for one loss function and testing on another.
I don't know if this is a standard thing, but you could try reweighting the NLL by the prior probability of each class (that is, multiply the NLL for each instance of class $k$ by \frac{N}{n_k}); that should help in basically the same way that adding class weights helps for accuracy or whatever other evaluation metric.
