2
$\begingroup$

This is an earlier discussion about LR producing well calibrated models:

Some people equate neural net based prediction models (even deep NN or deep+sparse NN) to be equivalent to logistic regression. We train them with adagrad (or some other methods) but always update weights by optimizing cost function on limited batch.

Q1: Is it true that Neural Nets have properties of logistic regression?

Q2: When we train with batches, weights that were calculated in the first batch and produce 'well calibrated' output, probably will be updated quite differently when processing subsequent batch. Intuitively, I think after a few updates this 'well-calibrated' property wouldn't hold (and especially for sparse neural nets where some embedding table is used). Is this correct?

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ A neural network with sigmoid activation and without hidden layers is a logistic regression. However, a simple counterexample shows that the logic used in the answer you linked does not apply to neural networks: Training a model on data with a low incidence rate without supplying weights often causes the neural network to predict all observations to have the same class. In such a case the sum of predicted class probabilities obviously does not equal the sum of the outcome. $\endgroup$
    – Frans Rodenburg
    May 3, 2019 at 23:19
  • $\begingroup$ @FransRodenburg is it though? What about probit regression, or cloglog? All of those link functions also have the shape of a sigmoid (as do most CDFs), but why specifically logistic? Or was it because the OP asked about logistic? $\endgroup$
    – runr
    May 22, 2019 at 8:54
  • $\begingroup$ @Nutle In neural networks, sigmoid almost always refers to the logistic function $\frac{1}{1 + e^{-x}}$, as opposed to other (possibly S-shaped) activation functions like softmax, tanh, ReLU, etc. $\endgroup$
    – Frans Rodenburg
    May 22, 2019 at 11:54
  • $\begingroup$ @FransRodenburg Thanks. You're right, my mistake. It's interesting how in neural networks a specific case of a specific sigmoid (logit) function is called as the whole family of curves, but that's off topic. $\endgroup$
    – runr
    May 22, 2019 at 12:29

1 Answer 1

Reset to default
2
$\begingroup$

I thought this may answer your question:

We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration.

But they found that temperature scaling can effectively calibrate predictions and in a new study some researchers found ways to calibrate the results of reinforcement learning. Since modern neural nets seem be trained all in batches so the aforementioned two papers are all about batch based training.

Reference: On Calibration of Modern Neural Networks

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.