I have a dataset $D = (x_i, y_i)_{i=1}^n$ where $x_i\in \Bbb R^d$ and $y_i\in\{0, 1\}$. Suppose that $y\sim\mathrm{Bernoulli}(p(x))$ for some probability function $p:\Bbb R^d \to [0,1]$ and I would like to learn this function the best way I can.

  1. Would that be correct to say that logistic regression is actually focused on approximating $p$ with some predictor $q$ which minimizes the cross-entropy loss $\mathrm{CE}(p,q)$?

  2. What are the other useful methods to approximate $p$ based off the data $D$? Of course I can replace a linear model in logistic regression with some non-linear counterpart $q_\theta:\Bbb R^d \to [0,1]$ and look for $\theta$ that minimizes $\mathrm CE(p,q_\theta)$, searching for example over NNs or random forests or whatever other non-linear point estimators come to mind. Perhaps, some Bayesian frameworks would come handy here?

  3. If I need to learn the entropy of $p(x)$ at a given $x$ of course I can just approximate it with the entropy of $q(x)$: if $p\approx q$ then we should expect $H(p) \approx H(q)$. I wonder however whether there is a direct way to learn $H(p)$ as a function of $x$, perhaps with me wanting to minimize $\mathrm{MSE}(H(p), f_\theta)$ where $f_\theta$ is some parametrized positive-valued function.

  • 2
    $\begingroup$ This seems a bit broad; I count 3 questions. Can you edit it to focus on just 1 issue? Regarding (1), you're correct that logistic regression uses the model $\frac{1}{1+\exp(-X\beta)}$ to approximate $p$ and estimates the $\beta$ that minimizes cross-entropy. Regarding (2), every probabilistic classification method is an approach to estimating $p$; some are Bayesian. (Non-probabilistic methods include SVMs, which measure distance to a hyperplane, not a probability.) $\endgroup$
    – Sycorax
    Oct 5, 2021 at 13:40
  • $\begingroup$ You are likely to be interested in calibration. $\endgroup$
    – Dave
    Oct 5, 2021 at 22:02
  • $\begingroup$ @Sycorax: thanks, I think indeed the last part deserves a separate question $\endgroup$
    – Ilya
    Oct 6, 2021 at 8:11
  • $\begingroup$ @Dave could you please provide a link where the concept of calibration you've referred to is explained in details? maybe with math $\endgroup$
    – Ilya
    Oct 6, 2021 at 8:12
  • $\begingroup$ That blue text indicates a link. That “rms” package is related to a book, “Regression Modeling Strategies”, by Frank Harrell, who, in addition to being an influential statistics professor, is a high-reputation member on here. $\endgroup$
    – Dave
    Oct 6, 2021 at 9:35

1 Answer 1

  1. Yes, this is true. $q$ is chosen here from a specific family of functions, parametrized by the intercept and slope of the logistic regression.
  2. Any binary probabilistic classification model would work well. The best method would depend on the nature of $p$, so the more methods you try, the better it would be. Usually, one of the following methods works well enough: logistic regression / neural networks (multilayer perceptron, in your case) / gradient boosting with small decision trees / KNN. Bayesian methods can work with any of these frameworks as a way of regularizing (= assigning priors to) them.
  3. I don't know about methods of direct evaluation of entropy, and I think that it is perfectly OK to evaluate it using the entropy of q, given that q itself is well calibrated.
  • $\begingroup$ Thanks, indeed as @Sycorax suggested, the last part deserves a separate question. I accept this as it addresses the core, and make a new question regarding my problems with the entropy estimates. $\endgroup$
    – Ilya
    Oct 6, 2021 at 7:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.