Currently I'm using high dimensional logistic regression to predict the probability of a rare event. I use this probability for both ranking and for other calculations which need it to be well-calibrated, ie, unbiased.

The problem is that different samples have different levels of uncertainty. And as I rank more samples, the chances for selecting an outlier increase.

I was thinking it would be nice to obtain a beta distribution for each event, rather than a single point estimate. This would let me experiment with different approaches, e.g., lower-confidence bound, quantile, etc.

There's another nice property to Beta distribution, but I need to frame things more precisely to explain.

Suppose I have $n$ iid samples each possessing different evidence or context, i.e., a binary $d$-dimensional vector $x_i$. The goal is to predict the rate of a positive event, ie, $p_i:=P(y_i=1|x_i)$. (Note the samples are probably not really independent, but they are exchangeable.) After selecting the item with largest $p_i$, we observe $y_i\in\{0,1\}$.

As indicated, I'm currently using logistic regression and for various reasons need to stay in this "realm."

Beta distribution can be made to meet this criterion since: $p_i \sim \textsf{Beta}(\alpha_i:=e^{w_1^T x_i}, \beta_i:=e^{w_0^T x_i})$ implies that $\textsf{E}[p_i]=\frac{e^{w_1^T x_i}}{e^{w_1^T x_i} + e^{w_0^T x_i}}=\frac{1}{1+e^{-(w_1-w_0)^T x_i}}$ which is identical to the prediction of a reparameterized logistic regression with weight vector $w=w_1-w_0$. Moreover this reparameterization is well-understood; it is the two-way latent-variable model interpretation of logistic regression.

Hence, it can be argued that logistic regression entails predicting the mean of a particular parameterization of a Beta distribution.

Is it possible to unambiguously predict $\alpha_i,\beta_i$, not just the mean?

Specifically, what's the most reasonable way to determine the latent variables $\alpha_i, \beta_i$? Can this objective be framed as finding a particular MLE (or MMLE)? Or is there some other optimization problem that finds the "best" $w_1,w_0$? I worry about non-identifiability and don't know how to make this goal well-defined.

Since the stationary condition of logistic regression is: $w_j:\sum_i^n y_i x_{ij}=\sum_i^n p_i x_{ij}$ (omitting regularization) I was thinking maybe: "fit beta regression subject to first moment agreement" (ie previous stationary condition. Specifically, I could learn $w$ (which I already have), then conditioned on this $w$ find vector $w_1$ with the interpretation: $p_i \sim \textsf{Beta}(\alpha_i:=e^{w_1^T x_i}, \beta_i:=e^{(w_1-w)^T x_i})$. And as required, $\textsf{E}[p_i|x_i]=\frac{1}{1+e^{-w^T x}}$, i.e., the mean remains the prediction from the first fit of logistic regression. In the second-layer beta-regression, the first moment condition is automatically satisfied by way of the parameterization of $\alpha_i,\beta_i$. The catch is that I'm not sure how to tie other predictions across an active feature, ie, $\{p_i:x_{ij}=1\}$.

An alternative idea is to predict the evidence subject to logistic regression constraint. That is, for all $j$: $\sum_i^n x_{ij} = \sum_i^n (e^{w_1^Tx_j} + e^{w_0^Tx_j}) = \sum_i^n \frac{1}{\textsf{E}[p_i|x_i]} e^{w_1^T x_j}$ where the last equality follows from the fact that $w=w_1-w_0$.


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.