Currently I'm using high dimensional logistic regression to predict the probability of a rare event. I use this probability for both ranking and for other calculations which need it to be well-calibrated, ie, unbiased.
The problem is that different samples have different levels of uncertainty. And as I rank more samples, the chances for selecting an outlier increase.
I was thinking it would be nice to obtain a beta distribution for each event, rather than a single point estimate. This would let me experiment with different approaches, e.g., lower-confidence bound, quantile, etc.
There's another nice property to Beta distribution, but I need to frame things more precisely to explain.
Suppose I have $n$ iid samples each possessing different evidence or context, i.e., a binary $d$-dimensional vector $x_i$. The goal is to predict the rate of a positive event, ie, $p_i:=P(y_i=1|x_i)$. (Note the samples are probably not really independent, but they are exchangeable.) After selecting the item with largest $p_i$, we observe $y_i\in\{0,1\}$.
As indicated, I'm currently using logistic regression and for various reasons need to stay in this "realm."
Beta distribution can be made to meet this criterion since: $p_i \sim \textsf{Beta}(\alpha_i:=e^{w_1^T x_i}, \beta_i:=e^{w_0^T x_i})$ implies that $\textsf{E}[p_i]=\frac{e^{w_1^T x_i}}{e^{w_1^T x_i} + e^{w_0^T x_i}}=\frac{1}{1+e^{-(w_1-w_0)^T x_i}}$ which is identical to the prediction of a reparameterized logistic regression with weight vector $w=w_1-w_0$. Moreover this reparameterization is well-understood; it is the two-way latent-variable model interpretation of logistic regression.
Hence, it can be argued that logistic regression entails predicting the mean of a particular parameterization of a Beta distribution.
Is it possible to unambiguously predict $\alpha_i,\beta_i$, not just the mean?
Specifically, what's the most reasonable way to determine the latent variables $\alpha_i, \beta_i$? Can this objective be framed as finding a particular MLE (or MMLE)? Or is there some other optimization problem that finds the "best" $w_1,w_0$? I worry about non-identifiability and don't know how to make this goal well-defined.
Since the stationary condition of logistic regression is: $w_j:\sum_i^n y_i x_{ij}=\sum_i^n p_i x_{ij}$ (omitting regularization) I was thinking maybe: "fit beta regression subject to first moment agreement" (ie previous stationary condition. Specifically, I could learn $w$ (which I already have), then conditioned on this $w$ find vector $w_1$ with the interpretation: $p_i \sim \textsf{Beta}(\alpha_i:=e^{w_1^T x_i}, \beta_i:=e^{(w_1-w)^T x_i})$. And as required, $\textsf{E}[p_i|x_i]=\frac{1}{1+e^{-w^T x}}$, i.e., the mean remains the prediction from the first fit of logistic regression. In the second-layer beta-regression, the first moment condition is automatically satisfied by way of the parameterization of $\alpha_i,\beta_i$. The catch is that I'm not sure how to tie other predictions across an active feature, ie, $\{p_i:x_{ij}=1\}$.
An alternative idea is to predict the evidence subject to logistic regression constraint. That is, for all $j$: $\sum_i^n x_{ij} = \sum_i^n (e^{w_1^Tx_j} + e^{w_0^Tx_j}) = \sum_i^n \frac{1}{\textsf{E}[p_i|x_i]} e^{w_1^T x_j}$ where the last equality follows from the fact that $w=w_1-w_0$.
