Quantifying explanatory potential

Question

Suppose I have a random variable $T_j \sim Bernoulli(p_j)$ where $logit(p_j) = \theta x_j + \epsilon_j$ and where $\epsilon_j \sim \mathcal{N}(0,1)$. Suppose further that $\theta = 0.018$ and that I have 100 observations of $T$. Here is a simulation:

# Simulation of T
set.seed(2018)

n <- 100

X <- rpois(n = n, lambda = 10)
E <- rnorm(n = n, mean = 0, sd = 1)
theta <- 0.018

pj <- exp(theta*X + E)/(1 + exp(theta*X + E))
#hist(pj)

T <- rbinom(n = n, size = 1, prob = pj)

Now suppose I have the following 2 models:

• $\mathcal{M_B}$: this is the baseline model where $\hat{p}_j = \bar{T} = 0.6$ where the latter equality is based on the simulation above.

# Baseline model
table(T)
# T
#  0  1 
# 40 60 

• $\mathcal{M_R}$: logistic regression model with $\hat{\theta}=0.043$

# Logistic Regression model
theta.hat <- coef(glm(formula = T ~ X - 1, family = binomial(link = "logit")))
theta.hat
# 0.0430746 

The goal is saying something about the "potential" of $P(T=1)$. From $\mathcal{M_B}$ I know that my baseline is $\hat{P}(T=1) = 0.6$. What I would like to say is that we could reduce this chance by some amount $\gamma = \frac{P(T = 1 | \mathcal{M_R})}{P(T = 1 | \mathcal{M_B})}$ if we utilized optimally the explanatory power of $\mathcal{M_R}$.

I am having trouble conceptualizing $\gamma$ in a theoretical-correct way.

One way that seems plausible is to run $\mathcal{M_R}$ over the whole feature space of $X$ and note the minimal $\hat{p}_j$. Thus $p_{min} = argmin_{X_{opt}}P(T = 1|\mathcal{M_R}, X)$. Assuming I don't know the feature space of $X$ I can only estimate $\hat{X}_{opt}$ based on my observed $X$ as follows:

# Determination of minimal pj attainable when fully 
# utilizing the explanatory power of the LR model.
# Emirical estimation
pj.hat <- exp(theta.hat*X)/(1 + exp(theta.hat*X))
min(pj.hat)
# 0.5322611

Then I can estimate $\gamma$:

# Gamma estimation 
min(pj.hat)/0.6
# 0.8871018

Obviously $\gamma$ is an estimate and has an error. Error can be estimated by bootstrapping everything. But my main concern is whether my approach is sound and whether someone has other ideas for tackling this problem?

Some of my other failed approaches:

• Classical $\frac{Var(T|X)}{Var(T)}$, but I don't see a way to link it to $\gamma$.
• Define mistake $M = T \text{ xor } \hat{T}$ and compute $E[M|\mathcal{M_B}]$ vs $E[M|\mathcal{M_R}]$.

Answer 1

There are a few problems in the way you are framing your question, but it sounds like what you are asking about is predicted probabilities of outcomes under different models. You should be careful to distinguish between the true probability of an outcome, versus an estimate of this probability based on your model. (It is better to denote the latter using hats on your estimated quantities.) You should also make sure that you are not making "predictions" of data that is actually observed. In your case you refer to "predictions" relating to the values $T_1, T_2, T_3$, but these are in your observed data.

The estimated probabilities from your models are:

$$\hat{p}_{\mathcal{B},j} = \bar{t} = 0.6 \quad \quad \quad \quad \hat{p}_{\mathcal{R},j} = \text{logistic}(\hat{\theta} x_j) = \frac{\exp(0.28 x_j)}{1 + \exp(0.28 x_j)}.$$

Let $\mathcal{E} = \{ T_{n+1} = 1, T_{n+2} = 1, T_{n+3} = 1 \}$ be the predictive event of interest (correcting to refer to the first three unobserved data points). The estimated probabilities of this event are:

$$\begin{equation} \begin{aligned} \hat{\mathbb{P}}( \mathcal{E} | \mathcal{M_B}) &= 0.6^3 = 0.216, \\[8pt] \hat{\mathbb{P}}( \mathcal{E} | \mathcal{M_R}) &= \frac{\exp(0.28 x_{n+1})}{1 + \exp(0.28 x_{n+1})} \cdot \frac{\exp(0.28 x_{n+2})}{1 + \exp(0.28 x_{n+2})} \cdot \frac{\exp(0.28 x_{n+3})}{1 + \exp(0.28 x_{n+3})}. \end{aligned} \end{equation}$$

Notice that the first model predicts a fixed probability of each outcome and so the prediction for the event of interest does not depend on the explanatory variables for the predicted outcomes. The second model uses logistic regression and so the estimated probability of the event of interest does depend on the explanatory variables. If you add explanatory variables for the predicted values you can calculate the latter, and then it is trivial to figure out the difference in the estimated probabilities under the two models.