2
$\begingroup$

Suppose I have a random variable $T_j \sim Bernoulli(p_j)$ where $logit(p_j) = \theta x_j + \epsilon_j$ and where $\epsilon_j \sim \mathcal{N}(0,1)$. Suppose further that $\theta = 0.018$ and that I have 100 observations of $T$. Here is a simulation:

# Simulation of T
set.seed(2018)

n <- 100

X <- rpois(n = n, lambda = 10)
E <- rnorm(n = n, mean = 0, sd = 1)
theta <- 0.018

pj <- exp(theta*X + E)/(1 + exp(theta*X + E))
#hist(pj)

T <- rbinom(n = n, size = 1, prob = pj)

Now suppose I have the following 2 models:

  • $\mathcal{M_B}$: this is the baseline model where $\hat{p}_j = \bar{T} = 0.6$ where the latter equality is based on the simulation above.
# Baseline model
table(T)
# T
#  0  1 
# 40 60 
  • $\mathcal{M_R}$: logistic regression model with $\hat{\theta}=0.043$
# Logistic Regression model
theta.hat <- coef(glm(formula = T ~ X - 1, family = binomial(link = "logit")))
theta.hat
# 0.0430746 

The goal is saying something about the "potential" of $P(T=1)$. From $\mathcal{M_B}$ I know that my baseline is $\hat{P}(T=1) = 0.6$. What I would like to say is that we could reduce this chance by some amount $\gamma = \frac{P(T = 1 | \mathcal{M_R})}{P(T = 1 | \mathcal{M_B})}$ if we utilized optimally the explanatory power of $\mathcal{M_R}$.

I am having trouble conceptualizing $\gamma$ in a theoretical-correct way.

One way that seems plausible is to run $\mathcal{M_R}$ over the whole feature space of $X$ and note the minimal $\hat{p}_j$. Thus $p_{min} = argmin_{X_{opt}}P(T = 1|\mathcal{M_R}, X)$. Assuming I don't know the feature space of $X$ I can only estimate $\hat{X}_{opt}$ based on my observed $X$ as follows:

# Determination of minimal pj attainable when fully 
# utilizing the explanatory power of the LR model.
# Emirical estimation
pj.hat <- exp(theta.hat*X)/(1 + exp(theta.hat*X))
min(pj.hat)
# 0.5322611

Then I can estimate $\gamma$:

# Gamma estimation 
min(pj.hat)/0.6
# 0.8871018

Obviously $\gamma$ is an estimate and has an error. Error can be estimated by bootstrapping everything. But my main concern is whether my approach is sound and whether someone has other ideas for tackling this problem?

Some of my other failed approaches:

  • Classical $\frac{Var(T|X)}{Var(T)}$, but I don't see a way to link it to $\gamma$.
  • Define mistake $M = T \text{ xor } \hat{T}$ and compute $E[M|\mathcal{M_B}]$ vs $E[M|\mathcal{M_R}]$.
$\endgroup$
1
$\begingroup$

There are a few problems in the way you are framing your question, but it sounds like what you are asking about is predicted probabilities of outcomes under different models. You should be careful to distinguish between the true probability of an outcome, versus an estimate of this probability based on your model. (It is better to denote the latter using hats on your estimated quantities.) You should also make sure that you are not making "predictions" of data that is actually observed. In your case you refer to "predictions" relating to the values $T_1, T_2, T_3$, but these are in your observed data.

The estimated probabilities from your models are:

$$\hat{p}_{\mathcal{B},j} = \bar{t} = 0.6 \quad \quad \quad \quad \hat{p}_{\mathcal{R},j} = \text{logistic}(\hat{\theta} x_j) = \frac{\exp(0.28 x_j)}{1 + \exp(0.28 x_j)}.$$

Let $\mathcal{E} = \{ T_{n+1} = 1, T_{n+2} = 1, T_{n+3} = 1 \}$ be the predictive event of interest (correcting to refer to the first three unobserved data points). The estimated probabilities of this event are:

$$\begin{equation} \begin{aligned} \hat{\mathbb{P}}( \mathcal{E} | \mathcal{M_B}) &= 0.6^3 = 0.216, \\[8pt] \hat{\mathbb{P}}( \mathcal{E} | \mathcal{M_R}) &= \frac{\exp(0.28 x_{n+1})}{1 + \exp(0.28 x_{n+1})} \cdot \frac{\exp(0.28 x_{n+2})}{1 + \exp(0.28 x_{n+2})} \cdot \frac{\exp(0.28 x_{n+3})}{1 + \exp(0.28 x_{n+3})}. \end{aligned} \end{equation}$$

Notice that the first model predicts a fixed probability of each outcome and so the prediction for the event of interest does not depend on the explanatory variables for the predicted outcomes. The second model uses logistic regression and so the estimated probability of the event of interest does depend on the explanatory variables. If you add explanatory variables for the predicted values you can calculate the latter, and then it is trivial to figure out the difference in the estimated probabilities under the two models.

$\endgroup$
  • $\begingroup$ Thanks for the answer, but no, I am not asking about the predicted probabilities of outcomes under different models. That would be too easy. I want to know how much I could potentialy reduce $\hat{P}(\mathcal{E})$ if I utilize the explanatory power of $\mathcal{M}_R$ compared to the baseline. $\endgroup$ – Davor Josipovic Jun 12 '18 at 5:46
  • $\begingroup$ Wouldn't that just be the difference between the two estimates? $\endgroup$ – Reinstate Monica Jun 12 '18 at 5:54
  • $\begingroup$ Take $T_1$ for example. It might be that due to $X_1$ that $\hat{P}(T_1=1|X_1) = 0.8$. Which means $0.8-0.6 = 0.2$ is explained by the model. I want to reverse this and thus say that we "could" reduce $\hat{P}(T_1=1|X_1) = 0.8$ to $0.4$. Its the theoretically correct approach to quantify this reduction that I am interested in. Written like this it pretty much amounts to $R^2$ in a binary setting for which I found some papers like this one and am evaluating it. $\endgroup$ – Davor Josipovic Jun 12 '18 at 6:13
  • $\begingroup$ One of the theoretical questions I have is whether this difference of $0.2$ can be reversed just like that. Adding $0.2$ to $0.6$ is not the same as subtracting. If the baseline was $0.2$ and $\mathcal{M}_R$ predicts $0.5$, then subtraction would result in $-0.1$. Much depends on the model and approach. $\endgroup$ – Davor Josipovic Jun 12 '18 at 6:24
  • $\begingroup$ I updated the question, making it more clear. You might want to update your answer. $\endgroup$ – Davor Josipovic Jun 19 '18 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.