Logistic Regression with (Normal) Distributions for Independent Variables

Question

Consider the logistic regression where $Y_i \in {0,1}$ are dependent variable observations and $X_i \in \mathbb{R}$ are the independent variables.

However we do not observe the $X_i$ themselves. Instead we observe some vector of parameters $\boldsymbol{\mu}_i$ and we know the distribution $F$ s.t. $F(X_i|\boldsymbol{\mu}_i)$.

How can we perform a logistic regression where the $X_i$ are not themselves observed? For $F$ a general distribution, one way may be to sample for each $i$, many values from $F(X_i|\boldsymbol{\mu}_i)$ and put them all into the regression model as observations.

Can we do things more efficiently if we know for example that $F$ is a normal distribution?

Answer 1

I think you can also go for a maximum likelihood approach considering the $x_i$ are latent variables over which you marginalize a likelihood.

Let's say the likelihood of your usual logistic regression, if you oberved the $x$ values, is $\mathcal{L}(\beta, x, y)$ where $\beta$ is the vector of parameters (typically, $\mathcal{L}(\beta, x, y) = (\frac{1}{1 + e^{-\beta x}})^y (\frac{1}{1 + e^{\beta x}})^{1 - y}$).

Then the likelihood only observing $\mu$ and $y$ is $$\mathcal{L}(\beta, y, \mu) = \mathbb{E}_{X \sim F_{\mu}}[\mathcal{L}(\beta, y, X)]$$ And the total likelihood is just the product of the likelihoods over all observed $(y_i, \mu_i)$.

Unfortunately, these expectations may be untractable (maybe for a simple normal distribution it is not, but it is not obvious to me...), so you can estimate them by Monte Carlo. For instance, sample $x_i \sim F_{\mu_i}$ and take empirical mean of $\mathcal{L}(\beta, y_i, x_i)$. I don't think that this is equivalent to simulating data according to $F_{\mu_i}$ and put them into the model, but it would be nice to see the links...

Another way would be to go with an E-M algorithm (where $x_i$ are the latent variables) to maximize this likelihood, this would vertainly be more computationally efficient.

I hope this helps a little bit...

Answer 2

Generalizing the bootstrap method proposed in the question, in which the regression does not attempt to estimate $X$ but instead determines how the distribution $F$ leads to a distribution of logistic regression parameters, one could use marginal maximum likelihood estimation, a technique common in random effects linear models. The likelihood to be maximized is the marginal likelihood $$ \mathcal{L(\beta)} = \prod_i \int_X P(y_i|X,\beta) P(X|\mu_i) \, X $$ Such likelihoods can rarely be solved exactly, but the existing literature may give some inspiration -- and the idea of estimating this $\mathcal{L}$ (or rather the $\beta$ that maximizes it) by Monte Carlo could be a good one. In the case that the distribution is normal, there might be hope of doing something more exact. Assuming for notational simplicity that $E[X|\mu] = \mu$ (just sop that I don't need to make a new variable), $$ \log \mathcal{L(\beta)} \propto \sum_i \log \int_X \left(\frac{1}{1+\exp(-\beta X)} \right)^{y_i} \left(\frac{1}{1+\exp(\beta X)} \right)^{1-y_i} \exp \left( - \frac{1}{2} (X-\mu_i)^T \Sigma^{-1} (X-\mu_i) \right) $$ Since either $y_i = 0$ or $y_i = 1$, this integral is known as a logistic-normal integral, and there is some accessible literature on it:

https://books.google.com/books?hl=en&lr=&id=iaieM_3lcHQC&oi=fnd&pg=PR5&ots=CM9147oK0H&sig=SegYdLgH2UtTDmcspTix2fnBgRg#v=onepage&q=logistic-normal%20integral&f=false

This book, in fact, is probably a good reference in general, as it examines this integral in the context of logistic regression with random effects, probably directly applicable to the posed question.