Suppose we have some logistic regression modelling problem; $f(X) = Y$, where $Y$ is binary and $X$ is a vector of normally distributed variables.
In industry it is sometimes the case that practitioners will delete rows of data where 'nothing happens' (according to an algorithmic criteria), often to reduce the dimensions of the regression (data can get quite big) or because of the perception that this 'nothing happening' is not what we want to model - we want the model only to capture when interesting events occur.
Needless to say this can bias out of sample probability estimates (i.e., given some new $X$, our estimate of $P(Y=1)$ is biased). A quick R
demonstration will show this, where the true data generating process has $Y=1$ half the time and $Y=0$ the other half but our subsampling approach has this ratio to $\frac19$:
predictions <- rep(NA,1000)
y <- c(rep(1,5000),rep(0,45000))
x <- rnorm(50000)
#y <- c(rep(1,25000),rep(0,25000))
#x <- rnorm(50000)
fit = glm(y~x,family=binomial(logit))
for(i in 1:1000) {
newX = data.frame(x=rnorm(1))
predictions[i]=predict(fit,newX,type="response")
}
mean(predictions)
plot(predictions)
Question
What I would like to ask is whether our fit, $\hat{f}$, can be salvaged in the following way. For every new $X$ that we observe, we only estimate $\hat{P}(Y-1)$ if and only if it would have passed through our filter had this new $X$ been part of our in sample dataset. Intuitively I want to say yes because in-sample we have the data generating process that we artificially created, $f_\text{intrusion}(X) = Y_\text{intrusion}$, and this is the same DGP that we are drawing from out of sample to get our probability estimates.