Suppose the linear model is $y = \beta x + \epsilon$, where $X \sim \mathcal{N}(0, 1), \epsilon \sim \mathcal{N}(0, s^2)$. If we only observe the sign of the output $y_i$, and the number of observations is large, $i$ how can we estimate $\beta$?
1 Answer
Probit regression operates under the assumption that there is some latent linear model that has Gaussian errors, but we only see the signs. This is exactly the situation described in the question, so probit regression sounds like the way to go.
A simulation shows this to be pretty slick and to give estimated coefficients that are close to the OLS coefficients and close to the true values.
set.seed(2023)
N <- 10000
x <- rnorm(N, 0, 1)
b0 <- -1
b1 <- 1
y <- b0 +b1*x + rnorm(N)
# Set indicator for positivity of y
#
z <- y > 0
# Fit a linear model to the original x and y
#
L <- lm(y ~ x)
# Fit a probit model to x and the indicator for the sign of y
#
P <- glm(z ~ x, family = binomial(link = "probit"))
summary(L)$coef[, 1]
summary(P)$coef[, 1]
> summary(L)$coef[, 1]
(Intercept) x
-0.9939896 1.0080528
> summary(P)$coef[, 1]
(Intercept) x
-0.9853826 0.9783042
EDIT
Worth a mention is that the standard errors on the probit model are higher. Since there is less information available to the probit model (the z <- y > 0
line is lossy compression of the y
variable), I suspect that this is not just a fluke. Consequently, if you have the original y
, it seems that you can get tighter coefficient estimates by using the linear model, even though you can do something reasonable if you only have the lossy-compressed z <- y>0
.
-
$\begingroup$ @AdamO It seems pretty spot-on if I set
b0 <- 0
and use0 + x
in the regression calls to force the intercepts to be zero. Perhaps you can post an answer clarifying what happens when the intercept is omitted (though I do suspect this just to be sloppy notation in the original question). $\endgroup$– DaveMar 16 at 19:36 -
$\begingroup$ ah right I made a mistake editing the code and kept the intercept as non-zero. $\endgroup$– AdamOMar 16 at 19:47
-
1$\begingroup$ @AdamO Do you mean that you just made a coding error when you modified my code in your own R sessions? I was hoping to learn about some theoretical concern about intercept-free probit models. $\endgroup$– DaveMar 16 at 19:49
-
$\begingroup$ Yes, well the interesting bit is that you say probit is for when we only observe the sign (+/-) when, in fact, the response can be any threshold $z = y > \theta$ and the $\beta$ is still estimable but only when the intercept term is included. The interpretation of the intercept becomes a complicated mix of $\theta$ and the actual shift $a$ of the untransformed response $y = a + b x + e$. $\endgroup$– AdamOMar 16 at 19:56
-
$\begingroup$ @AdamO You mean if there is a nonzero intercept but we do not model it in the probit regression? $\endgroup$– DaveMar 17 at 3:47