2
$\begingroup$

Suppose the linear model is $y = \beta x + \epsilon$, where $X \sim \mathcal{N}(0, 1), \epsilon \sim \mathcal{N}(0, s^2)$. If we only observe the sign of the output $y_i$, and the number of observations is large, $i$ how can we estimate $\beta$?

$\endgroup$
2
  • 3
    $\begingroup$ via probit regression :) $\endgroup$ Mar 16 at 18:49
  • $\begingroup$ @JohnMadden Care to expand that into an answer? $\endgroup$
    – Dave
    Mar 16 at 18:50

1 Answer 1

1
$\begingroup$

Probit regression operates under the assumption that there is some latent linear model that has Gaussian errors, but we only see the signs. This is exactly the situation described in the question, so probit regression sounds like the way to go.

A simulation shows this to be pretty slick and to give estimated coefficients that are close to the OLS coefficients and close to the true values.

set.seed(2023)
N <- 10000
x <- rnorm(N, 0, 1)
b0 <- -1
b1 <- 1
y <- b0 +b1*x + rnorm(N)

# Set indicator for positivity of y
#
z <- y > 0

# Fit a linear model to the original x and y
#
L <- lm(y ~ x)

# Fit a probit model to x and the indicator for the sign of y
#
P <- glm(z ~ x, family = binomial(link = "probit"))

summary(L)$coef[, 1]
summary(P)$coef[, 1]
> summary(L)$coef[, 1]
(Intercept)           x 
 -0.9939896   1.0080528 
> summary(P)$coef[, 1]
(Intercept)           x 
 -0.9853826   0.9783042 

EDIT

Worth a mention is that the standard errors on the probit model are higher. Since there is less information available to the probit model (the z <- y > 0 line is lossy compression of the y variable), I suspect that this is not just a fluke. Consequently, if you have the original y, it seems that you can get tighter coefficient estimates by using the linear model, even though you can do something reasonable if you only have the lossy-compressed z <- y>0.

$\endgroup$
5
  • $\begingroup$ @AdamO It seems pretty spot-on if I set b0 <- 0 and use 0 + x in the regression calls to force the intercepts to be zero. Perhaps you can post an answer clarifying what happens when the intercept is omitted (though I do suspect this just to be sloppy notation in the original question). $\endgroup$
    – Dave
    Mar 16 at 19:36
  • $\begingroup$ ah right I made a mistake editing the code and kept the intercept as non-zero. $\endgroup$
    – AdamO
    Mar 16 at 19:47
  • 1
    $\begingroup$ @AdamO Do you mean that you just made a coding error when you modified my code in your own R sessions? I was hoping to learn about some theoretical concern about intercept-free probit models. $\endgroup$
    – Dave
    Mar 16 at 19:49
  • $\begingroup$ Yes, well the interesting bit is that you say probit is for when we only observe the sign (+/-) when, in fact, the response can be any threshold $z = y > \theta$ and the $\beta$ is still estimable but only when the intercept term is included. The interpretation of the intercept becomes a complicated mix of $\theta$ and the actual shift $a$ of the untransformed response $y = a + b x + e$. $\endgroup$
    – AdamO
    Mar 16 at 19:56
  • $\begingroup$ @AdamO You mean if there is a nonzero intercept but we do not model it in the probit regression? $\endgroup$
    – Dave
    Mar 17 at 3:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.