I am working with a two-step heckman selection model. In the first step the selection occurs based on a probit model, in the second step the mean equation is fitted with a linear model where the inverse mills ration is included.

Technically I assume the existence of a partly latent variable $Y\in \mathbb{R}$, where we only observe $\Delta Y \in \mathbb{R_+}$, with $\Delta=I(Y>0)$ beeing an indicator function that is one if $Y$ is greater than zero which is seen as equivalent to $Y$ beeing observed. Furthermore we assume that the latent variable $Y$ can be modelled using a linear model, using some covariates $X$: $Y=X\beta + \epsilon$ and that we can use covariates $Z$ in order to employ a probit model for the probability of response, i.e. $P(\Delta=1|Z)=\Phi(Z\gamma)$, with $\Phi(\cdot)$ denoting a standard normal CDF.

Using this, plus the assumption that the errors from both stages are jointly normal, as well as $Z\subset X$, we can arrive at $$E(\Delta Y|X,\Delta=1)=E(Y|X,\Delta=1)=X\beta + E(\epsilon|X,\Delta=1)=X\beta + \rho \sigma_\epsilon\lambda(Z\gamma) $$ with $\lambda(\cdot)$ representing the inverse Mill Ratio evaluated at $Z\gamma$.

The estimation is done based on maximum likelihood, hence I can compare different models based on the AIC and/or the likelihood ratio test. However I also want to evaluate the Goodness-of-Fit for both stages separately.

For the first stage, the probit model, I can employ ROC Curves and calculate the area under the curve (AUC). Additionally I could rely on Precision-Recall Curves. But what about the second stage? I cannot employ the measure $R^2$ since the mean equation is not a linear regression model if I am condition on the selection. At the moment I am relying on calculating the mean squared residuals of the mean equation given $\Delta=1$: $$MSR=n^{-1}\sum_{i=1}^n(\Delta Y_i -X_i \hat{\beta}-\hat{\rho}\hat{\sigma}_{\epsilon}\lambda(Z_i \hat{\gamma}) )^2$$ I have not seen this measure anywhere else but I would use it in the following way: if the $MSR$ of model A is smaller than the $MSR$ of model B, model A provides the better fit, conditional on the selection.

Alternatively I could probably use the $R^2$ based on the fitted latent model $Y=X\hat{\beta}+\hat{\epsilon}$ but then the idea of evaluating both stages separately, first the selection and then the volume given the selection, is gone. Do you have same suggestions or solutions?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.