Suppose that I am interested in selecting from one of the following model spaces:

$$\begin{align*} y &= \beta_0 + \beta_1 x_i + \epsilon_i \, \, \text{or} \\ \log (y_i) &= \gamma_0 + \gamma_1 x_i + \eta_i. \end{align*}$$

I intend to use cross validation for making my selection.

If I calculate the prediction errors on the validation set, the first model will give errors in $y$, while the second will give errors in $\log(y_i)$. Obviously, it doesn't make sense to compare these.

Instead, for the second model, suppose that I estimate

$$\begin{equation*} \text{E}[y_i \mid x_i] = \exp \left\{\hat{\beta}_0 + \hat{\beta}_1 x_i \right\} \times \exp \left\{ \hat{\sigma}^2 /2 \right\}, \end{equation*}$$

using an assumption that $\epsilon_i \mid x_i \sim N(0, \sigma^2)$ (alternatively, I could just use a Poisson model with robust standard errors/oversidsperion to get the same result; see here for a nice explanation). Then, I compare this predicted $y$ to the observed validation $y$ to get my prediction errors.

Is this procedure sensible? Will it generally have good properties?

  • $\begingroup$ I assume you use sum of squared error to compare 2 models. If so then your first model will win for sure because it directly minimizes sum of squared error in original scale. $\endgroup$ – FMZ Mar 7 '12 at 4:49
  • $\begingroup$ @FMZ, Not necessarily on the validation sample. Plus, the first model will be the best linear fit, but not necessarily the best among all possible models. $\endgroup$ – Charlie Mar 7 '12 at 5:27
  • $\begingroup$ Agreed. Let's see how other people respond to your questions. $\endgroup$ – FMZ Mar 8 '12 at 3:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.