6
$\begingroup$

I have the following linear model:

$$w^*=\text{arg min}_w\sum_{i=1}^N \bigg(Y_i-\sum_{j=1}^M X_{i,j}\times w_j\bigg)^2$$

Let $T \in N^*$ and $e_i=|Y_i-\sum_{j=1}^M X_{i,j}\times w_j|$.

It's possible using logistic regression to predict which errors will be less than $T$ (i.e., $e_i<T$) and greater or equal with $T$ (i.e., $e_i \ge T$)?

Here is more information to make the question clearer:

$N$ represent the number of observations. My data has the following property: the histogram of errors using multiple linear regression has a Laplace distribution. My data come from digital images represented on 8 bits. The $Y_i$ are current pixels and $X_{ij}$ are neighborhoods pixels. I want to predict which pixels produce errors less than $T$. I want to know what R functions can I use to make a test? $T$ is not very large, it has the values between 1 and 15 in general.

$\endgroup$
2
  • $\begingroup$ What do you mean by "$N^*$"? What kinds of values can $Y_i$ have? To what data, precisely, do you propose applying logistic regression? $\endgroup$
    – whuber
    Commented May 29, 2012 at 17:29
  • $\begingroup$ It would be more usual to define $e_i$ without the $|.|$ and then compare $|e_i|$ with $T$. What's $N^*$? $\endgroup$
    – Glen_b
    Commented Jul 5, 2014 at 5:04

4 Answers 4

4
$\begingroup$

I think this would only work if the logistic regression model had access to relevant covariates that are missing from the OLS model. In such a situation (i.e., model misspecification), there could be regions in which the observed response values diverge strongly from the predicted values and the logistic regression model would have the requisite information to detect them.

$\endgroup$
1
  • 1
    $\begingroup$ Such an approach might instead be picking up variance misspecification, perhaps the variance changes according to some covariates that are in the OLS model for the mean. $\endgroup$
    – Glen_b
    Commented Jul 5, 2014 at 5:06
4
$\begingroup$

You are trying to use logistic regression to find a possible structure to your residuals. Residuals should be unstructured. If logistic regression picks up something the model is misspecified. Great.

Important: logistic regression looks for a very specific structure. Your method works only if residuals are broken in a very specific way.

For this sort of detection use non-linear regressions. Kernel or Additive regression on the squared/log residuals are far more useful.
Even better: they will allow you to model the whole variance. This answers your question of where the large errors really are.

$\endgroup$
2
  • $\begingroup$ What means that "Residuals should be unstructured"? $\endgroup$ Commented May 29, 2012 at 19:22
  • 1
    $\begingroup$ +1, I think what he means is that standard regression assumes the residuals are independent & identically distributed as a Gaussian w/ constant variance. If this is true, then by definition the logistic regression won't be able to find a relationship b/t $P(e_i\geq T)$ & your covariates. This is what I was trying to get at, although CarrKnight has given a clearer & more comprehensive version. $\endgroup$ Commented May 29, 2012 at 20:36
3
$\begingroup$

It would be possible in the case where the original model was heteroskedastic and the heteroskedasticity was related to the covariates. For example,

$y_i \sim \text{N}(x_i^T\beta, \sigma^2x_{i,1}^2)$

where the variance of the $i^{th}$ observation is proportional to the square of the first covariate.

One can imagine, in non-Normal regression situations, similar structures that don't require heteroskedasticity per se. However, the default assumptions of regression models in general involve the errors being independent of the regressors, and the variance of the errors being constant.

On the other hand, if doing, for example, Poisson regression, you're out of the linear model world, but since the variance of the "error" is proportional to the mean, it follows that it is related to the covariates, and such a logistic regression would work - although it would convey no information not already conveyed by the results of the Poisson regression, which fully specify the conditional distributions of the $y_i | x_i$. In the generalized linear / additive model framework, where the likelihood is fully specified, the only way you'll be able to add information to the initial regression by using the logistic regression you suggest is if the initial regression has misspecified (usually by ignoring) the structure of the residuals, e.g., ignored the heteroskedasticity in the linear model presented above.

Nonetheless, your suggestion might reveal something about the structure of the residuals in an exploratory analysis. I suspect, though, that effectively discretizing the residuals by $< T$ or $\ge T$ would usually decrease the information content of them more than it would help clarify the analysis - unless it was an outlier analysis, perhaps.

$\endgroup$
1
  • $\begingroup$ +1 for heteroskedasticity, which I should have thought of myself. $\endgroup$ Commented May 29, 2012 at 18:31
2
$\begingroup$

There are circumstances where such a thing might be made to work - assuming you can supply suitable predictors to your logistic regression, but

(i) dichotomizing the spread may be less useful than leaving it continuous

(ii) I think formal hypothesis testing is a bad idea for assessing model assumptions, since it doesn't answer a useful question (imagine, in large samples, a very small trend in spread - it might be highly significant, but really quite unimportant in terms of its impact on the inference in the original model). More important to think about the effect-size (how much impact on our inference?) rather than the significance (is our sample size large enough to detect it?). We almost never satisfy the assumptions exactly, and we don't gain anything from testing what we already know - either we reject, which tells us nothing more than what we knew before, or we don't, which only tells us our sample size was to small to detect what we already know to be the case. Neither tells us how bad the failure of the assumptions might be for us.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.