# Optimizing regression coefficients to predict the largest outcomes

What is a sound methodology to improve the efficiency of the regression coefficients when we are interested in predicting the larger values of the marginal distribution (tails)?

For example, we want to predict seismic waves based on a number of covariates recorded by our probes. Data can not be assumed to be strictly linear, given that most earthquakes develop abruptly after a certain covariates threshold is reached. The vast majority of observations are not considered harmful and should be down-weighted in our analysis. What we are really interested in estimating are the more extreme outcomes.

[thoughts...] Weighted least square comes to mind, but how should the weights be calculated? Is quantile regression with, say, $\tau = [0.2, 0,8]$ a better approach? [/thoughts]

• I would second quantile regression, but be careful to not go into the very extreme quantiles. If the larger values are qualitatively different, my suggestion would be to use a finite mixture or latent class model. – tchakravarty Jun 10 '12 at 12:12
• @FgNu Thanks. Why the warning on extreme quantiles? Which $\tau$ levels would you consider extreme? – Robert Kubrick Jun 10 '12 at 13:34
• Well, the mathematical principle is easy to state - assume that you are trying to estimate the $100p$th quantile, where $p\in(0, 1)$, and define $k_n = \lfloor np_n \rfloor$ to be the $k$th sample order statistic ($n$ is the sample size). Then, the sample order statistic is called extreme if $k_n - n = o(1)$. In practice, I would be wary of modelling any above the 90th quantile with quantile regression for moderate sample sizes. – tchakravarty Jun 10 '12 at 22:32

## 2 Answers

It seems that you either need to relax linearity assumptions for covariates (e.g., use restricted cubic splines, i.e., natural splines) or to define a new optimality criterion (other than maximum likelihood) and then to optimize the fit to that. I'm glad you clarified that you don't want to discard observations. The optimality criterion will define the effective weights; you don't have to.

Least squares will give automatic emphasis to fitting more extreme $Y$ values at the expense of higher mean absolute error or median absolute error.

• Doesn't least square minimizes the overall sum of squared residuals? Let's say I have 1,000 observations < 2 and only 1 observation at 10: wouldn't least square optimize the coefficients to minimize the errors of the 1,000 points, rather than the single error of the larger point at 10? – Robert Kubrick Jun 10 '12 at 15:35
• Correct, but the "far out" points are effectively given much more weight because of minimizing the sum of squared residuals rather than, for example, the sum of absolute residuals. With heavy-tailed $Y$ you will find accuracy for predicting in the middle sometimes sacrificed so that major errors are minimized. – Frank Harrell Jun 10 '12 at 16:38
• Got it. Another subtle difference to quantile regression where, if I'm not mistaken, most regressors minimize the sum of absolute errors, rather than squared. – Robert Kubrick Jun 10 '12 at 20:12
• Also, should I choose to use a robust MM estimator I would loose the natural least square adjustment towards the larger observations, right? – Robert Kubrick Jun 10 '12 at 20:21
• @Frank Harrell, are there methods that deliberately weight or scale Y for heavy-tail estimation ? – denis May 29 '13 at 13:20

What you have written suggests that logistic regression, with attention to diagnostics and residuals, will be your best bet. It can help you take into account nonlinearity in the relationship between predictor and outcome You'll want to test multiplicative effects (interactions), as the thresholds you talk about may be joint thresholds involving multiple predictors. I am concerned, though, by your statement that "The vast majority of observations are not considered harmful and will be discarded from our analysis." In order to explain what causes the events of interest, it is absolutely necessary to know something about the conditions that do not produce it, just as it is to know about the ones that do.

• Here we're not after what causes the event, but the outcome of a similar rare event. In this setting, aren't the majority of observations considered 'outliers'? – Pardis Jun 10 '12 at 11:26
• What I meant by "discarded" is actually down-weighted. In robust regression, the observations showing larger residuals are given lower weight to improve the efficiency of the coefficients. Here the focus is on large outcomes, so I was thinking to apply the same methodology (the larger the outcome, the larger the weight) to improve the accuracy of the coefficients. – Robert Kubrick Jun 10 '12 at 13:44
• @rolando2 I think the comment above makes what I was asking clear. In robust regression, we don't consider outliers to be "conditions that do not produce the desired outcome" but observations we'll rather have removed in a pre-processing step. What I was asking was that in this setting, should the observations with small outcomes be considered outliers? – Pardis Jun 10 '12 at 15:17
• @Pardis You're right, that's what I had in mind when I posted the question: if robust regressors are used to down-weight observations as the residuals grow larger, why wouldn't there be a technique to down-weight the smaller to null observations. As Frank observed in his answer, least square naturally overweights the larger outcomes. But I'm still a bit puzzled on why I haven't found this problem stated more often. It seems to me that focusing on large outcomes is a common problem, like in the example I described. – Robert Kubrick Jun 10 '12 at 20:54