A contrast suggested by the data

One approach to testing hypotheses suggested by the data is Scheffé's method. According to the Wikipedia article, the contrast $C = \sum_{i=1}^r c_i \mu_i$ (where $\sum_{i=1}^r c_i=0$) is estimated by the corresponding sample contrast $\hat C$, and the variance of the latter is estimated by $s^2_{\hat C} = \hat\sigma^2_e \sum_{i=1}^r c_i^2/n_i$ (where $n_i$ is the size of the sample from the $i$th population and $n=\sum_{i=1}^r n_i$), and with probability $1-\alpha$ all confidence intervals of the form $\hat C\pm s_C\sqrt{(r-1)F_{\alpha,r-1,n-r}}$ are valid (they all contain the parameters that they estimate). "All" means for all contrasts.

So I'm looking at a variation on this situation. Instead of multiple comparisons, I've got this simple linear regression: $y_i = \alpha+\beta x_i+\text{error}_i$, with 22 data points, no two of which have a common $x$ value. 20 of them look as if fitting a line through them is reasonable. The other two are set apart from that line, and the line through those two points is very close to the same in slope as is the line you get from fitting the model with only the other 20 points.

So I've hazarded a quick guess as to how to emend Scheffé's method for this situation. (Maybe later I'll work through the details carefully and see if I can prove that my guess makes sense.)

• Introduce an indicator variable $w$ equal to $1$ in the 20 cases and equal to $2$ in the other two. Fit the model $y_i=\alpha + \gamma w_i+ \beta x_i+\text{error}_i$.
• Instead of $s^2_C$ use the usual mean squared error from this fit.
• Instead of contrasts among $Y_i$, use contrasts among the RESIDUALS from the model WITHOUT THE INDICATOR.
• Instead of $(r-1)F_{\alpha,r-1,N-r}$, use $(22-1)F_{\alpha,22-1,22-3}$ since there are $22-3$ degrees of freedom for error.

Fools rush in . . . . Will this still look plausible after I've checked everything?

PS: Note my comment below. And if this is the wrong thing to do, is there a right thing?

• I have a considerable qualm, but there's a reason not to retreat from the things that incites it: the indicator variable I've chosen was of course itself suggested by the data. BUT in the situation that Scheffé considered, the validity of the estimate of error variability was undisturbed by the fact that one of those contrasts might actually be far from zero. Not so in this example. Is that a major difficulty that arises in this problem but not in the problem that Scheffé solved? – Michael Hardy May 24 '13 at 21:45
• What are you trying to compare? In what way are the two set apart? Is the slope and intercept similar or just the slope? And in general it's unwise to even consider testing patterns in data based solely on the pattern in the data. – John May 24 '13 at 22:59
• Perhaps a better starting point than effectively fitting two different models (one for 20 points, another for only 2) would be to actually try and figure out why these two points differ from the mass, and perhaps why they appear to relate in similar fashion to your predictor. In trying to figure out why 2 of your 22 cases are different--why they're outliers--maybe you'll stumble across something more interesting than overfitting your data with post-hoc adjustments to your hypothesized model. – Patrick Coulombe May 24 '13 at 23:31
• @PatrickCoulombe : OK, so how would you propose to test whether those two are outliers and whether they're the only ones? – Michael Hardy May 25 '13 at 4:31