1
$\begingroup$

In the post The role of validation in estimation and hypothesis testing Frank Harrell wrote:

If the adjustment variables were not pre-specified but determined by data dredging, you really need to use a bootstrap procedure to penalize the target variable's confidence interval. Otherwise it is been shown that all the variables in the model will have confidence intervals that are too narrow.

Would this statement apply in the case where we have improved a model fit by e.g. addding interaction terms suggested by residuals plots? Im not sure whether this is considered "data dredging"? Clearly we have changed the model so it is no longer "pre-specified".

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

At the risk of agreeing with myself :-) I'll reinforce that any process that is supervised (i.e., makes use of $Y$), results in a bias towards something you want to brag about at the end. Consider the simple case where there is one continuous predictor, you assume it's linear, and learn from the residual plot that linearity looks OK, so you continue to model the predictor with one coefficient. This is completely equivalent to looking at a raw data scatterplot and looking for linearity (and other features such as constant variability of $Y$ across $X$). If you model the variable with 1 d.f. but had given it the chance to have 2.5 degrees of freedom (e.g. your eye was able to detect something as complex as quadratic or cubic on the average), your residual variance will be underestimated and p-values and confidence interval coverage will suffer if you compute them correctly by simulation.

All of this is why Bayesian modeling has an advantage and more truthfully reports uncertainty: have a linear term with a wide prior and nonlinear terms with priors tilted towards zero, if you believe linearity was a bit more likely than nonlinearity in $X$. Bayes will then allow nonlinearity to the extent the data will allow you to reliably estimate the nonlinear effect, and the credible intervals are accounting for the uncertainties.

Very short answer: if you can't assume the model is simple up front, and you entertain the notion that it's not simple, you have to either (1) just allow it to be flexible and stick with that flexible model, (2) be Bayesian, or (3) be aggressive but bootstrap yourself to adjust for the aggression.

Improve this answer
$\endgroup$
4
  • $\begingroup$ thanks for your prompt reply. I would be most grateful for some advice on how to go about the procedure of boostrapping to adjust for the aggression. I am referring to e.g. penalising confidence intervals, rather than e.g. bias corrected optimism of prediction error. Perhaps you can suggest a reference or you have an example somewhere? $\endgroup$
    – Daniel Smith
    Commented May 27, 2018 at 6:46
  • $\begingroup$ Here's an outline, although full model specification (and avoiding bootstrapping) is better. Write code to, for a single data sample, execute every look at the data you take that uses $Y$ and that encodes the decision you make. At the end of the code have it estimate the quantity of interest. If that quantity involves parameters not in the model at that moment, insert zero for the parameter estimate. Now run this code hundreds of times and compute the sample standard deviation of the quantity of interest over the bootstrap resamples. $\endgroup$
    – Frank Harrell
    Commented May 27, 2018 at 11:20
  • $\begingroup$ Thank you very much Frank, I think I get the idea. Nevertheless, if you do have some example R code demonstrating this I would be grateful if you could point me to it or send me by email. I tried to accept the answer but it wont let me due to low reputation points but it does the job...thanks again. $\endgroup$
    – Daniel Smith
    Commented May 29, 2018 at 1:17
  • $\begingroup$ The process you're using to develop the final model, since you do not believe in pre-specifying the entire model up front, has not been programmed yet so you'd need to write the R code yourself to execute all the data looks. $\endgroup$
    – Frank Harrell
    Commented May 29, 2018 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.