0
$\begingroup$

I fitted a weighted regression model to predict age as a function of several DNA methylation markers (expressed in percentages). I used weighted regression because the variance of my original OLS model increases with age.

When using the predict function to generate prediction intervals for a set of new samples,

predict(fGLS, newdata = Testset, interval = "prediction", level = 0.95)

I get the following warning:

Warning message:
In predict.lm(fGLS, newdata = Testset, interval = "prediction",  :
  Assuming constant prediction variance even though model fit is weighted

I tried adding the same weights I used to fit the model and this no longer yielded a warning;

predict(fGLS, newdata = Testset, interval = "prediction", level = 0.95,
        weights = 1/hhat)

I have two questions:

  1. Am I correct in simply adding the same weights I used to fit the weighted regression model, to the predict function? What does this effectively do?

  2. In the first situation, my prediction intervals are roughly the same size throughout the data in my test set. In the second situation, the prediction intervals become larger with increasing age. Does this mean my prediction intervals in the first situation are wrong? Or is it okay to have equal interval sizes since I "corrected" for heteroskedasticity by using weighted regression? In other words, can I afford to simply ignore the warning?

$\endgroup$
  • 1
    $\begingroup$ I have applied edits to this to make this not a question about R or statistical programming. $\endgroup$ – AdamO Nov 15 '16 at 19:44
2
$\begingroup$

There seems to be some confusion about the purpose of a prediction interval.

If I have frequency weights, then if my weights vector has some element Weights[i] = 10, this indicates for the i-th factor level, there were 10 such people/observations having a similar distribution of characteristic factors.

That weight is endemic to the model and the model alone. When you calculate prediction intervals, it is for an independent 11th person or observation: the uncertainty of the prediction interval is a sum of the uncertainty in your estimates (confidence interval) as well as their individual uncertainty (sampling error).

If in a contrived way, you assume you conduct an independent study and resample another 10 or even 20 people for that i-th factor level and you are interested in prediction intervals for their aggregate mean, you can simply calculate this yourself using a (1/sqrt(10) + 1/n)*se scale for the prediction interval.

Your problem is easily understood by trying to replicate results obtain from predict commands with interval='confidence' and interval='prediction' arguments.

However, it seems in your case that the purpose of weighting here was precision weighting. In that case, you are correct to re-apply the weights, this should yield wider prediction intervals for more highly varied factor levels (higher age and more varied methylation). You can easily check this result for yourself.

$\endgroup$
  • $\begingroup$ I am definitely interested in obtaining prediction intervals, not confidence intervals, since the ultimate goal is to be able to predict the age of a single person based on their DNA. So when I add the weights, is it okay if I use the weights I got from fitting the model to my trainingset? Or would it be better to use the estimated weight of the new datapoint (so the sample I am actually trying to predict the age for)? For example, if I used wi = 1/xi to fit my model. Should I use the xi from the training set, or the xi from the new sample, to get the prediction interval for that sample? $\endgroup$ – I. Smeers Nov 21 '16 at 12:05
0
$\begingroup$

In hindsight this seems like a rather dumb question, but instead of deleting it I will just answer it myself since I have since figured out the solution.

The weights used for fitting the model based on the training set, are obtained from the predictors through an auxilary model, and are specific for those samples only. When applying the fitted model to new data, weights for those new samples can be obtained through the existing auxilary model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.