2
$\begingroup$

I'm constructing a model to predict the weight of a species of insect given a set of other variables. The plot below shows the performance of my model using a set of test data where the true weight of the insects is known. The x-axis is the true weight of the insect and the y-value is the associated error of my model—the absolute value of the predicted weight - true weight:

enter image description here

From this visual, you can see that I have many insects with relatively low weights. In these cases, my model has a relatively lower predictive error. In contrast, I have relatively few insects which are heavy and the error associated with these predictions displays much more variance along with higher error.

Given this model and the test data, I'd like to find a way to construct confidence intervals for new predictions. For example, if my model predicts a given insect is relatively heavy, the confidence intervals around this prediction would be large. My questions is, how can I do this? A linear model seems inappropriate for this data since most of the points are clustered near the origin. I'm at a loss for how I can construct confidence intervals for my predictions.

$\endgroup$
8
$\begingroup$

Bootstrap sample your training data many times (let's say, N times) and train a model from each bootstrapped sample (giving N models). Calculate a prediction on your test set using each model (giving N predictions for each point in your test set). This will allow you to calculate a confidence band for each test point's prediction. For plotting purposes, you may find it useful to fit a LOESS curve (or some other type of smoothing method) to the plot of insect size vs. CI for each element of your test set, giving an estimated mean confidence as a function of insect size.

$\endgroup$
  • $\begingroup$ Bootstrapping is naturally a versatile and useful method, but getting the model qualitatively right must be done first. It doesn't sounds as if that has been done yet. $\endgroup$ – Nick Cox Aug 4 '13 at 15:26
  • 2
    $\begingroup$ Whether the model is correct or not, you can still generate confidence intervals for the predictions it gives. The confidence bands will just further evidence that the model you have isn't a good one. The question was specifically how to construct confidence bands for the given model. $\endgroup$ – David Marx Aug 4 '13 at 15:41
  • $\begingroup$ Thanks for the help. I was thinking of the bootstrap, but wasn't quite sure how to set it up. I will use both of your suggestions. $\endgroup$ – turtle Aug 4 '13 at 16:38
1
$\begingroup$

There is no exact detail here on what model you used, but you are right in suspecting linear models for this kind of application.

The most appropriate kind of model is likely to be a generalised linear model with logarithmic link which will ensure that positive weight is predicted, even for very small insects, a desideratum here. Even with logarithmic link, error in the first instance means observed weight $-$ predicted weight.

You should indicate what software you are using so that people who also use it can tune in and indicate precisely which functions or commands or routines might help.

$\endgroup$
  • $\begingroup$ Thanks for the help. The log-transformation is a good suggestion; I didn't think of that for some reason. $\endgroup$ – turtle Aug 4 '13 at 16:37
  • $\begingroup$ Generalised linear model with log link is not the same as log transforming the response. $\endgroup$ – Nick Cox Aug 4 '13 at 22:04
  • $\begingroup$ Do you have any reference to the log link GLM? I tried searching, but didn't find any resources to what you are referencing. $\endgroup$ – turtle Aug 5 '13 at 12:00
  • $\begingroup$ It is covered in every account of GLMs that I have ever encountered, including en.wikipedia.org/wiki/Generalized_linear_models The book by Dobson and Barnett there referenced is good. Assuming that you are a ecologist, or find their literature fairly easy to read: most intermediate survey texts on statistics for ecologists include a chapter or so on GLMs. amazon.com/Ecological-Models-Data-Benjamin-Bolker/dp/0691125228 should be more congenial than most. $\endgroup$ – Nick Cox Aug 5 '13 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.