1
$\begingroup$

I am fitting a linear model to predict a variable which is a type of performance of animal behaviour. Let's call it performance.

When the model makes a prediction of performance for an animal, I want to use this to perform Monte Carlo simulation from a normal distribution. My performance prediction will be the mean of the sampling distribution, but I'm getting stuck on how I should calculate a variance for each animal's performance. It's quite important, because the variability of an animal's performance is equally if not more important than the performance prediction itself.

What I'd like is if I could get some sort of confidence interval for the predictions from the linear model. Then I could choose a variance based on the width of the interval. I've already tried predict(model, data, type=confidence) in R but it returns tiny intervals for each point that don't really make sense for what I need.

How can I "predict the unpredictability" of the performance response variable?

$\endgroup$
1
$\begingroup$

Let's say you have a simple linear regression model relating an outcome variable Y to a predictor variable X. The predict() function has two options for this type of model model:

  1. type = "confidence";
  2. type = "prediction".

The first option is to be used when you are interested in estimation and the second when you are interested in prediction.

For example:

Use the first option (type = "confidence") when you are interested in estimating the mean value of Y for all the subjects in the target population who are have an X value equal to x, where x is known and falls within the range of observed X values for the sample of subjects who generated the data used to fit the model.

Use the second option (type = "prediction") when you are interested in predicting the individual value of Y for a randomly selected subject from the target population who has an X value equal to x, where x is known and falls within the range of observed X values for the sample of subjects who generated the data. (This subject is NOT part of the sample of subjects whose data were used to fit the model.)

Comment:

If I understand things correctly, you have multiple animals and possibly multiple "runs" per animal and it seems that, for each of these "runs", you are measuring "performance" as well as 10 different variables which may predict this "performance".

For simplicity's sake, assume for a moment that all animals have 50 "runs" and that you only care about one predictor variable. A mixed model is akin to fitting a collection of linear models - one per animal - such that each model relates "performance" to the predictor variable expected to predict it using the data from the 50 "runs". The mixed model can assume that the effect of the predictor variable under consideration is different across animals. The mixed effects model will estimate the effect of this predictor variable for the "typical" animal. However, you can then assess this effect for all animals in your sample. Some will have higher effects than the one corresponding to the "typical" animal and some will have lower effects. I am not sure what you mean by "inconsistent" performance? Perhaps you mean that the predictor variable of interest might not have an effect on performance? Or might have a negligible effect?

$\endgroup$
  • $\begingroup$ Thanks for your reply - I have tried type="prediction" but I have found that the size of the interval returned is the exact same for every point. This contradicts what I need - the intuition is that some animals have a very wide interval because they are inconsistent performers, while some have very narrow intervals because they are less prone to inconsistency. What I'm looking for is an interval for each prediction that varies in size depending on the predictor variables for that observation. Does that make sense? $\endgroup$ – John F Oct 30 '18 at 18:47
  • $\begingroup$ Can you explain more about your study design? How many animals do you have? How many observations of what type for each animal? $\endgroup$ – Isabella Ghement Oct 30 '18 at 19:11
  • $\begingroup$ There are about 30k animals. For each animal I have between 1 and 50ish "runs" i.e. how they've performed in the past. I'm trying to predict their performances in the next run based on about 10 variables from their past runs. i.e. animals with a high value for variable x tend to perform better. But what I'm stumped with is: how can I quantify if animals with a low value for y tend to perform inconsistently ? For these animals I want to set up the Monte Carlo sampler to use a more suitable (greater) variance. $\endgroup$ – John F Oct 30 '18 at 19:56
  • 2
    $\begingroup$ Interesting - why not use some kind of mixed effects modeling to accommodate all animals? It seems like linear modeling might not be sufficient in this context. $\endgroup$ – Isabella Ghement Oct 30 '18 at 21:09
  • 1
    $\begingroup$ I'm going to look into that. Thats for the idea! $\endgroup$ – John F Oct 31 '18 at 1:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.