0
$\begingroup$

I am working with a set of historical data that I did not collect myself. I cannot add to this data set in any way. For a 350-year period I have many thousands of data points, each associated with one of ~50 time points. Some of the time points have hundreds of data points associated with them, so I'm sure that those time points can be used in the linear regression. However, others have very few, even just one.

So my question is, how can I determine a cut-off for the time points that have enough data (and should be included in my regression) and the time points that do not have enough data (and should be excluded)? I'd like to do this in R.

Also, perhaps this is relevant - I'm interested in the minimum for each time point, not the mean. Thank you!

$\endgroup$
1
$\begingroup$

There may be no need to exclude any of the data completely. Time points with few associated observations will necessarily be weighted less in the analysis than will the time points with many observations. And if that intrinsic correction in the linear regression is insufficient, based on your understanding of the underlying subject matter, the lm() function in R has an option to include weights for individual observations. So if you have independent information on data reliability at different observation times, you can use that option in R to incorporate that information.

$\endgroup$
  • 1
    $\begingroup$ Great, thank you. I've looked up the weights function and it seems that I can simply create a vector with the number of data points for each time point, and that will correctly weight them. Is that the best way to do this? $\endgroup$ – Charcha Sep 17 '13 at 15:34
  • 1
    $\begingroup$ Since you are interested in the minimum value at each time point rather than the mean, this is a little tricky. The weight should be related to the expected variance of the value at each time point. Minimum values are not as nicely distributed as mean values, so the weight by number of observations might not be the best, and analyses on minimum values are highly prone to errors from outliers. If you are interested in reliable low values at each time point, you might want to consider analyzing 5th or 10th percentiles at each time point instead of the minimum value. $\endgroup$ – EdM Sep 17 '13 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.