# What tool can we use to assess if there are enough data?

Imagine some paired data ${(x_i,y_i)}_{i=1}^n$ representing the results of two different measurement methods and the question is about the quantification of the bias between the two methods. Let us assume that the normality assumption is reasonable for $x_i-y_i$ hence we simply compute an estimate and a confidence interval of the mean difference.

Of course the width of the confidence interval is an indicator of how precise is the quantification of the bias, but what other tool(s) can we use to assess if there enough data for the quantification to be reliable ? I have in mind a "cross-validation" (I quote because I am ignorant on this topic): for example we could assess whether the result changes when we drop a data value. Is there a standard way to perform such an assessment ?

I am also interested in performing this assessment in the R software.

-
 What do you precisely mean by "bias"? Note that this is an statitical concept and if you estimate $\mu_x-\mu_y$ with $\bar{x}-\bar{y}$ then the bias is $0$. – user10525 May 31 '12 at 13:46 Yes I meant $\mu_x-\mu_y$ (the "bias" is not only the well-known statistical concept in the common language). – Stéphane Laurent May 31 '12 at 15:33

This doesn't sound like a classification problem. I am not sure why you feel the need to do anything else besides getting a confidence interval. If it is that you think there may be one or two outliers that have an inappropriste influence on the estimate of the bias of the mean difference you can use the influence function for the mean difference parameter to identify such outliers.

-

You can plot the learning curves: cross validate with 10%, 20%, ... 100% of the training data and see the evolution of the training and test errors. If the test error reaches a plateau then you have enough data. If the training and test errors reach the same value, then you model is (too) biased and a richer model with more degrees of freedom might perform better and might benefit from more data.

A good overview blog post on the topic:

A gist in python using scikit-learn and matplotlib:

https://gist.github.com/1540431

-
 Are you sure this is adapted to my case ? What is a biased model ? My model is just a random sampling from a normal distribution. – Stéphane Laurent May 31 '12 at 10:52 +1 This is relatively simple to do and very handy! – CarrKnight May 31 '12 at 12:03 In your case you can use the width of the confidence interval as the "error" function. The consideration about the difference between the training error and the testing error is not really interesting here as your model is simple and I don't see how a more complex model would be any more useful indeed. – ogrisel May 31 '12 at 15:31

What about plotting a histogram / density plot of the pairwise differences?

Yes, you could botostrap or jack-knife (cross-validate) your results, but I don't really see the point here.
Unless you have a particular reason that you want to know how much the estimate changes with x % less data.

About the learning curve: that would here be e.g. width of confidence interval over no. of samples. But as long as the CI is calculated from normal or t distribution, the "learning curve" is most probably determined by that assumption assumption rather than by the particular behaviour of your data...

-