Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 0 down vote favorite

Imagine some paired data ${(x_i,y_i)}_{i=1}^n$ representing the results of two different measurement methods and the question is about the quantification of the bias between the two methods. Let us assume that the normality assumption is reasonable for $x_i-y_i$ hence we simply compute an estimate and a confidence interval of the mean difference.

Of course the width of the confidence interval is an indicator of how precise is the quantification of the bias, but what other tool(s) can we use to assess if there enough data for the quantification to be reliable ? I have in mind a "cross-validation" (I quote because I am ignorant on this topic): for example we could assess whether the result changes when we drop a data value. Is there a standard way to perform such an assessment ?

I am also interested in performing this assessment in the R software.

share|improve this question
What do you precisely mean by "bias"? Note that this is an statitical concept and if you estimate $\mu_x-\mu_y$ with $\bar{x}-\bar{y}$ then the bias is $0$. – user10525 May 31 '12 at 13:46
Yes I meant $\mu_x-\mu_y$ (the "bias" is not only the well-known statistical concept in the common language). – Stéphane Laurent May 31 '12 at 15:33

3 Answers

up vote 3 down vote

This doesn't sound like a classification problem. I am not sure why you feel the need to do anything else besides getting a confidence interval. If it is that you think there may be one or two outliers that have an inappropriste influence on the estimate of the bias of the mean difference you can use the influence function for the mean difference parameter to identify such outliers.

share|improve this answer
I am a statistician consultant. The client asks me to evaluate the bias and asks if there are enough data. "The confidence interval is narrow" is not a very satisfactory answer. Rather we could wonder whether the estimate of the bias could change if there were more data. Assessing the influence of outliers seems to be a good point. Do you know a link describing this so-called "influence function" ? Using Google I only find things related to semi-parametric or nonparametric statistics. – Stéphane Laurent May 31 '12 at 11:18
The OP asked for something other that a CI to assess the bias. Certainly a larger sample size will help with the accuracy of the confidence interval and the estimated variance used to obtain the confidence interval would help decide on an adequate sample size but that does not answer the oPs question either. Influence functions were developed by Hampel as a measure to assess robustness of estimators you can find it in numerous sources. (1) My paper "The influence function and its application to data validation" AJMMS 1982. Books on robustness. – Michael Chernick May 31 '12 at 12:13
Peter Huber has one, Hampel coauthored one and there are others. Gnanadesikan discusses it in his multivariate book. Searching on Google will give you loads of references including a description in Wikipedia. – Michael Chernick May 31 '12 at 12:14
Thansk for the references. What is the OP ? the client ? Why do you say he asks something other than a CI ? He wants to know the bias with precision, so a CI is a good way. – Stéphane Laurent May 31 '12 at 13:02
My understand is that on this site we commonly use the term OP for the person asking the original question. I think it stands for original poser. The OP wrote Of course the width of the confidence interval is an indicator of how precise is the quantification of the bias, but what other tool(s) can we use to assess if there enough data for the quantification to be reliable ?" That indicates that he is looking for another tool. I agree that there is nothing much else to do short of taking more samples. – Michael Chernick May 31 '12 at 14:23
show 4 more comments
up vote 2 down vote

You can plot the learning curves: cross validate with 10%, 20%, ... 100% of the training data and see the evolution of the training and test errors. If the test error reaches a plateau then you have enough data. If the training and test errors reach the same value, then you model is (too) biased and a richer model with more degrees of freedom might perform better and might benefit from more data.

A good overview blog post on the topic:

http://digitheadslabnotebook.blogspot.com/2011/12/practical-advice-for-applying-machine.html

A gist in python using scikit-learn and matplotlib:

https://gist.github.com/1540431

share|improve this answer
Are you sure this is adapted to my case ? What is a biased model ? My model is just a random sampling from a normal distribution. – Stéphane Laurent May 31 '12 at 10:52
+1 This is relatively simple to do and very handy! – CarrKnight May 31 '12 at 12:03
In your case you can use the width of the confidence interval as the "error" function. The consideration about the difference between the training error and the testing error is not really interesting here as your model is simple and I don't see how a more complex model would be any more useful indeed. – ogrisel May 31 '12 at 15:31
up vote 1 down vote

What about plotting a histogram / density plot of the pairwise differences?

Yes, you could botostrap or jack-knife (cross-validate) your results, but I don't really see the point here.
Unless you have a particular reason that you want to know how much the estimate changes with x % less data.

About the learning curve: that would here be e.g. width of confidence interval over no. of samples. But as long as the CI is calculated from normal or t distribution, the "learning curve" is most probably determined by that assumption assumption rather than by the particular behaviour of your data...

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.