I have seen many places where they have input/output datasets where they first create a linear regression line, correct the bias, and then only use that data for their model. I didn't get what this bias correction is?

  • 6
    $\begingroup$ I think you may need to supply a reference or an explicit example so we can know precisely what it is you are reacting to. $\endgroup$
    – whuber
    Jun 27 '12 at 22:14
  • $\begingroup$ @naught101, please do a few at a time, don't spam the main page. $\endgroup$ Sep 22 '15 at 1:33
  • $\begingroup$ @gung: ah.. you mean tag a few and then wait a bit? Sorry, too late. I only found 10 or so, and I just did them all. Forgot about the front page effect :/ If only SE had a nice mass-tagging feature. $\endgroup$
    – naught101
    Sep 22 '15 at 1:35
  • $\begingroup$ @gung: Maybe today can be the inaugural bias-correction day :D $\endgroup$
    – naught101
    Sep 22 '15 at 1:38
  • 1
    $\begingroup$ @naught101 unilateral mass retagging is a bit of a no-no, especially on a tag you just made. Broadly it's best to engage on meta where feasible (to explain what you intend), and if it seems uncontroversial, then to do some retagging but only a few at a time. $\endgroup$
    – Glen_b
    Sep 22 '15 at 5:48

Although the problem statement is not precise enough to know exactly what type of bias correction you are referring to, I think I can talk about it in general term. Sometimes an estimator can be biased. This merely means that although it may be a good estimator, its expected or average value is not exactly equal to the parameter. The difference between the estimator's average and the true parameter value is called the bias. When an estimator is known to be biased, it is sometimes possible, by other means, to estimate the bias and then modify the the estimator by subtracting the estimated bias from the original estimate. This procedure is called bias correction. It is done with the intent of improving the estimate. While it will reduce the bias it will also increase the variance. So for it to be useful the improvement in bias must be large relative to the loss in the variance.

A good example of successful bias correction is the bootstrap bias correction estimates of classification error rate. The resubstitution estimate of error rate has a large optimistic bias when the sample size is small. The bootstrap is used to estimate the bias of the resubstitution estimate and since the resubstitution estimate underestimates the error rate the bias estimate is added to the resubstitution estimate to get the bootstrap bias corrected estimate of the error rate. When the sample size is small 30 or less combining both classes in a two class problem certain forms of the bootstrap estimate (particularly the 632 estimate) provide more accurate estimates of the error rates than leave-one-out cross validation (which is a very nearly unbiased estimate of error rate).

  • 2
    $\begingroup$ "While it will reduce the bias it will also increase the variance." - can you explain that a bit more? Doesn't it depend on the method? Do you basically mean that reducing the bias of an RMSE-optimal linear regression will necessarily increase the RMSE, or is it something else? $\endgroup$
    – naught101
    Sep 22 '15 at 23:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.