When comparing a regression estimation method (Y vs X) I currently use a weighted squared loss function:

$$ \int_{-\infty}^{\infty}(\hat{f}(x)-f(x))^2 \, \hat{p}(x) \, dx $$

Where $\hat{f}(x)$ is the estimated function, $f(x)$ is an estimation method I'm comparing against, and $\hat{p}(x)$ is the estimated X density.

This works well when I assume the X observations are randomly sampled from the true density. However if X is not a random sample then weighting by the estimated X density does not make sense.

If I have some knowledge that some intervals of X are oversampled and some intervals are undersampled how should I adjust the weights of my loss function?

  • $\begingroup$ @glen Is there a way you know about the variance or s.d. of the sampled Xs? $\endgroup$
    – suncoolsu
    Aug 1, 2011 at 16:15
  • $\begingroup$ @suncoolsu Yes, I would have knowledge of what the variance of the X's are at various intervals, including the intervals that were oversampled/undersampled. $\endgroup$
    – Glen
    Aug 1, 2011 at 17:07

1 Answer 1


Your problem can be solved by borrowing ideas from weighted regression and/or survey sampling.

In your case for each non-randomly sampled $x_i$, you have an estimate of variance $\sigma_i^2$. Let $n_i$ be the number of replicates for $x_i$, then the probability of sampling from $x_i$ is $\frac {n_i}{\sum_i n_i}$.

If you use an idea similar to the Horvitz-Thompson Estimator, the estimator of error in $\hat f(x_i)$ must be weighted by $\big(\frac {n_i}{\sum_i n_i}\big)^{-1}$.

Therefore the estimate of squared error or loss function will be:

$$\displaystyle \frac {\sum_i w_i (\hat{f}(x_i)-f(x_i))^2} {N}$$

where $N=\sum_i n_i$ and $w_i=\big(\frac {n_i}{\sum_i n_i}\big)^{-1}$.

However, if you think having more information, that is higher $n_i$, implies better information about $(\hat{f}(x_i)-f(x_i))^2$, then the approach to solve your problem would be more like the ideas used in weighted linear regression. As you say, you have an estimate of variance $\sigma_i^2$ for each $x_i$, then the estimate of squared error or loss function will be:

$$\displaystyle \frac {\sum_i w_i (\hat{f}(x_i)-f(x_i))^2} {\sum_i w_i}$$

where $w_i=\frac {1}{\sigma_i^2}$. Note that here $x_i$s can have duplicate values.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.