# Weighted loss function for non-random sample

When comparing a regression estimation method (Y vs X) I currently use a weighted squared loss function:

$$\int_{-\infty}^{\infty}(\hat{f}(x)-f(x))^2 \, \hat{p}(x) \, dx$$

Where $\hat{f}(x)$ is the estimated function, $f(x)$ is an estimation method I'm comparing against, and $\hat{p}(x)$ is the estimated X density.

This works well when I assume the X observations are randomly sampled from the true density. However if X is not a random sample then weighting by the estimated X density does not make sense.

If I have some knowledge that some intervals of X are oversampled and some intervals are undersampled how should I adjust the weights of my loss function?

• @glen Is there a way you know about the variance or s.d. of the sampled Xs? Aug 1, 2011 at 16:15
• @suncoolsu Yes, I would have knowledge of what the variance of the X's are at various intervals, including the intervals that were oversampled/undersampled.
– Glen
Aug 1, 2011 at 17:07

Your problem can be solved by borrowing ideas from weighted regression and/or survey sampling.

In your case for each non-randomly sampled $x_i$, you have an estimate of variance $\sigma_i^2$. Let $n_i$ be the number of replicates for $x_i$, then the probability of sampling from $x_i$ is $\frac {n_i}{\sum_i n_i}$.

If you use an idea similar to the Horvitz-Thompson Estimator, the estimator of error in $\hat f(x_i)$ must be weighted by $\big(\frac {n_i}{\sum_i n_i}\big)^{-1}$.

Therefore the estimate of squared error or loss function will be:

$$\displaystyle \frac {\sum_i w_i (\hat{f}(x_i)-f(x_i))^2} {N}$$

where $N=\sum_i n_i$ and $w_i=\big(\frac {n_i}{\sum_i n_i}\big)^{-1}$.

However, if you think having more information, that is higher $n_i$, implies better information about $(\hat{f}(x_i)-f(x_i))^2$, then the approach to solve your problem would be more like the ideas used in weighted linear regression. As you say, you have an estimate of variance $\sigma_i^2$ for each $x_i$, then the estimate of squared error or loss function will be:

$$\displaystyle \frac {\sum_i w_i (\hat{f}(x_i)-f(x_i))^2} {\sum_i w_i}$$

where $w_i=\frac {1}{\sigma_i^2}$. Note that here $x_i$s can have duplicate values.