2
$\begingroup$

I am thinking of running a regression of $Y ~ Xi$, but for each $X_i$, I have a pre-determined "score" $S_i$ in terms of quality of the data. The higher the score is, the more confidence I have in $X_i$.

So I want to run a modified regression of finding $w_i$ to minimize

$$| y - \sum w_i X_i |^2 + \lambda \sum w_i^2/S_i^2$$

Basically the penalty terms want me to give preference to the weights on $X_i$ that have "high score of confidence".

I wonder is there a problem with this formation? The closed form solution is

$w= (X^TX + \lambda/S^TS )^{-1}X^Ty.$ if I am not mistaken but I am not sure how to interpret this.

I'd also like to know if there are similar formation of this kind of "data quality regression" problem already.

$\endgroup$
6
  • 2
    $\begingroup$ Could you tell us quantitatively what "quality" is and how you arrive at the scores? Your problem sounds like an errors-in-variables regression. $\endgroup$
    – whuber
    May 19, 2023 at 17:53
  • $\begingroup$ @whuber quality can just be any kind of score, but it is "external" and known prior to regression. $\endgroup$
    – Matt Frank
    May 19, 2023 at 18:19
  • $\begingroup$ Unfortunately, that gives you no basis for using the scores in a regression. Moreover, if an $X_i$ happens to be useful for the regression -- imagine, as an extreme case, that it perfectly predicts $Y$ -- then why would you care about a low "confidence"? $\endgroup$
    – whuber
    May 19, 2023 at 19:11
  • $\begingroup$ @whuber confidence is probably not a good word. Think of it as a cost I guess. Its just more costly to assign more weights to a regressor in my use case. $\endgroup$
    – Matt Frank
    May 19, 2023 at 22:19
  • $\begingroup$ It is difficult to conceive of any way that would make sense for this kind of model. If, for instance, the "cost" were the cost to measure the value of the regressor, then so what? If it's too expensive to measure then you simply don't have very many observations of it. But the economics of your data collection procedure have no bearing on your statistical modeling. $\endgroup$
    – whuber
    May 20, 2023 at 15:26

1 Answer 1

0
$\begingroup$

Typically if we are more confident about certain data points than others, this is because there is more error in some of the measurements than in others. We can easily incorporate this into a linear model in at least one way. What you have done seems to be quite different. I will explain how this is different in a second. Most of this is pretty standard and you can find in most textbooks.

Let's start where I like to start, from Bayes' rule:

$p(\theta | D) = \frac{p(D | \theta)p(\theta)}{p(D)} = \frac{likelihood\hspace{0.1cm}* \hspace{0.1cm}prior}{marginal\hspace{0.1cm}likelihood}$

where $\theta$ is the parameters and $D$ is our data.

Very frequently, rather than finding the posterior, we just want to find the set of parameters for which $p(\theta | D)$ is at a maximum, the maximum of the posterior distribution -- the most probable set of parameters. This is called MAP (maximum a posteriori) estimation. If doing MAP estimation, you can disregard the denominator, which is a normalization constant for a given model, and just maximize the numerator.

For ridge regression, we are doing MAP estimation with an independent and identically distributed (iid) Gaussian likelihood and a Gaussian prior. In other words, if each datapoint $i$ has an associated feature vector $x_i$ and an associated real-valued label $y_i$, and there is a vector of weights we want to find $w$, then we say that:

$p(D | w) = \prod_\limits{i}^{N} N(y_i\hspace{0.1cm}|\hspace{0.1cm}x_i^T w,\hspace{0.1cm}\sigma_i^2)$

$p(w) = N(w\hspace{0.1cm}|0,\hspace{0.1cm}\lambda^{-2})$

If we can include a dummy feature that is always 1 as the last element of the input vector for each datapoint, then the y-intercept is just the last element of the weight vector.

Our prior here expresses a "prior belief" that the weights will be normally distributed and for the most part close to zero. $\lambda$ is in some sense quantifies the strength of our prior belief.

Notice $\sigma_i$, which is the standard deviation of the likelihood for a given measurement. As we'll see in a second, datapoints with a large associated standard deviation will end up being downweighted, so we will tolerate a larger absolute value in the error for those datapoints. If your "score" is the inverse of the variance in each measurement, i.e. if the larger the score, the smaller the measurement error, you could use $1/score$ as the standard deviation for a given datapoint, and this would downweight the datapoints with larger variance (smaller score) as we'll see shortly.

We can rewrite the likelihood as:

$p(D | w) = \prod_\limits{i}^{N} N(y_i\hspace{0.1cm}|\hspace{0.1cm}x_i^T w,\hspace{0.1cm}\sigma_i^2) = N(Y\hspace{0.1cm}|\hspace{0.1cm}Xw,\hspace{0.1cm}\Sigma)$

if X is a matrix where each row $i$ is vector $x_i$ and $Y$ is a vector containing all the Y values from $1...N$, and $\Sigma$ is a diagonal matrix where $\Sigma_{ii}=\sigma_i^2$ (this is where your confidence scores would go).

If we decide to minimize the negative log of the likelihood times the prior (equivalent to maximizing the likelihood times the prior, but easier), we find that we want to find theta such that:

$argmin_w\hspace{0.2cm} constant + \frac{1}{2}(Xw - Y)^T\Sigma^-1(Xw - Y) + \frac{1}{2}\lambda^2w^Tw$

Notice that the last term -- the ridge regression penalty -- comes from the prior. Because of the way we set up the prior, it basically acts to penalize large weight values. Thus what I am calling $\lambda$ and you are calling $K$ measures how strongly we want to penalize large weight values. Interestingly, you then divide $\lambda$ (or $K$) by the score, which doesn't really make sense -- I'm not sure how you got from there to your last equation. It $is$ possible to set up a prior with a diagonal covariance matrix (or even a dense covariance matrix), in which case our prior basically indicates that we think certain features are "more important" or more likely to have larger associated weight values than others, but that's not what you're trying to do. It looks like you've started out by modifying your prior for the weights with the score values which should actually reflect your confidence for the datapoints, so that may be where you went wrong.

But ok. Back to our derivation. Dropping the constant, eliminating the 1/2 and rearranging, we find:

$argmin_w\hspace{0.2cm} w^TX^T\Sigma^{-1}Xw - 2w^TX^T\Sigma^{-1}Y + Y^T \Sigma^{-1}Y + \lambda^2 w^T w$

Now differentiate w/r/t $w$ and set equal to zero (to find the minimum), which obtains:

$0 = 2X^T\Sigma^{-1}Xw - 2X^T\Sigma^{-1}w + 2\lambda^2w$

which is easily rearranged to yield:

$w = (X^T\Sigma^{-1}X + \lambda^2I)^{-1}X^T\Sigma^{-1}Y$

where I is the identity matrix. This I think is the expression you're actually looking for; confidence value for each data point goes on the diagonal of $\Sigma$. Just be careful as pointed out in the comments about interpretation if the score really corresponds to something quite different from the variance of the corresponding measurement.

Pretty much any statistical software you're using, whether statsmodels in Python or R or something else, will have an implementation of weighted linear regression like this. Hope that helps...

$\endgroup$
6
  • $\begingroup$ looking at the final solution, if we assume $\lambda = 0$, then the $\Sigma^{-1}$ cancel out? and we arrive at the ordinary linear regression solution? $\endgroup$
    – Matt Frank
    May 20, 2023 at 2:05
  • 1
    $\begingroup$ If you set each element of the diagonal of the sigma-matrix to be 1, and if lambda is zero, then the final expression simplifies to plain old ordinary least squares, where all the datapoints are unweighted and there is no regularization. $\endgroup$
    – HappyDog
    May 20, 2023 at 2:08
  • $\begingroup$ Ok i see your point ... finally. thanks ! $\endgroup$
    – Matt Frank
    May 20, 2023 at 2:16
  • $\begingroup$ But what if I really want to weight the regressor, not the data points though? like I mentioned in the comments, treat the regressor with a "cost". As in it costs me more to assign more weights to a regressor. $\endgroup$
    – Matt Frank
    May 20, 2023 at 2:17
  • $\begingroup$ So for linear regression, the number of weights (what I think you mean by regressors) is equal to the number of features associated with each datapoint (plus a dummy feature that is always 1 -- this way the weight vector automatically includes the y-intercept). So let's say for each datapoint, I have a single feature, x1, and then dummy feature 1; the vector associated with datapoint i is then [x1_i, 1]. To make a prediction, I'm going to take the dot product of this vector with a vector containing two weights. All I have is two weights, regardless of whether there are 100 or 1000 datapoints. $\endgroup$
    – HappyDog
    May 20, 2023 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.