1
$\begingroup$

I have a standard data set for linear regression (X - 2D array with values of features and Y - the corresponding 1D array with targets). I can use this data set to perform a standard linear regression.

Now we modify the setting slightly, namely we say that many targets are not available. In other words, for some features-vectors we have values of targets and for same feature-vectors we don't. For example:

1.2 0.4 3.5 -> 1.4
0.7 2.4 5.7 -> NaN
...
7.8 9.4 0.2 -> 3.9
2.7 6.6 8.7 -> 2.7
3.3 4.3 5.3 -> NaN

As you can see, some targets are missing. A simple approach would be just to remove all the rows (data points) for which targets are not present.

However, if we look at analytical solution for linear regression:

$ C = (X^T \cdot X)^{-1} \cdot X^T \cdot Y = \frac{X^T \cdot Y}{X^T \cdot X} $

we see that the first term ($X^T \cdot X$) is calculated without targets, so I thought that the complete X can be used (even those rows, that do not have any targets).

Of course we should first introduce a normalization in the nominator and denominator to count for the fact that in dot products we have different number of summation. For example if means of features and targets are equal to zero (we can always get it). In nominator we would just have covariance between features and target and in denominator we would have covariance matrix between features.

ADDED:

To make it more precise, I would use the following solution:

$ C = (X^T \cdot X)^{-1} \cdot X^T_{small} \cdot Y_{small} \cdot \frac{N}{N_{small}} $

Where "small" indicate cases where target is present.

Target is absent really by chance. There are not systematic differences between the statistics of features with and without the targets.

$\endgroup$
7
  • 1
    $\begingroup$ I am skeptical about any benefits because before any normalization is done your method is equivalent to setting NaN to zero. And if one just puts NaN to zero and then normalize it afterwards it seems more efficient and less complicated to start of by considering putting other values of NaN to begin with. $\endgroup$ Commented Jul 4, 2023 at 10:35
  • 1
    $\begingroup$ You don't explain the mechanism driving the NaNs, an important piece of your puzzle. Why are they missing? $\endgroup$
    – user78229
    Commented Jul 4, 2023 at 10:54
  • $\begingroup$ Just a minor note, NaN means ‘not a number’ which is different from being missing. $\endgroup$
    – utobi
    Commented Jul 4, 2023 at 11:43
  • $\begingroup$ @utobi, in my case missing values are represented by NaN. $\endgroup$
    – Roman
    Commented Jul 4, 2023 at 11:44
  • $\begingroup$ Pairwise deletion seems what you are considering stefvanbuuren.name/fimd/sec-simplesolutions.html 1.3.2 $\endgroup$
    – seanv507
    Commented Jul 4, 2023 at 12:27

1 Answer 1

0
$\begingroup$

In the analytical solution for linear regression C = ..., you cannot just change the first two occurrences of X to be something different than the third occurrence of X. Because then C = ... will no longer be the analytical solution for linear regression.

So the answer is "no".

$\endgroup$
10
  • 1
    $\begingroup$ I do not do this. $\endgroup$
    – Roman
    Commented Jul 4, 2023 at 11:39
  • 1
    $\begingroup$ I perform a normalization. In the nominator I have covariance between features and the target in the nominator I have covariance between features. I just estimate these two object with different number of observations. $\endgroup$
    – Roman
    Commented Jul 4, 2023 at 11:44
  • $\begingroup$ OP consider the case y=x $\endgroup$
    – seanv507
    Commented Jul 4, 2023 at 12:17
  • 1
    $\begingroup$ How y can be equal to x? y is a number and x is a vector. $\endgroup$
    – Roman
    Commented Jul 4, 2023 at 12:26
  • $\begingroup$ Consider univariate case $\endgroup$
    – seanv507
    Commented Jul 4, 2023 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.