Can linear regression be improved by data without targets?

I have a standard data set for linear regression (X - 2D array with values of features and Y - the corresponding 1D array with targets). I can use this data set to perform a standard linear regression.

Now we modify the setting slightly, namely we say that many targets are not available. In other words, for some features-vectors we have values of targets and for same feature-vectors we don't. For example:

1.2 0.4 3.5 -> 1.4
0.7 2.4 5.7 -> NaN
...
7.8 9.4 0.2 -> 3.9
2.7 6.6 8.7 -> 2.7
3.3 4.3 5.3 -> NaN


As you can see, some targets are missing. A simple approach would be just to remove all the rows (data points) for which targets are not present.

However, if we look at analytical solution for linear regression:

$$C = (X^T \cdot X)^{-1} \cdot X^T \cdot Y = \frac{X^T \cdot Y}{X^T \cdot X}$$

we see that the first term ($$X^T \cdot X$$) is calculated without targets, so I thought that the complete X can be used (even those rows, that do not have any targets).

Of course we should first introduce a normalization in the nominator and denominator to count for the fact that in dot products we have different number of summation. For example if means of features and targets are equal to zero (we can always get it). In nominator we would just have covariance between features and target and in denominator we would have covariance matrix between features.

To make it more precise, I would use the following solution:

$$C = (X^T \cdot X)^{-1} \cdot X^T_{small} \cdot Y_{small} \cdot \frac{N}{N_{small}}$$

Where "small" indicate cases where target is present.

Target is absent really by chance. There are not systematic differences between the statistics of features with and without the targets.

• I am skeptical about any benefits because before any normalization is done your method is equivalent to setting NaN to zero. And if one just puts NaN to zero and then normalize it afterwards it seems more efficient and less complicated to start of by considering putting other values of NaN to begin with. Jul 4, 2023 at 10:35
• You don't explain the mechanism driving the NaNs, an important piece of your puzzle. Why are they missing? Jul 4, 2023 at 10:54
• Just a minor note, NaN means ‘not a number’ which is different from being missing. Jul 4, 2023 at 11:43
• @utobi, in my case missing values are represented by NaN. Jul 4, 2023 at 11:44
• Pairwise deletion seems what you are considering stefvanbuuren.name/fimd/sec-simplesolutions.html 1.3.2 Jul 4, 2023 at 12:27

In the analytical solution for linear regression C = ..., you cannot just change the first two occurrences of X to be something different than the third occurrence of X. Because then C = ... will no longer be the analytical solution for linear regression.