Imputing data using covariance?

Question

Suppose I have some samples of sensor data, where each row has ten measurements from various sensors. And suppose I know what the covariances are among these sensor measurements. Are there any techniques of using my covariance matrix to imputing missing sensor data if a given row only has five out of ten sensor measurements for example?

Or are there better techniques that don't use covariances directly but use some other relationships in the data?

Answer 1

Gaussian model

Suppose there are $k$ sensors and let $x \in \mathbb{R}^k$ be a vector representing sensor readings for a single measurement. I'll assume that measurements are i.i.d. Gaussian with known mean $\mu$ and covariance matrix $C$:

$$p(x \mid \mu, C) = \mathcal{N}(x \mid \mu, C)$$

You may not know the true form of the joint distribution. But, if sensor readings take unrestricted real values and you know only the mean and covariance matrix, then it makes sense to use a Gaussian model because this is the maximum entropy distribution consistent with this knowledge.

Imputation

Let $x$ be partitioned as $x = \begin{bmatrix} x_m \\ x_o \end{bmatrix}$ where $x_m$ is a vector of missing values and $x_o$ is a vector of observed values. All we've done here is permuted the indices so that that the missing values are listed first (you can do this separately for each measurement where you want to run imputation, since each will have different missing values). Similarly, let the (permuted) mean and covariance matrix be partitioned as:

$$\mu = \begin{bmatrix} \mu_m \\ \mu_o \end{bmatrix} \quad \quad C = \begin{bmatrix} C_{mm} & C_{mo} \\ C_{om} & C_{oo} \end{bmatrix}$$

Inference of the missing values is based on the conditional distribution of $x_m$, given the observed values in $x_o$. Since the joint distribution is Gaussian, the conditional distribution is also Gaussian:

$$p(x_m \mid x_o) = \mathcal{N}(x_m \mid \mu_{m \mid o}, C_{m \mid o})$$

with mean and covariance matrix:

$$\mu_{m \mid o} = \mu_m + C_{mo} C_{oo}^{-1} (x_o - \mu_o)$$

$$C_{m \mid o} = C_{mm} - C_{mo} C_{oo}^{-1} C_{om}$$

If you wanted to impute a single value for $x_m$, then it would make sense to use the conditional mean $\mu_{m \mid o}$, since this is the expected value (having observed $x_o$). However, this doesn't account for uncertainty about the missing values. Ideally, you'd want to take advantage of the full conditional distribution $p(x_m \mid x_o)$, since it captures the full knowledge and uncertainty about the missing values. How to do this depends on your goals and downstream analysis. For example, multiple imputation can be performed by sampling from $p(x_m \mid x_o)$.

Unknown parameters

Above, we assumed the mean and covariance matrix were known, since this was stated in the question. However, it may be necessary to estimate these parameters from the data. In this case, the type of missing data must be carefully considered (e.g. does the missingness of measurements depend on the missing values?). The validity of various inference procedures depends on this. Assuming the data are missing at random (MAR), the EM algorithm could be used to jointly estimate the parameters and infer the missing values.

More complicated models

As mentioned above, the Gaussian model makes sense given only knowledge of the mean and covariance matrix. It assumes linear relationships between the sensors, and additive Gaussian noise. If a reasonably large dataset of sensor measurements is available, it may be possible to use other models and techniques that capture more complex forms of dependence. There are too many possibilities to go into detail here. But, one example would be more complicated probabilistic models fit to the data (e.g. using the EM algorithm to handle missing values). Alternatively, there are many imputation methods based on nearest neighbors and various forms of regression.

Answer 2

The short answer is yes - that is indeed possible. You have to first set the mean vector of your sample to zero:

1. $\mu^T=(0,\ldots,0)^T$, your covariance matrix is $\Sigma$,
2. Compute the eigen vector/eigen values $\Sigma=B\Lambda B^T$, with $B$ the orthogonal eigenvectors and $\Lambda$ the eigenvalues of $\Sigma$,
3. Order the eigenvectors in $B$ according to the size of the respective eigenvalue: $\lambda_g \geq \lambda_{g-1} \ldots \lambda_{1}$,
4. Define the partial diagonal matrix $E^{(h)}$ with on the nonzero diagonal entries the $h$ largest eigen values, and zeros on the remaining diagonal entries (all off-diagonal entries of $e$ are also zero), compute $W^{(h)}=B E^{(h)} B^T$, which equals

$ \begin{split} W^{(h)} = \sum_{j=g-h+1}^g W^j \end{split} $

where $W^j = {\bf b}_j {\bf b}_j^T$

5. The closed form solution to your missing value problem is now defined from

$ {\hat {\bf x}}^{m} = (I_m - W_m^{(h)})^{-1} W_{mk}^{(h)} {\hat {\bf x}}^{k} $

where $I_m$ is the $m \times m$ identity matrix, and $W^{(h)}$ had been partitioned into the four submatrices

$ \begin{split} W^{(h)} = \left[ \begin{matrix} W_{k}^{(h)} & W_{km}^{(h)}\\ W_{mk}^{(h)} & W_{m}^{(h)} \end{matrix} \right] \end{split} $

Here $k$ is associated with the known variables and $m$ with the missing variables.

The smaller $h$ is the more noise is filtered out - you can use this as a data regularization term.

Later, I will add a numerical example to this answer.