7
$\begingroup$

ZCA whitening can use regularization, as in

$$ \tilde{X} = L\sqrt{(D + \epsilon)^{-1}}L^{-1}X, $$

where $LDL^\top$ is an eigendecomposition of the sample covariance matrix. What's a good choice for the regularization parameter $\epsilon$?

I suppose that one could separately do unregularized ZCA whitening on the held-out data $X'$:

$$ \tilde{X'} = L'\sqrt{D'^{-1}}L'^{-1}X' $$

and then choose $\epsilon$ that minimizes the difference between such held-out whitened data and the held-out data whitened using the regularized ZCA developed using the training data:

$$ \tilde{Y}(\epsilon) = L\sqrt{(D + \epsilon)^{-1}}L^{-1}X' $$

$$ \epsilon^* = \mathrm{argmin} \|\tilde{Y}(\epsilon) - \tilde{X'}\| $$

I wonder though if there are easier or more principled approaches to choosing $\epsilon$ or regularizing PCA/ZCA in general.

Improve this question
$\endgroup$
5
  • $\begingroup$ I guess this is a special case of a more general problem: how to best choose $\epsilon$ in the shrinkage estimator of the covariance matrix $$\boldsymbol \Sigma = \mathbf{XX}^\top/(n-1) + \epsilon \mathbf I.$$ ZCA whitening is just one possible application of such a shrinkage estimator of covariance matrix. Regularized linear discriminant analysis (rLDA) and ridge regression are some other common examples. There people usually cross-validation to choose optimal $\epsilon$, and this can indeed be more tricky for whitening (because there is no response variable here)... $\endgroup$
    – amoeba
    Commented Apr 6, 2015 at 19:31
  • $\begingroup$ Why do you expect a cross-validation approach described in your question to be reasonable? The problem with unregularized whitening is that sample covariance matrix can have very low (or even zero) eigenvalues, making its inverse highly unstable. If this happens on the held-out data, then its unregularized whitening can be completely off and should not be considered as a target for cross-validation. $\endgroup$
    – amoeba
    Commented Apr 7, 2015 at 20:32
  • $\begingroup$ @amoeba $X'$ lies in the subspace spanned by the non-zero eigenvalue PCs, so exploding the orthogonal components should not affect $\tilde{X'}$, but I see your point w.r.t. the numerical stability. It's probably better to use a small $\epsilon' << \epsilon^*$. $\endgroup$
    – MWB
    Commented Apr 7, 2015 at 21:13
  • $\begingroup$ Even then, I don't get your cross-validation approach at all. As far as I can see, it won't work. +1, good question though. $\endgroup$
    – amoeba
    Commented Apr 7, 2015 at 21:17
  • $\begingroup$ @amoeba my last comment was more intuition than rigorous analysis. I'll try to think about this some more later. $\endgroup$
    – MWB
    Commented Apr 7, 2015 at 21:17

1 Answer 1

Reset to default
7
$\begingroup$

If the data was Gaussian distributed with mean $0$ and unknown covariance $\Sigma$ and we put an inverse-Wishart prior on $\Sigma$, \begin{align} \Sigma &\sim \mathcal{W^{-1}}(\Psi, \nu), \\ x &\sim \mathcal{N}(0, \Sigma), \end{align} the posterior expectation of $\Sigma$ would be $$\frac{XX^\top + \Psi}{n + \nu - p - 1},$$ where $n$ is the number of data points and $p$ is the dimensionality of the data. Choosing $\Psi = I$ and $\nu = p + 1$, for example, we would get $$\frac{XX^\top + I}{n} = C + \frac{1}{n}I = L\left(D + \frac{1}{n}I\right)L^\top,$$ where $C = XX^\top/n$. A sensible choice for $\epsilon$ therefore might be $1/n$.

You could go one step further and properly estimate the covariance using a normal-inverse-Wishart prior, i.e., taking the uncertainty of the mean into account as well. Derivations for the posterior can be found in (Murphy, 2007).

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ +1. Can you think of any reasonable approach to use cross-validation for choosing $\epsilon$? E.g. in ridge regression or in linear discriminant analysis the covariance matrix is also estimated with a shrinkage estimator $XX^\top/n + \epsilon I$, but there we can find an optimal $\epsilon$ by cross-validating. But I don't see how one could do it for whitening purposes, as there is no response variable here. $\endgroup$
    – amoeba
    Commented Apr 7, 2015 at 9:47
  • 1
    $\begingroup$ I guess one could evaluate the likelihood of the held-out data under a Gaussian model with covariance $C + \varepsilon I$, where $C$ is the sample covariance of the training data. But since whitening is only used as a preprocessing step (be it for classification, regression, or something else), it would be best to cross-validate with respect to the objective function one is ultimately trying to optimize. $\endgroup$
    – Lucas
    Commented Apr 7, 2015 at 11:14
  • $\begingroup$ Thanks! AFAICT though, there is a great deal of arbitrariness in the choice of $\Psi$, or even its scale, as well as in the choice of $\nu$. Wouldn't that affect the choice of $\epsilon$? $\endgroup$
    – MWB
    Commented Apr 8, 2015 at 1:47
  • $\begingroup$ ... I guess $\nu$ doesn't matter for large $n$, but if $\Psi = 100 I$, then we must choose $\epsilon = 100/n$? $\endgroup$
    – MWB
    Commented Apr 8, 2015 at 3:06
  • $\begingroup$ it would be best to cross-validate with respect to the objective function one is ultimately trying to optimize -- the later stages are often much more expensive, and this essentially adds a hyperparameter. It would be best if $\epsilon$, or some other regularization, could be chosen independently. $\endgroup$
    – MWB
    Commented Apr 8, 2015 at 3:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.