23
$\begingroup$

We usually use PCA as a dimensionality reduction technique for data where cases are assumed to be i.i.d.

Question: What are the typical nuances in applying PCA for dependent, non-i.i.d. data? What nice/useful properties of PCA that hold for i.i.d. data are compromised (or lost entirely)?

For example, the data could be a multivariate time series in which case autocorrelation or autoregressive conditional heteroskedasticity (ARCH) could be expected.

Several related question on applying PCA to time series data have been asked before, e.g. 1, 2, 3, 4, but I am looking for a more general and comprehensive answer (without a need to expand much on each individual point).

Edit: As noted by @ttnphns, PCA itself is not an inferential analysis. However, one could be interested in generalization performance of PCA, i.e. focusing on the population counterpart of the sample PCA. E.g. as written in Nadler (2008):

Assuming the given data is a finite and random sample from a (generally unknown) distribution, an interesting theoretical and practical question is the relation between the sample PCA results computed from finite data and those of the underlying population model.

References:

Improve this question
$\endgroup$
  • 14
    $\begingroup$ Just for note. PCA itself is not an inferential analysis. It is a transformation of multivariate dataset of numbers; its core is just svd or eigendecomposition. Therefore it does not make observation independence assumption. Assumptions arise when we use PCA as a statistical tool to analyze samples from populations. But they are not PCA's assumptions. For example, testing for sphericity to decide if PCA is justified to reduce the data does require the indepencence, and the test may look as if a "within-PCA" assumption test, but actually it is an "outside" test. $\endgroup$ – ttnphns Aug 2 '16 at 11:15
  • $\begingroup$ @ttnphns, very good points, thank you. If you see a neat way to edit my post, feel free to. I will think about it myself as well. $\endgroup$ – Richard Hardy Aug 2 '16 at 11:41
  • 1
    $\begingroup$ Richard, your question is fine and important (+1). Just maybe I'd rather re-word it a bit in a manner like "We usually use PCA as a dimensionality reduction for data where cases are i.i.d. assumed... What are typical nuances in applying PCA for time series data where cases (time points) are lag-interdependent...?" $\endgroup$ – ttnphns Aug 2 '16 at 12:21
  • 1
    $\begingroup$ @amoeba, right. But we hardly ever stop at just obtaining the loadings of the PCs. In the steps that commonly follow PCA, what should we be aware of under non-i.i.d'ness? I hope an answer could be better than the question (in its current formulation). If you look at it loosely/creatively, perhaps you could come up with some good points. $\endgroup$ – Richard Hardy Aug 2 '16 at 13:08
  • 2
    $\begingroup$ Plain PCA respects only "horizontal" associations (i.e. between columns) and ignores "vertical" (between cases): covariance matrix of columns is the same if you shuffle order of cases. Whether this can be called "no assumptions for case serial relations" or "assumption for independent cases is made" is a matter of taste. The i.i.d. assumption is the default in data analysis, and so methods which simply do not pay special attention to case order, like PCA, could be imputed the "silent support" for the i.i.d assumption. $\endgroup$ – ttnphns Aug 2 '16 at 13:38
1
$\begingroup$

Presumably, you could add the time-component as an additional feature to your sampled points, and now they are i.i.d.? Basically, the original data points are conditional on time:

$$ p(\mathbf{x}_i \mid t_i) \ne p(\mathbf{x}_i) $$

But, if we define $\mathbf{x}_i' = \{\mathbf{x}_i, t_i\}$, then we have:

$$ p(\mathbf{x}'_i \mid t_i) = p(\mathbf{x}'_i) $$

... and the data samples are now mutually independent.

In practice, by including the time as a feature in each data point, PCA could have as result that one component simply points along the time feature axis. But if any features are correlated with the time feature, a component might consist of one or more of these features, as well as the time feature.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ Thanks for the answer. That would be a very special case where time enters linearly. A more widespread phenomenon is, for example, autocorrelation where time itself does not play a role as a feature. $\endgroup$ – Richard Hardy Mar 16 '17 at 10:06
  • $\begingroup$ Ok, I see. So, you mean that, eg example $x_t$ is not just a function of some parameters $\theta$, but depends also on $x_{t-1}$? Therefore, $x_t$ is Markov, given $x_{t-1}$ and $\theta$? So, can we then add $x_{t-1}$ as a feature into the PCA? (I'm not saying we can or cant, just thinking through the problem really...) $\endgroup$ – Hugh Perkins Sep 4 '17 at 15:45
  • $\begingroup$ Something like that, yes, but without adding $x_{t-1}$ as a feature because I am interested in PCA that is defined on the original variables.. $\endgroup$ – Richard Hardy Sep 4 '17 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.