0
$\begingroup$

I am using sklearn IPCA decomposition and surprised that if I delete duplicates from my dataset, the result differs from the "unclean" one.

What is the reason? As I think, the variance is the same.

Explained variation per principal component with duplicates:

8.08489473e-01 1.42712763e-01 3.40174443e-02

Explained variation per principal component on "clean" dataset:

5.23586790e-01 2.62231842e-01 9.43627488e-02

$\endgroup$
6
  • 1
    $\begingroup$ Duplicates give more weight to your data, more certaintly, so it is natural that removing them removes some certainty. $\endgroup$ Dec 14, 2018 at 8:00
  • $\begingroup$ @user2974951 I cannot understand "weight". In my opinion, PCA roughly just "rotates data". How does the result depends on duplicates? $\endgroup$
    – cuga
    Dec 14, 2018 at 8:59
  • $\begingroup$ This is not related to PCA at all, this is about data. $\endgroup$ Dec 14, 2018 at 9:12
  • $\begingroup$ @user2974951 And how does the variance change deleting duplicates? $\endgroup$
    – cuga
    Dec 14, 2018 at 9:45
  • $\begingroup$ The variance increases if you delete duplicates. As for what this means for PCA, that is another matter. $\endgroup$ Dec 14, 2018 at 9:47

1 Answer 1

2
$\begingroup$

Quick example:
mean and variance of c(1,2,3) is 2 and 1,
mean and variance of c(1,1,2,2,3,3) is 2 and 0.8.

Duplicate values do not change the mean value in this context (if we duplicate every single value), but the variance does change, because of the formula for variance, which divides the estimate by $n-1$, that is the sample size. Even though we did not introduce any new information in the sample, we have increased its size, so the variance changes because we are more "sure" in our results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.