2
$\begingroup$

PCA gives me loadings with different signs. I understand that I may simply revert them (as explained in this thread), but I need an explanation (my boss insists) of why Matlab implementation gives me these signs, but not reverted.

UPDATE I found empirically, that in Matlab the loading with highest absolute value is always positive.

Improve this question
$\endgroup$
10
  • $\begingroup$ Also relevant: stats.stackexchange.com/questions/34396. $\endgroup$ – whuber♦ Feb 1 '17 at 19:30
  • 1
    $\begingroup$ @whuber I don't think it's a duplicate. This question specifically asks how Matlab chooses the sign. The fact that the sign is arbitrary seems to be clear to OP. But Matlab does not choose it randomly: if one runs pca() several times on the same data matrix, one gets the same signs (at least usually). I suppose that it has little to do with Matlab, but depends on some specific initialization choices in ARPACK. $\endgroup$ – amoeba Feb 1 '17 at 19:35
  • 1
    $\begingroup$ It is arbitrary choice. But it should be done. So, how is it done in Matlab, or in any other software (R or Stata) ?? $\endgroup$ – zlon Feb 1 '17 at 19:58
  • 1
    $\begingroup$ @whuber I think OP's question is meaningful. It might be considered off-topic (even though if it's not only about Matlab but about LAPACK library, then I'd say it's on topic), but it's not a dup. See the above comment by OP too. $\endgroup$ – amoeba Feb 1 '17 at 19:59
  • 1
    $\begingroup$ @whuber Yes, it's definitely a question about software. That's why I said it might be considered off-topic. $\endgroup$ – amoeba Feb 1 '17 at 20:06
3
$\begingroup$

Given any solution to PCA, a sign-flipped version of it is an equally valid solution. A numerical solver breaks this symmetry by finding one of these equally valid solutions. Implementation details and initial conditions determine which solution the solver will produce.

One way to think about PCA is that it maximizes the sum of the variance of the data projected onto the weight vectors, subject to the constraint that the weight vectors are orthonormal. Say the data set contains $n$ points in a $d$ dimensional space with mean $\mu$. We seek a set of orthonormal vectors $\{v_1, ..., v_p\}$ that solves the following optimization problem:

$$\max_{v_i, ..., v_p} \quad \sum_{i=1}^p \frac{1}{n} \sum_{j=1}^n \left [ (x_j - \mu)^T v_i \right ]^2 \quad \quad s.t. \quad \begin{array}{ll} \|v_i\| = 1 & \forall i \quad \\ v_i^T v_j = 0 & \forall i \ne j \\ \end{array} $$

Say we have a set of weight vectors that solves this problem. You can see that flipping their signs gives the exact same value for the objective function, and the constraints remain satisfied. Hence, the sign-flipped solution is equally valid. There are various other ways of thinking about PCA and writing it as an optimization problem. Some of these ways sound conceptually different, but they all yield the same set of solutions, and the same reasoning holds for all of them.

The reason a particular function implementing PCA returns any one of these solutions over the others comes down to implementation details and initial conditions. Say we're trying to solve the above problem using a standard optimization solver. It starts from some (possibly random) initial set of parameters, then iteratively updates them to increase the value of the objective function, while respecting the constraints. Imagine the objective function as a hilly landscape, where each location corresponds to a particular choice of parameters and the height at each location is the value of the objective function for those parameters. The constraints define particular regions the solver is allowed to go. Each solution to the problem is the highest allowed location on some surrounding hill. There are multiple hills, and the solutions all have the same height (i.e. are equally good). The solver starts from some initial location in this landscape and generally tries to move uphill, eventualy stopping when it can't make any further uphill progress. So, the solution it finally attains is determined by the hill it starts on and how it chooses to step around the landscape.

Of course, one wouldn't typically solve PCA this way because there are more specialized, computationally efficient ways to do it. For example, one popular method is to obtain the weights as eigenvectors of the covariance matrix. But the eigenvalue solver is itself an iterative algorithm, and is subject to the same kinds of issues.

Improve this answer
$\endgroup$
1
$\begingroup$

An eigenvector remains an eigenvector when you change its sign. Thus, an eigenvector does not have a "sign". As loadings as just eigenvectors multiplied by the square root of the associated eigenvalue, they have no "sign" either.

So now, why MAtlab gives you a specific sign I guess you would have to see how the decomposition is made behind. As far as I have seen on different topics (there or there) the sign is chosen arbitrarily.

Maybe this can help you. Otherwise, there has been some attempt to define what a "right" sign would be here.

EDIT :

Here is a website listing the methods chosen to solve the problem depending on the type of input (type of data in the matrix). I did not want to put it as I cannot find the sources the author of the page uses. But as far as I have seen, it seems to be still true:

http://matlab.izmiran.ru/help/techdoc/ref/eig.html

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks. "MATLAB chooses to normalize the eigenvectors to have a norm of 1.0, the sign is arbitrary.". Could you explain me what does it means? $\endgroup$ – zlon Feb 1 '17 at 19:14
  • $\begingroup$ zlon, suppose you were to ask two surveyors to mark off a one-kilometer segment of a road. Surveyor A marks it from east to west, while surveyor B marks the same stretch from west to east. Who is correct? Obviously they both are--and there is no way to choose between them that is not arbitrary. PCA finds one-dimensional eigenspaces: these are like roads. The choice of a sign is a choice between the two (equally correct) surveyors. $\endgroup$ – whuber♦ Feb 1 '17 at 19:33
  • $\begingroup$ whuber, I understand that it is arbitrary. My question is how Matlab choose "positive" direction. In Your example I may say let's east-west direction will be positive, because sun goes in this way. I found, that "MATLAB chooses to normalize the eigenvectors to have a norm of 1.0". I have terrible math background to understand it. What does it means "norm of 1" for eigenvectors? $\endgroup$ – zlon Feb 1 '17 at 19:56
  • 1
    $\begingroup$ Let's try this, then: because the sign makes no difference to PCA, you are free to impose whatever convention you wish. For instance, by switching signs as necessary, you can arrange to make the first nonzero component of each eigenvector be positive. What Matlab chooses is meaningless. Even if you could figure out how it chooses, that would only give you information about arbitrary, irrelevant aspects of the underlying numerical algorithms it uses. $\endgroup$ – whuber♦ Feb 1 '17 at 20:06
  • $\begingroup$ I understand it. But my problem is: I found, that loading A is positive, and loading B is negative. Then i decide to test the stability of loadings. I perform sampling with replacement for observations. And for each loading I found perfect bimodal distribution. But, variables just contrcorrelated. Thus, for one observations sets have +/- for others -/+. Now i just say ok, sign is arbitrary. I fix loading 1 to be always positive. I obtain perfectly stable loadings. But I should explain how my software choose this arbitrary direction. Does not matter which software. It is important to explain. $\endgroup$ – zlon Feb 1 '17 at 20:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.