Is PCA unstable under multicollinearity? I know that in a regression situation, if you have a set of highly correlated variables this is usually "bad" because of the instability in the estimated coefficients (variance goes toward infinity as determinant goes towards zero).
My question is whether this "badness" persists in a PCA situation.  Do the coefficients/loadings/weights/eigenvectors for any particular PC become unstable/arbitrary/non-unique as the covariance matrix becomes singular?  I am particularly interested in the case where only the first principal component is retained, and all others are dismissed as "noise" or "something else" or "unimportant".
I don't think that it does, because you will just be left with a few principal components which have zero, or close to zero variance.
Easy to see this isn't the case in the simple extreme case with 2 variables - suppose they are perfectly correlated.  Then the first PC will be the exact linear relationship, and the second PC will be perpindicular to the first PC, with all PC values equal to zero for all observations (i.e. zero variance).  Wondering if its more general.
 A: PCA is often a means to an ends; leading up to either inputs to a multiple regression or for use in a cluster analysis. I think in your case, you are talking about using the results of a PCA to perform a regression.
In that case, your objective of performing a PCA is to get rid of mulitcollinearity and get orthogonal inputs to a multiple regression, not surprisingly this is called Principal Components Regression. Here, if all your original inputs were orthogonal then doing a PCA would give you another set of orthogonal inputs. Therefore; if you are doing a PCA, one would assume that your inputs have multicollinearity. 
Given the above, you would want to do PCA to get a few input variable from a problem that has a number of inputs. To determine how many of those new orthogonal variables you should retain, a scree plot is often used (Johnson & Wichern, 2001, p. 445). If you have a large number of observations, then you could also use the rule of thumb that with $\hat{ \lambda_{i} }$ as the $i^{th}$ largest estimated eigenvalue only use up to and including those values where $\frac{ \hat{ \lambda_{i} } }{p}$ are greater than or equal to one (Johnson & Wichern, 2001, p. 451).
References
Johnson & Wichern (2001). Applied Multivariate Statistical Analysis (6th Edition). Prentice Hall.
A: The answer might be given in even simpler terms: the multiple regression has one step more than the pca if seen in terms of linear algebra, and from the second step the instability comes into existence:      
The first step of pca and mult. regression can be seen as factoring of the correlation-matrix $R$ into two cholesky factors $L \cdot L^t$ , which are triangular -and  which is indifferent to low or high correlations. (The pca can then be seen as a rotation of that (triangular) cholesky-factor to pc-position (this is called Jacobi-rotation as far as I remember)    
The mult. regression procedure is the to apply an inversion of that cholesky factor $L$ minus the row and column of the dependent variable, which is conveniently in the last row of the correlation-matrix.
The instability comes into play here: if the independent variables are highly correlated, then the diagonal of the cholesky factor $L$  can degenerate to very small numeric values - and to invert that introduces then the problem of division by nearly-zero.
