New answers tagged

0

In my opinion, yes you can use the variables that are most significant in constructing the respective factor scores, especially if you data is data-centered (minus the respective mean). The most obvious advantage is, unlike factor analysis produced constructs, your explanatory variable is readily understandable. However, the standard regression derived beta ...


0

I wouldn't think of this as a "small sample size" for PCA. The PCA example in Section 10.2.1 of An Introduction to Statistical Learning is for a data set with 50 items and only 3 continuous variables. The standard ways of performing PCA will have no inherent difficulty. The problem you face, as is typical of small data sets, is how well your ...


7

Your confusion seems to be about the notation being used here. You seem to be assuming that $x^{(i)}$ is a matrix. Whereas, in fact, $x^{(i)}$ is a column vector: $$ x^{(i)}= \begin{bmatrix} x_1^{(i)} \\ x_2^{(i)} \\ \vdots \\ x_p^{(i)} \end{bmatrix} $$ Thus, every $x^{(i)}$ contains a single observation of all the $p$ variables that you're computing the ...


5

Assuming zero-mean samples, both formulations give covariance estimates. I assume $m$ denoted in the upper-most formula is either $n$ or $n-1$. The first one uses individual samples, $x_i$ and the second one uses the data matrix, $X$, where the samples are the rows. The typical formulation of the data matrix $X$ is as follows: $$X=\begin{bmatrix}x_1^T\\ x_2^...


0

If you are plotting the LDA points calculated from PCA scores, you just use the percentage of LDA explained variance. In the figure caption or in the methods you describe the choice of the principal components and the details about the explained variance. You can see PCA as a feature engineering process, then LDA as a classification model based on them.


0

When I read the paper A Generalized Linear Model for Principal Component Analysis of Binary Data, I realized that the paper itself just gave us the solution. To compute the scores matrix U for the testing data we should use the same transformation, and this transformation is V as we previously trained. So we just need to fix V and update U for testing data. ...


1

It's possible your data has a lower or upper boundary. For example, if your dataframe has a Poisson distribution, then you would expect many 0s in your data. The diagonal line may indicate the 0s. Here is a better explanation. What does a wedge-like shape of the PCA plot indicate?


0

This Answer is inspired by the following thread: Relationship between SVD and PCA. How to use SVD to perform PCA? (@ameoba's excellent explanation for PCA and SVD. If you are totally unfamiliar with PCA and SVD. Give this a read first) In order to plot PC1 vs PC2, I made a scatter plot (V1:V2). V1 and V2 are first and second column of V This is wrong! $\...


2

You are not getting the expected orthogonal projection because you are only projecting the first 2 features. In order to see the orthogonality of the projection, you would need to plot also the other 2 features. You can try to run your code with X=iris[, 1:2]. Then, you will see the expected orthogonal projections. You can imagine that the projections on the ...


1

It depends of what you call an outlier. If for you, an outlier for categorical data is a category that appear less than, say, 1% of the time then there is a really easy algorithm to detect those: just count the number of values for each category (for example with pandas value_counts) and threshold this to find which category are abnormal in your sense. This ...


0

Assuming that you do PCA on correlation matrix, you will not need any normalization prior to it. Also, one hot encoding is performed on the target, not on feature matrix. So, it is completely independent from all other operations. I think you can follow something like that: Distinguish feature and target matrices Separately do PCA on features and apply one ...


0

One hot encoding its just aplicable to categorical data, so there is no need to "normalize" what is already categorical. Although, the rest of your numerical data should be normalized. I reccomend to do the one hot encoding of your categorical data first, cause if you normalize with min-max a 0-1 one hot encoding, they stay the same.


2

For prcom, the principal component is implemented via SVD, where the matrix $X$ can be decomposed into: $\mathbf{X} = \mathbf{U}\mathbf{\Sigma}\mathbf{W}^T $ where $\mathbf{U}\mathbf{\Sigma}$ is the principal component score and $\mathbf{W}^T$ is the transpose of your eigenvectors, or transpose of your loadings as returned by the prcomp() function. So to ...


2

Let me reformulate your question: you want to do a PCA on the subspace that is orthogonal to a given direction PC1 $\vec{p}_1$. You can project every data point $\vec{x}$ on that subspace by $$\vec{x}_{proj} = \vec{x} - \langle \vec{x},\vec{p}_1\rangle$$ where $\langle.,.\rangle$ denotes the scalar product. Then simply do a PCA on the projected data. Note ...


0

The principal component scores along any principal component loading vector average to 0 (because of centering - see page 384 of the pdf). If the variance is also 0, then all the scores must also be 0. Hence we can take only the principal components with nonzero variance score vector.


0

I skim read through the comments above and I believe quite a few have pointed out that PCA is not a good approach to feature selection. PCA offers dimensionality reduction but it is often misconceived with feature selection (as both tend to reduce the feature space in a sense). I would like to point out the key differences (absolutely open to opinions on ...


0

If anyone cares about performance difference in a specific case, I compared SVD and Eigen-analysys on a 632x632 real symmetric matrix L using C++/Eigen. I used BDCSVD // compute full V, U *not* needed Eigen::BDCSVD<Eigen::MatrixXf> svd(L,Eigen::ComputeFullV); and SelfAdjointEigenSolver Eigen::SelfAdjointEigenSolver<Eigen::MatrixXf> eigenSolver(N)...


1

The reason seems to lie in this remark of the prcomp documentation: The calculation is done by a singular value decomposition of the (centered and possibly scaled) data matrix, not by using ‘eigen’ on the covariance matrix. With eigen(cov(test)) five components are returned (with three eigenvalues pracically zero), but svd(test, nu=0) only returns min(...


0

The answer is that the way I call PCA from sklearn results in the covariates being centered to have $0$ mean (but not unit variance). Linear dimensionality reduction using Singular Value Decomposition of the data to project it to a lower dimensional space. The input data is centered but not scaled for each feature before applying the SVD. (The emphasis is ...


1

You need adequate dimension reduction to avoid overfitting. Overfitting comes from maintaining too many fully weighted features in a model, regardless of the particular modeling approach used. I'll focus here on regression techniques. LDA and SVM might or might not work better than regression in your case, but that's not because of differences with respect ...


1

Quite a staggering bit, apparently. I'm just getting into this myself, but you may want to check out the work of Brunton, Kutz and co-workers at the University of Washington, in particular HAVOK analysis (which uses exactly this.) I'm sure a little digging will turn up vastly more. Here is a YT: https://www.youtube.com/watch?v=831Ell3QNck Presenting this ...


3

Most likely you are dealing with a data set of independent variables, i.e. the covariance matrix of your dataset is an identity matrix.


1

The effect of g is relatively small compared to the error of N(3,1). So it will be really hard to estimate what goes into the intercept and what goes into g. I re-ran it with beta_1 = np.array([2, 1, -1, -1]) err = np.random.normal(0, 1, N) And maybe got somewhere closer to what was the actual estimate. Regarding why the coefficients are off, I saw in the ...


Top 50 recent answers are included