New answers tagged

0

If your goal were simply dimension reduction, then it would make sense to retain the first (and maybe second) components. The first component accounts for the largest share of the joint variance of the set of variables, while the second components accounts for the second largest share. The hope is that the first and maybe second components may account for ...


3

$w^T\Sigma w$ is convex function, you're right. As far as I see, whuber's answer defines concave functions as what we usually know as convex functions. It's also pointed out in the comments. Take two points on the unit sphere, and connect them with a line. Is the line (all of it) inside the domain? No, then the domain is not convex. Because, you maximize ...


1

You could do that, or you could use a method that weights the composites (on both sides) based on the relationships between predictor and outcome variables.Back in the day, Jacob Cohen (1982) described what he called "set correlation," a generalization of regression that allowed the LHS to include a set of variables. However, there are a range of techniques ...


1

I've encounter some problems with pipelines (for example, if I want to apply my own custom function, it is a real hazard) so here is what I use instead: X_train, X_test, y_train, y_test = train_test_split(X, Y, stratify=Y, random_state=seed, test_size=0.2) sc = StandardScaler().fit(X_train) X_train = sc.transform(X_train) X_test = sc.transform(X_test) pca = ...


5

For the benefit of possible readers who don't use the scikit pipeline: Centering and scaling the training subset does not only result in the centered and scaled training data but also in vectors describing the offset and scaling factor. When predicting new cases, this offset and scale is applied to the new case, and the resulting centered and scaled data ...


1

For anyone who might stumble upon this question, I have a solution using scikit-learn's Pipeline, as recommended in the accepted answer. Below is the code I used to get this to work for my problem, chaining together StandardScaler, PCA and Ridge regression into a cross-validated grid-search: pipe = Pipeline([("scale", StandardScaler()), ("...


5

You need to think feature scaling, then pca, then your regression model as an unbreakable chain of operations (as if it is a single model), in which the cross validation is applied upon. This is quite tricky to code it yourself but considerably easy in sklearn via Pipelines. A pipeline object is a cascade of operators on the data that is regarded (and acts) ...


2

Let's assume $X$ as $n\times m$, where $n$ is number of data samples and $m$ is number of features; and assume $X$ is mean-centered. You can't find an eigen decomposition for non-square matrices, as you also pointed out. But, of course, we don't take $X^TX$ just because it is square. It's the scatter matrix, i.e. proportional to the sample covariance. PCA's ...


3

Great question - you've hit a great realization about PCA. Even though the important feature in the model is X2 because of the large coefficient, because X2 has such low variance, the PCA is essentially ignoring it by giving it a low weight. PCA is great for capturing the greatest variance among variables, but that is a different task than regression, ...


0

I was looking for a solution that works for PCA performed using ade4. Please find the function below: library(ade4) irisX <- iris[,1:4] # Iris data ncomp <- 2 # With ade4 dudi_iris <- dudi.pca(irisX, scannf = FALSE, nf = ncomp) rotate_dudi.pca <- function(pca, ncomp = 2) { rawLoadings <- as.matrix(pca$c1[,1:ncomp]) %*% diag(sqrt(...


2

This question appears to focus on two things: How can principal components regression fail badly? Could the failure be due to not standardizing the variables? Let me elaborate on the meanings of these terms. The setting is where you have $(x_1, x_2, \ldots, x_k, y)$ data. You view the values of the $y$ as $n$ independent observations of random variables $...


1

The beauty of (unrotated) principal components is that they are mutually orthogonal. There is no problem in just adding them up. There is also no surprise in how they will behave. The behavior of the sum will be exactly the same as if you included the three components as predictors but with equal regression weights, rather than allowing their regression ...


1

The first principal component is - by construction - the best single feature (or score, if you like) to explain your data. If you only want one element, you should simply use the First Principal Component. The first approach you propose will return you exactly the same principal components you already have, since these are already built to maximise variance ...


0

It probably means PLS outperforms OLS even in the presence of unrelated variables. I would definitely check regression coefficients of PLS model to see whether the the unrelated variables have relatively low absolute magnitudes. However, if the unrelated variables are related to each other, then the regression coefficients can be deceiving. The lower RMSE ...


1

First, why are you doing logistic regression on the average? Once you take the average, the result is no longer discrete and you could try linear regression. Second, whether you can average Likert scale items at all isn't completely agreed. Technically, you can't. But a lot of people do it and it seems to "work" for some not-too-precise definition of "work"....


1

Yes, you can do CFA with cases that are countries (or companies, or...). Be aware that the variables may have been grouped together for conceptual reasons, rather than because of their statistical behavior, so rejecting a factor model will not invalidate the grouping. Moreover, n = 33 is a very small sample size for CFA. With 33 observed variables, your ...


1

CFA can absolutely be used on country-level data. One of the most famous examples of CFA, Bollen's political democracy analysis, is on country-level data. See Bollen (1980).


1

I'm assuming the other 7 features are continuous or binary, and this 8th feature is categorical with 6000 different categories - otherwise one hot encoding only it wouldn't make sense. What you can do, if you have some prior knowledge, is aggregate some of these 6000 categories together. It depends on your data, but there might be a meaningful way to make ...


0

Not the second largest variance but the second largest unique variance. The variance of a correlated second component would be larger than the variance of an orthogonal second component, but much of the second component's variance would be shared with the first--the stronger the correlation, the more the shared variance. The more shared variance, the more ...


0

Whenever we choose a component during PCA, it is ensured that we find the complete variance of the data on that particular component. If the 2nd component is not orthogonal to the 1st one, then the 2nd component will have some dependence on the 1st one. This means that the variance of the data on the 1st component hasn't been captured completely yet on the ...


0

Without knowing what all your variables are, we can't possibly name a component. When one PC explains a large proportion of the variance and all its loadings are roughly equal, then that is some sort of general PC. What are all your variables measuring? For instance, with physical measurements, such a PC is often an overall size variable.


0

Let the dimension be denoted by $d$. There are $d$ eigenvectors of dimension $d\times1$. In PCA< we choose $k<d$ of them an obtain an underrepresentation. Typically, the data matrix, $X$, as also in your case has dimension $n\times d$, where $n$ is number of data points in the dataset. So, each row is a sample. In order to convert a sample, say $x$ of ...


2

In PCA, we choose the projection direction(s) that captures the most variance; this is well explained in the linked posts. In some sources, inertia is described as variance. So, the directions with most variance are the directions with most inertia. I can't review the definitions in the book to see if they're talking about the same concept or not since it's ...


Top 50 recent answers are included