A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Hot answers tagged

7

You need to think feature scaling, then pca, then your regression model as an unbreakable chain of operations (as if it is a single model), in which the cross validation is applied upon. This is quite tricky to code it yourself but considerably easy in sklearn via Pipelines. A pipeline object is a cascade of operators on the data that is regarded (and acts) ...


7

For the benefit of possible readers who don't use the scikit pipeline: Centering and scaling the training subset does not only result in the centered and scaled training data but also in vectors describing the offset and scaling factor. When predicting new cases, this offset and scale is applied to the new case, and the resulting centered and scaled data ...


3

$w^T\Sigma w$ is convex function, you're right. As far as I see, whuber's answer defines concave functions as what we usually know as convex functions. It's also pointed out in the comments. Take two points on the unit sphere, and connect them with a line. Is the line (all of it) inside the domain? No, then the domain is not convex. Because, you maximize ...


3

Great question - you've hit a great realization about PCA. Even though the important feature in the model is X2 because of the large coefficient, because X2 has such low variance, the PCA is essentially ignoring it by giving it a low weight. PCA is great for capturing the greatest variance among variables, but that is a different task than regression, ...


2

This question appears to focus on two things: How can principal components regression fail badly? Could the failure be due to not standardizing the variables? Let me elaborate on the meanings of these terms. The setting is where you have $(x_1, x_2, \ldots, x_k, y)$ data. You view the values of the $y$ as $n$ independent observations of random variables $...


2

For anyone who might stumble upon this question, I have a solution using scikit-learn's Pipeline, as recommended in the accepted answer. Below is the code I used to get this to work for my problem, chaining together StandardScaler, PCA and Ridge regression into a cross-validated grid-search: from sklearn.model_selection import GridSearchCV from sklearn....


2

Let's assume $X$ as $n\times m$, where $n$ is number of data samples and $m$ is number of features; and assume $X$ is mean-centered. You can't find an eigen decomposition for non-square matrices, as you also pointed out. But, of course, we don't take $X^TX$ just because it is square. It's the scatter matrix, i.e. proportional to the sample covariance. PCA's ...


2

I've encounter some problems with pipelines (for example, if I want to apply my own custom function, it is a real hazard) so here is what I use instead: X_train, X_test, y_train, y_test = train_test_split(X, Y, stratify=Y, random_state=seed, test_size=0.2) sc = StandardScaler().fit(X_train) X_train = sc.transform(X_train) X_test = sc.transform(X_test) pca = ...


1

You could do that, or you could use a method that weights the composites (on both sides) based on the relationships between predictor and outcome variables.Back in the day, Jacob Cohen (1982) described what he called "set correlation," a generalization of regression that allowed the LHS to include a set of variables. However, there are a range of techniques ...


1

The beauty of (unrotated) principal components is that they are mutually orthogonal. There is no problem in just adding them up. There is also no surprise in how they will behave. The behavior of the sum will be exactly the same as if you included the three components as predictors but with equal regression weights, rather than allowing their regression ...


1

The first principal component is - by construction - the best single feature (or score, if you like) to explain your data. If you only want one element, you should simply use the First Principal Component. The first approach you propose will return you exactly the same principal components you already have, since these are already built to maximise variance ...


1

Yes, you can do CFA with cases that are countries (or companies, or...). Be aware that the variables may have been grouped together for conceptual reasons, rather than because of their statistical behavior, so rejecting a factor model will not invalidate the grouping. Moreover, n = 33 is a very small sample size for CFA. With 33 observed variables, your ...


1

CFA can absolutely be used on country-level data. One of the most famous examples of CFA, Bollen's political democracy analysis, is on country-level data. See Bollen (1980).


1

I'm assuming the other 7 features are continuous or binary, and this 8th feature is categorical with 6000 different categories - otherwise one hot encoding only it wouldn't make sense. What you can do, if you have some prior knowledge, is aggregate some of these 6000 categories together. It depends on your data, but there might be a meaningful way to make ...


1

First, why are you doing logistic regression on the average? Once you take the average, the result is no longer discrete and you could try linear regression. Second, whether you can average Likert scale items at all isn't completely agreed. Technically, you can't. But a lot of people do it and it seems to "work" for some not-too-precise definition of "work"....


Only top voted, non community-wiki answers of a minimum length are eligible