5
$\begingroup$

Context

I have a dataset of 8 categorical variables. And I want to apply Categorical Principal Component Analysis (CatPCA).

Before doing that, I have been advised to look at the transformation plots of all these variables after transforming them into nominal variables. This shows that some variables should be transformed into ordinal variables (when non-decreasing) and some into numeric (linear trend).

Now when doing the CatPCA with all nominal variables and comparing this to the CatPCA with newly transformed variables, there is a slight decrease in variance explained in the dependent variable.

Question

  • So if variance explained decreases after the transformations, what is the advantage of the transformation?
$\endgroup$

1 Answer 1

4
$\begingroup$

The main reason why applying the transformation is important is to avoid over fitting. In some contexts, there is a related issue of model parsimony.

Nominal transformations permit a variable to be scaled in any way that maximises variance explained in the sample. Thus, applying constraints on how that scaling can occur will reduce the variance explained in the sample. Such constraints include ordinal scaling where transformed values have to preserve the order of the values of the untransformed variable. Numeric scaling increases the constraint further by requiring equal numeric distances between categories.

However, all this freedom in nominal transformations can lead to serious over-fitting. If over fitting occurs, the model will not predict well to data not included in the model.

A simple way to train your intuition about over-fitting is to split your sample into two, then build a model in one sample, and examine the fit in the other half. Given the large amount of freedom in most optimal scaling models, it is particularly important when evaluating the fit of a model to adopt some form of cross validation.

Combining theory and an initial nominal level analysis allows you to add constraints to the data. While these may reduce variance explained in the sample data, they should increase variance explained to external samples.

The degree to which overfitting is a problem in optimal scaling varies based on several factors:

  • Smaller sample sizes have more problems with over fitting.
  • Models with many variables have more problems with over fitting.
  • Variables with more categories have more problems with over fitting.
  • Transformations that provide fewer constraints have more problems with over fitting (e.g., nominal < ordinal < spline ordinal < numeric).
$\endgroup$
4
  • 2
    $\begingroup$ (+1) There's also some discussion of the overfitting issue with nominal scaling, which is the least restrictive level, in Meulman's papers, Prediction Accuracy of Regression with Optimal Scaling Transformations: The .632 Bootstrap with CATREG (p. 55), and PCA with nonlinear optimal scaling transformations for ordinal and nominal data. $\endgroup$
    – chl
    Apr 10, 2012 at 9:22
  • $\begingroup$ @Jeromy Anglim: Should we use, in all cases, nominal as the initial scaling level? If the categories can clearly be ordered (like poor, middle, rich for example) by theory then should I use nominal scaling level initially and then only change the scaling level for those variables whose transformation plots show ordinal nature? Or, I should use ordinal beforehand? If you kindly clarify this point, it would be great! $\endgroup$
    – Blain Waan
    Aug 27, 2012 at 0:33
  • $\begingroup$ I guess it depends on how much you want to be theory driven versus data driven. Also, the more data you have, the safer it will be to be data-driven (i.e., less chance of over-fitting). $\endgroup$ Aug 27, 2012 at 0:37
  • $\begingroup$ I was asking you about this because I am working on a data that has most of the variables 'ordinal' by theory. The number of observations is 178. But when I take 'nominal' as a scaling level initially then some of the plots are roughly U-shaped, some are monotonically increasing and some are zigzag! Here is the data. So, I am a little confused with what I should choose. $\endgroup$
    – Blain Waan
    Aug 27, 2012 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.