I am attempting to produce models on a dataset with approximately 60 variables and 800 subjects. Other datasets are also being considered for use. The datasets were not originally constructed for this type of analysis. The dataset also comprises a mixture of variables. Some are continuous and some discrete. Also some of these are correlated with one another as can be seen on a correlation matrix and also just by eye/common sense.
Currently clustering variables has been used with the aim to remove any correlated components being used to generate models to avoid over-fitting and reduce the number of variables. By choosing the most representative variable from each cluster however I am unclear if this is being done correctly and not confident this is correct and will result in incorrect models.
Previously what has been done is to just load all the variables into the platform and produce a cluster summary. The most representative member variable is picked out of each cluster and models made. In many clusters there is a variable present that shows a low R squared with the cluster it is in, yet it has been included. The variable also does not show correlation to the other group members on reference to the correlation matrix. It's almost as if it has been forced in where it will fit best rather than left on its own. (Sometimes a cluster does include 1 variable.) Also, sometimes two separate clusters appear to contain variables which are correlated.
Thus I feel at times correlated variables are being used for models and at the same time uncorrelated variables are being missed.
I am very new to this level of stats and I am struggling with wrapping my head around eigenvalues etc. It was implied that this is an aspect of PCA but I have found that this is not really the case.
Information on the web tends to get into some quite complex math on the subject.
Here is my understanding, it may well be off.
I understand that PCA is a transformation of the dataset so that the first component explains the most variation in the variables, the second PC is orthogonal to this (uncorrelated) and that there is one PC for each variable. Each PCA is given a total eigenvalue that = number of variables (due to the transformation of the original axis where the variance of the data is made to =1 and mean =0). The PC that explains most of the variation in the data takes the greatest share of this followed by the second etc. This is due to variable weighting coefficients where each variable is given a weighting in respect to its influence on that PC. Eigenvectors are where I fall apart!!
Clustering involves looking at all the variables and performing iterative splits. A split involves a PCA analysis on the cluster members? then.. from a guide.. 'the cluster with the largest second eigenvalue is chosen to be split into two new clusters' Again I fall apart due to not really understanding eigenvalues.
I think the tools I have been using have been suggested to perform a dump and click solution which I can't really believe..but I cannot argue the case due to limited understanding. Long post, but I'm getting a bit lost!