Why linear transformation can improve classification accuracy when the dimensionality of data is high? Let $X$ be an $m\times n$ ($m$: number of records, and $n$: number of attributes) dataset.  When the number of attributes $n$ is large and the dataset $X$ is noisy, classification gets more complicated and the classification accuracy decreases. One way to over come this problem is to use linear transformation, i.e., perform classification on $Y=XR$, where $R$ is an $n\times p$ matrix, and $p<=n$. I was wondering how linear transformation simplifies classification? and why classification accuracy increases if we do classification on the transformed data $Y$ when $X$ is noisy?
 A: Merely doing a linear transformation is not going to help much in the classification. There two ways of linear transformation that you can do.
PCA - Principal Component Analysis - Do a principal component analysis and take only few directions components that explains most of the variability in the data. Say you take 5 principal components that accounts for 95% variability in the data. You can remove the other components that together explains 5 % of the variability in the data. There is a high chance that this 5 % is noise. Either you can work with these 5 variables from the 5 principal components or you can reconstruct your data back with only these 5 principal components and then do a classification. This method is only going to reduce the noise but it doesn't give you any information as to which principal components are significantly useful for the classification in hand.
Linear Discriminant Analysis - This method tries to find linear projections in the form of components that maximizes the inter-class variance and reduces the intra-variance and hence it tries to find out components which helps to discriminate the two classes. You can take few components that gives you good discrimination between the two classes. These components can be used as variables to other classification algorithm like logistic regression,SVM,etc.
Note that - Linear discriminant analysis works well when within each class the data points more or less follow Gaussian distribution.
