Rotate PCA components to equalize the variance in each component I'm trying to reduce the dimensionality and noise of a dataset by performing PCA on the dataset and throwing away the last few PCs.
After that, I want to use some machine learning algorithms on the remaining PCs, and therefore I want to normalize the data by equalizing the variance of the PCs to make the algorithms work better.
One simple way is to simply normalize the variance to unit values.
However, the first PC contains more variance from the original dataset than the following ones, and I still want to give it more "weight".
Therefore I was wondering: is there a simple way to just split its variance and share it with the PCs with less variances?
Another way is to map the PCs back to the original feature space, but in that case the dimensionality would also increase to the original value.
I guess it's better to keep the resultant columns orthogonal, but it's not necessary at this moment.
 A: In his perspicacious and comprehensive answer @amoeba has shown - as part of the answer - how one can rotate two uncorrelated variables (such as principal components for example) to achieve the wanted variances for them (while at expense of losing uncorrelatedness, of course). Let orthogonal variables $X$ and $Y$ have variances $\sigma^2_{max}$ (a larger) and $\sigma^2_{min}$ (a smaller), respectively. Rotate them so that $X$ will get arbitrary, diminished variance $\mu^2$ (while $Y$, consequently, will become of the variance $\sigma^2_{max}+\sigma^2_{min}-\mu^2$).
@amoeba shows the formula from which we can compute the angle of such rotation, $\cos\theta$:
$$\mu^2 = \cos^2\theta (\sigma^2_{max}) + \sin^2\theta (\sigma^2_{min})$$
but has not demonstrated where this equation comes from; probably thinking that it is obvious without explanation. Obvious or not, I believe it is worth elucidating - some way. My answer presents one way.
And so, we have an ellipsoidal, centered data cloud in the space of uncorrelated variables $X$ and $Y$. We have to rotate the axes by angle $\theta$. A data point in the cloud (such as shown as green spot on the picture) with $X$ coordinate $x$ will have this coordinate as $x^*$ after the rotation.

Observe that projection of the coordinate $x$ notch onto the rotated axis $X^*$ is given by $x'=x\cos\theta$ (cathetus as the hypotenuse and the angle between them). Observe also that $x^*$ is less than $x'$ by the cut of length $x'-x^*$ computable from coordinate $y$: $y\sin\theta$ (another cathetus and hypotenuse). And so,
$$x^* = x' - (x'-x^*) = x\cos\theta-y\sin\theta$$
We know (see the beginning) the variances (or sums-of squares) of the two variables and the variance (sum-of squares) $\mu^2$ of $X^*$. Then it follows:
$$\mu^2=\sum x^{*2} = \sum(x\cos\theta-y\sin\theta)^2 = \sum(x^2\cos^2\theta+y^2\sin^2\theta-2xy\cos\theta\sin\theta) = \cos^2\theta\sum x^2 + \sin^2\theta\sum y^2 - \underbrace{ 2\cos\theta\sin\theta\sum xy}_{\text{=0 (X and Y are uncorrelated)}} = \cos^2\theta (\sigma^2_{max}) + \sin^2\theta (\sigma^2_{min})$$
From which you estimate $\cos\theta$, as @amoeba has shown, and perform the rotation.
A: If I interpret things correctly, you mean that the first principle component (eigenvalue) explains most of the variance in the data. This can happen when your compression method is linear. However, there might be non-linear dependencies in your feature space.

TL/DR: PCA is a linear method. Use Autoencoders (non-linear pca) for dimensionality reduction. If the machine learning part is supervised learning then simply monitor your loss function while adjusting the (hyper)parameters for the autoencoder. In this way you will end up with a far better compressed version of your original data. 

Here's a scikit example where they do grid search to find the optimal number of principal components to keep (hyper-parameter) using PCA. Finally they apply Logistic Regression on the lower dimensional space:
http://scikit-learn.org/stable/auto_examples/plot_digits_pipe.html#example-plot-digits-pipe-py
Protip: Autoencoders do not have a closed form solution (afaik) so if your context is streaming data, this means you can continuously update your autoencoder (compressed representation) and can thus compensate for things such as concept drift. With pca you have to re-train batch mode from time to time as new data comes in.
As to giving some features more "weight", see regularization ( I'd start from norms https://en.wikipedia.org/wiki/Norm_(mathematics) ). You might also be surprised how similar logistic regression is to the perceptron.
