Should I only select two Principal components for a regression analysis? Why are two principal components usually selected for principle regression analysis? I read somewhere that since it is 2 dimensional data, there should only be 2 principal components extracted from the dataset.  Even if 3 Principal components explain more variance (say 80%) compared to two components which explain (68%), we should still select only 2 components for the regression.  Can someone confirm that for me?
 A: What dimension is your original data? Certainly if you are starting with 2D data, I wouldn't go less than 2D. You may still find a better set of axises though PCA though - your data may just be skewed 45 degrees or whatever.
The simplest explanation about choice of dimensions is simply that graphing 2D data after PCA is easiest to see visually. Just a classic X/Y graph. In the ideal case for most folks, you go about your PCA fitting your high dimensional data, project your data into 2D, plot it, and boom -- you have wonderful little clusters, or the data has a nice relationship of some sort. You write your paper or understand something lovely about the data and move on. But...you don't always get so lucky.
So it depends on your usecase. If you transform & plot your transformed dataset in 2D and it reveals what you are trying to determine or show, it doesn't really matter if it accounts for 99% of the variance or 68%. It does what you need it to do.
If, however, you are doing something where the variance captured truly matters (ie: you have a large dimensional dataset that is too large to fit in memory for an ML model you are training), you probably DO want to try to get as close to capturing all the variance as you can before you run out of memory. Because obviously, your NN or SVM model really doesn't care how "interpretable" the data is, it just needs more variance to draw boundaries in high dimensional space to get a better accuracy or F1 score, etc.
As a final note, including all the variance isn't necessarily the best for ML purposes either. Performing PCA and training over the data after removing some portion of the variance can actually be a form of regularization -- it removes some of the noise that would only serve to distract our classifier and cause it to overfit to things in our data that don't actually matter, or aren't representative of the data at large. As always, good cross valuation and clean training, validation, and test sets are your friend! They can be your ultimate guide as to your choice of PCA dimension.
