For example let's say we have two variables X and Y
A two variable problem isn't great for illustrating what PCA is about, there are easier to understand methods for handling correlation between two variables. While the maths is easiest to understand for a few variables, the usefulness is easier to understand with a few hundred or thousand variables.
What is PCA doing
PCA simplifies complex datasets by summarising them in terms of basic building blocks that are found to repeat over and over across the dataset. These building blocks all have unique characteristics and each sample contains different mixtures of the building blocks. The aim is to identify these building blocks and use them to explain the data as succinctly as possible. PCA is predicated on the assumption that a set of observations can be described as a linear combination of underlying data generating processes.
$$ Obs_1 = A_1*x + B_1*y + C_1*z \ldots\\Obs_i = A_i*x + B_i*y + C_i*z \ldots\;\;\;\;$$
where $Obs_i$ is the $i^{th}$ observation and $x,y,z$ are hidden (or latent) processes.
Datasets with a large number of variables present a number of challenges. It is difficult to inspect of datasets for quality (‘sanity’ checking), to perform initial trend analysis to ensure relevant information is present, to avoid high false positive rates as a result of multiple hypothesis testing (family-wise error rate and per family error rate), and to avoid generating spurious correlations through overfitting. Modelling using original variables effectively carries out multiple hypothesis testing on hundreds or thousands of variables. Overfitting is a high risk as the risk of false positives (spurious associations based on chance) is high, especially for classification problems where the number of states is very limited. A model is overfitted when it exploits features unrelated to the variation of interest (i.e. 'noise' in a generic sense).
This problem is linked to the assumed independence of noise between variables, but the inverse problem is true for many datasets which have a high level of redundancy through high variable-wise correlation between variables. PCA actively exploits the correlation present to retain information while reducing the variables needed to explain the variation in the dataset, resulting in new variables with no redundancy of information.
What is PCA good for?
PCA is used for simplification, data reduction, modelling, outlier detection, variable selection, classification, prediction, unmixing see Wold paper here (paywall), denoising, digital signal processing (baseline correction, sample-wise intensity standardisation and interferent correction).
but I'm curious if there is any human readable benefit to doing this.
There is none for 2 variable datasets. If you have 1000s however, fewer variables while eliminating redundancy opens new options for visualization and interpretation of data. All Principal Components (PCs) are orthogonal (uncorrelated, PCs are at right angles to each other when plotted). In higher dimensions, any two PCs have an inner product equal to zero. If there are more variables than observations the full rank PCA contains all the information from the dataset, but with the informativeness strongly weighted in a smaller number of dimensions. This means undertanding the majority of what goes on in your dataset only requires evaluating the top few PCs, rather than 1000s of variables.
PCA exploits a trick based on how the variables share information; variables (that correlate together are grouped into Principal Components each describing one set of correlated information from the dataset. To loop back to the concept of building blocks in the dataset, what PCA is looking for is blocks made out of correlation. If we consider that perfectly correlated variables give the same information, this means that all the variables contributing to one piece of information are corralled into a single vector.
I have read that this sort of thing is good for machine learning algorithms
In statistical analysis it is highly desirable to use input variables that are as independent as possible to ensure that coefficicents are reliably weighted (if a model sees two variables giving the same information then the coefficicents can be arbitrarly adjusted and give the same result). While machine learning is not statistics, good ML will be based on sound statistical principles. Minimizing crosstalk is essential to allow cleaner and simpler statistical analysis. Note that I referred to independence as ideal, PCs are not independent, but they are uncorrelated which is often close enough.
]The PCs summarise the variation patterns more succinctly than the original variables simplifying statistical modelling. For this reason PCA is used as an initial step prior to a very wide range of analyses such as classification, neural networks, and decision trees. Furthermore, PCA can dramatically simplify the complexity of a dataset and reduce computational load, and for cloud based services the number of bytes to be transmitted can be culled.
PC1 axis no longer means anything real
Based on context I am guessing here you are referring to the plot of the scores. The PCs themselves are model coefficients, so we can arbitrarily treat them as dimensionless (if you prefer to have the scores dimensionless just switch round the reasoning). Is true for datasets where no care has been taken to ensure consistency of units across the variables (typically for pragmatic reasons). However, the scale of the score is directly related to the dataset as it was analysed by PCA (so if the dataset was mean centered and scaled to unit variance, then it is this processed dataset it transforms to). Scores are calculated by the inner product of the data and the PCs (PCs are unit vectors), so each score is the sum across the product of the relevant loading, i.e. there is a direct connection between the score and the data. This means that if your dataset scales are interpretable then your scores scale will be interpretable. If your dataset scales are not interpretable it is not the fault of PCA.
