In addition to the answers that already focus on the mathematical properties, I'd like to comment from an experimental point of view.

Summary: data generation processes are often optimized in a way that makes the data suitable for principal compoent (PCR) or partial least squares (PLS) regression.

I'm analytical chemist. When I designing an experiment/method to measure (regression or classification) something, I use my knowledge about application and availabe instruments to get data that carries a good signal to noise ratio with respect to the task at hand. That means, the data I generate is designed to have large covariance with the property of interest.
This leads to a variance structure where the interesting variance is large, and the later PCs will carry the (small) noise only.

I'd also prefer methods that yield redundant information about the task at hand, in order to have more robust or more precise results. PCA concentrates redundant measurement channels into one PC, which then carries much variance and is therefore one of the first PCs.

If there are known confounders that will lead to large variance that is not correlated with the property of interest, I'll usually try to correct for these as much as possible during the preprocessing of the data: in many cases these confounders are of a known physical or chemical nature, and this knowledge suggests appropriate ways to correct for the confounders. E.g. I measure Raman spectra under the microscope. Their intensity depends on the intensity of the laser light as well as on how well I can focus the microscope. Both lead to changes that can be corrected by normalizing e.g. to a signal that is known to be constant.
Thus, large contributors of variance that does not contribute to the solution may have been eliminated before the data enters PCA, leaving mostly meaningful variance in the first PCs.

Last but not least, there's a bit of a self-fulfilling prophecy here: Obviously PCR is done with data where the assumption that the information carrying variance is large does make sense. If e.g. I think that there could be important confounders that I don't know how to correct for, I'd immediately go for PLS which is better at ignoring large contributions that do not help with the prediction task.

In addition to the answers that already focus on the mathematical properties, I'd like to comment from an experimental point of view.

Summary: data generation processes are often optimized in a way that makes the data suitable for principal component (PCR) or partial least squares (PLS) regression.

I'm analytical chemist. When I designing an experiment/method to measure (regression or classification) something, I use my knowledge about application and available instruments to get data that carries a good signal to noise ratio with respect to the task at hand. That means, the data I generate is designed to have large covariance with the property of interest.
This leads to a variance structure where the interesting variance is large, and the later PCs will carry the (small) noise only.

I'd also prefer methods that yield redundant information about the task at hand, in order to have more robust or more precise results. PCA concentrates redundant measurement channels into one PC, which then carries much variance and is therefore one of the first PCs.

If there are known confounders that will lead to large variance that is not correlated with the property of interest, I'll usually try to correct for these as much as possible during the preprocessing of the data: in many cases these confounders are of a known physical or chemical nature, and this knowledge suggests appropriate ways to correct for the confounders. E.g. I measure Raman spectra under the microscope. Their intensity depends on the intensity of the laser light as well as on how well I can focus the microscope. Both lead to changes that can be corrected by normalizing e.g. to a signal that is known to be constant.
Thus, large contributors of variance that does not contribute to the solution may have been eliminated before the data enters PCA, leaving mostly meaningful variance in the first PCs.

Last but not least, there's a bit of a self-fulfilling prophecy here: Obviously PCR is done with data where the assumption that the information carrying variance is large does make sense. If e.g. I think that there could be important confounders that I don't know how to correct for, I'd immediately go for PLS which is better at ignoring large contributions that do not help with the prediction task.