0
$\begingroup$

I have a PCA addin for MSExcel. One of the results it gives for each component is the R², cummulative R², Q², and cummulative Q². It looks like the cummulative R² is the sum of all previous R² values, while the cummulative Q² is:

$$ cummulativeQ^2_n = cummulativeQ^2_{n-1} +\left(1-cummulativeQ^2_{n-1}\right)*Q^2_n $$

Can anyone tell me what this Q statistic is, how it is calculated, and what it means?

$\endgroup$
  • $\begingroup$ "$Q$" could mean anything in general: doesn't the documentation tell you? $\endgroup$ – whuber Mar 21 '18 at 13:02
1
$\begingroup$

What $Q^2$ is

For PCA $Q^2$ is a measure of the residual variation after applying the model to samples that have been held out, i.e. how much of a sample cannot be explained by the model. The difference with $R^2$ is that $R^2$ is used on the training set samples included in the current round of cross validation.

How is it calculated

With each component you expect to fit progressively more of the variation (you do achieve this in included samples, but is not guaranteed in held out), so the $Q^2$ can decrease - you can see progressively higher residuals. This is why the cumulative function is not a sum of all previous. The metric has a range of 0 to 1 (see below for exception), so each component can only account for 1-previous $Q^2$, hence the $1−cumulativeQ^2_{n−1}$ term.

You need to calculate $Q_i^2$ for the $i^{th}$ PC directly, and the reference you give in the comments (http://wiki.eigenvector.com/index.php?title=T-Squared_Q_residuals_and_Contributions) does indeed indicate how this is done. Some others are

https://www.rdocumentation.org/packages/pcaMethods/versions/1.64.0/topics/Q2

https://umetrics.com/sites/default/files/books/sample_chapters/multimega_parti-3_0.pdf (Page 58)

It is calculated the same as $R^2$ but only on held out data.

What it means

The $Q^2$ is an indicator of how much variation is accounted for in your held out dataset. You hope to see this rise similarly to $R^2$, but as your model starts to overfit the two diverge more and more. More critically $Q^2$ is a very useful metric as it starts to drop again in held out samples if the model is overfitted. This indicates that the model PCs are accounting for variation that is not stable in the held out samples (which is the definition of an overfitted model). Lower order components of overfitted models contain non-reproducible variation, which actually then imputes this noise onto the residual after accounting for the model.

Like one version of $R^2$, it is possible to get a negative number indicating a very bad model that is adding noise to your held out samples When is R squared negative? http://forum.smartpls.com/viewtopic.php?t=15444

$\endgroup$
  • $\begingroup$ Do you know how Q² is calculated for each component? I found this which explains how to calculate a Qᵢ value for each sample. Is this correct? How do I combine the Qᵢ to give the total Q² for a given k? $\endgroup$ – Kevin Nowaczyk Mar 21 '18 at 16:48
  • $\begingroup$ I reread my linked document, and it clearly defines i as the index for each sample, NOT each variable. It says Qi is calculated as the sum of the squares of a particular row from E, and E is effectively the residuals for each sample in X. If I have a data set of 250 samples, and 10 variables, I will expect to have 250 Qi values for a given subset of PCs. I’ll dig into your other links next. $\endgroup$ – Kevin Nowaczyk Mar 23 '18 at 11:24
  • $\begingroup$ I used the definitions for xhat and Q² as defined in the rdocumentation.org site. When I used a P matrix with 1 eigenvector, the first component loading, the result for Q² was the same as R² calculated as (Eigenvalue/variables) for that component. If more than one vector is used in P, the Q² is the cumulative R² summed over the same components. The Excel addin has different values for R² and Q². $\endgroup$ – Kevin Nowaczyk Mar 23 '18 at 17:46
  • $\begingroup$ I get it now. I have to regenerate the model at every sample, leaving that sample out, and sum the squares of the differences between the two predictions. $\endgroup$ – Kevin Nowaczyk Mar 23 '18 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.