0
$\begingroup$

Let $A$ be a $k \times 1$ random vector, and $\mathbf{A}$ be a $n \times k$ matrix of observations.

Letting $t \in \mathbb{R}^{k}$ be a vector of weights s.t. $||t||_2 = 1$, suppose we are interested in maximizing the variance of a linear combination of the columns of $\mathbf{A}$, i.e. $\mathbf{A}t$. I have read that this can be written as solving $$ \max_{||t||_2 = 1} \frac{t'(\mathbf{A' A})t}{t't}$$


However I'm a bit confused about where this came from. Would the problem not be solving the following instead? $$ \max_{||t||_2=1} \quad \text{Var}(\mathbf{A}t) = t'\text{Var}(\mathbf{A})t = t'(\mathbb{E}[AA'] - \mathbb{E}[A]\mathbb{E}[A'])t $$ Replacing the expectations with the sample analogue yields, \begin{align*} \max_{||t||_2=1} \quad &t'\left(\frac{1}{n}\sum_{i=1}^n A_i A_i' - (\frac{1}{n}\sum_{i=1}^n A_i) (\frac{1}{n}\sum_{i=1}^{n} A_i') \right)t \\ &\propto t'\left( \mathbf{A'A} - \frac{1}{n} \sum_{i=1}^{n}A_i \sum_{i=1}^{n}A_i' \right)t \\ &= t'\left( \mathbf{A'A} - \frac{1}{n} \sum_{i=1}^{n}\sum_{j=1}^{n}A_i A_j' \right)t \end{align*}

where $A_i$ is the $i$th row of observed $\mathbf{A}$. In the formulation above, a) what happened to the term involving the double summation and b) where did the $t't$ in the denominator come from?

If we further assumed $\mathbf{A}$ has mean 0, then I can see how a) drops out though that doesn't address b), and the formulation above also does not seem to mention the requirement of de-meaning.

$\endgroup$
4
  • 1
    $\begingroup$ people tend to assume that data are centered when discussing these kinds of things, and this is the resolution to your (second) point (a). Regarding your (first) point (b), we need to divide by the length of $t$, or find some different way of normalizing, otherwise we can get arbitrarily large variances by just making $t$ very big. $\endgroup$ Commented Jan 30, 2023 at 21:06
  • 1
    $\begingroup$ The $t^\prime t$ was never in the denominator, because under your restriction $||t||_2=1,$ it was identically equal to $1.$ $\endgroup$
    – whuber
    Commented Jan 30, 2023 at 21:28
  • 1
    $\begingroup$ Thanks! I have figured it out with your hints. The original formulation should've been written as $\max_{t \neq 0}...$, and only applies if $\mathbf{A}$ is mean 0. The point (a) then drops out, and to get the denominator, we can for instance apply a Lagrangian to the constraint $||t||_2=1$. $\endgroup$
    – Adam
    Commented Jan 30, 2023 at 21:33
  • 1
    $\begingroup$ For further discussion of some the points raised here, see my post at stats.stackexchange.com/a/301561/919 (on whether the PCA optimization problem is convex). $\endgroup$
    – whuber
    Commented Jan 31, 2023 at 18:29

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.