In Andrew Ng's lectures he introduces Locally Weighted Linear Regression (page 15) for which he describes the weight as:

$$ \begin{align} w^{(i)} = exp\Big(-\frac{(x^{(i)} - x)^2}{2 \tau^2}\Big) \end{align} $$

However in the problem assignment for Locally Weighted Logistic Regression, the weight is described as:

$$ \begin{align} w^{(i)} = exp\Big(-\frac{||x^{(i)} - x||^2}{2 \tau^2}\Big) \end{align} $$

where $x^{(i)}$ had dimensions $1 \times n$ and $x$ has dimensions $m \times n$.

Looking at the solution set, it seems that they first get the difference between $x$ and $x^{(i)}$ (giving an $m \times n$ matrix) followed by element-by-element squaring (using the Matlab/Octave .^2 operator), and then sum along the rows to give a $m \times 1$ vector. In the linear algebra review notes (pages 7-8), multiple types of norms are described, but the sequence of operations performed doesn't seem to be consistent with any of the descriptions of norms. What is going on here?


First off, because I am not sure that this wasn't a part of your question, the norm they are using is defined as $|| \mathbf{x} || = \sqrt{\sum_{i=1}^n x_i^2}$, so if we take the norm of $\mathbf{x} - \mu$ (where $\mu$ is our mean or center) and square it, we get $|| \mathbf{x} - \mu ||^2 = \sum_{i=1}^n (x_i - \mu)^2$. The two expressions are equivalent.

It sounds like they are assigning a different center, $\mu_{i,j}$ for each $x_i$ and each hidden node $h_j$ for $j = 1, 2, \ldots, m$. In that case, for each input value $x_i$ and hidden node $h_j$, you would have:

$$ w_{i, j} = \exp \bigg(\frac{(x_i - \mu_{i,j})^2}{2\tau_j^2} \bigg) $$

The input to node $h_j$ is then the sum of these values over all inputs $x_i$ for that node: $\sum_{i = 1}^n w_{i, j}$. I think what they are doing in the code is just the vectorized form of this.

| cite | improve this answer | |
  • $\begingroup$ Ah feel so stupid! Didn't write it out and so didn't see that the outer square was cancelling out the square root. Thank you! $\endgroup$ – oort Mar 11 '17 at 18:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.