I was reading the following paper on hyper basis function (HBF) (similar to radial basis function RBF network) and was trying to figure out how one learns the movable centers of the hyper basis functions (less importantly in this question, but also the rest of the parameters). Recall that we trying to find the optimal setting of the parameters of the predictor function $f^*(x) = \sum^n_{\alpha=1} c_{\alpha}G(\| x_i -t_{\alpha} \|^2_{\bf{W}}) $ such that we minimize:

$$ H[f] = \sum^{N}_{i=1} ( y_i - f(x_i) )^2 + \lambda \| Pf\|^2 $$

where $c_{\alpha}$ is the set of weights for each center, $t_{\alpha}$ are the centers itself (that are NOT necessarily the same as the data points) and $\bf{M} = W^TW$ is the weight matrix indicating how much we weight each coordinate i.e. $\bf{W}$$x$ indicates how much we weight each coordinate.

One way that they suggest to do this in the paper is by running gradient descent (i.e. standard gradient descent for the parameters $c_{\alpha}$, $t_{\alpha}$ and $\bf{M}$ w.r.t. to the function $H[f]$ but they only mention the simple case $\lambda = 0$). Which seems totally reasonable to me. However, they also mention other settings of the parameters that makes it slightly confusing to me if its equivalent to trying to minimize $H[f]$ or not.

For example on page 12 they have the following setting of the parameters:

$$ t_{\alpha} = \frac{ \sum_{i} P^{\alpha}_i x_i }{ \sum_i P^{\alpha}_i }, \alpha =1, ..., n$$

where $P^{ \alpha}_i = \Delta_i G'( \| x_i - t_{\alpha}\|)$ and $\Delta_i = y_i - f^*(x_i)$. I do know that they mentioned that equation only applies when one uses the identity function for $\bf{M}$. Therefore, that obviously means in that case the parameter $\bf{M}$ is fixed and doesn't have to be learned or included in the minimization procedure. Furthermore, sine $P^{\alpha}_i$ depends on the center $t_{\alpha}$ being learned itself, it seems to me the equation is cyclical and not clear how to compute. How does one event compute this?

Even if it was not cyclical, it is not 100% clear how we have to learn the rest of the parameters $c_{\alpha}$. What I am guessing is that in this case the parameters $t_{\alpha}$ and $\bf{M}$ are considered fixed, then one just uses equation 20:

$$ c = (G^TG + \lambda g)^{-1} G^T \bf{y}$$

where $(\bf{y})_i = y_i$, $(\bf{c})_{\alpha} = c_{\alpha}$m $(G)_{i\alpha} = G(x_i; t_{\alpha})$ and $(g)_{\alpha \beta} = G(t_{\alpha} ; t_{\beta})$.

So why question is how does one learn the movable centers?

To my understanding is that, if nothing is fixed apriori, then we can only find them via standard gradient descent. Otherwise, if we have the conditions that $\bf{M} = I$ is fixed, then $ t_{\alpha} = \frac{ \sum_{i} P^{\alpha}_i x_i }{ \sum_i P^{\alpha}_i }, \alpha =1, ..., n$ and $ c = (G^TG + \lambda g)^{-1} G^T \bf{y}$ minimizes $H[f]$.

Is that correct or am I missing something?

Also, I was watching the lecture following lecture and they mentioned that the way to find the movable centers is by k-means clustering (in fact a weird non-standard stochastic gradient descent like clustering). Is finding the weight through that weighted sum the same as the standard k-means algorithm or is it slightly different? When do you use k-means clustering to find the centers? Is k-means always the way to go or only when $\bf{M}$ is fixed?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.