I'm reading Rasmussen's GPML text and I am having trouble understanding the likelihood function and the underlying function itself.

For C classes, it first treats the latent function as $$f = (f^1_1, \dots, f_n^1, f^2_1, \dots, f_n^2, \dots, f_1^C, \dots, f_n^C)$$

prior as: $$f\tilde{} N(0,K)$$ Likelihood as: $$p(y_i^c|f_i) = \pi_i^c = {exp(f_i^c) \over \sum_{c'} exp(f_i^{c'})}$$ From here, given data, I can compute $$p(f|X,y) \propto p(y|f) * p(f)$$

Here are my questions:

  1. What should $K$ be in the prior? Do I need to provide some istoropic gaussian?

  2. What is the likelihood function? Given $M$ number of samples, I wish construct the likelihood function. For binary, it was simply $$\prod_{i=1}^M \sigma(a)^t \cdot (1-\sigma(a))^{1-t}$$ where $t \in \{0,1\}$ and $\sigma(a)$ is sigmoid function, and $a=f(x)$. For multi class do i need to construct likelihood for each class separately?

  3. The underlying function is a combination of all classes. A test data would be a vector of length $n$(and not $Cn$ like the underlying function). How do I make prediction on my test data?


1 Answer 1

  1. $K$ is found using the covariance functions you've selected for each class. These can be the same or different. The treatment in the book assumes that the latent functions are independent though, so K is block-diagonal.

  2. The likelihood for multiclass classification (assuming i.i.d. samples) is the product of the likelihood for each $y_i$. $P(y|f) = \prod_{i=1}^{M}\frac{\exp(f_i^{t_i})}{\sum_{c\in C} \exp(f_i^{c})}$, where $t_i$ is the true class for the ith example. This is just the softmax likelihood function you defined above, so I'm not sure where your confusion is.

  3. To make a prediction, you find the most likely value of each latent function at your new input, then softmax those to get class probabilities. I think you're confused here because $n$ is the number of training examples. Each of those examples will be an input, which is a vector of dimension $d$, and an output $y$, which is a vector of dimension $C$. Your new inputs would therefore also be of dimension $d$, not $n$. Algorithm 3.4 in the book describes a numerically stable way to do this.

  • $\begingroup$ 1. Right. But the book seems to use the notation $p(f|X)$. Does this mean that it uses data to construct block matrices $K_c$ for each class? 3. If $n$ is number of training samples, what is $f=(f_1^1,…,f^1_n, f^2_1,…,f^2_n,…,f^C_1,…,f^C_n)$? I thought each $f$ was $x^Tw$ where $x$ is our input. But what confuses me is that $f$ must be $nC$ in length which can't happen if I have an uneven number of data for each class. $\endgroup$
    – MoneyBall
    Commented Apr 23, 2017 at 1:10
  • $\begingroup$ 1. Yes. 3. For each data point, you have an $\mathbf{x}$ (d-dimensional vector) and a one-hot vector $\mathbf{y}$ that identifies its class. However, during training and inference, you infer $f^1$ through $f^C$ for every input, regardless of its output. So, for example, if you had 3 classes, $\mathbf{f}_i$ for each input would be a vector of length 3. Therefore, no matter how many of each class you have, you always end up with $nC$ entries in $\mathbf{f}$. $\endgroup$
    – Kevin Yang
    Commented Apr 23, 2017 at 5:47
  • $\begingroup$ Okay, things are making more sense to me now. One more question: to approximate the posterior $p(f|X,y)$, let's say I use Laplace approximation. Once I found the mode and covariance of $p(f|X,y)$, I can create a gaussian $q(f | X,y)$. Now I can compute the predictive mean and covariance. When constructing covariance matrix $K$ with the new datum $x_*$, do i need to use the hyperparameter obtained from finding the maximum likelihood of marginal likelihood of $p(y|X)$? $\endgroup$
    – MoneyBall
    Commented Apr 24, 2017 at 17:54
  • $\begingroup$ Yes. If you think my answer is sufficient, could you select the checkmark to indicate that the question is answered? $\endgroup$
    – Kevin Yang
    Commented Apr 24, 2017 at 23:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.