Is there something like softmax but for top k values? I have a dataset with binary labels of which exactly k outputs are 1, on which I want to train a neural network. If k=1, softmax can do the job of representing the output distribution. I am interested in k > 1.
One approach that comes to mind is to use multiple sigmoid units and pick the top k outputs. With this, however, we cannot force the number of positive outputs to be exactly k. So, I was wondering if there is a better method.
 A: I assume you want to approximate the binary representation for picking k-out-of-n, here's an idea using tempered & weighted softmax for k times.
Let me define the tempered & weighted softmax as
$$\text{softmax}(x,w, t)_i = \frac{w_i\exp(x_i/t)}{\sum_{i=1}^n w_i\exp(x_i/t)},$$
where $w_i$ is the weight, with $w_i\ge 0$, and $t>0$ is the temperature, say, $t=0.0001$ to make the result close to a binary vector.
If we do softmax for the first time with all $w_i=1$, this produces a top-1 vector, call it $x^{(1)}$.
Now let $x^{(2)}= x^{(1)} + \text{softmax}(x, 1-x^{(1)}, t)$. The weight $1-x^{(1)}$ zeros out the top-1 element, hence gives us the second largest in the softmax. Doing this recursively for $k$ times gives us an approximate to top-k binary.
Here's the R code.
softmax_w<- function(x,w, t=0.0001){
  logw = log(w+ 1E-12) #use 1E-12 to prevent numeric problem
  x = (x + logw)  /t
  x = x - max(x)
  exp(x)/sum(exp(x))
}

top_k<- function(x, k){
  y = rep(0, length(x))
  
  for(i in c(1:k)){
    x1 = softmax_w(x, w= (1-y))
    y = y+ x1
  }
  
  y
}

x = rnorm(10)
y = top_k(x, 5)

y


