0
$\begingroup$

I'm trying to implement deep speaker embeding system and after getting voice embeddings I need to somehow calculate accuracy. But there is no mention in deep speaker paper about how they calculated accuracy for person identification (at least I've not found it).

So in general I have array of embeddings (shape [N, 512]), corresponded labels (shape [N]) and some function f(a, b) which calculate distance between embeddings. In order to calculate accuracy (and other metrics) we need to find array of predictions.

Currently I'm using nearest neighbor approach:

for each emb1 find nearest emb2 and set label of emb2 as label for emb1. I also tried to use mean of embeddings of specific class instead of emb2, but this approach yields worse result.

Also come is mind some clustering approach:

divide embeddings into k clusters and check if all items in each cluster have same label. Where k is number of unique classes.

$\endgroup$
0
$\begingroup$

See the last paragraph in the first column of page 5:

Speaker verification and identification trials were constructed by randomly picking one anchor positive sample (AP) and 99 anchor negative samples (AN) for each anchor utterance. Then, we computed the cosine similarity between the anchor sample and each of the non-anchor samples.

So, it sounds like they run multiple trials (one per test utterance). In each trial they take one of the test utterances (let's call it $x_t$) and create 100 pairs $(x_t, x)$ where $x$ is a randomly chosen utterance. However, they make sure that out of the 100 pairs one of them is positive (i.e. both $x_t$ and $x$ are from the same speaker) and the rest are negative (i.e. $x_t$ and $x$ are from different speakers).

You can assign a value of 0 or 1 to each trial by computing the cosine similarity between the two utterances in each pair (i.e. $\frac{x_t \cdot x}{||x_t|| \times ||x||}$) and checking weather the positive pair got the highest similarity score among all the 100 pairs (1) or not (0).

Finally, you can run multiple trials to compute the accuracy. In the first column of page 5 in the paper they specify how many trials they ran for each dataset. If you run $N$ trials and $M$ of them evaluate to 1 then the accuracy of your method is $M/N$.

$\endgroup$
  • $\begingroup$ have read it and forget it... thank you for your time :) $\endgroup$ – Aiven Dec 28 '17 at 16:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.