Say I have created some numerical embedding for each word in a text corpora -- that is, for each word I have some n-dimensional vector representing the word. Now say I would like to perform some modelling task using those embeddings on various documents. As such, I'd want to aggregate the word embeddings per document, based on their frequency in the document, to create a single vector representing the document.

A simple way to do this would be to do a weighted average of the word embeddings based on their frequency. However, I might also want to capture the extremes of the vectors by capturing the min and the max (and concating that vector to the weighted average as new features). The min or max word embedding however could just be of extremely low frequency and not really representative of the extremes of the document.

Is there then a way to account for the vectors' weight in coming up with a min or max similar to average? Or restated, is there a way to represent the proportional "extremes" of the vector?

I imagine there must be some arbitrariness involved in such, but I figure there is a principled approach. I've thought perhaps of a weighted avg of upper/lower quantiles but suspect there might be a better method.

  • 1
    $\begingroup$ hmmm not sure about min and max, but for other extreme quantiles, like 95% and 5%, you could define weighted quantiles via a weighted tilted loss stats.stackexchange.com/questions/251600/… (Whuber's answer provides a nice derivation and jjet has a clear form for the loss term) $\endgroup$ Jan 21, 2023 at 16:05
  • 1
    $\begingroup$ @JohnMadden My trouble with thinking of it from the quantile perspective is I don't perceive the obvious means of finding guidance on where to draw the quantile cut. For instance, it seems just as valid to extract my measure from the top 25% or the top 5%. It may well be arbitrary or require some context knowledge, but I'm hesitant to claim so without search. $\endgroup$
    – Josh
    Jan 21, 2023 at 16:12
  • $\begingroup$ why not try some different ones and see how they compare? $\endgroup$ Jan 22, 2023 at 2:14
  • $\begingroup$ @JohnMadden This would be my default stance pending any better method. My query is to see if there is a more principled approach out there. $\endgroup$
    – Josh
    Jan 22, 2023 at 14:36
  • $\begingroup$ another option if your modeling task is supervised: treat the quantile as a parameter to backpropagate wrt to. $\endgroup$ Jan 22, 2023 at 15:45


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.