1
$\begingroup$

For context, I've been using feature hashing for a rapid text classifier with a very small number of features (2000, it is very small on purpose). I noticed that some of the results were a bit wonky due to collisions and was wondering if it would be possible to select a better hash (varying the hash function and random key) to have more "natural" collisions.

To select a better hash empirically I've settled on, for a given dataset, measuring the similarity of the distribution of the hashing features versus neural features Specifically I want to capture:

  1. How well are neighborhoods preserved?
  2. How well preserved are distances between neighborhoods?

Here's what I've tried so far:

  1. Kendall's Tau, captures 1. but not 2. It's also relatively expensive to compute (~20 seconds on an older computer)
  2. Gromov-Wasserstein, looks at distribution rather than specific words. Similar time to 1
  3. Solving for a linear transformation between the two (Least Squares). Not sure how to transform that into a meaningful metric (residual, magnitude?). Cost is variable.

Beyond a method to measure embedding space similarity I'd also be open to other methods of picking better hashes

Improve this question
$\endgroup$
4
  • $\begingroup$ Because there are so many plausible solutions, please clarify--as quantitatively as possible--what aspects of "similarity" are of importance to you. $\endgroup$
    – whuber ♦
    Commented Aug 13 at 21:01
  • $\begingroup$ @whuber are preserving neighborhoods (k nearest items) and conversing distances between items across the two embedding spaces not quantitative descriptions of similarity? If not could you give me an example of quantitative definitions of similarity or a reference? $\endgroup$
    – Felix Labelle
    Commented Aug 16 at 17:37
  • 2
    $\begingroup$ "Quantitative" refers to how one measures something. So far you have offered only qualitative descriptions of what you are looking for. Usually, one's application will provide guidance concerning how to quantify a distance. But, because you don't describe your application and how you intend to use these measures of distance "preservation," we don't have adequate information for answering. $\endgroup$
    – whuber ♦
    Commented Aug 16 at 18:29
  • $\begingroup$ Fair point about the purpose, I've added context for the underlying problem I'm trying to solve. LMK if that looks good now $\endgroup$
    – Felix Labelle
    Commented Aug 16 at 20:12

0

Reset to default