1
$\begingroup$

I'm sorry if this is vaguely worded, but I'm looking for a way to score each row in a dataset by, essentially, the amount of information that that row adds - or the uniqueness of that row in the data set. For example, say I have a table with 3 rows and two columns:

  1. col1: a, col2: b
  2. col1: c, col2: d
  3. col1: a, col2: d

I want a metric that basically says that rows 1 and 2 are equally weighted and 3 has a lower weight. My thinking behind this is somewhat along the lines of - if I wanted to list all the sets that contained all the values for each column, then I would have 2 sets {1,2} and {1,2,3}. Rows 1 and 2 are elements in both sets, but row 3 is only in one of the sets, so in some way it should have half the weight.

Is there a way to scale this or a different sort of metric around the uniqueness of each row when all of the fields are categorical?

Thanks!

$\endgroup$
8
  • $\begingroup$ Maybe clustering based on cosine distance can help. After you cluster all rows, you can post analyze the ones most distant from their clusters centeroids. $\endgroup$
    – yoav_aaa
    Feb 7, 2020 at 19:47
  • $\begingroup$ In a dataset it is usually understood that different columns can have different datatypes, so that it won't make sense in general to compare a or c to either of b or d. Your question, however, uses set notation to designate the rows, as if the columns did not matter and a row is just a set (which has no order). Could you please clarify your meaning? Some additional examples of the values of the intended metric would also be helpful, because it will help narrow the (currently huge) range of possible solutions. $\endgroup$
    – whuber
    Feb 7, 2020 at 20:05
  • $\begingroup$ I didn't mean to imply that the row is a set - I just didn't know a better way to display the data. In the example, a,c,a is one column and b,d,d is another column. I edited it, please let me know if it is clearer $\endgroup$
    – dfmoore
    Feb 7, 2020 at 20:38
  • $\begingroup$ Now your description appears to be at odds with your calculations. The subsets of the rows {1,2,3} containing all values of column 1 are {1,2} and {2,3}, while the subsets containing all values of column 2 are {1,2} and {1,3}. Your calculation still seems somehow to be throwing all the values in both columns together, rather than operating on "each column" as you write. Presenting a set of examples, rather than one very limited example, may help clarify your meaning and overcome any ambiguities in your description. $\endgroup$
    – whuber
    Feb 7, 2020 at 20:53
  • $\begingroup$ This seems to match with my description... Your sets are looking at column by column which is fine. And since {1,2} is in both of your set-of-subsets, then it contains all values for both columns. Only other subset that does that is {1,2,3}. $\endgroup$
    – dfmoore
    Feb 7, 2020 at 21:39

0

Browse other questions tagged or ask your own question.