1
$\begingroup$

I have a big set of subsets of 2 or more documents (maximum 10). The documents from each subset should describe the same product. One document contains pairs attribute - value:

<description>present box</description>
<length>76 cm</length>
<width>54 cm</width>
<height>0.80 m</height>
<weight>0.387 kg</weight>

For each subset I want to verify if the documents indeed describe the same product.

After extracting and converting quantitative values I ended up with a matrix like this:

           length width height weight
document1      76    54     80    387
document2      77    53     82    385
document3      85    65     87    411

Searching the web I found out that this could be a clustering problem but I couldn't figure out what clustering algorithms should I use. For k-means I have to provide the number of desired clusters and in this case I don't know it. For hierarchical clustering how should I interpret the dendrogram?

Because the question can be simplified to "There is one or more than one cluster?" where one cluster means the documents describe the same product and multiple clusters -> multiple products, which clustering algorithm should be applied for maximum accuracy? Or I am on the wrong direction from beginning and I have to differently address the problem?

$\endgroup$
0
$\begingroup$

Clustering usually builds on similarities.

You have vectors now, but not yet similarities. How do you do convert cm into grams to make the attributes comparable?

Also, do you really want to cluster boxes? There are standard sizes for certain product classes, so do you need anything beyond a tolerance threshold?

$\endgroup$
  • $\begingroup$ hi @Anony-Mousse! From vectors I can build the distance matrix. Some clustering algorithms use the distance matrix as a similarity measure. Anyway, I saw that some implementation build the distance matrix as part of the algorithm without need to build it myself. Why I need to convert cm into grams? I compare one document's length with another document's length. I don't compare length with weight. I do not cluster boxes, but documents that should describe same product. The present box is just an example. $\endgroup$ – Jorj Feb 5 '19 at 15:47
  • $\begingroup$ Distances are not that easy! What is the reason to compute them this way? Look at the distance computations! There are infinitely many distances, and Euclidean likely is not the right one here. Do the distance computation with units, to understand the problem! $\endgroup$ – Has QUIT--Anony-Mousse Feb 5 '19 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.