I have log data and I encode the data for clustering purpose.

For example, I have one data column and I represent this unique data in numerical values or binary to be as one column as below.

Example 1

col1: Failure, Error, NonError, Wrong, Failure.

col1_encoded: 2, 1, 5, 3, 2

Example 2

col1: Failure, Error, NonError, Wrong, Failure.

col1_encoded: 101, 010, 1001, 1101, 101

Then, the encoded column used for data clustering.

If I do clustering, I will get different clustering for first or second type of encoding.

How do you evaulate that encoding is the proper method?. The examples upp present these types of encoding. Different encoding methods give different clustering results.

Which one is more correct?

Any help

  • $\begingroup$ Could you explain what "correct" might mean? $\endgroup$ – whuber Jan 8 at 18:16
  • $\begingroup$ I hope I defind it better. $\endgroup$ – YAcaCsh Jan 8 at 18:29
  • 1
    $\begingroup$ What do you mean by encoding "representing" the data? If you use random labels, or single label for all categories, then it does not represent anything, otherwise it represents something, but I don't know what kind of answer you are looking for. $\endgroup$ – Tim Jan 8 at 18:42
  • $\begingroup$ @Tim there are many different encoding methods. How do we know that the selected method represents the data correctly?. For example, if we use the first example encoding then we cluster the data, we might get different clustering results if we encode the data in binary. Binary method of integers represent the real data through encoding technique. $\endgroup$ – YAcaCsh Jan 9 at 7:59
  • $\begingroup$ @Tim I have clarrified. Let me know what you think. $\endgroup$ – YAcaCsh Jan 9 at 8:35

You should not get different results. You should treat any encoding of this variable as categorical, if you are treating it as numeric, then both codings (and any others like that) are wrong.

To do clustering, you need to create a distance matrix or similarity matrix. There are methods of doing this for categorical variables and they should work out the same for any encoding, as long as you don't treat it as a number.

  • $\begingroup$ Thank you for the answer. What do you mean by categorical?. If you use k-means, you need to calculate the sequare distance between the selected cluster and the data point. In this case, your numerical values that encoded will be used for calculating the distances and clustering data. $\endgroup$ – YAcaCsh Jan 9 at 12:36
  • $\begingroup$ Categorical means having categories. You don't have numbers. No numerical encoding is correct here. You need to create a distance matrix from the categorical data; there are methods for doing that, as I said, but just picking some encoding is not one of them. $\endgroup$ – Peter Flom Jan 10 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.