I have a bunch of data of the form (x0, x1, x2) -> y (everything is categorical data), and I want

  • a decision/classification rule for predicting y
  • a metric on y's space (there is no "natural" metric)

I won't be able to do perfect prediction, (I have instances where the same feature vector corresponds to a different y, and expect outliers and errors). So, I want the decision rule that has the "least error", according to the metric.

But since I don't have a natural metric, I'm defining the distance (ya, yb) as "how often the decision rule would predict ya when the value is yb (in the training data)".

Now, this gets a bit circular, but it seems to me I can make a decision rule, use that to get a metric, use that metric to get a slightly more refined decision rule, etc. and eventually converge on something stable (possibly with some extra steps of identifying outliers in the training data and manually removing them if they are indeed wrong).

Does this seem sensible? Is there a better/more general approach to this? Is there a name for this kind of approach? ("classification with a distance/error function"?)

  • $\begingroup$ So what you are looking for is the appropriate metric for evaluating any decision rules on your data, right? $\endgroup$ – TaoPR Jun 3 '15 at 9:10
  • $\begingroup$ I already use Gini impurity, but in addition/instead of that I'd like to mark certain miscategorisations as "less wrong" than others - and also I want to find outliers with a distance function, to clean the data. $\endgroup$ – Emile Jun 3 '15 at 10:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.