I have a bunch of data of the form (x0, x1, x2) -> y (everything is categorical data), and I want
- a decision/classification rule for predicting y
- a metric on y's space (there is no "natural" metric)
I won't be able to do perfect prediction, (I have instances where the same feature vector corresponds to a different y, and expect outliers and errors). So, I want the decision rule that has the "least error", according to the metric.
But since I don't have a natural metric, I'm defining the distance (ya, yb) as "how often the decision rule would predict ya when the value is yb (in the training data)".
Now, this gets a bit circular, but it seems to me I can make a decision rule, use that to get a metric, use that metric to get a slightly more refined decision rule, etc. and eventually converge on something stable (possibly with some extra steps of identifying outliers in the training data and manually removing them if they are indeed wrong).
Does this seem sensible? Is there a better/more general approach to this? Is there a name for this kind of approach? ("classification with a distance/error function"?)