I'm working on a multi-class problem which I have redefined as a series of binary problems (i.e. a one vs all classification problem). However, each observation can belong to more than one class. For example, if my observations where different kinds of fruit my classes might represent different characteristics such red and round. In some cases a fruit is both red and round.
My question is: what should I consider when evaluating my binary models? Can one simply use metrics such as accuracy to understand the performance of the model. If I have three different classes (i.e red, round and sweet) is it acceptable to merely take the mean accuracy of the three binary classification tasks as the accuracy of my model as a whole?
This is a little different than most multi-class classification problems I've seen where all the classes are independent.