Performance does not englobe a formal definition, it is mostly dependent on, but not limited to:
- the context of the model you are trying to implement
- the metrics you are using to evaluate the model's output
- the objective you are pursuing
The use of one metric or another will depend whether you are trying to predict a continuous or a discrete variable. Some of these are: Accuracy, Recall, Precision, R2, F-Measure, Mean Square Error, etc.
To make this clear, say for instance you are working on a credit card fraud machine learning algorithm, where you want to predict the number of fraud transactions. To evaluate how well the algorithm works: a) understand the context b) understand the metrics that are applicable to the problem
a) We are dealing with a classification problem; a transaction can be fraud or not (target is a discrete variable). We will be most-likely facing a highly imbalanced dataset since most of the credit card transactions are non-fraudulent.
b) Since it is a classification problem, we can discard all the metrics associated to continuous variables (for example, R2 and MSE). Moreover, due to the possible imbalance, we can also discard Accuracy metric. This leaves us with two metrics: Recall and Precision. We know by theory that these two metrics present a trade-off (if I increase one, the other decreases). Recall would help detect the major possible transactions that were fraud. Precision would help with avoiding misclassifying frauds.
Concluding, how well our it works, after considering the previous things, will ultimately depend on what is our objective:
- Is our goal to detect the highest amount of fraudulent transactions
as possible? If it is, our metric to evaluate the model's
performance, and further improvements, will be Recall.
- Is our goal to avoid classifying a transaction as non-fraud when it
was a fraud? If it is, our metric to evaluate the model's
performance, and further improvements, will be Precision.
Hopefully this provides you a better insight.