3
$\begingroup$

I am self-teaching Machine Learning, so excuse me if this is a stupid question. I am trying to understand MLP Classifiers, but would like to know what the best way to create a score is. e.g. preferably a normalized score between 0 and 1.

For instance, I looked at Scikit-learn's MLP Regressor which uses a score of

$1 - u / v$, where $u = \sum(TRUE - PREDICT)^{2}$ and $v = \sum (TRUE - AVGTRUE)^2 .$

see: here

I can't see a way of extending this to Classifier problems. Scikit-Learn's documentation is (to me at least) not very clear on what the Classifier score actually is, as in what the calculation is.

I would like to know a standard, or recommended, function for calculating a Classifier score (not necessarily Scikit-Learn's).

$\endgroup$
  • $\begingroup$ If you don't write out what you mean by "MLP", I'm going to assume you're trying to classify My Little Pony characters. It's a hard problem, now that we have 7 seasons and several movies. $\endgroup$ – Kodiologist Sep 26 '17 at 19:56
1
$\begingroup$

You are essentially asking the loss function in classification setting. You may read this page in wikipedia. Here is a short version:

  • If the classifier is outputting a binary label in binary classification problem, 0-1 loss can be used.

  • If the classifier is outputting a probability number in binary classification problem, logistic loss can be used.

$\endgroup$
0
$\begingroup$

The value you give is the Coefficient of determination which is by no means the best choice for optimizing a regression problem. There are multiple options and you must decide based on your problem. I tend to use RMSE or MAE if I want to be less sensitive to outliers. The Coefficient of determination is mainly used because it is normalized.

For binary classification the situation gets more complicated. I would read this to get some idea or your options. Many are easily availiable in sklearn. I would recommend you think about which type of error you are most concerned about. For example in many problems the most important issue is Sensitivity or Precision so you want to evaluate based on one of them alone. A good combined metric is the Matthews correlation coefficient but as with regression you must decide what your goal is.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.