Fitness metrics for a classification problem I am new into ML and trying to play around and find an expression that yields a binary (Yes/No) output for a simple classification problem (with 7 inputs). For one solution (obtained using genetic algorithm by applying logistic regression), R^2 is around 0.4, MSE is 0.11 and ROC curve is almost perfect. I guess I am asking if somebody can explain the differences between these two fitness metrics and which would apply best to this problem. Thanks.
 A: *

*$R^2$ is a measure of how close the observations lie to the fitted
line and effectively indicates (as a percentage/proportion) how much
variability the model explains out of the total variability in your
response around it's mean. This blog gives a more detailed break
down of it. Here is an excerpt: 

R-squared is always between 0 and 100%:
0% indicates that the model explains none of the variability of the
  response data around its mean. 100% indicates that the model explains
  all the variability of the response data around its mean.
In general, the higher the R-squared, the better the model fits your
  data.

$R^2$ has limitations though  - it can't tell you if a model is bias
or not, and does not really tell you how adequate the model is. Moreover, it will always increase with additional terms, so can be bias to overly complex models that are really just fitting noise.

*Mean squared error (MSE) is exactly as it's name suggests, and it is used to tell you about how well your data predicts new
data. It essentially describes the average squared error you would
expect if you tried to predict new data. 
However it comes with a few issues. Since the data is squared it is
sensitive to outliers (big errors lead to big increases in MSE, even
if they are only one off). Moreover, since the errors are squared,
and then never square rooted the metric is not on the same scale as
the original data. For this reason many people prefer root mean
squared error (RMSE) which does this last step for you. In your
situation (I.e. fitting a logistic regression model where the
response follows a binomial distribution) MSE falls a little short
because your response variable is bounded (between 0 - 1) which means
that the MSE (or RMSE) might suggest that a prediction has an error
+/- greater than 1 or less than 0.... This is were brier scores come in which is more meaningful as it was derived specifically for
bounded outcomes.

*Receiver-operator curves (ROC) plot the true positive rate against
the false positive rate. However, people more often use the area
under the curve (AUC) of a ROC, as this creates a single metric
describing how often the model correctly classifies negatives and
positives. These metrics aren't really all the great though because
they don't tell you individually about the models ability to classify
as they only give you one metric... For instance a high AUC might be
the result of a really high specificity (true negative rate) and
a mediocre sensitivity (true positive rate) meaning the model is
great at identifying false hoods but poor at identifying "successes".
The same AUC could equally be indicating the exact opposite.... As
such, it is better to report specificity and sensitivity separately,
or presenting a two by two confusion matrix. You can also look
at the true skill statistic.


An extra note on all of this - when calculating any of these metrics it is often a good idea (where possible) to use something like cross-validation to calculate them as they give you a much better idea about the models true performance when it encounters new data. If you are using R then this can be quite easily done with the caret package.
Let me know if anything is unclear.
