1
$\begingroup$

I have a fairly reasonable understanding of the theory behind neural networks, regularization, and cross-validation, but I am lacking in the actual experience department.

In a nutshell, I am using a neural network regression model to make predictions of prices based off of several variables, most of them being categorical (34 categorical, 4 continuous). Using about 4000 examples. I am using scikit-learn to do this analysis.

When it comes time for me to tune my hyperparameters, the "best" score I get from using GridSearchCV doesn't necessarily give me the "best" score from scoring the Neural Network. I assume this is okay because it is doing cross-validation and the bias will be slightly off.

Am I correct in assuming I should listen to what GridSearchCV tells me are the "optimal" hyperparameter tunings?

How much should I rely on the r^2 as an evaluation of my neural network? Higher is better, but is there anything more to it than that?

Are there any graphs I can use to help me make some judgment?

Thank you all, and let me know if I am leaving out any pertinent information!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Well, scoring the neural network as-is is not going to give you the right clues about how the neural network will perform on out-of-sample data, because it's too biased an estimation. So ideally you should perform grid search to find the best neural network, and then evaluate its results on a different data set altogether (or better, use nested cross-validation).

As far as the plotting part goes, you should use the learning curves to see if the network is overfitting (scikit has a function that does that for you); you should also plot a train-validation curve, which is useful to control how the net is learning (not sure if scikit has support for it, assuming you are using MLPClassifier).

About R^2, I used to read this article that argues it's not that useful, but it seems the images are off. See if you can make sense of it, otherwise I'll have to find a paper I read some time ago where R^2 was criticized as well. Moral of the story, my advice is don't trust R^2 too much.

Improve this answer
$\endgroup$
4
  • $\begingroup$ To your first point, I did split my data 75/25 for train/test sets, and I am using r^2 to score the performance on the test set. To your second point, I am using MLPRegressor, not MLPClassifier since I am doing a regression as opposed to a classification. $\endgroup$
    – actuary_meets_data
    Commented Nov 22, 2016 at 20:00
  • $\begingroup$ If I do not use R^2, can you propose an alternative metric to select one model over another? $\endgroup$
    – actuary_meets_data
    Commented Nov 22, 2016 at 20:01
  • $\begingroup$ Yeah sure, I wrote MLPClassifier but intended MLPRegressor, obviously. I think MSE would be more appropriated than R^2, but that is kind of my opinion. $\endgroup$
    – mp85
    Commented Nov 22, 2016 at 20:16
  • $\begingroup$ First point:, you are selecting your model based on its evaluation on a single test set. It might be the case that you got "lucky" with that particular choice for a test set, and that could explain the higher performance (which would probably be not quite as high using another test set) $\endgroup$
    – mp85
    Commented Nov 22, 2016 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.