2
$\begingroup$

In an supervised learning approach, the training data set is already labelled with correct values. So, what is the purpose of evaluating the accuracy of the classifier function after it gets trained with the training data set ?

Wouldn't the accuracy be 100% since all the training data that is being fed will be correctly classified for it to learn from it.

example:

classifier = tf.estimator.LinearClassifier(feature_columns=feature_columns,n_classes=3,model_dir="/tmp/iris_model")
classifier.train(input_fn=input_fn(training_set),steps=1000)
accuracy_score = classifier.evaluate(input_fn=input_fn(test_set_, steps=100)["accuracy"]
$\endgroup$
2
  • 3
    $\begingroup$ Not if you reduce model-complexity by regularization and co. E.g. by adding a l1-norm penalty on the weight in linear-regression. Then you can observe quickly, if this penalty is too strong as you don't even get a good accuracy on the train-data. (actually: a linear-model cannot achieve 100% on data which cannot be seperated in a linear way; achieving 100% on train is in the real-world not what you want as this nearly always leads to overfitting) $\endgroup$
    – sascha
    Dec 7, 2017 at 5:24
  • 1
    $\begingroup$ I like to look at in-bag/training versus out-of-bag/validation model performance as an indicator of generalization. If there is a significant gap between the two, even if the absolute values are not huge, then it can tell me about a failure to generalize. I use it as a defining contrast to interpret the test performance. It can help me detect the presence of pathological data, or even pathological design issues (if I have missed them) in my model/learner. note: Sick <--> mean error, pathological <--> variance in error. $\endgroup$ Dec 7, 2017 at 15:27

3 Answers 3

3
$\begingroup$

Well actually you are not deducing the values of y from the training set by using features X1, X2,..... Xn. You are actually training your model to get theta. Theta is the vector which is used to get predicted y. Hence, you don't get accuracy of 100% on your training set. This is because predicted y may be different from actual y.

However, it is advisable to use a separate training, cross-validation and test set for generating real time result of your model

$\endgroup$
4
  • $\begingroup$ ah. now i can relate. So, this theta is the hypothesis that is used to predict the y for an unseen value of x. An when measuring accuracy, does the program uses the derived theta to predict and compare the values of y for the training data and hence derive the accuracy ? $\endgroup$
    – yathirigan
    Dec 7, 2017 at 5:36
  • $\begingroup$ Training = deriving the theta from training set , Measuring Accuracy = Use the theta to predict output and compare prediction with actual output in training set. Is this understanding correct ? $\endgroup$
    – yathirigan
    Dec 7, 2017 at 5:37
  • 1
    $\begingroup$ Yes. Now you are thinking correctly. However, there is more to machine learning than this. You must explore a lot more if you need to get into the depth. $\endgroup$
    – Akshay Bahadur
    Dec 7, 2017 at 5:42
  • $\begingroup$ after too many theory videos, i just started looking at some code/implementation of those concepts. Thanks for the clarification. This helps. $\endgroup$
    – yathirigan
    Dec 7, 2017 at 5:51
3
$\begingroup$

No, most certainly not! In fact, if you got 100%, you should not trust your model, because you are most definitely overfitting, which is a bad thing.

Machine learning models learn from the data that you provide them, as you'd stated. However, what you want for them to do is to learn so that they can fit data beyond the data you supplied. This is typically done via some sort of assessment, like the accuracy that you mention.


Often, but not always, this is done by removing some of the data, then trying to fit that data and measure the accuracy. There are different ways to do this, but 2 common ways are:

  1. Bootstrapping
  2. Cross-Validation
$\endgroup$
2
  • $\begingroup$ thanks, these are some more new information. will read these as well. $\endgroup$
    – yathirigan
    Dec 7, 2017 at 5:52
  • 1
    $\begingroup$ You're welcome! Machine learning is all about probabilities and estimations. Even worse, machine learning itself does not (yet) have much of the statistical rigor that older stats fields have. So, learning the "art" of properly estimating future error is hugely important. Otherwise, you can very easily find yourself promising the world and hugely underperforming. This causes a world of disappointment if you are working with people who do not understand machine learning and its limitations. $\endgroup$ Dec 7, 2017 at 6:18
1
$\begingroup$

A slightly different perspective.

As the other answers have noted, training error is not a useful measure of the predictive performance of a model, as it has very little to say about how your model will react to unseen data. Simulating the model seeing new data is what cross validation is for.

On the other hand, training error satisfies a very useful constraint:

As the model complexity increases, training error always decreases.

Constraints are useful for debugging, as all experienced programmers know. This means that whenever you have a modeling pipeline, which sometimes takes a significant amount of engineering work, it is useful to still observe the training error. If the complexity constraint is not satisfied, i.e. if the training error goes up unexpectedly as model complexity increases, then it points to a bug or error in your engineering work.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.