I am working on a classification problem where I need to classify objects based on a visual data. There are a couple hundred different classifications to be made and I have around a million plus observations to draw upon currently. The data has 49 features and 1 label. The features are all continuous.

In order to begin working on my model I decided to focus on the top four most popular classes. My training data was about 100k observations scattered over time fairly evenly. When I fit and test my model on this data, I get very good performance(99% accuracy).

I was highly skeptical of this performance so I decided to pull some newer observations of those same four classification. When I ran the model on these observations, my performance dropped to something around 60%.

What could I be doing wrong? I am new to Machine Learning and this data. What could be some troubleshooting techniques to solve this? I am using both R and Python/Sklearn.

Thanks in advance

  • 1
    $\begingroup$ Agreed with Imo, it will likely be moved. But if you're trying to train a model for real world applications for observations coming down the pipe, you should rely on the RF algorithm's internal bagging and variable sampling to reduce bias/overfitting, and increase your training sample size considerably. In fact I would invert your validation and training data sizes. RF can only predict for combinations of "features" and "labels", using your terminology, for which it has in the training data. You likely are just running into areas of your data space for which you haven't trained the model. $\endgroup$ Commented Apr 27, 2016 at 13:47

2 Answers 2


There are many reasons why your model is not accurate, just because, you did not perform model fit and validation in a correct manner. There are basic steps I want to share with you:

  • Step 1. Split data.
  • Step 2. Train your model.
  • Step 3. Test your model.
  • Step 4. Compute the accuracy of a model.

Step 1: There are many methods, use K-fold cross validation (K-CV), e.g., usually, K=10 (10-fold CV) your data is split into 10 folds.

Step 2: You use your classifier here. For example, nearest neighbors classifier (KNN).

Step 3: Test the model using the rest of the data.

Notice that Step 2 and Step 3 use data from Step 1. For example, if Step 2 uses 10% of data then 90% is for Step 3. This is repeated 10 times since K=10.

  • Step 5. Validate the model. You can validate your model using unseen data
  • Step 6. Still not accurate?. The problem here comes to re-address the Step 2. This is called parameter tuning. Let's say you have KNN classifier. It has two parameters: N-number of neighbors, weights - weight function, e.g, Euclidean or cosine or uniform distance. I will write possible range here: N = range(1, 50), weights = ('Euclidean', 'Cosine', 'Uniform').

This Step 6 is the key point to choose the best model in your classifier ^^. Hope it helps.


You should not fit and test the model on the same data. Try completing this course https://www.coursera.org/learn/machine-learning you will have a get a very good understanding of Machine Learning. I am 95% confident about this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.