Random Forest model good train and test performance but bad "real world" performance I am working on a classification problem where I need to classify objects based on a visual data. There are a couple hundred different classifications to be made and I have around a million plus observations to draw upon currently. The data has 49 features and 1 label. The features are all continuous.
In order to begin working on my model I decided to focus on the top four most popular classes. My training data was about 100k observations scattered over time fairly evenly. When I fit and test my model on this data, I get very good performance(99% accuracy).
I was highly skeptical of this performance so I decided to pull some newer observations of those same four classification. When I ran the model on these observations, my performance dropped to something around 60%.  
What could I be doing wrong? I am new to Machine Learning and this data. What could be some troubleshooting techniques to solve this? I am using both R and Python/Sklearn. 
Thanks in advance
 A: There are many reasons why your model is not accurate, just because, you did not perform model fit and validation in a correct manner. There are basic steps I want to share with you:


*

*Step 1. Split data.

*Step 2. Train your model.

*Step 3. Test your model.

*Step 4. Compute the accuracy of a model.


Step 1: There are many methods, use K-fold cross validation (K-CV), e.g., usually, K=10 (10-fold CV) your data is split into 10 folds.
Step 2: You use your classifier here. For example, nearest neighbors classifier (KNN).
Step 3: Test the model using the rest of the data.
Notice that Step 2 and Step 3 use data from Step 1. For example, if Step 2 uses 10% of data then 90% is for Step 3. This is repeated 10 times since K=10.


*

*Step 5. Validate the model. You can validate your model using unseen data

*Step 6. Still not accurate?. The problem here comes to re-address the Step 2. This is called parameter tuning. Let's say you have KNN classifier. It has two parameters: N-number of neighbors, weights - weight function, e.g, Euclidean or cosine or uniform distance. I will write possible range here: N = range(1, 50), weights = ('Euclidean', 'Cosine', 'Uniform').


This Step 6 is the key point to choose the best model in your classifier ^^.
Hope it helps.
A: You should not fit and test the model on the same data. Try completing this course https://www.coursera.org/learn/machine-learning you will have a get a very good understanding of Machine Learning. I am 95% confident about this.
